I played a lot of Call of Cthulhu way back when, and enjoyed it immensely. Back then we weren’t terribly concerned about losing sanity, because in general our characters were killed by horrific monsters far before it happened. Now I’m playing Pathfinder, and my DM in introducing insanity rules. I’m interested but cautious.
There are two sets of advice regarding sanity rules in Pathfinder. Hypertext d20 SRDTM has the most extensive rules, but not the ones I would want to play. Players can lose sanity in a lot of ways, including casting spells (so much for mages!), having fear spells or illusions cast at them, or running into various monsters, including giants, unicorns, and humanoids! Way too severe. 1d6 sanity is regained by gaining a level, which is definitely a losing battle, unless the DM also allows magical healing of Sanity. Still, there’s a lot of ideas in that set of rules, and worth looking at. Paizo’s rules are much shorter, but not terribly useful.
For my money, insanity derives from stress, horror, and perhaps anger (although that is certainly related to stress). For adventurers, stress derives from the death of comrades, the near death of oneself, and the inability to resolve the adventure or stop whatever is going on. Sarah Lynne Bowman’s “Social Conflict in Role-Playing Communities” comes to mind. I referenced it in one of my old blogs.
Horror has to do more with the scale of the challenges encountered. The sudden introduction of bizarre and unexpected creatures (doppelgangers, beholders, etc.) could do the trick, or encountering gruesome deaths or tortures. One can become appear to become inured to gruesome deaths, and then one day go off and start killing innocents. Or one can obsess about such things, feel guilt, and become depressed and suicidal. Of course, if the person is absolutely convinced they were justified, they will not be troubled as much by it. I think paladins would not feel badly about any gruesome deaths they caused, nor would terrorists, since both feel justified. I remember a former black ops operative and sniper I knew. He had no problem with the things he had done, because he had an absolute belief in the United States and that he was doing the right thing for the country. I also remember a guy who grew up in a mob family and was a sadist, perhaps as a result. The only thing he regretted was cutting his mom and laughing about it. Of course, one might class him as psychotic because of his sadism, but if so it was a functional psychosis and didn’t show to people on the street.
Any way I look at it, insanity should be part of the roleplaying experience, but not simply relegated to rolling on the insanity table when one’s sanity points get too low. Sanity points are the easiest way to keep track of the character’s state, but the manifestation of the insanity should be commensurate with the causes. If a character goes temporarily insane but recovers, then the pattern of his insanity probably sets the stage for deeper neuroses later on (what COC calls Indefinite Insanity), and perhaps identifies the future pattern of his permanent insanity should he reach a Sanity of 0. There is, by the way, a COC supplement dedicated to insanity, namely The Taint of Madness (1995). I don’t have it at hand, but it would probably be worth a look.