The New York Times has an article on the simple but effective idea that a statistical analysis of word frequency in written text can be a guide to the psychological state of the author. It’s a technique that’s been pioneered by psychologist James Pennebaker who has conducted a considerable amount of intriguing research to back up his technique.

In fact, he’s completed a huge number of studies looking at word frequency in everything from bereavement to suicidal and non-suicidal poets.

However, some of his most impressive work has focused on the benefits of getting distressed or ill people to write, finding that it benefits recovery from trauma, but perhaps more surprisingly seems also to boost immune system function in HIV patients.

The evidence and theory behind the work was described in a great 2003 review article which notes that the importance lies not so much in the subject or action words, but in the ‘bitty’ parts of speech, such as the use of pronouns (I, you, we and so on).

These seem to relate to the focus of the thoughts and Pennebaker was asked by the FBI to apply the technique to the communications of Al Queda:

Take Dr. Pennebaker‚Äôs recent study of Al Qaeda communications ‚Äî videotapes, interviews, letters. At the request of the F.B.I., he tallied the number of words in various categories ‚Äî pronouns, articles and adjectives, among others. He found, for example, that Osama bin Laden‚Äôs use of first-person pronouns (I, me, my, mine) remained fairly constant over several years. By contrast, his second-in-command, Ayman al-Zawahri, used such words more and more often. ‚ÄúThis dramatic increase suggests greater insecurity, feelings of threat, and perhaps a shift in his relationship with bin Laden,‚Äù Dr. Pennebaker wrote in his report [pdf], which was published in The Content Analysis Reader (Sage Publications, July 2008).

Interestingly, the FBI have their own in-house text analysis technique but I’m damned if I can remember the name or find it on the net. Answers on an encrypted telegram please…

Link to NYT piece ‘He Counts Your Words (Even Those Pronouns)’.

Link to review article ‘Psychological aspects of natural language’.

Link to PubMed entry for same.