Mysticism is a crucial aspect of the study of religion. "One may say truly, I think, that personal religious experience has its roots and centre in mystical states of consciousness," William James writes in The Varieties. That said, it's important to be clear about what he means by phrases like "states of consciousness".

Our view is coloured by a psychologising tendency that's grown since James. It can be associated, in particular, with Abraham Maslow's notion of "peak experiences" – the ecstatic states that satisfy the human need for self-actualisation. This exaltation of feelings of interconnectedness is questionable on two counts.

First, Maslow's analysis is scientifically dubious. As Jeremy Carrette and Richard King put it: "Sampling disillusioned college graduates, Maslow would ask his interviewees about their ecstatic and rapturous moments in life." No offence to students, but they probably do not provide the best samples of mystics.

A second critique of Maslow's work is found in the writings of the great spiritual practitioners themselves. The author of The Cloud of Unknowing, for one, explicitly argues that, whatever the mystical might be, it is hidden from experience. Or, as any decent meditation teacher will tell you, clinging to oceanic experiences will hinder your progress quite as much as clinging to anything else.

This is not to say that mystical experience has nothing to do with feelings, James continues. Rather, it is a state both of feeling and of knowledge, of wonder and intellectual engagement. The two faculties must be deployed when weighing any insight. "What comes," James explains, "must be sifted and tested, and run the gauntlet of confrontation with the total context of experience." Mystical states can, therefore, be assessed for their truth value. But how?

Not, James explains, in the way advocated by the "medical materialists" – those for whom mysticism signifies nothing but "suggested and imitated hypnoid states, on an intellectual basis of superstition, and a corporeal one of degeneration and hysteria".

More controversially, James also believes that the truth of mystical experience has little to do with the specificities of time and place – which puts him at odds with scholars today, who generally emphasise the opposite: the need for a dialogue between the experience and its historical context. For example, the Jewish scholar, Gershom Scholem, coined the term "historiosophy" to capture the mix of real-life events and metaphysical response that he argued was implicit in mystical experience. A case in point is the origins of Lurianic Kabbala, which Scholem interpreted as a reaction to the expulsion of the Jews from Spain in 1492.

James, though, is mostly interested in the individual, not communal, significance of mysticism and, further, he inclines to the view that it is real and positive. "In mystic states," he writes, "we both become one with the Absolute and we become aware of our oneness." In short, there are two characteristic outcomes: optimism and monism. This, he believes, is amply demonstrated in copious accounts of mystical experience. Generally speaking, they move "from a less into a more, as from a smallness into a vastness, and at the same time as from an unrest to a rest. We feel them as reconciling, unifying states".

Of course, such perspectival shifts could be deluded. The experience may be "nothing but [a] subjective way of feeling things, a mood of … fancy". James also considers what he calls "lower mysticisms", a category that includes states of consciousness that are the product of chemical not spiritual stimulation. An example is alcohol: "The sway of alcohol over mankind is unquestionably due to its power to stimulate the mystical faculties of human nature, usually crushed to earth by the cold facts and dry criticisms of the sober hour."

Nonetheless, there is, for James, such a thing as genuine mystical experience, providing a pointer to a reality that is more likely true not false. The monism and optimism that is their product have such a demonstrably positive impact upon those who have them. And, James concludes, "that which produces effects within another reality must be termed a reality itself".

James's logic is faulty on this point. Robert Segal, of Lancaster University, has called it the "functional fallacy": delusions can lead an individual to act in positive ways, too. Moreover, James can be accused of putting the cart before the horse because the great mystics tend not to claim that good deeds prove the truth of an experience. Rather, they make the lesser claim that misdeeds invalidate them.

So, James's discussion of mysticism is not unproblematic. Alongside the functional fallacy, he is vulnerable to charges like that of sidelining historical context. But there is significant value in the way he frames the subject, particularly his recognition that experience is an interaction of both feeling and cognition. He seeks to assess mysticism's veracity not merely on account of its "luminousness". He also takes account of its reasonableness, judged by its fit within a well-articulated and philosophically defensible system of beliefs. And he stresses its overall moral consequences for the individual, the quality known as saintliness.