If you can be sure of one thing, then surely it is that you exist. Even if the world were a dream or a hallucination, it would still need you to be dreaming or hallucinating it. And if you know nothing else about yourself, surely you know that you have a mind, one perspective on the world, one unified consciousness?

Yet throughout history, there has been no shortage of people claiming that the self doesn't exist after all, and that the individual ego is an illusion. And such claims are no longer the preserve of meditators and mystics. The only disagreement many scientists would have with philosopher Thomas Metzinger's claim that "modern philosophy of mind and cognitive neuroscience together are about to shatter the myth of the self" is that the destruction has already occurred.

There is a wide range of scientific evidence that is used to deny "I think, therefore I am". In René Descartes' famous deduction, a coherent, structured experience of the world is inextricably linked with a sense of a self at the heart of it. But as the clinical neuropsychologist Paul Broks explained to me, we now know the two can in fact be separated.

People with Cotard's syndrome, for instance, can think that they don't exist, an impossibility for Descartes. Broks describes it as a kind of "nihilistic delusion" in which they "have no sense of being alive in the moment, but they'll give you their life history". They think, but they do not have sense that therefore they are.

Then there is temporal lobe epilepsy, which can give sufferers an experience called transient epileptic amnesia. "The world around them stays just as real and vivid – in fact, even more vivid sometimes – but they have no sense of who they are," Broks explains. This reminds me of Georg Lichtenberg's correction of Descartes, who he claims was entitled to deduce from "I think" only the conclusion that "there is thought". This is precisely how it can seem to people with temporal lobe epilepsy: there is thought, but they have no idea whose thought it is.

You don't need to have a serious neural pathology to experience the separation of sense of self and conscious experience. Millions of people have claimed to get this feeling from meditation, and many thousands more from ingesting certain drugs.

While some people experience lack of self, some seem to have more than one. Most obviously there are sufferers of dissociative identity disorder, the preferred term these days for multiple personality disorder. Perhaps even more interesting are the "split-brain" patients of Roger Sperry and Michael Gazzaniga. As a last resort in an experimental procedure to treat severe epilepsy, Sperry and Gazzaniga severed the connection (the corpus callosum) between the two hemispheres of the brain. The results of this operation, called a commissurotomy, was that the epilepsy was indeed much reduced. But then Sperry and Gazzaniga conducted some experiments that revealed a remarkable, unforeseen side effect.

Patients were asked to focus on a dot in the centre of a screen. Words and images were then flashed up for a few seconds on either the right or left side of the screen. When these appeared on the right side of the screen, the patients were easily able to say what they were. But when they appeared on the left of the screen, they claimed to have seen nothing. However, if asked to draw an object with their left hand, they would draw what they had just seen, all the time denying they had seen any such thing. They could also manipulate or use the object normally with their left hands. So what was going on?

The way in which vision works is that information from the right visual field is processed by the left brain hemisphere, while information from the left visual field is processed by the right hemisphere. But it is the left hemisphere that (in most people) controls speech. Because normally the corpus callosum allows the two hemispheres to communicate, this presents no practical difficulty for most people. But after a commissurotomy, this information exchange cannot occur. That means that if you control carefully which side of the brain receives information from the environment, you can effectively make one hemisphere aware of something that the other is not. What is astonishing about this is that for this to be possible, there would have to be two centres of awareness in the individual concerned. Commissurotomy therefore seems to show that selves can be divided – at least temporarily – or that they needn't have just one centre of consciousness after all.

Intriguingly, however, in normal life, such patients experience the world in the normal, unified way. Gazzaniga's explanation of this is that "we don't miss what we no longer have access to". Consciousness of self emerges from a network of thousands or millions of conscious moments. This means that when we lose bits, the way a split-brain patient does, we don't sense anything as lost at all. Gazzaniga explains this thought with a metaphor of a pipe organ. "The thousands or millions of conscious moments that we each have reflect one of our networks being 'up for duty'. These networks are all over the place, not in one specific location. When one finishes, the next one pops up. The pipe organ-like device plays its tune all day long. What makes emergent human consciousness so vibrant is that our pipe organ has lots of tunes to play."

Gazzaniga's metaphor holds for healthy as well as damaged minds. In other words, what the numerous pathologies of self-experience expose is that even in normal cases, there is no unified "I" behind experience. Rather, to use another musical metaphor, the mind is like a jazz orchestra that usually plays with sufficient harmony to disguise the fact that it lacks a single player, a score, or even a conductor. A few bum notes or absent musicians, however, and the illusion is shattered.

This is not an idea most are comfortable with. "People do not want to feel that the continuity of their life is just a matter of which molecules are arranged in which order inside their brain," the psychologist Susan Blackmore told me. Similarly, Broks said, "we have this deep intuition that there is a core, an essence there, and it's hard to shake off – probably impossible to shake off, I suspect. But it's true that neuroscience shows that there is no centre in the brain where things do all come together."

Astonishing though this thesis can sound, there is a danger here that we could overstate just what this lack of a unified pearl of self at the heart of us really means. Many writers, such as Blackmore and Metzinger, draw the conclusion that the self is an illusion. This is true in the sense that it is not what it seems to be. But that is not to say that the self doesn't exist.

This can be most simply explained by thinking about what it means for anything other than fundamental particles to exist. In Buddhist philosophy, there is an analogy attributed to Sister Vagira, a contemporary of the Buddha's, which compares a person to a cart. There is no cart, she says, only the wheel, the axle, the flat bed and so on. In the same way, there is no self, only experiences, thoughts, and sensations. But, of course, there is a cart – it's just that it is nothing other than the ordered collection of parts. In the same way, there is a self – it is simply no more than the ordered collection of all our experiences.

To take an even simpler example, water is not something that has two hydrogen atoms and one oxygen atom: it just is two parts hydrogen and one part oxygen. In the same way, we are not things that have experiences – we just are our experiences.

Neuroscience and psychology provide plenty of data to support the view that common sense is wrong when it thinks that the "I" is a separate entity from the thoughts and experiences it has. But it does not therefore show that this "I" is just an illusion. There is what I call an Ego Trick, but it is not that the self doesn't exist, only that it is not what we generally assume it to be.