The Amazing Johnathan pretends to cut his arm with a big knife. ‘What you are about to see,’ he says, ‘is just a trick.’ He only scratches the surface, but you can already see some blood. ‘It is stage blood. It is riding on the surface of my arm, it is not penetrating my arm, that would be real, this is an illusion.’ And then, all of a sudden, he exclaims: ‘This is real!’ and penetrates his arm with the big blade. You can see it right in front of your eyes, it sure looks as real as anything ever did. Just for good measure, he zealously mutilates his arm a bit more, and then withdraws the knife, showing that the arm is unsevered. ‘It is a trick,’ he says nonchalantly, as if the carnal and graphic butchery you just witnessed never happened. But now you have a hard time believing him, because you saw it happen with your own eyes.

According to Teller, the master magician of Penn and Teller fame, magic is best conceived of as ‘a form of theatre that depicts impossible events as though they were really happening’. Thus, he points out in an interview with the Smithsonian, ‘you experience magic as real and unreal at the same time’. By this account, the Amazing Johnathan’s performance is magic at its best. You just saw that the knife really cut though his arm, but you know it cannot be true, hence the whole thing feels utterly unreal at the same time.

Trying to figure out the secret behind the knife-through-arm trick is extremely difficult. It is as if all the doors to the secrets behind the tricks are closed. Maybe the blood was fake blood (it was), but still, you did see the blade penetrating the arm, and you also saw that the arm was unsevered afterwards. What are the options? Was your attention manipulated in any way? Maybe it was, but how would that explain what you just saw? Maybe the magician made you see something that did not happen? Yes, in fact he did, but how is that possible? And if so, doesn’t that imply the existence of magical powers after all, like extraordinary powers of suggestion?

It is well-known that magicians can make spectators’ question their own sense of reality by misdirecting their attention. The vanishing cigarette trick used in intriguing research by the magician-psychologist Gustav Kuhn at the University of Durham and his colleagues is a case in point. Here, the magician is about to light a cigarette, but notices that he is trying to light up its wrong end. He turns the cigarette around and tries to light it again, but now the lighter seems to have disappeared. He looks with surprise at the hand in which the lighter is supposed to be, flicks it open with a snap of the fingers and reveals that it is gone. He then looks back at the other hand holding the cigarette, but now the cigarette has disappeared as well. An investigation of how people react to this trick shows that almost nobody is able to figure out how the cigarette disappeared when they view it for the first time, but if you let people watch it a second time, it’s blatantly obvious to all of them. The magician simply dropped the cigarette into his lap, right in front of their eyes.

Of course, the magician’s gesturing is skilfully orchestrated to direct your attention to the other hand and away from the dropping cigarette. The interesting point, however, is that when you view the trick for the first time, you don’t really have the impression that you are blind at the location where the cigarette falls. Quite to the contrary, you have the impression that you see everything that is going on with utmost clarity. When you watch the trick a second time, however, you see the cigarette fall down as if to mock your sanity, and you wonder how on earth you could have missed it the first time around.

The vanishing cigarette trick provides an excellent example of a startling and highly counterintuitive phenomenon called ‘inattentional blindness’, a phenomenon studied in the 1990s by the American psychologists Arien Mack and Irvin Rock. This term is aptly chosen, because we are indeed dealing with a kind of sighted blindness. The basic phenomenon is that you will typically fail to see something you’re looking at directly if you’re attending to something else. This sighted blindness of ours is routinely exploited by magicians and, as the vanishing cigarette trick illustrates, it enables magicians to perform their secret moves right before our eyes.

Inattentional blindness is just one example of a more general feature of our visual experience known to cognitive scientists as ‘the grand illusion’. When we look at the world around us, almost everything in our visual field appears clear, vivid and rich in detail but, in experiments, our objective ability to detect change is more suggestive of an observer with a bag on his head, with just a small hole through which to see anything. This observation hole can be moved around by the observer himself or it can be manipulated automatically when interesting events occur in the environment. But at any given moment, the observer sees the world only through a small hole in a bag. The essence of the grand illusion is that you have the impression of a clear view, while in reality you are limited by what you can see through the little hole in the bag over your head.

The grand illusion, rather than the misdirection of attention by itself, is the crucial factor that enables the magician to create the experience of magic. A friend might misdirect your attention by claiming that Bob Dylan just entered the room, and grab your last french fry while you are looking for His Bobness. But when you turn back to your empty plate, you will at best feel slightly amused. Magic is the illusion of impossibility, and there is nothing impossible about not noticing something that you are not looking at. The real magic occurs when you don’t notice something that you are positive that you would have noticed if it really happened. Thus, it is not the inattentional blindness by itself that creates the magic, but rather your blindness to your inattentional blindness. Magic requires boldness. The magician must trust not only his or her own skill, but also in the spectators’ blindness to their own blindness.

It is well-known that magicians can use attentional misdirection to make something that happens right in front of the spectators’ eyes invisible. But attentional misdirection cannot explain the knife-through-arm trick, because here you don’t fail to see something that actually does happen, such as the cigarette falling into the magician’s lap. Rather, you see something that isn’t actually happening. You see a knife penetrating the arm but, in reality, that is not what happens. So how does the magician make you hallucinate a blade penetrating the arm? The answer is so simple that it’s almost embarrassing: the blade has an opening that makes it fit nicely around the arm.

Would you expect many people to be fooled by such a simple trick? Probably not, but magicians know from experience that this trick works like a charm. The question, then, is why such a stupid trick can create such a strong and robust experience of magic.

Since it is difficult to question what you see with your own eyes, the visual system closes the door to the right solution

With my colleagues, the psychologists Bilge Sayim and Johan Wagemans, I recently argued that this magic trick and many others rest heavily on a quirky aspect of our mind, analogous to the grand illusion in being a dramatic and pervasive misperception of our own perception – but quite the opposite in terms of the direction of the misperception. Rather than being a case of sighted blindness, this is a case of unsighted vision. When you look at the knife ‘penetrating’ the arm, you have the impression of seeing only the part of the knife outside the arm, but an intriguing line of research pioneered by the Belgian psychologist Albert Michotte in the 1940s, and continued in the 1980s by Gaetano Kanizsa at the University of Trieste in Italy, suggests that it actually makes more sense to say that you also see ‘the part of the blade penetrating the arm’ than to say that you just think it must be there.

The argument draws on a plethora of experimental findings showing that our mental experiences of the occluded parts of objects behave in much the same way as real visual percepts. A characteristic property of visual percepts is that they persist even when we know that that they are illusory. As the developmental psychologist Alan Leslie at Rutgers University in New Jersey put it in ‘The Necessity of Illusion’ (1988), the essence of visual illusions ‘is that a bit of the world appears to us in a way we know is not or cannot be the case but which, despite such knowledge, appears this way repeatedly and incorrigibly’.

In a well-known trick called multiplying balls, the magician starts off by holding a ball between thumb and index finger. Then, with a flick of the wrist, a second ball magically appears between the index and the middle finger.

As the psychologist Amory Danek at the Ludwig Maximilian University of Munich and colleagues have shown in experiments on problem-solving in magic, people find it very difficult to figure out the secret behind this trick, though it is almost ridiculously simple. I will give you a hint: as in the case of the knife-through-arm trick, you see something that is not there. Are there really two balls? No, just one. The other ball is an empty semi-spherical shell. In the beginning, the real ball is hidden within that shell. When the magician flicks his wrist, he just flips it out of its hiding place, resulting in the illusion of two complete balls, where in fact there is just one and a half.

A central point in our argument is that the spectators do not merely think that the shell is a complete ball, but actually see it as a complete ball. And since it is difficult to question what you see with your own eyes, the visual system has essentially closed the door to the right solution.

If you are not convinced that the experience of the backside of a halved ping-pong ball (or the complete one, for that matter) is more than just imagery, try this: balance a halved ping-pong ball on your fingertip and look it at from above. Although you know that it is a halved ping-pong ball, and you have your finger in it to prove it, it still looks like a complete ball. Still more bizarre, you might experience the illusion that your finger feels shorter than normal, as if to make space for the illusory volume of the ‘ball’.

My colleagues and I documented this illusion by asking people to point to the location where they felt that their fingertip was located. But an easy way to notice the illusion for yourself is to balance the halved ping-pong ball on your middle finger while extending all your fingers towards your eyes and asking yourself whether the tip of your middle finger is closer to or further away from your eyes than the other fingers.

The middle finger is longer than the other fingers, so it should feel like it’s closer, but you will probably feel that it’s further away – which means that your finger must feel shorter than normal.

This illusion illustrates the power of visual experience to impact your sense of reality. Surely, the mere thought that the halved ping-pong ball is a complete ball would not make you feel that your finger is shorter than normal. The feeling that your finger has become shorter is a product of the internal logic of the perceptual system rather than the logic of conscious reasoning: the perceptual impression of the backside of the ‘ball’ demands that your finger must feel shorter – whether your conscious thinking likes it or not.

Our failure to figure out what the magician does results from the fact that we are unable to imagine it

The visual system’s tendency to fill in the invisible parts of the objects we see before us can be exploited to create a wide range of compelling magical illusions: spoons bending, knives cutting through flesh, and solid rings magically linking and unlinking. The principle underlying all these tricks, as well as many others, is that the visual system convinces you that something is there, whereas in fact there is just a gap or a missing piece.

But our visual system doesn’t just fill in the gaps. It can also create a compelling impression that the space hidden behind an object in the foreground is empty. In most situations, this impression is correct, but in some cases, it is illusory and misleading. To magicians, this perceptually empty space is the perfect hiding place for the things they don’t want you to know about. It is a no-brainer that hiding things behind something else makes them invisible, but the illusion of perceptually empty space entails more than just invisibility: it makes you ‘see’ that that there’s nothing there, although there sometimes is – particularly when you are watching a magician at work.

The philosopher Jason Leddington at Bucknell University in Pennsylvania says the experience of magic results from a failure of imagination. Our failure to figure out what the magician does results from the fact that we are unable to imagine it. This view neatly captures the essence of the illusion of empty space: it makes it difficult for us to even imagine that there is anything hidden behind the object in the foreground – be it the magician’s thumb, hand or anything else.

The illusion of empty space is a new concept in cognitive science, so we do not yet know for certain how the brain creates it. We can speculate, however, that the driver is ‘a general principle of perceptual processing’, in which the brain avoids perceptual interpretations involving suspicious coincidences. Preliminary observations suggest that the positive illusion of empty space – as opposed to mere invisibility – occurs whenever the hiding and the hidden object are aligned along the line of sight in a way that would be an unlikely accident in real life. The reason why such unlikely accidents occur in magic shows, of course, is that the magician sets them up on purpose.

Magicians rely heavily on our inattentional blindness and particularly on the grand illusion associated with it – our blindness to our own inattentional blindness. But my team’s work reveals that the art of magic also relies on an analogous, but opposite grand illusion, in which we are blind to the prodigious clairvoyance of our visual system – which makes us see hidden things. Exploiting either of these grand illusions not only requires skill and knowledge on the part of the magician, but also chutzpa, because he or she must place absolute faith in the counterintuitive quirks of the spectators’ visual systems, and allow them to produce the real magic.