The magician picks up a coin, conceals it in his hand and, after a magical gesture, it mysteriously disappears, only to reappear from behind your ear. As you watch this performance, you fully understand that objects cannot simply materialise from thin air, yet this is exactly what you have just experienced. Conjuring is one of the oldest forms of entertainment and throughout history, tricksters have amazed audiences by performing illusions of the impossible.

The art of magic has never lost its appeal, and even in our modern lives, which are dominated by science and technology, we are still captivated by experiencing things we believe to be impossible. This universal appeal can be traced back to a deep-rooted psychological drive to explore things we do not understand. Indeed, from an early age, infants are captivated by events that confound their understanding of the world, and the same is true for adults. Most people simply think of magic as just another form of entertainment, but the ancient art of conjuring is now helping scientists uncover some of the mysteries of the human mind.

Magic deals with some of the most fundamental psychological and philosophical questions. What do you believe to be possible? What is consciousness? How much control do you have over your thoughts and your actions? And yet, until recently, the art of magic has received little scientific attention.

I have always been captivated by magic and I dedicated most of my teenage years to it. As a kid, I borrowed every book on magic from the local library and spent my free time practising new tricks. I was particularly interested in understanding why magic works, so I read books on psychology, which I hoped would give me a deeper understanding of how to trick the mind. It was this desire to discover more powerful ways of hacking the mind that led me to study psychology at university.

For most magicians, this link between magic and psychology is obvious. Magic relies on powerful psychological illusions and magicians create their tricks by exploiting gaps and errors in our conscious experience. For example, magicians use misdirection to manipulate what you attend to and this allows them to control what you see – and what you miss. However, as I enrolled on my degree course, I was surprised to learn scientists were not particularly interested in magic. None of my textbooks on cognitive psychology talked about misdirection, and there was only a handful of research papers that had investigated magic scientifically – and most of them had been published more than 100 years ago. I was disappointed, but as I started to engross myself in learning more about the mysteries of the brain, I replaced my passion for magic with psychology. To put it simply, I exchanged my cape for a lab coat and embarked on a career in cognitive science.

I exchanged my cape for a lab coat in order to be a cognitive scientist

Back in 2003, as I was completing my PhD on consciousness, I had the opportunity to use eye-tracking equipment to investigate how magicians misdirect people’s attention. We developed fun experiments in which we used the tracker to measure people’s eye movements while they watched me performing simple tricks. The results were astonishing – the misdirection was remarkably effective at manipulating people’s conscious experiences. It was also the first time we had scientific data that helped us understand how misdirection works, and we were surprised that people often failed to see things that were right in front of their eyes. The misdirection was so effective that some people were looking at an object, yet they simply did not see it. We soon realised magic could provide a useful tool to study visual attention.

These early scientific experiments were a turning point in my career. I’d found a way to combine two of my main interests: magic and psychology. I am now a reader in psychology and the director of the MAGIC laboratory (Mind, Attention and General Illusory Cognition), and spend most of my time studying human cognition. Rather than performing magic to entertain people, I study magic in the lab.

Our scientific approach is based on the following logic: magicians have spent hundreds of years developing the art of deception and by doing so they have discovered powerful tricks that capitalise on cognitive errors. Scientists regularly study cognitive errors, often by looking at psychological impairments caused by brain damage. For example, lesions to particular parts of the cortex can prevent some people from recognising faces, while other lesions can result in specific memory failures.

Magicians are not concerned with understanding the anatomy of the brain, but their experience in tricking people has helped them identify profound errors in cognition. Indeed, most magic tricks rely on exploiting surprising and powerful cognitive errors, and magicians have informally learned to understand psychological principles that push our cognitive processes to breaking point. By understanding these conjuring techniques and their underlying cognitive mechanisms, we can then gain valuable knowledge of how the mind works.

Since my early scientific studies on misdirection, there has been an explosion of interest in studying magic scientifically, and the science of magic has now become a field in its own right. Magicians and scientists have started to collaborate and are investigating cognitive processes that underpin magic to explore a wide range of psychological phenomena.

Our true perception is more removed from reality than you think

Much of our work on misdirection reveals that the gaps in our conscious experience are bigger than most of us had assumed. As you look at your surroundings, you experience the world as a rich and complete sensory experience. However, our research on misdirection illustrates that this conscious experience is a powerful illusion. Our true perception is full of gaps and holes, and much more removed from reality than most of us imagine. I spend much of my time studying these types of illusions, and even though I know my brain is being tricked, I still struggle to appreciate just how little I am truly conscious of. It’s a very compelling illusion and one that is very difficult to break.

This research on misdirection has important real-world implications. It is often important to accurately judge our own cognitive abilities, and misjudgments can have fatal consequences. For example, most people underestimate the extent to which their attention is misdirected by a phone call. Research has shown that talking even on a hands-free phone has the same detrimental impact on your driving as being over the drink-drive limit. However, since we overestimate our own abilities, we don’t notice the impact this technological misdirection has on our performance.

Research on magic highlights that we are not only wrong about the amount we see, but also about the extent to which we can trust the things we see and remember. As we are learning more about the mind, it has become apparent that most of our experiences are an illusion. Of all of these illusions it’s the illusion of free will that I find most unsettling.

We like the feeling of being in charge of our thoughts and actions, and abandoning our sense of free will feels rather uncomfortable. However, magicians have developed powerful ways of manipulating your thoughts, and they can influence many of the choices you make. For example, the magician may ask you to choose a card from a deck of playing cards, and while you feel you have an entirely free choice, the magician made you choose one particular card. This is known as forcing and is a principle by which magicians covertly guide you towards a predetermined choice.

We are now studying the psychological mechanism that underpins these forcing techniques, and the ease by which we can covertly manipulate people’s decisions is intriguing. Most importantly these findings illustrate that even our sense of free will may prove to be a powerful illusion. Studying the ease with which a magician can manipulate our conscious experience is providing intriguing and, at times, unsettling new insights into the human mind.

Experiencing the Impossible: The Science of Magic by Gustav Kuhn is published by MIT Press at £20. Buy a copy for £17.60 at guardianbookshop.com

Smoke and Mirrors: The psychology of Magic opens at London’s Wellcome Collection on 11 April. Gustav Kuhn’s research features in the exhibition