I was a junior doctor when I encountered Yvonne. She was 40 and worked in a supermarket. One Tuesday she was reorganising produce in a refrigerated unit while a colleague cleaned the glass. Yvonne turned to talk to her just as she released a small spray of window cleaner. Yvonne felt the liquid splatter on her face and a burning in both eyes. Another colleague led her to the bathroom and washed her eyes with water. Yvonne was taken by ambulance to hospital, where she was examined and had her eyes bathed. Her husband was called, and by the time he arrived she was feeling better. Her eyes were red and full of tears, but she could see normally and the pain had lessened. Her husband took her home.

He told her not to go in to work the next day, and during the morning Yvonne noticed her vision blurring. By lunchtime, she was struggling to read the digital clock. Her husband and children came home that evening and encouraged her to have an early night. When Yvonne woke up and opened her eyes, everything was black. She cried out for help and heard the noise of people entering the room, but could not see them. She was completely blind.

When I met her, six months had passed and, despite several hospital visits and tests, doctors had told Yvonne they could find nothing wrong. As a last resort she was referred to a neurologist.

At the time, I was a trainee. Since then, I have been a consultant for more than 10 years, first at the Royal London Hospital, now at the National Hospital for Neurology and Neurosurgery.

I knew from an early stage that I wanted to be a neurologist. I enjoyed the detective drama of the job, unravelling the mysteries of how the nervous system communicates its messages. But I did not predict how far I would be drawn into the care of people whose illness originated not in the body but in the mind. Some examples of how the mind affects the body are commonplace: we are familiar with the shake in our hand as we pick up the pen to sign the marriage register, or the bead of sweat on our brow as we stand up to give a presentation. These are the body’s physiological responses to stress. But we are less familiar with the frequency with which people can unconsciously think themselves ill. Up to one third of people seen in an average general neurology clinic have symptoms that cannot be explained by medical tests or examinations. In those people, an emotional cause is often suspected.

Yvonne was sitting on the bed, a tiny woman, shoulders hunched, elbows pressed tightly to her sides, fingers intertwined in her lap. She looked so much older than 40. Her blank eyes stared far beyond us. Her husband sat beside the bed, his arms folded over an expansive chest, his eyes vigilant.

There were nine of us standing by – a mixture of junior doctors, medical students and a nurse – as the consultant began reviewing Yvonne’s story. Her husband made frequent corrections. “No, it wasn’t just window cleaner, it was a bleach-based detergent. Yes, a colleague had tried to rinse her eyes but had wiped them with a wet towel, not used running water.” He said he had contacted the company who made the detergent and they had said it could cause nerve damage if absorbed by the eye. He’d like that documented. I exchanged a small smile with one of the other junior doctors.

The consultant asked Yvonne if her vision was good enough to distinguish light or dark, or make out shapes. She said she could sometimes tell that a light had been switched on, but that was all. He shone a torch into her eye and asked if she noticed any change. She said she couldn’t be sure.

At first, as she answered the consultant’s questions, Yvonne appeared to stare to his right, as if she could not tell where he was. But as the conversation developed, her eyes began to dart about, sometimes to his face or to her husband, then flitting back to a point in the middle distance.

We watched as the consultant worked through his examination. Yvonne’s pupils reacted briskly to light, just as they should. But when he asked her to follow the light with her eyes, she couldn’t do it. He took from his bag a small rotating drum painted in black and white stripes. He held it in front of Yvonne and spun it quickly. Her eyes flickered from side to side in response to it, involuntarily drawn to the spinning stripes.

As the consultant lifted the ophthalmoscope to Yvonne’s eye for the final time, she blinked. I heard a small thrill of a laugh somewhere in the room. Yvonne must have heard it, too, as she was startled. “Is there someone else here with you?” she asked.

The giggles were less stifled this time. Out in the corridor, one person whispered a shared thought: “There’ll be no Oscars for that performance.”

***

Yvonne was in hospital for a week, waiting for her tests. I spent a little longer with her than the other patients, because I was interested in her. She had grown up in rural Ireland, in a loving but sheltered home. At 20 she had married her first boyfriend, Gerald, who was 10 years older. Within 10 months of the wedding she’d had her first child. By the age of 30 she had six children under 10.

Yvonne had spent most of her adult life devoted to the care of her family. Her husband worked full-time. “We are a good team,” she said. It was only when the oldest children had left home and the youngest were all in secondary school that she first experienced what it felt like to have time on your hands. Encouraged by her eldest daughter, she decided to take a job in a local supermarket. Gerald had been against the idea, worried she’d have less time for the family. She assured him she would work only in school hours and that they would not even know she had been anywhere else.

I noticed increasingly that when she spoke, she looked me in the eye. It was more than a glance; I felt our eyes connect

During Yvonne’s stay in hospital, she came to recognise my voice and step, and greeted me warmly when I called to see her. As the days went by, I noticed increasingly that when she spoke, she looked me in the eye. It was more than just a simple glance; I felt our eyes connect.

Yvonne stayed in hospital for two weeks. On the day she was to be discharged, Gerald was there as the consultant gave her the results. Each test had been normal. The integrity of her visual pathway was intact, her brain was normal and we could not find any neurological cause for her problem. The only possible explanation left for her loss of sight was functional blindness, caused by stress.

“My wife hasn’t had a day of stress in her life. You’re talking nonsense,” Gerald replied.

It was explained again that all the avenues had been explored and there was no other possible explanation. Finally, her husband reluctantly agreed that Yvonne would meet a psychiatrist; if only to prove us wrong, as he put it.

I returned to the ward with Yvonne’s discharge letter. “I have something for you,” she said, and handed me a card. On the front was a flower-filled field overlooked by a single tree. It was drawn in coloured pencil. The words inside said thank you, it was nice to have somebody to chat to every day.

“I made the card,” Yvonne said.

“You made it!” I could not hide my surprise.

“Yes, I borrowed pencils and paper from the woman in the bed next to me,” Yvonne replied.

“But if you can’t see, how could you draw?”

“I can feel the pencil marks on the paper,” she answered. She did not seem in the least affronted.

Gerald appeared and led her away. I looked at the picture again. All the colours were correct – the tree green, the bark brown. Not a single outline was broken, not a single leaf or flower out of place.

***

One of the greatest challenges for most doctors is the struggle to believe in the truly subconscious nature of their patients’ psychosomatic symptoms. Pierre Janet, a French philosopher and psychologist in the late 19th century, was pivotal in the development of what we call the subconscious. He described consciousness as those sensory experiences and thoughts of which we are actively aware, whereas our subconscious is a place to store information not immediately available to the conscious mind.

Janet thought our consciousness could expand and contract, and choose what we perceived and what we ignored. There are examples of this in everyday life – looking for a friend in a crowd, for example. They are right in front of you, waving, but somehow you look right past them. “You must have seen me,” they say afterwards. “You looked right at me!” But you didn’t. For a moment, your mind employed selective attention and blocked something from your view.

Janet said that a separation between the subconscious and conscious could see memories and feelings exist in parallel parts of the mind, neither knowing of the other. This he referred to as dissociation, arguing that a psychological trauma could cause the subconscious to slink away so it was no longer available. In his model, it would be possible for Yvonne both to see and be unaware of seeing at the same time.

I saw Yvonne just once more, to arrange her psychiatry assessment. What little I would learn later would come in correspondence from the psychiatrist. These letters were factual, plainly written but always slightly cagey, as they have to be. Yvonne’s accident at work had resulted in an argument at home, the first letter said. Gerald had been called away from work to collect her from hospital. When he learned what had happened, he insisted she resign instantly. Yvonne, whose vision was not affected in the incident’s immediate aftermath, tried to plead otherwise, but he called her employer and offered her resignation. Later that evening the argument became moot when Yvonne discovered she had lost her vision.

The letter stated that Yvonne was struggling at home. Gerald had hired a housekeeper who did all the housework and helped with the children, but would not take responsibility for Yvonne. This left her feeling both superfluous and frightened.

After meeting the psychiatrist, Yvonne agreed to an admission to the psychiatric ward for intensive rehabilitation and talking therapy.

The final letter came months later:

I am happy to let you know that Yvonne’s vision has returned to normal. She has engaged well with treatment, although both she and her husband still struggle a little to accept the diagnosis in full. She has recently moved back to the family home and reports that things are going well. So much so that she has decided it would not be right to return to work.

Yvonne has crossed my mind several times since then, and I feel I did her a disservice. I was a fool to question her motives and insight because she had, suspectingly or unsuspectingly, told me exactly how things were when she handed me the card she had made. A woman who wishes to lie and fake wears dark glasses , carries a cane and stumbles about. That woman certainly does not draw a picture. Yvonne’s drawing was evidence not of guilt but of innocence, and, at the moment she handed it to me, it was I who could not see.

***

Shahina’s illness had begun six months before we met, following an incident at university. She had turned up late to her lecture one day, to find all the outer seats of the long, tiered rows were taken. Rather than pushing past her classmates, Shahina took off her coat and sat on a step behind a row of other latecomers, her hands behind her on the floor for support. For the next five minutes the door of the lecture theatre swung open from time to time as other students ducked in.

Shahina was leaning to one side, craning to see the lecturer, when she felt a crushing pain. She let out a loud cry and a titter went up from students nearby. The red-faced boy who had stood on her hand muttered an embarrassed apology. A tear fell as she held her hand to her chest. For the rest of the lecture Shahina could not concentrate, as she watched a large, dark bruise develop.

Shahina’s family had little sympathy for her that evening, when she complained of the pain and refused to help with household chores. They reacted differently when they saw how swollen and bruised her hand was the following morning. Shahina’s mother took her straight to A&E, where it was discovered that she had a hairline fracture of a metacarpal bone. Her hand was splinted and her arm was put in a sling. For three weeks Shahina could not use her right arm. She typed with one hand and used a Dictaphone to record lectures.

When the splint was finally removed, underuse had left Shahina’s hand thin and useless. The doctor arranged for her to meet a physiotherapist, who advised a series of exercises. She was glad to be able to use her hand again.

About two weeks later, Shahina was sitting in a lecture when she felt a cramp in her hand. Her pen slipped from her fingers and rattled noisily to the floor. She bent to pick it up, but found herself pawing uselessly at the pen as it slid between her fingers and rolled away. When the lecture was over, Shahina showed her friends her hand. The index and middle fingers were bending inwards. She could easily stretch them out flat with her other hand, but as soon as she let go, the fingers curled again, slowly. Her friends laughed when they saw it and for half an hour played the game of curl-and-uncurl with Shahina’s fingers. Shahina also found it funny, but only for a while.

When we met in the outpatient department one month later, her arm was in a sling and all four fingers were curled inwards, the index and middle ones completely folded, their nails hidden from view. When I tried to straighten them, there was resistance. It was painful but possible to draw the fingers outwards, so that the palm of the hand was bared. Four red welts were visible where the nails had burrowed into the skin. When I released the fingers, they sprang back into their coiled position.

“Have you ever seen anything like that? Do you know what it is?” her mother asked.

“It looks as if Shahina has developed focal dystonia,” I said. “It’s a condition in which people’s muscles go into spasm. It can be triggered by trauma, but we need to look for other causes.”

When Shahina and I met again, she had been admitted as an inpatient on the ward. Her hand was just as it had been when I saw her last. Blood tests, genetic tests and brain scans designed to look for an underlying neurological disease were all normal. But Shahina’s dominant hand was useless to her now, and she needed her mother to do up her buttons and cut her food.

A specialist neurologist recommended that she be given a therapeutic trial of botulinum toxin. Botox isn’t just for cosmetic use: it has long been used as a treatment for neurological disorders. In people whose muscles go into painful spasm, for whatever reason, it can paralyse the muscles so they relax. The paralysed muscle may then prove useless, but if it results in an improvement in pain and deformity, it can be worth it anyway.

I accompanied Shahina to have the procedure. A small needle electrode was placed in the muscles of her forearm. The electrode recorded the excess electrical activity that was produced by the furiously over-contracting muscles leading to her fingers. The computer converted the electrical activity to a noise, so that when the needle was inserted into Shahina’s arm, the room filled with a wild crackling sound. The doctor running the test leaned over and turned down the volume.

“Is that my arm making that noise?” Shahina asked.

“Yes.”

“Is that what it is supposed to sound like?”

“Not if you are trying to relax. If the muscles going to your fingers were able to relax, there would be silence.”

The doctor took a small syringe filled with botulinum toxin, attached it to the needle, and slowly injected. Shahina watched the computer screen. We were all listening. The static crackling that had been present since the electrode had been inserted was dying down. Shahina’s gaze moved from the screen to her hand. Her eyes were transfixed as her fingers slowly unfurled.

Do you really think my daughter could hold her hand like this for weeks? She's in pain. I know my own child

“It worked!” she cried.

The doctor who had given the injection looked at me with raised eyebrows.

That afternoon I went to the ward to see if Shahina’s improvement was sustained. I found her tapping away at the keyboard of her computer.

“It hurts and my hand feels a bit weak, but look how good it is.” She opened and closed her fist. “I’m cured, can I go home now?”

What I did next I would regret many times.

“I need to explain something to you, Shahina.” I sat beside her on the bed as I spoke. “Botulinum toxin poisons the nerve ending and the result is that it relaxes the muscles. But it doesn’t usually work instantaneously. It takes a day or two.”

Shahina looked at me, puzzled. She had not understood the full implications of what I’d said.

“But it worked straight away for me. That’s a good sign, right?”

“Yes, which is all that really matters.”

“OK, so I can go home.”

Shahina had offered me an opportunity to retreat and I ignored it.

“What I’m trying to say is that I don’t think it could have been the botulinum toxin that made you better. The recovery was too quick.”

Shahina was staring at her hand, watching her fist opening and closing in front of her.

“You think I’m mad?”

“Of course not.”

The mood in the room had turned. People passing heard the raised voice and looked in.

“I can’t fucking believe it! I came to this hospital with muscle spasm in my hand and now I’m being told I’m doing it on purpose.”

The next 10 minutes were delicate. I explained psychosomatic symptoms to her.

“Where does that leave me? Will my hand stay better? Will I need botulinum toxin again?”

“I think that since your hand is better now, it will probably stay better.” I was offering hope because expectations matter.

“I’m sorry I shouted.”

“I’m sorry you’ve had such a hard time.”

I left, understanding that Shahina would go home and we’d meet again in the clinic in the future. An hour later I was called back to the ward.

Shahina’s mother stood squarely in the door.

“My daughter has told me you said she was doing this deliberately. Do you really think a young girl could hold her hand in that position for weeks? There are welts on her hand. She’s in pain.”

“Shahina’s spasms of her hand are involuntary and very disabling, we all agree on that.”

“I know my own child. She is a bright girl, halfway to being a lawyer. She works hard, she is never ill. I guarantee you that if she had any control over this, it would not be happening.”

A week later, a letter of complaint arrived. It was printed on the headed stationery of the law office where Shahina’s mother worked. It ended by saying that Shahina had met another doctor, who had assured her that there was no possibility that the problem could be psychosomatic.

More than a year later, I received a letter from another neurologist at a different hospital, who informed me that Shahina’s dystonia had recurred and that she had responded well to a further administration of Botox, but that later the dystonic contraction had moved to her left arm and spread to her trunk. “I am beginning to wonder whether some, if not all, of her problem may be psychological in origin,” it said.

Even when a diagnosis of psychosomatic illness is delivered carefully, anger is a common response and it can prevent a patient getting the help they need. I have often wondered if the outcome for Shahina could have been different had I managed to communicate my suspicions in a better way.

***

In 2011, three GP practices in London identified 227 patients with the severest form of psychosomatic symptom disorder. These 227 constituted just 1% of those practice populations – but estimates suggest up to 30% of GP encounters every day are with patients who have a less severe form of the illness. If psychosomatic symptoms are so ubiquitous, why are we so ill equipped to deal with them?

Doctors should be less afraid of this diagnosis, more willing to confront it and more compassionate to sufferers. But for us to consider a psychological cause for serious illness, it is vital that we believe such a thing is possible. Maybe if we understood better the way our own bodies lose control, triggered only by a feeling inside, then more extreme reactions might not seem so unacceptable. Think about laughter: it is a physical display of emotion whose mechanism is ill understood; it is not always under our control, it affects our whole body, it stops our breathing and speeds up our heart; it releases tension and communicates feelings. If we can collapse with laughter, is it not just as possible that the body can do even more extraordinary things when faced with even more extraordinary triggers?

***

Matthew was a product of the internet age. When he came to me, his research had utterly convinced him that he had multiple sclerosis. Throughout our first conversation he kept using the words “my multiple sclerosis”. “Is my multiple sclerosis more severe than other people’s? How will my multiple sclerosis affect my life insurance?”

Matthew’s problem began with a feeling of pins and needles in one foot. At first it affected him only if he sat for prolonged periods. After having the symptoms for nearly two weeks, Matthew went to see his doctor. He was advised to take regular breaks at work and to avoid sitting for too long.

Matthew noticed his symptoms change and spread. The pins and needles were now moving around his body: one day in an arm, the next over the bridge of his nose, then in the back of his head, then his lower lip. He researched possibilities online. The internet advised him that diabetes could damage the nerves and lead to pins and needles. The doctor told him his blood sugar was normal. He read that trapped nerves were a common cause of his symptoms. A chiropractor wondered if he might have a disc out of place in his neck, but the treatment helped only briefly. Matthew began to exercise regularly. When this did not help, he tried resting as much as possible.

Nothing Matthew did made him any better. By now he was finding it difficult to work. Sitting for prolonged periods was impossible. His workplace had assessed his office space and made changes, but it made no difference. He cut down his working hours. At the same time he intensified his research. That was when he discovered that multiple sclerosis could cause sensory abnormalities that moved around the body.

Model: Sharifa Safiya at Ugly. Hair and makeup: Johanna Dalemo

Then, one day, Matthew awoke to find that he had lost all strength in his legs. He had no feeling and no movement. He was admitted to hospital, where he underwent a series of investigations which offered no explanation. A lumbar puncture took a sample of his spinal fluid, and that was normal. Blood tests and electrical studies of his nerves and muscles showed nothing wrong.

Three weeks later, after a fortnight in hospital with no answers, Matthew’s wife wheeled him into my office. Smartly dressed, with a sheaf of papers on his knee, he greeted me cheerily as his wife pulled up a chair and sat beside him. I asked him to tell his story from the start.

“I know I have multiple sclerosis,” he began.

“Let’s not make any assumptions,” I said.

His story was detailed, but he was strangely casual about his degree of disability for someone who three months earlier had been active and in full-time work. I examined him. Although he was in a wheelchair, he could walk short distances and moved, with difficulty, to the couch. As he lay down, I tested the strength of his muscles one by one. I asked him to lift his leg off the couch. He couldn’t. His face contorted with the effort. His right leg moved a few centimetres, then he cupped his hands under his thigh, to lift it further.

I can't believe it! I came to this hospital with muscle spasm and now I'm being told I'm doing it on purpose

“Point your toe,” I asked him next.

Matthew’s foot lay inert, but his face told me he was trying. When I pressed a blunt pin to his leg, he could not feel it. When I applied a vibrating tuning fork to his skin, he felt nothing below the waist. But the power to move or perceive sensations are things over which both the conscious and subconscious mind have some control, so next I tested the things where this is not the case. Despite the lifelessness of his legs, the reflexes reacted as they should. The tone of his limbs felt normal.

Matthew stood again and I asked him to walk. He locked his knees tightly and walked with straight legs and great difficulty. I asked him to sit on a chair and stand with his arms folded, and after several tries he managed it. He didn’t realise I had tested the same muscle groups in different ways: each time they had behaved differently.

He went on to have a further MRI scan of his brain and spine – the standard tests for MS – and an electrical study of his nerves. His neurological pathways were intact. When all the results were in, Matthew, his wife and I met again. I knew his concerns and tried to address them from the start. I explained how we make a diagnosis of MS and said none of his tests had shown any evidence of it. What fantastic news, I suggested; MS is a serious illness, but it had been ruled out. I could see Matthew’s face darken and, to his left, his wife’s shoulders heaved and I saw her roll her eyes.

“I know you are suffering, Matthew. I don’t want to detract from that. But you do not have MS.”

I said I wondered about a psychological cause.

“How can you say that? Just because the tests are normal, you assume I’m mad. That’s what doctors say when they don’t know what’s wrong.”

“There is more than just the tests,” I answered. “The weakness in your legs doesn’t fit with neurological disease. It should come with other clinical signs, altered reflexes or wasted muscles.”

“But it feels so real, it can’t be nothing.”

“It feels real because it is real. Your paralysis is not imagined, but that does not necessarily mean that it is a primarily physical disorder.”

“I don’t feel confident MS has been ruled out.”

I didn’t answer, and he shook his head

“For heaven’s sake, Matt, how many times does she have to say it?” his wife said, suddenly. “You don’t have MS. You came here for the doctor’s opinion, so why don’t you try listening to her?”

“OK.” Matthew agreed, reluctantly.

***

Six weeks later I met him again and was greeted by a transformation. I hadn’t yet had the psychiatrist’s letter and I was not sure what to expect, but, even from a distance, he looked more content. His wife walked behind the wheelchair as Matthew came into the room and, although she didn’t smile or acknowledge me, I thought I could detect a glimmer of something positive in her, too.

When Matthew told me how he was doing, there was only good news. His wife had encouraged him to meet the psychiatrist and carry out his own research, and somehow he managed to bring himself to an understanding that there might actually be some truth in my odd diagnosis.

Fantastic news: none of his tests had shown evidence of MS. I could see his face darken and his wife's eyes roll

“How did the meeting with the psychiatrist go?”

“Well, I didn’t like it at first but I realised he made a lot of sense. He said the nervous system is like a computer, that my hardware is intact, but I have a software problem that stops my legs receiving the instruction to move.”

Matthew had been given a label and an explanation he could relate to. For any illness, the first step to getting better is to accept the diagnosis, and Matthew had learned to do that.

With intensive physio and occupational therapy, Matthew was learning to walk again. He was keen to return to work and his employer allowed him to work from home until he had recovered. And with the intervention of the therapists, little pieces of him had been revealed. He was one of three brothers and they were a successful family, but Matthew never quite felt that he made the grade. He was as successful as his brothers but felt he had to continually work harder to hold on to his success. He was a man on a treadmill looking for a way off.

Now he had a prognosis. He could anticipate recovery. He could contact his workplace and tell them he had a functional neurological disorder, that he had to undergo intensive physiotherapy, but that he would get better.

Matthew had found his way out.

• Names have been changed. This is an edited extract from It’s All In Your Head, by Suzanne O’Sullivan, published on 4 June by Chatto & Windus at £16.99. To order a copy for £13.59, go to bookshop.theguardian.com or call 0330 333 6846.