‘I don’t like to take medications,’ Hernando (not his real name) told me when I recommended a strong dose of ibuprofen for the pain in his thumb during an even busier day than usual in the primary care clinic where I work. I’m not his regular doctor so he tried to explain that he preferred natural treatments to manufactured pharmaceuticals. We can work with that, I thought. But then he said: ‘I’d prefer not to wear a wrist brace,’ a key component of treatment for his tendonitis.

This visit was not going how I had hoped. I was rushed and already 40 minutes behind schedule, hoping to catch up. But my appointment with Hernando required time to navigate his preferences and wrestle with the clunky electronic record system when he requested a letter excusing him from work. By that point, I was even more stressed than before the visit and visibly frustrated. I saw he was not pleased either. Ashamed, I hurried to see my next patient, thinking: ‘That didn’t go well.’

Like Hernando, most people want an empathetic doctor, someone who can prescribe medications but also someone who listens and understands. Yet patients rightfully complain that there just seems to be less empathy from doctors in clinics and hospitals. Researchers have found that medical students’ empathy scores drop through the course of medical education, and medical journals lament that physician empathy is on the decline. Most research looking to understand and solve the ‘empathy gap’ focuses on training individual physicians, but it could be a lost cause if the work environment remains unchanged.

The key resides in the nature of clinical empathy, which requires that the practitioner be truly present. That medical professional must be curious enough to cognitively and emotionally relate to a patient’s situation, perspective and feelings, and then communicate this understanding back to the patient.

At times, empathy’s impact seems more magical than biological. When empathy scores are higher, patients recover faster from the common cold, diabetics have better blood-sugar control, people adhere more closely to treatment regimens, and patients feel more enabled to tackle their illnesses. Empathetic physicians report higher personal wellbeing and are sued less often.

If the case for empathy is clear, the way to boost it remains murky indeed. New research shows that meditation and ‘mindful communication’ can increase a physician’s empathy, spawning a niche industry of training courses. Yet this preoccupation has missed the glaring deficits in the work environment, which squelch the human empathy that doctors possess.

This is the story you find when you talk to frontline physicians or read the myriad ‘quit lit’ that doctors and medical students write when they finally leave the field.

Most lament how the practice environment prevents them from caring for their patients in ways they had hoped and their patients deserve. ‘It became impossible to be empathetic at the bedside … while constantly being interrupted by nurses, physician call-backs, requests to get orders written using archaic and non-physician written EMR [electronic medical record] systems,’ said one commenter. They blame the time pressure created by a billing system that promotes quantity of patients seen over quality, the lack of control over the chaotic work environment, and endless time spent on administrative tasks. Perhaps unsurprisingly, many of these are the same factors contributing to physician burnout.

It’s the same story in my time with Hernando. We met on the toughest day of my week in clinic, and running behind was added to the usual clinic stress. I didn’t know him at all, requiring us both to spontaneously navigate our preference for alternative versus allopathic medicine.

Finally, burnout was a factor. I saw Hernando during the busiest time of my residency, a stretch of nine days off over 12 weeks, working close to 80 hours per week. I had no reserve to take care of myself, let alone my patients. Struggling with chronic sleep deprivation, my empathy circuits were shut down, a known crisis in current medical education. It’s not an excuse – I certainly hold myself to a higher standard than that appointment with Hernando – but I needed rest more than a course in mindfulness.

Despite anecdotal data from physicians, you won’t find studies in medical journals testing the empathy effect of less chaotic hospital environments or longer primary care appointments. It isn’t that empathy researchers ignore these environmental factors. In fact, many papers mention them in passing. There are just few examples of empirical studies in medicine.

But outside of medicine, a link between our environment and empathy has been known for four decades. The famous 1973 Princeton study of seminarians tested altruistic behaviour in a controlled stressful situation. The men were assigned to give a talk about either the Good Samaritan or about getting good jobs. Some were told that they were already late for the talk, others that they had plenty of time to arrive at the talk location. Along the way they had to pass through an alley where a fake victim was lying in distress, coughing and appearing to be unconscious. The question was: would they stop to help?

Seminarians – men who chose a religious life to help people; men who clearly knew the story of the Good Samaritan, and some of whom were primed to think about it – should all stop to help. But they didn’t. In fact, the main factor associated with providing assistance was not their individual personality or even being assigned to talk about the Good Samaritan. It was how much time they were given to make the trip.

Because of time pressure, some seminarians ‘did not perceive the scene in the alley as an occasion for an ethical decision’. Others did, and were noticed to be ‘aroused and anxious’ after their encounter with the victim. They had been asked to help and now were caught in a conflict between their commitment to give a speech or helping the victim in the alley. As the researchers put it, ‘conflict, rather than callousness, can explain their failure to stop’.

The implications are profound – time pressure can cause either a suspension of ethical decision-making or an internal conflict that overwhelms a person’s natural tendency to help those in need. This happens every day in the clinic and in the hospital, where medical professionals, many of whom were driven to medicine by a desire to help people, are tasked with helping two masters – the distressed patient in their office and a system that demands seeing more patients, faster. It creates an internal conflict that might explain what appears to be callous behaviour, where doctors appear to be missing numerous cues from patients seeking empathy. It creates an environment where, as the Princeton study concluded, ‘whether a person helps or not is an instant decision likely to be situationally controlled’.

This is no surprise after the recent discovery of ‘mirror neurons’, thought to be the neurobiological centres for empathy. Mirror neurons fire when we observe people, and can unconsciously reflect emotions we perceive in someone else.

But mirror neurons are fragile. Exposure to indifference or violence in childhood blunts their growth. Even adults with fully functioning pathways can have their mirror neurons shut down in high-stakes situations governed by fear and stress.

Recent psychology and anthropology studies are only beginning to shed light on how social context affects our capacity for empathy. Empathy is not a constant, inherent trait but one that varies over the course of a day or a month, fluctuating with our social situation, mood and self-esteem.

But these developments in social science have yet to make their way to medicine. We need to understand the contexts in which clinical empathy naturally blossoms. We should study the impact of lengthening appointments or providing higher reimbursement for time-intensive counselling discussions. Instead, we require busy clinicians to train for empathy without changing their work stress, and we blame doctors alone for the empathy deficit. Until hospitals, clinics and insurance companies take environmental factors seriously, patients will suffer and physician empathy will continue to decline.