Tom Nicol thought he had a problem with sleep. He could never get enough. He took “a very disciplined, stripped approach” to his routine. He drank water only at premeditated times, ate according to schedule, avoided caffeine, exercised (but not close to bedtime) and shut down all screens at 9pm. Nicol, a PhD student, was recounting this long list of sleep settings to his student counsellor after yet another bad night, when she told him: “You have perfectionism.”

“I’m not good enough to have perfectionism,” Nicol replied.

It was “one of the most perfectionist things you can say”, he says now. At the time, though, the discovery took Nicol by surprise. He shared his surprise with his partner. “She was like: ‘Well, duh!’” But he needed to be convinced.

On the phone, Nicol, 25, has a persuasive and clear-eyed sense of his own averageness. He is “not particularly” industrious, “quite” messy and has “never been the kind of person who was seen as one of the top achievers”, he says. He recounts these perceived shortcomings with an amiable ease that sounds a lot like contentment. But maybe this, too, is a perfectionist sleight of hand, to present persistent self-criticism as casual self-deprecation. I arrange to visit Nicol at the University of York, where he is in the second year of his doctorate in theoretical chemistry.

Perfectionism can affect people of all ages and lifestyles, but it is increasingly prevalent among students. Earlier this year, research involving 40,000 students at universities in the UK, the US and Canada found a 33% increase since 1989 in those who feel they must display perfection to secure approval. The report’s lead author, Thomas Curran of the University of Bath, fears a “hidden epidemic of perfectionism”.

Perfectionism is a personality trait rather than a mental health condition. There is no World Health Organization diagnosis code for perfectionism and it is not listed in the American Psychiatric Association’s Diagnostic and Statistical Manual of Mental Disorders. It can fly under the radar and masquerade as the pursuit of high standards, yet it overlaps with a plethora of disorders from eating to obsessive-compulsive disorder, anxiety, body dysmorphia, depression and suicide.

‘Even exercise doesn’t sound like a total escape for Tom Nicol: he is training to beat his dad’s personal best mile time.’ Photograph: Gary Calton/The Guardian

Nicol’s corner of the library is a corridor with a table pushed against the wall. There is a fire extinguisher at his elbow and a mystifying electrical hum emanating from the book racks. But there is none of the mess that Nicol promised. The desk is tidy. The notebook is neat. The to-do list has subsections. “There’s a difference between orderliness and industry,” Nicol says.

He defines perfectionism as “a fear of failure”. His fear is all the greater not only because he believes in perfection, but also because he thinks he has experienced it. In 2014/15, Nicol had what he calls “my quote unquote perfect year” in which he lived perfection so thoroughly that he could tell it by the minute. Each day, he hit the gym at 8am, worked from 9am to 6pm, fixed his food for the next day and went to bed. He did this five or six days a week. On the seventh, he graded his consistency and productivity by calculating the completion rate of his to-do list. The reward for all this endeavour was a score of 86% in his master’s dissertation (although his sister has just got a 95).

However, since starting his PhD, Nicol has been unable to replicate the conditions that underpinned his success. He is sleeping poorly; he cannot hit the same stride.

“Every day feels suboptimal,” he says. Nicol knows that one of the reasons he excels – not a word he would choose – in chemistry is his ability “to pick things apart conceptually”. Applied to his own behaviour, though, his talent turns against him. A merciless “mental punishment” ensues.

On one hand, Nicol is philosophical about the elusiveness of perfection. “The chaotic nature of our world means that it’s going to be extremely transient and even if you attain it for a moment, if you’re a striving kind of person, you’ll want the next thing,” he says. On the other, he castigates himself for falling short.

“There is no excuse. If I slept for two hours the night before, I still berate myself for not being able to work, even if my brain feels like treacle.” Nicol battles “the mental difficulties” of perfectionism, “the cyclical nature of ‘You’re terrible!’”. Sometimes he will be “thinking about things” while cleaning his kitchen “and they’ll boil up and out loud I’ll say: ‘I’m so fucking useless!’” The thought impedes productivity, which in turn incites the thought. Even exercise, which Nicol enjoys as “an escape from my brain”, doesn’t sound like a total escape: he is training to beat his dad’s personal best mile time of four and a half minutes – a habit, perhaps, learned growing up in “an air of competitiveness”.

All this sounds familiar to Roz Shafran, the co-author of Overcoming Perfectionism and the chair in translational psychology at University College London’s Institute of Child Health. She sees self-beration as a defining feature of perfectionism and often detects a “flipping point” among students: starting university, for instance. “The domain changes, or reaction to failure changes. For many people, it starts off healthy,” she says.

But there is no consensus on this. Paul Hewitt has worked in the field for 30 years. In a groundbreaking article in 1991, he laid out, with fellow psychologist Gordon Flett, three types of perfectionism: socially prescribed (in which a person believes that others require them to be perfect); other-oriented (in which a person requires others to be perfect); and self-oriented (requiring oneself to be perfect). Hewitt sees perfectionism as a “personality style”.

Research found a 33% increase since 1989 in those who feel they must display perfection to secure approval

He disagrees that conscientious striving can turn into perfectionism. “That’s not perfectionism,” he says. “They are two very different things ... Perfectionism is about attempting to correct or deal with a defective, flawed, not-good-enough sense of self.”

Kirsty Schafer, 23, is two years into a degree in social work at Bournemouth University, having switched from Bath. We meet in a cafe near Bournemouth train station. Over a coffee, she explains how she is “on the cusp of leaving perfectionism behind”.

For long periods of her degree, Schafer has exited the library at about midnight, with barely 100 words to show for hours of work. “My brain feels like it’s been punched and punched and nothing comes out,” she says. The problem is that the words are not perfect. Each could be better. Hours pass as she retypes a single one. The words she takes home “are blue in the face from where I’ve forced things”.

On the worst nights, “I’ll wake up and my head is whirring”, she says. In the most intense weeks of work, she forgoes sleeping, eating and washing. It may be out of self-preservation that she switches to the second person to talk about this.

“You neglect absolutely everything. You become absorbed by your own brain. Physical things – eating, showering, not going to sleep at the normal time, not looking after yourself …” She trails off. “When I’m in that state of mind, I just can’t see anything on the outside. I’m in my head constantly.”

And yet – eventually – Schafer produces essays that meet her idea of “a perfect 10”. While many students with perfectionism don’t meet or are derailed by their targets, Schafer often scores a first. At some point in her writing, she senses that the job is done. She cannot say how. There is no line to demarcate an imperfect work from a perfect one. All she knows is “something inside releases its clasp”.

Looking back, Schafer can see that she was prone to perfectionism from a young age, although she is unsure why. “My parents were the most supportive parents ever. I’ve never been subjected to criticism. If anything, it’s always been really high praise.” When her primary school teachers called her a perfectionist, she took it as a compliment. But somewhere between the ages of 15 and 16 her experience intensified. University loomed. Friendships were changing. Schafer clicks her fingers. “I became very focused.” She began to build an increasingly punishing academic schedule. “I focused it all on myself, so the only person who could be disappointed was me. I was the only person I could let down, the only person I could punish.”

For Schafer, and many others, “perfectionism serves some kind of function”, Shafran says. “Whether it’s control, getting good results at school, whether you don’t have to socialise because you’re shy … It serves a purpose.”

This usefulness can make perfectionism all the harder to leave behind. “I really, really thank my perfectionism,” Schafer says. “I can see the value of it. But I don’t need it now. It’s just this thing I’m dragging along that is holding me back, because there’s nothing I need to control any more.”

Schafer speaks eloquently of her perfectionism as an attached but distinct entity. At times, it is a heavy appendage; at others, a persistent and insinuating parasite. “There is something in my head, like a little worm, that just talks to me constantly and is very critical. ‘You need to do better at this, you need to do more, you haven’t pushed yourself enough.’” The challenge is “to get control of that little worm”, she says. But how?

Liam O’Dell, who calls himself ‘a person of routine’, tracked every mark of the final two years of his degree on a spreadsheet. Photograph: Teri Pengilley/The Guardian

Linda Blair, a psychologist, recommends that her clients try “the best friend test”: offer yourself the advice you would offer to a friend with the problem. For Schafer, this is easy: a close friend also has perfectionist tendencies and Schafer knows exactly what to say. “‘Your best is your best.’ That is the only advice I would ever give.” Her friend says the same to her. “And it doesn’t work! We laugh about that. We think: ‘I could really do with listening to that advice.’ But … it just doesn’t go in.” Schafer thinks perfectionism is “a disease of the self. Although ideals and the media feed into it, it is ultimately controlled by me. And that silly little voice in my head. The worm.”

So, why is perfectionism thriving? Curran, Bath’s lead researcher into perfectionism, has experienced it himself (“more imposter syndrome, because I come from quite a poor background”) and largely holds neoliberalism responsible. “A marketised form of competition has pushed young people to focus on their achievements,” he says. In the UK, tuition fees have exacerbated the marketisation of education, along with a focus on results – even for seven-year-olds sitting Sats. And, of course, social media has made everything seem performative – and perfectable.

While Nicol emphasises his middling qualities and berates himself for his averageness, Liam O’Dell, who has just completed a degree in journalism at the University of Lincoln, strives for exactly that. For a long time, O’Dell, who is mildly deaf and has dyspraxia – which affects co-ordination, spatial awareness and sensory perception – felt uneasy in group situations. He spent school lunchtimes in the library. His heartfelt attempts to “be socially perfect” centred on not standing out.

“I was striving to put on my best impression of what was deemed normal,” he says. Just as Schafer knows there is no perfect, O’Dell knows “there is no normal”. But socialising was arduous. He was “always thinking: ‘What’s the tone in this conversation? Is this appropriate to say?’” A barrage of questions seems to be the perfectionist’s rhetoric.

O’Dell, 21, and I meet while he is waiting to learn his degree result. He has high hopes, having tracked every mark of his final two years on a spreadsheet. Like Nicol, O’Dell is “a person of routine”.

Perfectionism is about attempting to correct or deal with a defective, flawed, not-good-enough sense of self Paul Hewitt

On the occasions when his work has been given a 2:1 or 2:2, he has appealed. “I’ll be like: ‘It could be better, it could be better. What did I do wrong? What mistake did I make that I missed?’”

We are drinking coke by a canal in north London. In the middle of recounting all this, O’Dell looks up and says: “I’ve come to accept that perfection can’t ever be achieved. I’ve done my best. It may still not be perfect, but I know that, in trying my best, I’m as close to perfect as I can be.”

But that is the holy grail of perfectionism, I say. When did he get there, I ask?

“I think I’ve yet to reach it, to be honest,” he says.

The path ahead is unclear. Schafer has found cognitive behavioural therapy (CBT) helpful. Perfectionism “is me, my problem”, she says. “I’m the one who needs to seize it and say: ‘Right! Enough!’”

But I wonder if this determination to self-cure – the belief that, with the right information and application, one might master one’s own shortcomings – is not, in fact, perfectionist. Schafer and Nicol, for instance, are well read in – and perhaps a bit perfectionist about – perfectionism. They are not so much people with a problem as expert witnesses. The quest for a solution plays to their conscientiousness.

“That’s an approach that many perfectionist individuals have,” Hewitt says. While Shafran has seen clients benefit from CBT and Hewitt believes it can reduce cognitive elements of perfectionism, he think it is less useful for tackling “the more ingrained personality questions”. There is, he says, “a bit of a hint that there might be some genetic component to it”. He advocates a form of “psychodynamic psychotherapy” in which perfectionism “evolves as a result of relational experiences”.

“If I had to get right to the crux of the issue, it would be working with people to have self-acceptance,” Hewitt says. “This is who I am. I am on this earth once. If I can accept myself, with my strengths, my abilities, my liabilities and just get on with living my life rather than trying to evaluate myself and coming up short all the time ... That’s really what we focus on.”

Nicol has benefited from counselling. “I suggest facing it, in any way you can, and trying to understand it. Running away doesn’t help,” he says. Schafer, aided by a tutor, is trying an alternative composition method she calls “word vomit” – out they all come on to the page – and is producing “more authentic” work. O’Dell has been awarded a first for his degree. “I do feel my hard work and perfectionism have paid off,” he says. He learned his result only two days ago and already his happiness has been intercepted by a familiar thought. “There soon comes that feeling: ‘OK, now I need to focus on the next task,’” he says. “It’s a short-lived relief.”