There is a scene late in US author John Green’s new novel, Turtles All the Way Down, where his protagonist, Aza Holmes – a bright, troubled teenager, as Green’s heroines generally are – goes through an ordeal so distressing it is difficult to read. Aza has obsessive compulsive disorder, and the “tightening gyre” of her thoughts has taken over. Green catapults his reader right into the middle of Aza’s desperate mental state as she becomes increasingly panicky over the possibility of being infected with the bacterium C. diff – “do you want to die of this do you want to die of this because you will you will you will you will” – eventually scooping handfuls of hand sanitiser into her mouth, gagging and vomiting as her mother tries to stop her.

Green, who has OCD himself, says it was “very hard to write”. “I wrote it right at the last, in the last revision. That’s as close to my experience as I could get.”

Turtles All the Way Down, Green’s seventh novel and one that is long awaited by his massed armies of fans – he has more than 5 million followers on Twitter – is his “most personal”, says the young adult writer, “in the sense that it’s not entirely confined to the past for me. Having OCD is something that is an ongoing part of my life and I assume will probably be a part of my life for the rest of it.”

Green has “had this obsessive thinking problem” since childhood. For a long time it was hard “to talk directly about it, and I felt a lot of embarrassment”; part of writing it into his fiction was a desire “to find some kind of form or expression that would shake it out from within me and allow me to look at it”.

Aza – or Holmesy, as she is known to her best friend, Daisy – is Green’s narrator. The novel opens as the girls hear on the radio about the $100,000 reward which has been offered for a fugitive billionaire, Russell Pickett. Aza, it turns out, used to know his son Davis because they’d “gone to Sad Camp together … this place down in Brown County for kids with dead parents”. Daisy convinces Aza that they’ll be able to find Pickett and claim the reward, and the wannabe detectives hop into Aza’s old canoe and paddle down the river to the Pickett mansion.

While the novel is filtered through Aza’s perspective, OCD is only one theme. There’s also Daisy’s burgeoning fame as a Star Wars fan fiction writer, Pickett’s decision to leave his fortune to his pet tuatara (a kind of large reptile) and, of course, romance, friendship and high school.

It has been nearly six years since Green topped the charts with his sharp, funny, moving story of the romance between two teenagers with cancer, The Fault in Our Stars. That novel was inspired by the time Green spent working as a chaplain in a children’s hospital after university; it has now sold more than 23m copies, been adapted into a hugely successful film, and seen Green named one of the 100 most influential people by Time magazine.

When writing my second book, I was like, ‘I have no idea how to write a book.' I have to figure it out anew each time

No pressure, then, for the follow-up. “In one way it gave me the luxury of time, the ability to take as long as I wanted,” says Green. There were “a few abandoned novels along the way that got kind of scrapped for parts”. But “it’s also true that I felt very scared of writing, because I was in the shadow of this success”, he says, admitting to being “super nervous” about the next novel’s reception. “I remember thinking when I was writing my first book, ‘The great thing if I get the opportunity to do this again will be that I will then know how to write a book.’ Then when it came time to write my second book, I was like, ‘I have no idea how to write a book,’ and each time that was the case. I think that’s part of the reason my books have come out relatively slowly. I seem to have to figure it out anew each time.”

His first novel, 2005’s Looking for Alaska, was written while he was working as a publishing assistant and production editor at the book review journal Booklist, entering data and reviewing books and slowly realising this might be something he could do himself. He’d write at night and at the weekend, labouring for three years with his editor at Booklist, a children’s author herself, to turn it into something that was ready to send out to publishers. Drawing on his experiences at boarding school, the book is narrated by Miles Halter, who sets out to “seek a Great Perhaps”, in the words of Rabelais, at Culver Creek boarding school. There, he meets the beautiful, unsettled Alaska.

Breakout hit … Ansel Elgort and Shailene Woodley in The Fault in Our Stars. Photograph: James Bridges/AP

It didn’t make the bestseller list – that wouldn’t happen until 2012, seven years after it came out, once The Fault in Our Stars had boosted all Green’s backlist – but it did win him one of the US’s biggest young adult prizes, the Printz award. Green would go on to write novels including Paper Towns, about a boy’s quest to find the vanished Margo Roth Spiegelman, “the most fantastically gorgeous creature that God had ever created”.

His teenagers, whatever they’re dealing with, are bright, nerdy, articulate and funny, much like Green himself. He has no desire, at the moment, to write an adult novel – it’s clearly that period of flux, of burgeoning adulthood, which draws him in. “I was never very in touch with teenage culture ... what I’m interested in, or most interested in, is the emotional experience of being a teenager,” he says – how teens “approach those experiences without irony”.

Now 40, married with two children and living in Indianapolis, Green has memories “of really intense emotional experiences from high school that I draw on a lot. I focus on that and hope I will be forgiven for not understanding Snapchat.”

He also began, in 2007, a project that would end up bringing him one of the biggest followings among writers today. He and his brother Hank started sending each other video blogs on YouTube, talking about everything from haircuts to books to world affairs. One begins: “Good morning, Hank. It’s Friday, October 26th. Today I’m going to discuss the political situation in Nepal while eating five sheets of toilet paper.” They have made more than 1,000 videos, which have been viewed more than 600m times. The community call themselves the Nerdfighters – the name comes from Green’s misreading of an arcade game, Aero Fighters, as Nerdfighters – and take as their motto “DFTBA” (“Don’t forget to be awesome”). Through side ventures including Project for Awesome, which sees thousands of people posting videos advocating for charities, they have raised millions for charity.Green has always had “some kind of other job” while he writes, whether it was on Booklist, or writing for the trivia magazine Mental Floss, or more recently working on the educational video series Crash Course, which he launched with his brother in 2012. “I like not having to think of myself as only a writer because that’s sort of stressful,” he says. “I still don’t usually say when I meet people and they ask me what I do that I’m an author … I’d say that I make educational videos, that would be true, and also ends the conversation 99% of the time.”

Despite this, when it comes to the world of young adult fiction, it doesn’t get bigger than Green. With great success comes great scrutiny, and Green seems more aware of criticism than most authors. “I never think of myself as being very good,” he says. It’s hard, given his success, not to see this as disingenuous. But the way he picks over the issues in his books gives it the ring of truth: he knows perfectly well what critics dislike about his novels, and Turtles All the Way Down is, in some ways, an answer to them.

So what about the accusation that his teenagers are preternaturally smart? “First, I do think teenagers are smart, and we miss out by failing to listen to them or take them seriously,” he says. “I’m much more interested in how we sound to ourselves than how we technically sound.” And anyway, “in this book I try very hard not to [do this]. Aza hardly says anything. Her thoughts are very carefully articulated, but she can’t really articulate them in speech.”

He gives short shrift to those who accuse him of populating his books with a series of beautiful troubled girls. Paper Towns was, he says, written “very aggressively against” the trope of the manic pixie dream girl, while “I hope that Aza does not come across as a manic pixie dream girl – I don’t think she exists to solve Davis’s problems.” Her OCD, meanwhile, means it’s almost impossible for her to have a normal romantic relationship.

Green has addressed critics online, acknowledging what he sees as his weaknesses – “I tell too much and show too little. My plots are thin. I’m too fond of turning phrases” – while taking on those criticisms he sees as unfair: “all this crap about how I fetishise brokenness and lift up misogynistic constructions of young women and romanticise suicide is just, I think, totally unfair”.

And he has been very careful, he says now, not to romanticise Aza’s OCD. “I do think we stigmatise mental illness a lot in our culture … we don’t devote nearly enough resources to its treatment so people who can’t afford or access high-quality mental healthcare are doubly disadvantaged. I also think that we at times romanticise mental illness – the reason I wanted Aza to be nicknamed Holmesy is that, while I love Sherlock Holmes, my experience has been that my OCD would make me a terrible detective. In the first chapter, Aza doesn’t notice that her best friend has dyed her hair, she’s so deep inside herself. I do not feel my obsessiveness has come with a secret power … It comes with a complete inability to apprehend the world outside of myself unless I treat my mental health problems consistently and with care. So I really wanted to try and write against those stigmas and romanticisations.”

Aza’s core problem is one Green understands intimately. “She is forced to think thoughts that she doesn’t want to think and she’s unable to choose what she thinks about for extended periods of time. That can be really destabilising … If I am beholden to these thoughts, then what are the implications of that for me being a sovereign creature of free will, or at least enough of an illusion of free will that I can make my way through the day without having to feel this constant onslaught of fear that I’m just a passenger in this skin-encased bacterial colony that is my body?

“Aza’s problem is not only the obsessive thought spirals or the compulsive behaviour she uses to manage them,” he continues. “The problem is also that these thoughts really undermine her entire sense of self. That’s something I am pretty familiar with … I find it profoundly scary not to feel like I’m able to choose my thoughts, and it’s also just exhausting and infuriating: when it’s happening you feel like it’ll never end. I’m also really lucky that this is mostly treatable in my case and there are things I can do to make it better.”

“There have been periods when this has been a disabling part of my life and there have been periods when this has been a pretty small part of my life,” he says. Turtles All the Way Down is intended to show how “most people with chronic mental illnesses also live long, fulfilling lives”.

• Turtles All the Way Down is published by Puffin. To order a copy for £12.74 (RRP £14.99) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.