After each Olympics, golfers, tennis players and footballers return to their usual schedules, performing in front of vast and adoring crowds and often earning handsome sums. Swimmers, meanwhile, return to relative, or actual, obscurity after a few weeks among the world’s most scrutinised athletes – unless their personal problems or indiscretions become public knowledge, that is.

Michael Phelps, the most decorated Olympian in history, struggled with alcohol and went into rehab after two DUI arrests. Ryan Lochte’s 2016 went from a gold in the pool in Rio to a gas-station “robbery” scandal to a 10-month suspension to elimination in week eight of Dancing With The Stars. Ian Thorpe wrote a 2012 autobiography detailing his struggles with depression, while other Australian swimmers have ended up in prison or become mixed up in drugs. Whether this is due to the rigors of swimming is a moot point, but how much of a toll does the sport take on its athletes?

“The pressure is incredible,” says Matt Grevers, standing poolside at a meet in Austin last month. “It’s not like a basketball game where you have time to warm up, or you miss a shot you have the next one. Swimming? You miss your start, you have a bad breakout, even a couple of bad breaths or a bad turn, whatever it is, one mistake and you’re out of luck. You have to be so perfect and that can get to you. That’s the focus that can add a lot of pressure.”

Not that the heat was on the four-time Olympic gold medallist during the Arena Pro Swim Series event in Texas. There were perhaps a couple of dozen paying spectators in the stand at the University of Texas’s arena, suggesting the presence of more competitors than fans. This in a sports-crazed environment: a quarter-mile away across campus, the football stadium can seat more than 100,000.

But the absence of mainstream interest, in a way, helped enhance the 31-year-old’s point. Only 13 months earlier, Grevers set a world record in the 100m backstroke. The 6ft 8in American has six Olympic medals and plenty more prizes from world championships. In other words, he is a star. But the world only notices every four years.

And last summer, Grevers missed out on a trip to the Rio Games when he finished third in the US trials with a time that would have captured the silver medal in the 100m backstroke at London 2012. That hurt.

“You have to be at your best for that 52 seconds out of four years, otherwise you’re kind of out of luck, you don’t make the spot,” he said in Austin. “That one race every four years, you have to be perfect. Yeah, it adds a lot of pressure.”

Not only do elite swimmers face the mental challenge of having to peak for Olympic trials and finals every four years then cope with the drop-off, they have chosen perhaps the loneliest sport of all, doing grueling laps in the solitude of their lane, trapped in their thoughts as they perform rote actions for hours.

“I definitely think you have to be at peace with your own mind. If you have inner demons, swimming’s probably not the sport for you,” Grevers said.

“You have a lot of time to think about things. You’re obviously thinking about your stroke and that occupies a large part of your mental processing but there is a lot of time to think about life and what you’re doing outside. Swimmers have to have a good relationship with themselves, I think, to really perform well. You don’t really get to talk things through or distract yourself by talking to others. Even in the ready room usually it’s silence so even right before the race you’re not talking to someone.”

In 2008, the Independent reported that British researchers found aspiring Olympic swimmers as young as 11 were expected to spend up to 27 hours a week in the pool – five times as much training as children in other sports such as football. “A lot of people try to blame the burnout, or people dropping out of the sport early even though they have a great future, on the time consumption that it has. Swimming’s a low-impact sport so you can kind of punish your body more than other sports,” said Grevers. “I really think it’s more just being the mental side of things, not having as many social interactions. People get bored but I don’t, because I like my solitude in the water, it’s a meditative place for me.”

Eloquent, thoughtful and approachable, and a new dad, at least Grevers seems well-equipped to handle the pressures of swimming? “I don’t know I do deal with it that great,” he said. “I’d say this Olympic trials, 2016, my whole family said I was definitely acting a little different under that sort of pressure. It’s hard because at the professional level your livelihood, your family security, is now at stake. It’s not just accomplishing your goals, it’s supporting your family. There’s more and more pressure every year.”

Contrast that with the fact that swimmers’ sacrifices often go unnoticed. At the age of 19, Katie Ledecky is already one of the greatest athletes ever to dip a toe in the water. But, crude measure of celebrity and cultural penetration though it may be, her Twitter account has a relatively meagre 147,000 followers – about half as many as track the Brentford midfielder Josh McEachran. Last August, though, she was one of the most famous people on the planet.

“It is a bit weird. Me personally, I’m not a big fame guy, I don’t really want that. So the level you can attain if you’re not Michael Phelps in this sport is pretty acceptable to me,” said Josh Prenot, a 23-year-old American who won silver in the 200m breaststroke in Rio.

“I think that’s kind of a cool thing about our sport. Any other sport you’ve got like a big championship every year, the Super Bowl, World Series, what have you, so to have something every four that subjectively matters more but is still the same thing is kind of a unique aspect which I really appreciate.”

After the glare of the global spotlight and the fulfillment of a lifelong dream, it would only be human to experience some sort of post-Olympic lull. “It was a little hard for me because going into the Olympics, that’s just your ultimate long term goal is to be there. I’ve just been so hungry for that for so long and just pursuing every day, every workout, with the mentality of ‘I have to do this’. And then after achieving your goal, if you want to keep being hungry you have to reset your goal. And I did not do that for a long time,” Prenot said.

So he took a “six-month break not really from swimming, but from trying to get better at swimming – that’s given me the hunger again and made me realize how much I appreciate the process, the grind. It’s certainly not easy but it’s a fantastic job to have.”

Cullen Jones stepped away from the pool for six months after failing to reach his third Olympic Games. The 32-year-old New York native won a gold in the 4x100m freestyle in Beijing and another in the 4x100m medley in London four years later, along with two silver medals.

“After a couple of days, or weeks or months, [attention] does kind of drop off,” he said, speaking on a chilly morning at an open-air pool near Austin after giving a class to promote participation and safety among the African-American community.

“I think it’s a task for any governing body, for track, for swimming, to kind of keep us relevant, not only just for the sake of keeping the sport going, getting more people to continue the sport, but just to keep the fanbase and have people interested in it,” he said. “I think as long as social media’s out there we’re going to be OK.”

Jones keeps busy by teaching, studying and giving speeches, and stays motivated using psychological techniques: “I make things real for myself. In 2012, 50 freestyle, I wasn’t doing well going into that year, I wasn’t looked at as being even in contention to do that. But I put up pictures of some of the other swimmers around my house and whenever I didn’t feel like I wanted to practice or whenever I didn’t feel like giving it my all I’d either open up my locker or wake up and look at my ceiling and there was a picture of someone I’m gonna have to race against. And there they are in their proud moment, and it made it real for me – get up, do what you have to do, focus.”

He goes to the gym and took up boxing as a means of staying in shape without succumbing to the monotony of training – the 20-30 plus hours in the pool each week, typically early in the morning. “The mundane going back and forth staring at the black line was getting to me,” he said.

But Grevers believes that swimming’s special challenges make for a superior sense of achievement. “There’s such an individual accomplishment there, that you know you were responsible for that outcome. It wasn’t someone else’s abilities or plays or talents, it was yourself and you worked super hard for that,” he said. “I absolutely think there’s a higher high in swimming, or those individual type sports where you know you deserved it and you did all the work to make it happen and that goal you set four, eight, ten, 16 years ago, whatever it is, is finally accomplished. I can’t imagine anything else feeling as good as that.”