In the summer of 2013, Stephen Baxter and Terry Pratchett published The Long War, the second volume of their Long Earth science-fiction series, about parallel worlds that can be “stepped” into. By the end of that year, the two authors – both prolific by any standards – had completed drafts of the remaining three novels in the series. It was an astonishing rate of work, but there was a deadline that needed to be met: Pratchett had announced his diagnosis with a rare form of early onset Alzheimer’s in 2007. By the summer of 2014, he would pull out of a Discworld convention, citing “The Embuggerance”, which was “finally catching up with me”. He died in March this year.

“I think Terry was aware he was running out of time, and he wanted to do other things as well,” Baxter says. “So we rushed through it a little bit. Terry’s basic vision was the first step, but he also wanted to have a huge cosmic climax at the end, which would be book five ... We had no idea how to get there but we knew where we were going.”

The Long Utopia, the fourth in the series, sees settlers on an Earth more than “a million steps” west of ours stumble across a disturbing, insectile form of alien life. Like its predecessors, the novel is compelling not only for its central storyline of exploration and danger and humans doing foolishly human things – and in this case a particularly cataclysmic finale – but also for its slow, unhurried laying out of the minute differences between these empty-of-humanity Earths.

The concept of a chain of parallel worlds, each a little different from its neighbour, was one Pratchett originally had, and set aside, in the 1980s. He told Baxter, a long-time friend and one of the UK’s most respected science-fiction authors, about it over dinner one night, and they decided to collaborate.

“It was a great idea but Terry’s strength did not lie in landscapes and things,” Baxter says. “He’d get a story by having a basic idea, get two people in a room talking and see where it went from there.”

This is not how Baxter works. His fiction, whether about the colonising mission sent to a planet orbiting a nearby red dwarf star, in Proxima, or the exploration of different evolutions of humanity in the Destiny’s Children series, is meticulously planned and pinned down, rooted in the scientific background from which he comes. He has a degree in maths from Cambridge and a PhD in aeronautical engineering; he is a fellow of the British Interplanetary Society and applied for a guest spot on the Mir space station in 1989, making it through a number of stages on his quest to be a cosmonaut but eventually missing out because of his lack of foreign languages.

Whether Baxter decides to submerge the world (Flood), or make humanity live in the centre of a neutron star (Flux), or keep the sea off Doggerland in an alternative prehistory (Stone Spring), there’s always a hook into something real. “I try to get it right. If you can get the maths right, I figure you’re most of the way there,” he says.

Baxter is fiercely intelligent, in a generous way, sharing his enthusiasms and knowledge on everything from recently discovered exoplanets to the Mars project (he’s not hopeful, because he doesn’t think enough has been done on long-term life support systems). At the British Interplanetary Society, he’s been part of study projects on everything from designing star ships to extraterrestrial liberty, an issue explored in Ark, his follow-up to Flood, in which the scraps of humanity flee their devastated planet in “generation ships” for an uncertain future outside the solar system.

“It’s all very well to plan a five-generation mission to Alpha Centauri, but if you’re one of the middle generations, you live out your life with very little room for manoeuvre,” he says. “So what right do you have to submit your children and grandchildren to a life of slavery like that? You get some interesting ethical issues – do you have rights over people who don’t yet exist, do they have rights?”

Terry Pratchett and Stephen Baxter in 2012. Photograph: SFX Magazine/PR

The Long Earth, Baxter says, has “a sort of basis in science: it was Terry playing with the idea of quantum mechanics, quantum parallel worlds”. That’s where Baxter came in. “I tried to think that the further you go the stranger the worlds get, so I’d try to put numbers on that,” he says, adding, with a twinkle: “I wouldn’t say we clashed, but he’d like to joke about how I turned up with a box file full of spreadsheets. It’s a lie – but it was something like that. I did turn up with a big chart about how this might work, and a time scale ... So you had this mix of order and chaos. I think it worked well in the end. He needed the order from me but I needed the chaos from him to make it more chaotic overall and more interesting in the end.”

Baxter’s response to working on this gentler side of science fiction was to write Proxima, and its sequel Ultima. “It’s very hard SF,” he says. “That was partly in reaction to The Long Earth ... I thought I’d do something entirely different, so we’re off to the stars.”

The final time Baxter and Pratchett saw each other was late last summer when Baxter went down to Pratchett’s home in the Wiltshire countryside from Northumberland “and we just talked. Terry had buzzing round in his head a new plot strand for book five, and we talked through that – we didn’t do much writing that time,” he recalls. “It was giant trees. He had this vision of trees, five kilometres tall, as tall as Everest, on one of the parallel worlds. Just remember that.”

Softly spoken and courteous – you can picture him as the maths and physics teacher he used to be – Baxter has a mind so full of universe spanning ideas that, says his fellow science-fiction author Alastair Reynolds, “one minute he’s off rebuilding the entire cosmos, the next it’s mammoths or the Romans or ancient Britons”. “In a SF and fantasy world full of writers with a shtick: big hats, black jackets, big mouths, multiple piercings, fierce tattoos,” adds the author Jon Courtenay Grimwood, “Stephen Baxter’s shtick is to be so entirely normal he could pass for a fan among fans, which is weird for someone who casts so large a shadow.”

An SF reader since the age of 12, Baxter began writing his own stories a couple of years later. He carried on submitting his work to magazines all the way through his degree and PhD (he’d considered astronomy or astrophysics, and says relativity is “a good concept but deadly dull technically”), teaching in a sixth form college and a later career in computing and project management. In 1986, his story “The Xeelee Flower” was picked up by Interzone magazine. It features an astronaut stranded in space as a supernova approaches; the astronaut discovers an alien gadget, which grows, like a flower, very quickly, and he uses it to surf the supernova wave. The story focused on one astronaut but, Baxter says, “you had an alien race who put the gadget there, and another who’d stranded the guy there, so there was a sort of implicit universe”.

He sold more stories, and his first novel, Raft, in 1991, but it wasn’t until 1995 that he gave up his day job. By then, he’d published six novels, mostly in the Xeelee sequence, which now covers a mind-boggling time span starting more than 13bn years ago and running to AD5bn, when the Milky Way collides with Andromeda. Fans should note that he’s planning another in the series, and is promising big revelations. “You never see the Xeelee. They’re off stage, so at some point there has to be a pay off.”

The game changer was his sixth novel, The Time Ships, a sequel to HG Wells’s The Time Machine. Baxter had spotted the 100th anniversary of the original publication, and decided to write the sequel that he had longed for as a 15-year-old himself. Authorised by the Wells estate, it sees the Time Traveller embarking on another journey to the far future – precisely, to Day 292,495,940, when he lost his Eloi friend Weena to the Morlocks.

It won Baxter a host of awards and some very high praise. “I’m almost tempted to say (I know this is blasphemy) that the sequel is better than the original,” said Arthur C Clarke. New Scientist put him amid the ranks of Clarke, Robert Heinlein and Isaac Asimov, writing that “the reaction is that which CS Lewis referred to when he described science fiction as the only genuine consciousness-expanding drug”.

Clarke had supported Baxter from the start, providing a blurb for his first novel, and after The Time Ships they entered into regular correspondence. They collaborated on a short story, and went on to write four novels together. “With Clarke, his way of working seemed to be to have outlines, like the first two chapters of the book, but leave the ending open, so all the plot threads were coming out of the set-up or the gadgets,” says Baxter.

SF, Baxter reflects, is “like an expression of the concerns of the time, hopes or fears” – Wells’s, he points out, was the first generation to be taught evolution, by a generation that had grown up largely with the Bible’s version of creation. “It must have been a huge shock: you’re lost adrift in this mechanistic universe, as opposed to the security of a religious universe. So if you look at The Time Machine, you see the awful outcome of a future descent,” he says. “So it’s like therapy, you’re telling a story because you’re frightened of the future.”

Climate change is the issue being grappled with now – he’s tackled it himself in Flood, Ark and the Stone Spring books. He suggests that the plots of huge young adult bestsellers such as The Hunger Games and Divergence are metaphors for the topic, “sublimated and changed and embodied in these evil rulers”.

He smiles, before adding: “And sometimes it’s just escapism, let’s be honest.”