When he was shooting his latest movie, Good Time, in Queens last year, Robert Pattinson would start the day with a run. And he'd be recognised, as always. Such is life for the 31-year-old actor formerly known as Edward Cullen, the broody vampire in the Twilight movies. Over five years and five films, he inspired such a vast and hysterical following that more than any star of his generation he became a prisoner of his own celebrity. He was forced to sell his home in Griffith Park, Los Angeles, because of paparazzi at the gates. They trailed him everywhere, entailing all kinds of Jason Bourneism, like swapping clothes with friends and assistants in restaurant bathrooms, sending them off in decoy cars, up to five at a time. And if he failed, if just one tweet went out with his location, then armies of paps and Twi-hards, crazed and shrieking, would come galloping over the horizon like the Dothraki hordes.

But after each run, something extraordinary happened. He got into costume as his character in Good Time, Connie Nikas, a Greek-American criminal from Queens, and just like that, the staring stopped. He could walk down the street unmolested. This latest film is his best performance by some distance, an electric, adrenalin shot of a movie that will establish him as one of the most vital actors of the day, so there's that. But this gift of anonymity may be equally precious. Good Time will put Pattinson's name in lights while simultaneously helping him blend into the background. Shooting it gave him his life back. It's handed the prisoner a set of keys, because as Nikas, Pattinson could move through the world again. He was free.

"It was amazing. Invisibility cloak," he says. "I've always wondered what can you do, just a simple thing to your face so you can just… exist in the world. And now I know. Darken your beard and put on these acne scar things and people will look directly into your face, and not even a glimmer. It's fascinating. Also earrings, there's something about fake diamond earrings."

He looks a bit Connie Nikas today, actually. We're in a booth at a private member's club in West Hollywood, and he's wearing a sports jacket on top of a hoodie, never mind that this is the height of summer. The jacket's Lacoste; very hipster I tell him. And he laughs.

"Is anyone not a hipster now? I think it's just normal culture," he says. "Anyway, I found this on eBay so, you know... I'd be cool if I had it from school, like, 'I've had this for aaages. I still dress exactly like I did when I was 12.' Ha ha ha!"

He looks happy, energised, garrulous. The hands move around, the Lacoste rustles, he's chewing on a toothpick and tipping his head back to laugh and laugh. He looks like a guy who made a bet on himself and won, which he did. And this is what he's here to tell us: chase what you want in life, take the risk, who cares what people think in the end. This is your life, not theirs.

Black leather bomber jacket, £6,400; white cotton T-shirt, £360; white denim jeans, £490; black leather boots, £1,350, all by Berluti. Black cotton socks, £12, by London Sock Company Carin Backoff

Fashion by Catherine Hayward

The last time I saw Pattinson for Esquire, three years ago, he'd only just made that bet. He came over to my house for lunch, and we got the barbecue going, there were beers — things celebrities never do — and we talked about The Rover, a film he made with director David Michôd (Animal Kingdom). It was his first major step on the route away from Twilight and towards Good Time, a life that he actually wanted. He'd made a pact with himself to only pick roles that were unlike anything he'd done before, that would broaden him as an actor and human being, and to work only with film-makers he loved, with no compromise. So post-Twilight, his CV is just one auteur after the next, in a string of movies that don't make money but are always compelling. Besides The Rover, there's his second film with David Cronenberg, 2014's Maps to the Stars; The Childhood of a Leader directed by his friend Brady Corbet; The Lost City of Z with the film-maker's film-maker, James Gray, not to mention the Safdie brothers, Josh and Benny, who made Good Time.

Back in 2014, he was living next to rap impresario Suge Knight in a gated community on Mulholland Drive, still in hiding from Twilight fans. It was a secluded life, with just an inflatable boat and an assistant for company. "Aww, I miss my assistant," he says. "He's now a real estate agent in Phoenix. Couldn't take it any more. 'All you do is play video games!'" Most of Pattinson's time was spent in one room, watching films and reading books, much as it is today.

"Probably my fondest memory from that house is watching the first three seasons of Game of Thrones over four days." He laughs. "So lame that's my fondest memory!"

He dreamed of escape. #Vanlife on Instagram became an obsession, posts celebrating the nouveau hippy world of attractive young surfer types living the free-spirit life in camper vans, free of all material possessions beyond a hammock, a book of poetry and a mobile phone to upload selfies to madden people in cubicle offices.

"I nearly did it," Pattinson says. "I was 100 per cent going to live in a van, but not just any van — a stealth van! It's a special niche, not like living in a trailer. Stealth vans looks like a normal Transit van, so you can park on the street, put signs on saying you're a plumber or whatever and no one would notice."

Van life promised anonymity, freedom, mobility: all the things he missed and wanted.

"It's that thing, where you can just leave in the middle of the night and, like, drive to Nebraska," he says. "And with solar power, you're totally off the grid. I'd love that so much. And I was like, I'm still young, this is my chance..."

So he looked into it. The Mercedes-Benz Sprinter looked tidy; it had a toilet and shower in the back. But no.

"The Sprinter's too fancy. It draws attention. So I visited different companies to retro- fit Transit vans but it's complicated," he says. "Once you build [in] a toilet and shower yourself, you can't get it insured and blah blah blah."

Still, he hasn't ruled it out. One day, maybe. For now, though, instead of Nebraska, he moved five minutes down the road, to another secluded mansion in the hills. Only this time it's not quite such a Spartan existence. He lives with "Twigs", aka FKA Twigs, the British singer, and their little dog Solo. He won't talk about her, though they may be engaged after three years together. And one can't blame him; the Twi-hard fanbase has already subjected her to a torrent of racist abuse. Which is partly why they spend half their time in London, out east near Hackney Downs (hipster level: high). Pattinson gets hassled much less back home. "I go around on my bike," he says, "so I'm basically a ghost."

He was deep into #vanlife when he saw a still from the Safdie brothers' movie of 2014, Heaven Knows What. It was just a close-up of the actress Arielle Holmes in a pink/blue light, her eyes sunken and strung out as if on heroin; she was playing a homeless junkie, a life she'd led until Josh Safdie approached her in a Manhattan subway and asked to make a film about her. The realism was palpable. And Pattinson was hooked at once: he had to work with these people.

"It was so cool, this photo, it had an amazing vibe, but also they're American. Normally with an image like that, the director turns out to be Czech or something," he says. "And my agents hadn't heard of them either, so I knew I'd found something before anybody else." This is what Pattinson loves more than anything — making discoveries.

Without even seeing the movie, he wrote the Safdies an email rich with compliments, a tried and tested ploy. "I basically say, 'Look, I'm not playing. I like very little and I like this thing you did, I think you're good, and I just... know!' And after that I call repeatedly."

He's done this with James Gray, with acclaimed French film director Claire Denis (who's writing and directing his next film High Life). It's a winning strategy. "I realised about four years ago, this is the best way to do it. I don't even tell my agents."

At first, Josh Safdie was hesitant. He was working on a movie about New York's diamond district and Pattinson just wasn't right for it. But they clicked, and once they met up, Josh saw something: "He has a wounded war veteran vibe to him, like there's a major trauma in his life and he's constantly trying to hover, trying not to be seen. I thought that was perfect for a guy on the run." So the Safdies created a project for Pattinson, essentially writing him a movie.

"The thing about Josh and Benny," Pattinson says, "is their energy and drive. It's astonishing. And that's how their movies feel, like there's too much fuel in the car! I wanted that energy, something superkinetic. A lot of the stuff I'd done before was reactive, so I wanted to be forced into a situation. That's their tone: runaway train. Their genre is literally panic. And that's kind of who I am as well. So I said, 'Just push push push, be as audacious as possible.'"

The story centres around Connie, a sociopathic street criminal who can't stand the thought of his mentally challenged brother Nick — played brilliantly by Benny Safdie — being institutionalised. So Connie takes him on a bank robbery, the first of several terrible decisions, each one cascading chaotically into the next. It's a film that seizes you by the lapels and doesn't let go for 100 minutes.

Unlike anything else he's done, Pattinson was involved throughout the writing process. He was in the jungle in Colombia at the time, making The Lost City of Z, a gnarly experience by all accounts: he has stories of picking maggots out of his beard, and crew members being bitten by snakes. But at the day's end, he'd find a volley of emails (there's wi-fi in the Amazon, apparently) from the Safdies about Connie Nikas, about criminals, about the world of their movie.

They worked together painstakingly on Connie's backstory, and Robert read all the books the brothers were inspired by, The Executioner's Song by Norman Mailer and In the Belly of the Beast by Jack Henry Abbott. He watched the documentaries they sent over, notably One Year in a Life of Crime (1989) by John Alpert, and episodes of Cops, the Nineties reality TV that featured police chasing down and arresting a whole menagerie of street criminals. Josh calls it "America's greatest TV series". There would often be dialogue or behaviour that would be useful in building Connie Nikas. By the time Pattinson was ready to move to Queens, he was already halfway there.

Pattinson doesn't do method; he's more or less untrained, apart from a short stint in the Barnes Theatre Company aged 15. The Safdies introduced him to a new level of improvisation and research. They had Robert as Connie writing Nick letters as though from prison. Then they went on a tour of the Manhattan Detention Complex.

"Rob came as Connie, but he didn't have the accent yet so he just looked around and kept to himself," says Josh Safdie. They met people that Connie might be friends with. "My friends at Lucky's Automotive Repair in Yonkers, basically. We started bringing Benny in as Nick then." And from there, Rob and Benny took their characters out into the world, going to Dunkin' Donuts, even working at a car wash together for a week.

"We had Nick drive the cars off after they went through," says Benny. "But Nick has issues. He can't do what Connie wants him to, so there was tension between them, it almost got violent. And that's what we wanted. We wanted to give Rob a history of the emotions he would feel in certain moments."

Critically, though, no one clocked Pattinson through all this. The car wash manager knew who he was, but no one else did, and they didn't ask. It was a revelation. As Connie — with the clothing, hair and makeup — Pattinson could go unrecognised to such a degree that when they shot a scene toward the end in an apartment block, local residents didn't even see him as an actor. They knew a movie star was in their midst but had heard it was Bradley Cooper.

"So, I was in this packed elevator and people were like, 'Yo, you like Bradley Cooper's security guard?' It was amazing," Pattinson says.

Black wool jacket, £1,350; white cotton T-shirt, £230; black wool trousers, £490, all by Dior Homme Carin Backoff Esquire

One of the joys of Good Time is remembering just how different Robert and his character Connie actually are. Pattinson is from south-west London, where he went to The Harrodian, a nice public school in Barnes. The son of a vintage-car salesman father and a model-booker mother, he grew up middle class and comfortable, an artistic type who set out after a music career (his band's name: Bad Girls) before acting won out. He never came across characters like Connie Nikas in real life, so he imagined them; they were "fantasy figures", as he calls them. And as such, no less influential.

"Growing up, you see Pacino and you want to be that," he says, and then laughs. "I sound like a dick already, comparing myself to Pacino!"

But the point is sound; to Pattinson, Connie falls into the tradition of Pacino's Sonny in Dog Day Afternoon, or Robert De Niro's Johnny Boy in Mean Streets, the very characters who inspire people like Pattinson to become actors in the first place. Like all middle-class kids, he craved Connie's authenticity.

"Everyone wants to say, 'I've gone through hardships' or whatever. And some kids at school got so obsessed with looking tough that eventually they just were. They were mugging people. It's like, 'Why are you mugging people? You live in Wimbledon!' But you could see the progression," he says. "It was born out of desire, not necessity. It's fascinating."

As for Pattinson, he just lied.

"I decided the best way to be real is to fake it! I used to lie all the time when I was younger. Like even though I had a London accent, I'd tell people I grew up on a farm in Yorkshire. That was about as gritty as I could pull off."

His own life of crime was limited to stealing porn mags, aged 11, a story he told US shock jock Howard Stern. Eventually, he was caught, of course, the moment of humiliation seared into his memory as, in front of a line of old ladies collecting their pensions, the shop owner reached into his bag and pulled out one jazz mag after another.

"I turned on the tears and everything. I was desperate!" he says. "And when my mum heard, I totally threw one of my friends under the bus: 'Dan did it!' It's pretty terrifying when you're backed against the wall. When people ask how would you behave in an emergency, now I know. I'm a wimp! I guess that's pretty obvious!"

He says wimp, but there's a quiet strength behind that self-effacing, affable front. Not everyone would confess to being a cowardly kid, or lying about their background, as insecure people don't admit their flaws so freely. One of the reasons he was so drawn to the role of Connie, for instance, was the character's lack of fear or shame. "I'm the opposite. Shame is the most crippling thing. I don't even know what it is, it's not connected to any other emotion. So I choose work to directly combat elements of my own personality."

Josh Safdie spotted Pattinson's ambition early on. "There's a mania to him," he says. "A manic desire to conquer the world. I was very happy to see it."

And for all his self-deprecation, there's a pride there in what he's achieved post-Twilight. None of his subsequent film choices are obviously commercial, which suits him perfectly: low-budget indies, he says, have a lower bar to break even and with his international stardom, courtesy of Twilight in no small part, he can usually rest easy. Sometimes, his involvement is what makes these projects actually happen.

But artistically — and this is where he's definitely not a wimp — every project is a risk, a test, a leap, yet another opportunity to fail and land very publicly on his arse. But that's just how he likes it. The nerves, the threat of failure keep him interested.

"I like a big mountain to climb," he says. "Some parts no one would think of me for, and I don't blame them."

Why go for those roles though, if they're so against type? He shrugs.

"Probably just to prove I can, really."

Dark mustard wool blouson, £250; grey wool trousers, £245, both by Ami. Black leather boots, £375, by Frye. Black cotton socks, £12, by London Sock Company Carin Backoff

As the bill arrives for our meal, Pattinson chomps merrily through another round of toothpicks. It seems he's been entirely sensible this time around. Not even one beer. "If I drink I'll sound like a cock," he says. "Actually, I probably sound like a cock already!" Anyway, he's saving room for a cognac tasting later tonight with the Good Time producers. Not the kind of thing he does that often but these are heady times, what with the excitement around the movie, the critical acclaim. It's such a buzz that even the press tour isn't so painful. There's room for some mischief at any rate.

On Jimmy Kimmel Live!, he tried to make fun of Josh Safdie but it came out wrong. He told Kimmel that Safdie had asked him to jerk off a dog. "It got [animal charity] Peta angry… everyone. It was like a whole American uproar for a day-and-a-half," Josh says. "He's a little shit, I promise you. But I love that about him."

For the most part, though, Pattinson leads a fairly quiet life. It's just him, Twigs and Solo kicking around at home. When he's not working, he says, he's looking for work.

"I'm basically flicking through the pages of Loot every day. I live the life of an unemployed person." And for him that means watching art house movies, trawling film-geek websites and — so long as Game of Thrones isn't on — cold-calling directors.

In a couple of weeks, he's off to Germany for cosmonaut training for the movie he's making with Claire Denis. It's about another ex-con, this time in space as part of a human reproduction experiment. He mentioned it in a Q&A session in LA after a screening of Good Time, and no one in the audience had heard of Denis. Such is Pattinson's particular taste.

"I don't think Claire has made a bad movie in, like, 20, but I don't know if any have been commercially successful!" he laughs. "That's what it's like in France. There's a market there for less conventionally commercial movies, and that's the world I want to be a part of. I just want to do stuff that people are only making for themselves, because it ends up being, by definition, more singular."

The project that has him excited comes at the end of the year: The Devil All the Time, by Antonio Campos, who made Christine last year, a brilliant drama about a depressive Seventies news anchor in Florida. (For the record, Pattinson cold-called him too.) "There's this line in it — and sometimes that's all you need. And it's like, 'Ooh... that's scary to say'. Because it'll go down in posterity and I'll be the one saying it. You literally cannot get darker. It's fucking dark. This character is an evangelical preacher in the South in the Fifties, but he's gleefully bad and kind of funny and charismatic too. I know, it's irresistible."

Like, sexually repulsive, violent?

"Mmm... yes, all that. But you know when actors say, 'I refuse to play someone who does something bad.' I'm, like, why? That's fucking crazy. You can't do anything bad in your real life. I think if someone needs to play a hero all the time, it's probably because they're doing really gross stuff in their real life."

So you're telling me, this is the only chance you get to be bad?

He laughs, and gets up to put on his Lacoste jacket, his camouflage, and flips up the hoodie underneath. Now he's safe to leave our meeting without causing an incident. But it's impossible now not to see shades of Connie, the sociopath bank robber from Queens.

"Yeah," he grins. "The rest of the time, I'm an angel!"

Good Time is out on 17 November