James Franco, the stoner’s comedian inside a workaholic arthouse auteur trapped in a Hollywood leading man’s body, is a bewildering enough prospect as an actor, but that’s nothing compared with what he is as an interviewee. As I walk into his hotel room in San Sebastián, Spain, where he is at the film festival showing his latest effort, The Disaster Artist, which he directed and stars in, I wonder which side I’ll get today. (Please, God, not the pretentious-auteur one.) After all, what to expect of a man who, in one year, made eight movies including Eat Pray Love; the pretty good Allen Ginsberg biopic, Howl; the completely meh comedy, Date Night; the endurance movie, 127 Hours (for which he got an Oscar nomination); and a near unwatchable short movie – which he directed, wrote and starred in – called Masculinity and Me?

And my wariness isn’t just based on his career. The two previous times I had encountered Franco in person were baffling – not just because he was so odd, but because he was odd in such different ways that he seemed like two different people. The first was in 2009 at the New Yorker festival when, before an audience of increasingly bemused young women who had come to swoon in front of a Hollywood heart-throb, he read out Frank Bidart’s poem about a necrophiliac, Herbert White, which includes lines such as: “When the body got too discomposed/ I’d just jack off, letting it fall on her.” Don’t all swoon at once, ladies! Franco, on the other hand, liked the poem so much he turned it into a short film starring Michael Shannon.

The second encounter was two years later, when he hosted the Oscars with Anne Hathaway, which I covered for this paper. It’s hard to pick out a low point, but the sketch in which Franco, for no obvious reason, dressed up as Marilyn Monroe from Gentlemen Prefer Blondes would certainly make the top five.

“Oh my God, you have been around at some weird moments in my life – HA HA HA! Definitely some strange times!” Franco booms in his unexpectedly loud voice when I tell him about my previous audiences with him. That’s putting it mildly. After all, those two episodes happened during a period during which Franco seemed to be living his life as an art experiment. He was making, on average, seven Hollywood movies a year; studying for about a dozen different academic degrees; writing and directing endless po-faced short movies; teasing the media about whether he was gay or not; making bizarre appearances on daytime soap operas and satirising his own persona by playing himself in comedies including Knocked Up and 30 Rock.

This could all have been dismissed as the antics of another idiot celebrity were it not for the fact that Franco was always clearly very talented. Whereas the group of actors he emerged with on the cult 1999 TV show Freaks and Geeks, including his best friend and frequent collaborator, Seth Rogen, have coasted through by playing versions of themselves on screen, Franco immerses himself in a role to an occasionally absurd extent. When the results are good, as in 127 Hours and – one of my favourite Franco performances – 2012’s Spring Breakers, in which he played a creepy hick gangster, he is one of the most exciting actors of his generation. But it was hard to remember those roles during the strange times when he seemed determined to turn himself into a ubiquitous celeb.

In a suede jacket, tight T-shirt, jeans and retro-quiff hair, Franco, 39, still has the pretty-boy looks that got him cast as James Dean in the critically acclaimed 2001 TV biopic. But, while his handsomeness is as expected, he himself turns out to be lucid, engaged and self-aware – words I have never heard applied to Franco before, ever. This is the all-new Franco, one who has moved beyond the aforementioned “strange times”, and he even kicks things off by mentioning that his girlfriend, a US TV publicist, has come with him to Spain. He is, apparently, done with his previous coy game-playing about his sexuality, although his recent claim that, “I’m gay in my art and straight in my life,” is surely the most Francoesque way possible to confirm one’s heterosexuality.

The Disaster Artist reflects this change in him, and it is not just the most fun and accessible movie he has directed yet, it is also his most personally revealing. It tells the true story of the making of the oft-described “worst film ever” (and now cult movie) The Room, made in 2003 and directed, funded, written by and starring the peculiar Tommy Wiseau, a man whose pan-European accent is as unique as his acting style. Franco, buried beneath a mountain of prosthetics, plays Wiseau, who has always refused to reveal anything about his background or even where he is from, as a heroic figure; one who has total faith in himself even while everyone around him tells him he is devoid of talent. Franco clearly likes Wiseau – he even has a Tommy Wiseau case on his iPhone – and when he lets slip at one point that he’d wanted to shoot some scenes in Romania, suggesting that’s where Wiseau is from, his eyes widen in horror, as if he has given away a state secret.

Franco on the set of The Disaster Artist. Photograph: Justina Mintz

Franco, by contrast, grew up in California in an artsy, middle-class home and found success in acting in his late teens. But despite the obvious differences between him and Wiseau, Franco poured his life into The Disaster Artist in ways that verge on Freudian, or maybe just Franco-ish. His brother, Dave, a fellow actor, plays Greg Sestero, Wiseau’s longsuffering best friend and movie costar. Dave’s then girlfriend, now wife, Alison Brie, plays Sestero’s girlfriend, who Wiseau deeply resents. Judd Apatow, who discovered Franco for Freaks and Geeks, plays the big Hollywood director who tells Wiseau he will never go anywhere, while Rogen plays a straight-talking cameraman, watching Wiseau’s eccentricities with increasing frustration.

At heart, the movie is about the intense friendship between the straightforward Sestero and the oddball Wiseau, which, as it happens, is also what the many other movies Rogen and Franco have made together are about, such as Pineapple Express and This Is the End. This, as it also happens, is also pretty much the dynamic of their real-life friendship: Rogen has said in the past that Franco’s fearlessness has encouraged him to go further in his comedy than he might otherwise, and today Franco says Rogen’s pragmatism helped him make this movie:

“When I came across this story, I knew it had everything that I’m interested in examining with the artistic process, but also these insane characters and comedy that would allow it to go into a more commercial space. So the smart part of my brain said: ‘OK, get Seth to produce this.’ He knows how to work with a studio and still make very personal movies; the kind of movies he wants to make. He really pushed me to be more responsible, to slow down, to focus on one thing at a time. And that was a really big lesson to me.”

Given that Franco – who stayed in character during the shoot – directed The Disaster Artist as Wiseau alongside Rogen, while making a movie about Wiseau directing a movie with his best friend (stay with me here), it must have been hard for Franco to tell at times where he ended and Wiseau began.

“Yeah, yeah. The whole thing did start to feel really autobiographical,” Franco agrees with a laugh. “One of the other lessons I learned from the movie is I’m a bit like Tommy. He thought he was making this big dramatic movie and it turned out to be a comedy. I think there’s a bit of that in me.”

In that you’d been directing all these small literary movies and comedy is actually your strength?

“Yeah, yeah. You think you should be doing one thing and then you realise, ‘Oh wait, OK, over here is where my soul speaks best, actually.’”

So was he taking himself too seriously before?

“Certainly – and it’s something I still do a lot. That’s been one of my big issues all through my life. But who doesn’t struggle with that?” he asks, and then laughs again.

After Freaks and Geeks ended in 2000, Franco, determined to be the next Brando, essentially turned his back on comedy. “So I did a bunch of dramas I shouldn’t have done. I mean, that’s a whole story …” he says and then groans and covers his eyes, thinking back on the early-to-mid 00s when he made forgettable dross such as Tristan & Isolde and Annapolis. It wasn’t until he made 2008’s Pineapple Express with Rogen that he realised making comedies with his friends was actually what he enjoyed, and it’s no coincidence that it was in that movie he gave his greatest performance yet. He invested his potentially two-dimensional character, a permanently stoned dealer called Saul, with so much sweetness and soulfulness that he lifts the movie to a level of near grandeur – and when I tell him how much I loved him in that movie he gives a Buddha-like nod of humility and talks about how that movie “made me enjoy acting again”. And yet the awkward fact remains that it was during and after that movie that the “strange times” began. If he was so happy then, why was he behaving so oddly?

“Yeah, yeah, look,” he says, and his shoulders sag a little and his brow furrows. “There’s this podcast I listen to, You Must Remember This – do you know it?”

I love it, I say, and we talk a little about Karina Longworth’s fantastic podcast about 20th-century Hollywood.

127 Hours, for which Franco received an Oscar nomination. Photograph: Rex Features

“But almost every single story on that podcast is full of wreckage and tragedy, right? So what’s the lesson?”

That it’s hard to be a human?

“Yes, for sure,” he says, but in a tone that suggests I’ve slightly missed the point. “But also that fame magnifies a lot of problems. So in that period, I looked around and thought, ‘All of these other outlets are taking ownership of my public persona – why shouldn’t I?’”

It didn’t look like you were having much fun, I say.

“Well, it was interesting, until it ran its course. But look, I’ve always been earnest in everything I tried. I mean, we don’t have to go into the whole Oscars hosting thing …”

Oh, go on.

“Well, it was a weird time and a weird situation,” he sighs, furrowing his brow again. “And maybe I shouldn’t have been in that situation, but I agreed to do it.”

Surely most people would say yes to hosting the Oscars, if only for the experience.

“Mmm no, I don’t think so! I think a lot of people wouldn’t. But, you know, it was a fun experiment that turned into something else.”

Was there also an element of workaholism because you wanted to hide from something in your life?

“Exactly, any -ism is hiding from something,” he says and sighs again. “Yeah, it was a fear of failure, fear of criticism – it’s all ego. Like, if I work on many projects and get criticised for project A, it doesn’t matter because I’ve got Project B. It was that mentality. Also, there was a fear of facing the world, facing people, so I’ll busy myself with work because I understand the dynamics of a set, so if I’m on a set all the time I’ll be OK.”

But it’s all change now. Although Franco is now promoting The Disaster Artist, appearing in and directing the David Simon series The Deuce and has, by my count, about eight movies in post-production, this is downright slacking off compared with his previous form. In fact, he says with some pride, he has only worked as an actor for two weeks this year. Instead, he is filling his time with surfing, playing tennis, seeing friends, travelling with his girlfriend.

“Do you have kids?” he suddenly asks.

I have twins, I say.

He makes a soppy smile: “Aww, I really want twins!” he says moonily, which is the kind of thing only someone without kids would say.

Our time is up, but, before I go, I have one last question. What did he mean when he said he is gay in his art, but straight in his life? He furrows his brow again, as if he is about to make another ever so serious answer. But then suddenly his face clears and instead he looks up at me and just laughs. Even Franco can laugh at Franco these days, it turns out.

The Disaster Artist is in limited UK cinemas from 1 December and cinemas nationwide on 8 December