When Adrien Brody became the youngest winner of the best actor Oscar in 2003 for his role in Roman Polanski's The Pianist, he was the toast of the film world. With a cameo role in Wes Anderson's The Grand Budapest Hotel, he talks about how the accolade has shaped his career

The night after Adrien Brody won the Oscar for best actor in March 2003, he went to a restaurant and the entire room stood up and applauded. Eleven years later, he makes a far more low-key entrance at the fancy Bondi Beach restaurant Icebergs – no PR, no entourage, wearing flip-flops, and slightly miffed that I dropped his name in order to get a table. "I would never have done that," he frowns.

When Roman Polanski's film The Pianist made him the youngest actor ever to win the Academy award, he was 29; now, he's 40. Being a middle-aged actor in Hollywood is "different", he says after a long pause. Then again: "I think an actor hits his prime now. If you look at all the iconic leading roles and great actors, they're all in their 40s and 50s."

Brody is in Australia making a thriller directed by Michael Petroni called Backtrack, in which he plays a psychologist from Melbourne who finds out that all his patients are the ghosts of people who died in a terrible accident 20 years previously. Release-wise, his latest film is Wes Anderson's The Grand Budapest Hotel, his third collaboration with the director, which sees Brody play the villain in moustache-twirling style. Though Brody looks sensational rushing down a corridor in a double-breasted greatcoat, and destroys an Egon Schiele picture with considerable verve, it's a small role (one of several cameos by big-name actors).

Brody in The Grand Budapest Hotel: 'It was a chance to enter this fantasy world.' Photograph: Allstar

"It was a chance to play with friends," he says. "It was different from a lot of the roles that I've had a lot of responsibility with. It's not my movie, it wasn't the weight of the world on my shoulders, it was just a chance to enter this fantasy world. The key to playing the role was just to have a balance between playing the seriousness of the circumstances and the comedic aspect of it, not being too self-indulgent."

Two days previously, the film opened the Berlin film festival. Brody says he wishes he could have gone, but hasn't read the reviews, which were enthusiastic. "Obviously everyone is entitled to have an opinion and the more you're out there, the more opinions are formed," he says. "I'm a pretty good judge of seeing something that I've done and what I could have achieved that's different and better."

Instead of The Grand Budapest Hotel, over the course of lunch (Brody drinks mineral water and picks carefully at a piece of fish) the films we end up talking about are the ones that have defined his career: Terrence Malick's The Thin Red Line; Spike Lee's Summer of Sam, in which he played a punk; The Pianist; and Anderson's The Darjeeling Limited, which Brody loved because it allowed him to ride around India on a Royal Enfield motorbike, though he nearly came to grief when he crashed into a cow. "Riding's something indescribable and alluring, but dangerous, and rear-ending a cow with my face wouldn't have been a dignified death. I remember laughing as it was almost over – just the absurdity of: 'This cannot be the way to go out.'"

The rest of his back catalogue is all over the place, ranging from action films such as Predators to Detachment, a 2011 Tony Kaye-directed indie film about the failings of the American education system, in which Brody played a teacher, and which he regards as his most underrated film.

"That's a movie that we all made for the right reasons and no one saw," he laments. "That is upsetting. It was a wonderful role, and in playing that character it made me think a lot about our children's future and how frightening it is, and what they're up against." Does he want to be a dad himself? "Yeah, hope so," says Brody, whose girlfriend is the model Lara Lieto. "I'm not there yet, but I think I'd make a good father."

Since The Grand Budapest Hotel, Brody has made two films (The Third Man, directed by Paul Haggis and arriving in the summer, and American Heist, in which he co-stars with Hayden Christensen), plus his first TV mini-series, Houdini, in which he plays the title role. Brody, who used to perform magic tricks as a boy as The Amazing Adrien, doesn't see it as a comedown to be working for the History Channel. "I don't, because it wasn't a commitment to being on TV, it was an opportunity to give the most in-depth view of a man for whom I have tremendous respect. I don't think you could tell that in a film. You have the machine and the might of History putting it out there. And I like experimenting. I'm drawn by the role in all my choices, or the director, or what that experience is going to do with my life, how it will enrich my understanding."

Defining role: Brody in his Oscar-winning performance in The Pianist.

While all but the most fortunate actors usually end up veering between credible, heavyweight films and less classy fare, it's striking how Brody started in arthouse but somehow ended up playing the buffed-up lead in Predators. He thinks that he doesn't have a particular "thing" he does – IMDb's list of his trademarks suggests that his "calming voice" is particularly notable – though his singular looks must impress casting directors as much as his acting ability; certainly Polanski said that Brody had an extraordinary face. As on film, up close his nose is a mesmerising sight – long, elegant and bent to his left (it was broken during the filming of Summer of Sam). He's a fashion-world favourite, joining a lineup of actors on the catwalk for Prada for the designer's AW 2012 collection, a riff on themes of male power. "I got to hang out with Gary Oldman and I told Gary: 'I'm channelling you because I played a very Dracula-esque persona in the runway walk," he says. "This was 20 seconds of my life and it becomes a topic, it's funny." Does he care about clothes? "Nah," he says. "I'm not dismissive about clothes but I don't consciously think about what I should wear. Clothes are very interesting tools for an actor and that's where I give them a lot of consideration."

After winning the Oscar, in theory, Brody could have done anything. So has his career gone according to plan? "Does anything?" he asks. "Maybe for some, but I think very few people visualise what the endgame is and achieve it. Your hopes and aspirations are just that. There's no guarantee that you deserve anything in this life and there are an infinite amount of gifted people that do not ever receive any opportunity or recognition. So do I have standards and hopes that I could find a calibre of material on a par with The Pianist more frequently, and collaborations with film-makers like that more frequently? Yes. Those are out of my control for the most part.

"I focus, I'm very disciplined with the work that I do but there are definitely projects that I've been involved with that were very different from how I envisioned them," he continues. "It's remarkable to have a couple of amazing things in your repertoire – it's so hard to align those things in your life. It's how old you are, where you are in your life, what's in the ether, and at the same time you have to work and continue to cultivate your body of work."

When I ask which actor he thinks has got it right in terms of their choice of roles, he replies: "Meryl Streep. She navigated it quite well and continues to find things that are new for her and challenging."

Brody does take time off when he can't find an appropriate role. He makes music in his spare time – beats for hip-hop tracks. He is undoubtedly passionate about his music. He talks with huge enthusiasm about sampling the sound of popcorn hitting a pan and turning it into a beat, and when he complains about the "shit" he hears on the radio, it's the only time he swears in almost two hours. "I think there was a period in my life where had I met the right people to help me figure out how to pursue it, I would have loved to transition into producing an album," he says.

Severely cut: Brody in Terrence Malick's The Thin Red Line. Photograph: Allstar/Cinetext/20th Century Fox

So what happened? "I think nobody got it. That's the dilemma also with having immense success associated from one role, like The Pianist. That's people's introduction to who you are and what you must be like, and then they meet you and you're from Queens and you're making music that's inspired by a lifetime of hip-hop and urban culture and New York City and street life. Sequenced music is very hard for lots of people to grasp anyway. And then I didn't have the wherewithal, the focus I guess, to do something on my own. But you know, I just did a film with Akon [American Heist], we've discussed playing around with something; I'm still friends with RZA and we've talked about doing some things. I have a lot of ammo, it's just stockpiled."

Brody identifies himself as a New Yorker more than as an American. An only child, he's close to his parents; his mother is a photographer who used to work for the Village Voice (Brody used to hang out in the offices as a child); his father is a retired history professor. "They are the most kind, honest, decent people I know," he says fervently. His parents were with him on the night of his first major career setback. Having spent six months in Queensland, Australia, shooting The Thin Red Line, Brody, then only 23, attended a screening of the film to find that his lead role had been reduced in the editing suite to a bit-part. Malick hadn't even warned him – and even worse, though he'd been told it was a private screening, it was actually a press viewing.

"Not only the film but every kind of visual memory I had from the experience was lost," says Brody, whose video camera with footage of the making of the film was stolen on the way back home. He says that the experience gave him a glimpse of what soldiers coming back from the battlefield must feel like, though he has the grace to laugh – "coming home and being asked about it a lot, but no one could really get it. But it's OK," he adds. "In some sense it was necessary. I think I would have been too young if that film had been very successful and that role remained intact."

He says that he would work with Malick again – "it's water under the bridge" – although you wouldn't put money on it. When I ask Brody how he cries on film, he says: "I think of Thin Red Line. No … I'm kidding." He isn't though – he goes on to say that the misery of losing the role informed the way he acted as a Jewish musician who loses his entire family in the Holocaust in The Pianist, though he is careful to stress that the two situations were hardly comparable.

Being cut from Malick's film could easily have ended Brody's career. He was rescued by two more auteurs – first Spike Lee, then Polanski. Brody is great in Lee's Summer of Sam, and it seems to rankle somewhat that he was not more acclaimed for it. "If that movie had come out after The Pianist, there'd be a lot more celebration of the daringness and the uniqueness of the character," he grumbles. "You don't know who that actor is, you don't know that punk culture had no influence on him in his entire life and he had to create something that's entirely distant from himself."

Brody in Summer of Sam with John Leguizamo. Photograph: Sportsphoto Ltd/Allstar

On the upside, "those experiences enrich your life so much – I mean, I played a show at CBGBs. I got up onstage, I sang, I screamed loud, I had real street kids pulled off St Mark's Place in there spitting on me, throwing drinks on me, me throwing drinks, spitting back – it was great."

As for The Pianist, most artists, when interviewed, will say that their most recent work is the best they've ever done, but Brody says that Polanski's film is "by far" the most gratifying he's made. Playing the Polish-Jewish pianist Wladyslaw Szpilman, Brody starts off cocky and ends up traumatised and broken by the Holocaust and the second world war.

Brody and Polanski are still friends; Brody sees him every time he goes to Paris (Polanski, of course, would be arrested if he were to return to the US, after pleading guilty to unlawful sex with a minor). "There's always lots to talk about with Roman," says Brody. "Roman's given me some of my best jokes." Tell us one, I ask. "Some of them are dirty," says Brody, who is suddenly staring out to sea and wreathed in smiles. "No, they're not appropriate for this," he says tantalisingly. "But they're great." He laughs.

Brody clearly tries to avoid controversy – and stonewalls when asked about another director who has been accused of sexually abusing a child, Woody Allen (he played Salvador Dali in Midnight in Paris). Having been "in character mode", he says he hasn't followed the accusations flying between Allen and the Farrows.

"I don't have a view to share about people's lives. No way. It's not my place. Do you feel it's my place, or are you just looking to see if I have a view? I don't." Similarly, when asked about the state of the US, he says: "I think America is an amazing country, but it is very complicated and it's not really for me to delve into the complications."

So it's back to discussing acting, and the ups and downs of his career. He says that he won't be upset when someone beats his record of being the only best-actor winner aged under 30; and points out that there are many actors, including one of his best friends, who have struggled for work all their lives. (Brody tries to recommend his friend to directors and has got him bit parts "here and there".) There are also the considerable upsides to fame – not least the kindness of strangers.

"It's made the world a much smaller place," he says. "I was lost, hiking in New Zealand and these two girls literally picked me up and drove me home. There was nothing weird, there was no agenda, nobody asked for anybody's number, not even a photograph, but they recognised me and they felt safe to put me in their car – a complete stranger from New York, a grown man. The reason that resonated with me is that here, on the opposite side of the world, complete strangers kind of took me in."