Jack Black's latest film, Bernie, marks a real departure for Hollywood's favourite joker. He plays a loner whose intense relationship with an elderly widow ends in murder. He explains why the role appealed to him

As a general rule, few people look as if they're having as much fun in their day job as Jack Black. Whether he is solemnly lecturing 10-year-olds about the importance of rock'n'roll in School of Rock ("You guys have never got the Led out?"), quarrelling with Will Ferrell over a burrito in Anchorman, or sneering at other people's music taste in his break-out role in High Fidelity ("That is perverse! Don't tell anyone you don't own Blonde on Blonde!"), Black, with a face like a demonic cherub, has carved a happy niche for himself as Hollywood's cackling ball of deadpan, quip-laden fury. It is perhaps not too surprising that, in interviews, Black has tended to come across as a close approximation to his onscreen incarnations.

Unfortunately for him, when we meet in a London hotel, Black is not feeling quite his usual cackly, fun self, having had too much fun the night before. It was the final night of the European tour of his band, Tenacious D, and he "served up", he says, his eyes only just about open, "a generous portion of rock, and London was very appreciative". And then the lids droop fully closed.

However, Black's fatigue is my good fortune. I had been warned beforehand that getting a straight answer out of him is like attempting to pin a wave to the sand. But today he is too tired to adopt his usual interview technique of batting questions away with irony. In the main, he actually gives answers – exhausted but thoughtful answers, with only the occasional dodging joke and burst of song.

He is not a great fan of interviews at the best of times – "this is the job part of the job" – but it helps, he admits, that he is "very proud" of the particular movie we are here to talk about, and he should be. Bernie is a really great little film (I have now seen it twice and would happily see it again) and, while there are ace performances from Shirley MacLaine and Matthew McConaughey, this is very much Black's film and it could easily represent as much of a turning point in his career as High Fidelity did almost a decade ago. US critic Roger Ebert described it as "surely one of the performances of the year" and a Golden Globe nomination followed.

Black's eyes light up cheekily at the mention of this and that satyric grin gets a little wider: "Yes, there's talk of prizes and buzz and it's all very flattering. I like it. Keep that buzz buzzing," he says, pointedly directing his comments into my recorder (we are speaking before the ceremony).

Bernie, directed and co-written by Richard Linklater, who also worked with Black on School of Rock, is the true story of Bernie Tiede, a gentle and much beloved assistant funeral director in the small town of Carthage, Texas. Tiede was always especially kind to the older women in Carthage and one woman whose friendship he made pains to seek out was Marjorie Nugent, played in the film with vim by MacLaine. Nugent was the wealthiest widow in town and the meanest, yet she – momentarily – softened to Tiede and the two became companions, travelling around the world together on exotic holidays. However, Nugent began to abuse Tiede as she did everyone else and then, (spoiler) one day in 1996, this soft-spoken and kindly man shot the 81-year-old woman in the back. This, by the way, is not even the end of the film.

It's a delicate story to tell and Linklater does a good job of balancing the black comedy with tragedy, smartly leaving some things unspelled out – namely, whether Tiede and Nugent's relationship was ever more than platonic and why Tiede stayed friends with Nugent when she abused him so much. Black has the trickiest job, never tipping into camp parody or opacity. But he saw a different problem:

"The big hurdle for me was playing a guy who murdered someone, you know?" he says. "Someone who's not a lovable loser, who's not entirely sympathetic. That's difficult for me because I love to be liked. So to play a role where there's some grey area, that's, uh, a vulnerable terrain to tread."

Black has been veering towards this more interesting terrain for a little while. After the disappointing experience of making King Kong in 2005 ("I don't like these jobs with a big, bloated budget and you sit in the trailer all day"), he stayed largely in cultish comedies playing the aforementioned lovable loser such as in 2008's Be Kind Rewind, or doing voiceovers in the highly lucrative Kung-Fu Panda series. But he branched out into a darker, straighter kind of role in Margot at the Wedding, and says that, while he would still "like to make people laugh", he had been "getting tired of the old kind of frat-guy comedies and I was looking for something else".

Black offscreen: 'I like to be loved. So to play a role where there's some grey area, that's vulnerable terrain to tread.' Photograph: Beck Starr/WireImage

Black spent some time with Tiede to get a true sense of the man in order to play him properly.

"I talked to him about his friendship with Marjorie and they were, uh, sort of best friends. There was nothing sexual. Also, talking to Bernie, I realised that he and Marjorie were stuck together and that's why he couldn't get away. They had that kind of relationship – oh, what do you call it?" he asks sleepily.

Co-dependent?

"Co-dependent, exactly. Also, Bernie just bottles it up when people hurt him as he has no release valve and he's such a sweetheart. I think that was his fatal flaw."

Black sings in all of his most beloved roles (School of Rock, High Fidelity) and in Bernie he gets to do gospels ("They were strangely seductive") and renditions from The Music Man, as Tiede was a big fan of amateur dramatics. He is a revelation in the latter, dancing around like a loose-hipped professional. Has he ever considered going on Broadway?

"Actually," he says, waking up a bit and leaning forward excitedly, "Shirley [MacLaine] and I were saying we should take Bernie to Broadway and adapt it into a dark comedy musical. Sondheim's still alive; get him on the case!"

MacLaine sounds pretty much all that one would hope for of an aged Hollywood diva, and Black clearly adores her. He recently bumped into her at a photoshoot for Paramount's 100th anniversary where it was "just wall to wall famous people". So he went up to MacLaine and commented on the weirdness of the scene. She rolled her eyes.

"Jack," she said, "this is nothing. If you want to know Hollywood go back to 1961 and run around with me, Frank Sinatra and Jerry Lewis."

"And that was her world!" Black says excitedly, still a little starstruck. "If you were anyone in Hollywood then, you hung around with Shirley MacLaine." She hasn't read Black's past life yet (MacLaine is a firm believer in reincarnation) but she has invited him to her retreat in New Mexico for a reading and regression. "I don't really believe in that ooga-booga stuff, but I'll do it for the experience," he says eagerly.

Speaking of belief, Black, who grew up in California, was raised Jewish but is now an atheist. What prompted the switch?

"Well," he replies, palpably warming up into a joke, "when I was a kid I remember hearing that a barmitzvah was like your birthday times 10. And it was not like that at all. I got socks. So I turned to atheism."

Shirley MacLaine and Black in Bernie: 'We were saying we should take the film to Broadway and adapt it into a dark comedy musical.' Photograph: Deana Newcomb

Now that he is a father with two small boys, though, he has started going back to synagogue near his home in LA "and it's nice, you know? The kids play with other kids, and the singing …" And at that, he starts singing, with positive relish, "Chanukah, oh Chanukah!", a classic childrens' holiday song, with all the passion of a religious figurehead. When he finishes, he looks up with solemn pride.

One of the many joys of School of Rock was the way Black interacted with the children and never talked down to them. He and Linklater have been trying to figure out a way to do a School of Rock 2, but would his relationship with the kids be different now that he is a father himself offscreen?

"No, I don't think so. I talk to my boys as real people, with a lot of horsing around, too. But when I made School of Rock I was still thinking I'm never gonna have kids, I'm too much of a kid myself, I want to wander the streets, like a vagabond. Then, during King Kong, I saw Andy Serkis with his wife and kids and I thought, that looks fantastic! He's really making it work. I started to get the hunger then, and it's been fantastic."

It is when the talk turns to music that Black properly wakes up. He says he doesn't really separate singing and acting because it was only when he started doing the two together that both careers took off. Tenacious D is about to release its next record ("It will just have one song, but a very long song, and it's called Simply Jazz") and he is trying to organise "a rock comedy summit" called Festival Supreme, which will take place in LA next May, hopefully featuring the Lonely Island, Flight of the Conchords, Spinal Tap and Tenacious D, as well as Zach Galifinakis on keyboards and Jason Segel on the piano.

"I was also thinking of throwing in the Hoff – but just one song, though. Just one," he muses. And then, at last, he makes a long cackle.

Bernie is released in the UK on 26 April.