Dan Stevens looks shaken. We are in an empty restaurant in New York and I have just become the first outsider he has ever told about his death in Downton Abbey. “It is very odd,” he says.

After the Christmas special yesterday, the entire world knows that honourable, handsome, happy Matthew Crawley has died at the wheel of his car, reducing a nation to tears of dismay and disbelief. At the time of our interview last month, there was a lot of speculation, but no actual confirmation. “It is very strange to make it official especially since we are talking about it in the future perfect,” he says, with a laugh. “I am not sure exactly what tense it is, but it is something very weird.”

That is an almost perfect Dan Stevens joke. He is the charming, well-spoken, Cambridge-educated actor who has become as famous as any movie star thanks to his role as the romantic lead in Downton. But his literary aspirations, his desire to be more than just another TV sensation, meant that though his fans wanted him to stay, they knew that he would probably choose to go.

In fact, he made the decision in February before he even started filming the third series. “We were always optioned for three years,” he explains. “And when that came up it was a very difficult decision. But it felt like a good time to take stock, to take a moment. From a personal point of view, I wanted a chance to do other things.

“It is a very monopolising job. So there is a strange sense of liberation at the same time as great sadness because I am very, very fond of the show and always will be.”

As yet, he can’t say what future projects he will take on, though “there are some exciting opportunities”. Until February, he is on stage in New York, playing opposite Jessica Chastain and David Strathairn in The Heiress, an adaptation of Henry James’s Washington Square. Stevens is Morris Townsend, who may or may not be a fortune hunter. He has sideburns and an American accent and when he walks on stage, there is the strange frisson of seeing him play someone who is not Matthew Crawley – and convey the ambiguities of a darker character very well.

This ambition to do something different is what has spurred him on. “It is a desire for freedom really,” he says. “I don’t see money or a particular status as an actor as a goal but I want to do the best work I can in as interesting a range of roles as I can. And I think a moment like this is quite unique and presents those opportunities more than ever before.

“That may not be the case,” he adds, with another laugh. “I genuinely don’t know exactly what is around the corner but I hope it will be something a little bit different. Morris Townsend is a little bit different, and that for me is good enough.”

His voice trails away, and he looks down at his hands. When he is talking about books or theatre, there is no stopping him. When he talks about Downton, he is more cautious. At the time he signed up, Stevens was mainly a theatre actor familiar on television for his part in Alan Hollinghurst’s The Line of Beauty. But he was neither a household name, nor a heartthrob. Downton, playing on his boyish handsomeness and his passionate affair with the stand-offish Lady Mary, has made him a star, both in the UK and America. When he walks on stage in New York, he commands a round of applause just as great as that of his movie-making co-stars.

‘None of us had any idea of how successful Downton was going to be,” he says. “I thought I was signing up for another period drama that had a slightly modern feel. It had a freedom about it because it was coming out of the head of Julian Fellowes. Anything could happen and generally did.”

Its appeal, from the first, was its company feel – rare in TV. “There was no main character. Everybody owned their storyline. And it was fun. It had a tongue in cheek element which set it apart. I had done quite a few period dramas at that point and I was ready not to do another one and then these scripts came along and the Matthew/Mary relationship was just such fun – I am really glad I didn’t turn it down.”

Stevens’s affection for Downton is unmistakable – he generally tweets “Hound’s bum abbey time” as each episode starts, a reference to the dog’s bottom that opens the credits. “In terms of its popularity,” he explains, “there is a kind of ironic enjoyment as well as a serious enjoyment. One of my ways of coping with the attention that it has received is to join the ranks enjoying the mania of Downton rather than take the whole thing too seriously. But that is my way with most things. Not to take them too seriously.”

That much is clear. He laughs a lot while he talks, and makes rather good jokes. But he also uses this charm to deflect questions. If he has felt any frustration, he does not show it: but he did make his decision to leave after the second series, the one in which Matthew – apparently paralysed in the war – was forced by the plot to rise from his wheelchair like Lazarus from the grave.

“I think it was harder for the people who had to react to me getting out of the chair,” says Stevens, with a grin. “That was a particularly strange point in the narrative. I think there were some justifiable criticisms of series two and its pace. I think from what I have seen, series three has been a lot stronger. But from the actor’s point of view all the bombs and the mud and everything were great to film and I had a great time.”

Nevertheless, for so intelligent a man, it must have hurt when his peer Benedict Cumberbatch was quoted as describing that second series as “f------ atrocious.” “From what he has told me, and from what I understand, he was misquoted or certainly quoted out of context,” says Stevens, loyally. “But the thing that upset people was that there is a sort of unwritten rule that whatever you think of other people’s shows you don’t diss them to journalists.”

He insists, however, that it is Downton’s capacity to surprise – whether it is a Turkish diplomat expiring in Lady Mary’s bed or the unforeseen death of Sybil in childbirth – that sets it apart. “You think you are trotting along with a nice Sunday night drama and something happens. It wouldn’t be Downton if it wasn’t for all the big twists and shocks.”

Now Mathew’s death is the melodrama that has left viewers reeling. “It was very emotional shooting the end of this series, because those guys are like family. We have been living together for three years and have been on the most amazing journey. I don’t think any of us, with the possible exception of Maggie, have had this kind of explosion in our career paths, and may never again. It has been so bizarre, and only those who have been through it can understand it.”

His closeness to the Downton tribe is obvious: he describes Hugh Bonneville and Allen Leech (Branson) as being like “hilarious brothers” to him and the experience of working with Maggie Smith as a joy. “There are certain takes where you can see us still half-chuckling from some remark she has made just before ‘action’.” He will miss Lady Mary (Michelle Dockery) because “after everything we have been through it will be sad not to see the relationship continue”.

“On the other hand, I won’t be sorry to see the back of that dining room,” he says, with a roar of laughter. “It may have held some of the key plot points, but it is just a nightmare to shoot in. There are so many angles and edits and it gets very airless and stuffy – and it is blacked out so even at 10 in the morning we are in the dark.”

For all his expressed sadness, he almost glistens with the excitement of what is to come. He has left after what he describes as the “busiest year I have ever had professionally. I hope I never have another year that is quite like this.” Even listing what he has done is exhausting: seven months of shooting Downton, being a judge for the Man Booker prize, producing the film Summer in February, in which he also stars, co-editing an online literary magazine (thejunket.org), writing a column for the Sunday Telegraph, passing 30, becoming a father for the second time (a son, Aubrey to join Willow, aged three), starring on Broadway.

This may have had some effect on his choice. “I was in Cornwall producing my first film. I had 145 novels on my plate with the Booker, I was writing and editing, we had our second baby on the way [his wife is the jazz singer and teacher Susie Hariet] and things were getting kind of crazy.”

His natural curiosity is revealed by the way he talks about each aspect of the year. The Booker judging, in particular, allowed the bright boy who sailed out of Croydon via public school and Cambridge to fulfil his intellectual ambitions. “Whenever we met as judges it was like some of the finest supervisions I had at university, talking about literature with brilliant people.” On the other hand, in the early stages of trying to read so many novels, “I can’t even begin to describe the depths of despair I was in at some points.”

He feels he can reveal that thanks to his liking for the experimental, he was particularly fond of Will Self’s Umbrella. “It is without question an extraordinary novel but ultimately the question was, are we choosing the most ground-breaking book or the best work of literary fiction that year. Hilary Mantel’s Bring Up the Bodies is a brilliant, brilliant novel. And if you have written the best work of literary fiction you should win the prize, so in that sense it was unanimous.”

Next on the cards, I suspect, though he is guarded, is not theatre or literature, but a big film. “I haven’t done as many films as I would have liked,” he says. “A lot of my contemporaries have done more. I don’t have ‘I will be a movie star’ emblazoned on anything, but I’d like do a bit more screen stuff and then when the time is right come back to theatre. When it is good, theatre takes a lot of beating both to watch and perform.”

It is possible, I suggest, that his life will never again reach this high point of fame. “Oh it is quite possible that none of us in Downton will ever again get the ratings this has had,” he smiles. “But from a career point of view, it has opened so many doors.

“I genuinely don’t feel ‘I must play this role’ or ‘I must take this much at the box office’ in order to fulfil my happiness quotient. As long as I am given the opportunity to keep performing and keep exploring in whatever medium, I’ll be happy. As long as I get to spend time with my family, I’ll be happy. As long as I can write in some form, I’ll be happy. It is the essential things like that I equate with happiness.”

However sad the end of Matthew Crawley, the happiness of Dan Stevens is likely to grow and grow.

Downton Abbey: A Journey to the Highlands is out now on DVD and Digital Download