Back in 2001, seven years before he directed his first feature film, Steve McQueen made 7th Nov, an installation that features in his forthcoming Tate Modern retrospective. Visually, it is his most minimalist work: a projection of a single still photograph of the crown of a reclining man’s head, which is bisected by a long, curving scar. And yet it possesses a visceral charge that unsettles more than any other piece that will be in the exhibition. That power rests in the accompanying monologue in which McQueen’s cousin, Marcus, recounts in brutally graphic detail the terrible events of the day he accidentally shot and killed his own brother.

7th Nov can be seen in retrospect as a signal of what was to come as McQueen made the transition from artist to director, creating acclaimed feature films that merged formal rigour with a narrative style that is often unflinching in its depiction of human endurance.

“It was tough, I can tell you that,” he says, when I ask him about 7th Nov. “I knew what had happened, of course, but when I was sitting in that room recording my cousin, I was hearing the full story for the first time. I mean, I had no idea. No idea.”

He shakes his head furiously as if trying to erase the memory. “As far as I’m concerned, it’s all about the truth, the whole truth and nothing but the truth. End of. To get to that, you have to go in close, uncover what’s been hidden or covered over. Obviously, the easy thing is not to go there, but I have a need to go there.”

Facebook Twitter Pinterest 7th Nov, by Steve McQueen. Photograph: © Steve McQueen. Courtesy the artist, Thomas Dane Gallery and Marian Goodman Gallery

As a feature film director, McQueen has made an art out of going there, whether evoking the unyielding nature of Irish republicanism in his debut, Hunger (2008), sexual addiction in Shame (2011) or the long nightmare of slavery in the American South in 12 Years a Slave (2013). In the process, his rapid ascendancy as a filmmaker has all but eclipsed his reputation as an artist. Which is why his imminent retrospective is timely. Through 13 carefully selected pieces, it will trace his creative arc as an artist from 1999, the year he won the Turner prize, to the present day. That means that earlier signature works such as Deadpan (1997), in which he reenacted a famous stunt by silent film star Buster Keaton, and Drumroll (1998), which saw him push a barrel thought the streets of Manhattan, are not in the show.

There is one formative film included: Exodus, from 1992, which comprises Super 8 footage he impulsively shot of two elderly black men he spotted carrying potted palms though the streets of east London. It is a low-key metaphorical work that contrasts sharply with the intensity of later works such as Western Deep/Caribs’ Leap (2002), which evokes the long, dark shadow of colonialism, and End Credits (2012), in which he uses redacted FBI files to show the extent of surveillance of the black singer and activist Paul Robeson. For those who only know McQueen’s feature films, these non-narrative works may make for a challenging experience.

“Many younger people will go to Tate Modern because Steve, the Hollywood director, is an icon for them, but these works certainly require a different level of attentiveness and patience,” says the author and academic Paul Gilroy, who taught McQueen at Goldsmiths in the 1980s and has written one of the catalogue essays. “It is worth the effort, not least because they offer a wealth of signals and approaches that echo through the later feature films. This is someone who, from the start, had an encyclopaedic knowledge of film and an almost obsessive interest in its history, and that underpins his art practice in often playful and provocative ways.”

Across the river, in Tate Britain, McQueen’s epic project, Year 3, on show until May, brings his artistic journey up to date. The ground floor gallery walls are covered floor-to-ceiling with thousands of school portraits that add up to a panoramic vision of contemporary multicultural London. “It’s been incredible,” he says of the public reaction to Year 3 since it opened in November. “People have been bringing their grandparents into central London sometimes for the first time to see it. Ordinary people are being seen and seeing themselves on the walls of one of the country’s major art galleries. It’s about recognition, really – ‘Look! We are here!’ That’s a powerful thing.”

Facebook Twitter Pinterest Steve McQueen at the Year 3 exhibition at Tate Britain. Photograph: Tolga Akmen/AFP via Getty Images

It is 11 years since I last interviewed Steve McQueen. In that time, he has gone from being a Turner prize-winning British artist to an internationally acclaimed director, with an Oscar for 12 Years a Slave and a couple of Baftas under his belt. In this year’s New Year honours list, he was given a knighthood for services to British film, having already been the recipient of an OBE and a CBE.

This extraordinary success may have mellowed him somewhat, but his impatient, coiled-up energy is still evident when we meet over early morning coffee in the almost deserted Tate Modern members’ room. Once or twice during our conversation, he drums impatiently on the table or repeatedly slaps his hand as he searches for the right words to fully convey his meaning – and the full force of his conviction. When I ask him if his mainstream success as a director means that he now has to make time for his other job as an artist, he looks momentarily offended. “God no! I don’t think like that at all,” he says, shaking his head. “I mean, ‘success’? I don’t know what that is. With me, it’s about the work, whatever form it takes. I just want to do the work. Often it’s work that I don’t see being done elsewhere. It’s the dirty work, I suppose. And sometimes I want and welcome that burden.”

So, you feel there is a burden to what you do, the burden of truth-telling? “Yes. For sure. It’s about putting oneself in places that are not going to be comfortable but, by going there, you might uncover the truth of what is actually going on. Basically, my attitude is: we’re all going to die anyway, so let’s just go for it.”

The truth-telling will continue apace later this year when McQueen unveils his short non-narrative film about the Grenfell Tower fire, which claimed 72 lives on 14 June 2017. The date and location has yet to be decided and the film is still being edited, so he is reluctant to talk about it in any detail, but when pressed he admits, “It has been difficult, really difficult in so many ways. Until I was six, I lived less than a mile from there and, even after we moved out to Ealing, I’d go back there to hang around Ladbroke Grove because that’s where everyone I knew would hang out. So, going back there again in the circumstance was heavy, very heavy.”

You want to cause a bit of trouble, stir things up. We’re all a bit numb right now. Wake up!

Self-funded by McQueen, the Grenfell project is not a commercial work and will never be broadcast or shown on television, instead eventually being housed in a London space with free admission so that it is accessible to all. He describes it as “an artwork that is about keeping the tragedy in the collective consciousness”.

With the proviso that it would not be shown for two years, McQueen was granted permission by the local community to film the scorched shell of Grenfell Tower from a helicopter, before it was covered up with plastic sheeting. “It was very raw, but also very necessary,” he says. “It was not about being commissioned, it was about getting permission. Initially, it was the permission to talk to the people there, to tell them who I am and where I come from – I actually used to work on a stall in Ladbroke Grove under the Westway selling secondhand clothes. It was essentially about getting their trust.”

He pauses for a long moment. “That building was like a skull after the fire,” he says, finally. “Then, as soon as they covered it up, it was almost like they were saying it never happened. So, no matter that some people might not want to deal with what happened there, I’m saying, no, no. Let’s not forget.”

Death looms like a dark shadow throughout the Tate Modern retrospective, not just in 7th Nov, but also the elegiac Ashes (2002-15), which like Carib’s Leap is set in Grenada, where his father was born. It is a poignant meditation on the life and death of the charismatic young man whose nickname provides the film’s title. Using split-screen projection, it contrasts footage of a vibrant, carefree Ashes, balancing on the prow of his boat, with film of two older local men labouring in the outdoors. It turns out they are painstakingly creating a headstone for his grave.

Facebook Twitter Pinterest A still from Steve McQueen’s immersive installation Ashes. Photograph: Courtesy Thomas Dane Gallery, London. © Steve McQueen

“For many young black men, not just in the Carribean, but here as well, the choices are so limited,” says McQueen. “Ashes is working on his boat catching lobsters for rich hotels and he finds a big stash of weed. It’s an opportunity to earn some dosh and he grabs it, because he has so few choices. Same with my cousin, the reason he gets a gun in the first place is that he doesn’t want to end up on the buses. In a way, those films are about the cost of wanting some liberty, the liberty that other people take for granted.”

In both instances, though, the cost is high. In Ashes’s case, beneath the visual poetry of the film lies a cold, hard fact: life is cheap. McQueen nods. “Yes. And not just in the Caribbean, but here. People’s lives are cheap.” His voice rises in anger. “Look around you at what’s happening here – people getting stabbed every day of the month in London and no one seems to give a shit. If they did give a shit, it wouldn’t happen. It happens because it’s seen to be of no great importance. If it was, things would be done against it to prevent it happening.”

Though McQueen has lived in Amsterdam for more than 20 years, he still considers himself a Londoner – “absolute, total 100%!” The son of West Indian parents he was raised in Shepherd’s Bush and Ealing, and was continually encouraged by his father to learn a trade. “There were no examples of artists who were like me,” he told the Guardian in 2014. “When did you see a black man doing what I do?”

He describes his primary school experience as a happy one, but in our conversation his thoughts return more than once to his secondary school days at Drayton Manor in Ealing, where, at 13, he was placed in a class for children who were deemed to be not academically up to speed. A lazy eye and undiagnosed dyslexia accentuated his sense of isolation.

“What I do as an artist is, I think, to do with my own life experience,” he says at one point. “I came of age in a school which was a microcosm of the world around me. One day, you’re together as a group, the next, you are split up by people who think certain people are better than you. It was kind of interesting to observe that.”

Was there a stigma attached to that separation? “Oh for fucking sure. And it was informed by class and race and privilege. Absolutely. No ifs or buts or maybe about it.”

He recounts a recent meeting between himself and a woman whose young son has a major role in his forthcoming BBC series, Small Axe, which chronicles the everyday lives and struggles of black British people from the late 1960s to the mid 1980s. “She went to my school and had read some of the stuff I’d said about my time there. She told me that the exact same things had happened to her. It turns out that she had ended up home-schooling her son so that they wouldn’t happen to him. There is a lot of publicity about black kids underachieving, but that sort of thing goes unacknowledged.”

Facebook Twitter Pinterest Steve McQueen with his best picture Oscar for 12 Years A Slave in 2014. Photograph: Joe Klamar/AFP via Getty Images

He tells me another story about how a group of his old school friends bumped into one of their former teachers in a pub a few years ago. He told them that, back when they were at school, he had put forward the idea that the school should address the problem of underachieving black pupils. The teacher was told by his superior that, if they did so, it would inevitably mean that more black children would apply to the school. “They were in effect investing in black failure,” says McQueen, shaking his head in disgust.

There are some people, I suggest, who will read this and cite his own extraordinary achievements and his recent knighthood as evidence to the contrary. He shoots me a fierce look. “I am sitting here in front of you doing this interview not because of, but in spite of,” he says. “I had every obstacle thrown up against me on the way up. And a big part of the reason I am sitting here is because of the people who went before me and made sacrifices. They helped clear the path for me because they made some noise and pushed back against racism – in music, in film, in writing, in debates, in protests. I am sitting here because of them. That’s just a fact.”

I ask if he had any hesitation in accepting the honours conferred on him by the British establishment, given that the OBE and CBE are linked by name and history to the long colonial shadow cast by empire. He nods. “I can see that some people would feel hesitant absolutely and, don’t get me wrong, it wasn’t an easy decision. It wasn’t, ‘Oh, yeah, I’ll have that!’ But at the same time I was like, this [knighthood] is one of the highest awards the state gives out, so I’m going to take it. Because I’m from here and if they want to give me an award, I’ll have it, thank you very much and I’ll use it for whatever I can use it for. End of story. It’s about what you do, it’s about being recognised. If you don’t get recognition, it’s easier for them to forget you.”

McQueen was first given recognition when he was accepted for Chelsea School of Art, where he found a creative home of sorts. Likewise at Goldsmiths, where Gilroy remembers him “knocking on my door with a huge pile of Leni Riefenstahl photography books and talking nonstop about the history of cinema.”

At Goldsmiths and later at NYU, McQueen walked his own singular path, his fierce commitment to doing the work already evident. Gilroy reflects: “I think he went to New York with high hopes of finding a fertile environment for his ideas and energies, and it turned out to be a disappointing experience.” It was out of this context that Drumroll was created, and Gilroy muses that McQueen’s decision to doggedly roll a metal oil drum mounted with cameras through the crowded streets of midtown Manhattan might on some level have been a response to that increasing sense of frustration.

When people realise that you’re being open and you’re listening, that creates possibilities. They become drawn to you

As the YBAs made headlines and big money with their art provocations, McQueen remained an outsider. It is difficult, I say, to imagine him socialising with Damien, Tracey and co down the Groucho back in the day. “No, I didn’t,” he replies, matter of fact. “I went for a drink with some people once. That was it. It was” – he searches for the right word – “isolating.”

Moving to Amsterdam in 1997, he found a city more suited to his temperament, and has lived quietly there ever since with his wife, Bianca Stigter, and their two children. He told one interviewer that he cried when his daughter started school there because “it was so beautiful… so different.” For all that, he tells me, he still loves London, “but it’s hard, man, it’s fucking hard. When I was coming up, I got help from the Arts Council, from Channel 4, and without that I would not be here. All that has changed and we should be fighting to get it back. It’s just so bloody expensive to live in London now, but you know, there’s still some magic there.”

With Small Axe, a six-part TV series co-commissioned by Amazon and the BBC, and scheduled for broadcast in November, McQueen turns his attention to the London of a different time and to the lives of some of the ordinary British black people who went before him. It is a project he has been connected with for several years over a protracted script development period. Set in London between 1968, the year of Enoch Powell’s inflammatory “rivers of blood” speech, and 1986, and unfolding over six hour-long episodes, the series takes its title from an early Bob Marley song. McQueen described it as a reevaluation of “the journeys that my parents and the first generation of West Indians went on to deliver me here today, calling myself a black British person.”

Today, he is keen to emphasise that “it is not about the Windrush generation, but a later generation who were already established here”. The opening story, which unfolds over the first two episodes, revisits a defining moment in black British experience: the protests that erupted in West London in 1970 following the police closure of the Mangrove restaurant, a lively community meeting place on All Saints Road, and the ensuing trial of local activists dubbed the Mangrove Nine.

“There is politics in there from the start and the racism that people faced,” elaborates McQueen. “But it is also about their everyday lives; how people met, fell in love, how they danced and enjoyed themselves, the vibrancy of their everyday lives. One episode is about lovers’ rock, another about people meeting at a blues’ dance. So, it’s about the black British presence in a very real sense, how people found themselves through all those different circumstances.”

Small Axe is McQueen’s first foray into directing for television and, unsurprisingly, he has done it his own way. “Steve has a film sensibility and essentially he has made a series of individual films,” says one of its executive producers, Tracey Scoffield. “They remind me in a way of the great self-contained social dramas that the BBC’s Play For Today slot broadcast back in the 1970s. It was an extraordinary experience for all of us because you realise very quickly that you are working with someone at the absolute peak of their game. He was like a general on set, taking control, marshalling the troops. His energy and commitment makes everyone raise their game.”

McQueen’s work rate remains phenomenal. He has also just struck a deal with Amazon to develop a complex sci-fi thriller called Last Days, in which a female character tries to uncover a geopolitical conspiracy against a backdrop of ecological disaster and the rise of artificial intelligence.

When I ask Paul Gilroy what sets Steve McQueen apart, he says, “There are places that he wants to go in his art that are almost transgressive in their extremity: the sense of discomfort he evokes, the suffering in the body, the grief you cannot articulate. These are the very things he wants to articulate visually. That’s where he wants to be in his work.”

In many ways, Steve McQueen has set himself an almost impossible task, but that too may be part of his extraordinary drive, his singular pursuit of the essential truths. What, I ask, does he get from film-making? He thinks about the question for a long time. “There’s a power in it, but not the kind of power you might expect. There’s a power in being open, in being vulnerable, in being wrong, even, and admitting it and correcting it. When people realise that you’re being open and you’re listening, that creates possibilities. They become responsive. They are drawn to you. That’s one key part of being a director.”

Is there another key part? This time he doesn’t hesitate. “You want to cause a bit of trouble, stir things up a bit,” he says, grinning, then growing animated, the words tumbling out of him like a torrent. “We’re all a bit numb right now, so that’s even more important. It’s like, ‘Wake up! Wake up!’ Let’s make some noise. There’s a little bit of my youth in there, growing up in the late 70s and early 80s with punk, reggae, anti-racist protest. Anger is an energy and all that.”

Just as suddenly, he becomes quiet and reflective. “But it’s not just about anger,” he says. “It’s about seeing, contemplating, serious consideration. It’s about being seen, and heard and recognised, so as the years pass they can’t make you invisible. You want to ensure that what you do will have a lasting effect. That’s all you can hope for, really.”

• Steve McQueen opens at Tate Modern, London SE1, on 13 February