A few months ago, I found myself in a taxi from the airport in Paris, having returned from Japan, where I had been to promote my first novel, The End of Eddy. We had driven a few kilometres when the driver, a tall man from Ivory Coast, asked me what I did. I can never bring myself to answer that I’m a writer; I’m too afraid of seeming arrogant or of giving the impression that I’m trying to hawk my books. Most of the time, I’ll say I’m a literature student, and that’s how I answered on this occasion. “Oh,” he said, “so that’s the kind of thing that interests you, books?” I said yes. He went on, “You know, myself, I don’t read books. But I can tell you one thing, I’ve noticed that in France, when they give out literary prizes, the Goncourt and all the others, they pretty much only give them to white people who write about white people. Have you noticed? Everything is for white people.”

I nodded in agreement. I learned a long time ago to keep my face expressionless, and I’m sure that the driver could not see the effect that his remark had produced in me. He had stated something so obvious, and yet something I had never been able to articulate quite so clearly: even without reading, even without any contact with books, this man understood that literature and its institutions, the system it was part of, didn’t only know nothing about lives like his, it actively excluded them. His words had taken me abruptly back in time 15 years, to my childhood.

I was born in a small village in the north of France, where up until the beginning of the 1980s a local factory employed almost all the inhabitants. By the time I was born, in the 1990s, after several waves of layoffs, many inhabitants were out of work and doing their best to survive on welfare. My father and mother quit school at the ages of 15 and 16, as had my grandparents before them and as would my younger brother and sister. My father worked at the factory for 10 years, until a weight fell on him and destroyed his back. My mother didn’t work; my father insisted a woman’s place was at home taking care of the children.

Literature was not something we paid any attention to – quite the opposite. On television we would see that literary prizes went mostly to books that did not speak of us, and in any case, just like the taxi driver, we were aware that, prize or no prize, books in general took no interest in our lives. My mother would say it over and over: us, the little folks, no one is interested in us. It was the feeling of being invisible in the eyes of other people that drove her to vote for Marine Le Pen, as did most of my family. My mother would say: she’s the only one who talks about us. The Front National got more than 50% of the vote in the village where I was born, and that vote was above all, beyond racism, beyond anything else, a desperate attempt to exist, to be noticed by others.

Many of the authors who have meant the most to me, such as James Baldwin, Simone de Beauvoir or Didier Eribon, wrote about the liberating effect of literature in powerful ways that continue to affect new generations of readers. Yet however different Baldwin’s childhood was from De Beauvoir’s, mine was like neither of theirs: in my childhood, there were no books. My parents have never read a book in their lives; there wasn’t a single book in our house. For us, a book was a kind of assault: it represented a life we would never have, the life of people who pursue an education, who have time to read, who have gone to university and had an easier time of it than us.

As for school itself, that experience had driven my parents out of the education system and denied them access to culture at the age when middle-class children were just beginning their studies. Culture, the education system, books had all given us a feeling of rejection: in return, we rejected them. If culture paid us no attention, we would have our revenge. We despised it. It should never be said that the working classes reject culture, but rather that culture rejects the working classes, who reject it in turn. It should never be said that the working classes are violent, but rather that the working classes suffer from violence on a daily basis, and because of that they reproduce this violence by, for example, voting for the Front National. The domination comes first; those in positions of dominance are always responsible.

I am more aware than some of the violence that literature can represent, because at a certain point in my life, I made use of that violence to hurt the people around me. Thanks to a series of accidents and failures, I made it into a lycée and then to university. I was the first person in my family to do this. During the week I would board at school or stay with friends, so I would only spend weekends with my parents. As soon as I walked into the house, I would sit on the sofa with a book, one that, most of the time, I would only be pretending to read. I wanted to let my family know that I wasn’t like them, that I no longer belonged to the same world as them, and I knew that a book would be the most violent instrument I could use to do that.

Today, authors such as Zadie Smith work obstinately to invent a more welcoming and inclusive literature.

Today, all I feel is shame when I think back: shame in the face of my brutality and arrogance. But at the time I didn’t think, I was just trying to get away. I was too proud to have escaped from my family’s social circumstances; I was an obnoxious fool. In the evenings, our meals would nearly always end in arguments. I would be speaking and my mother would interrupt: stop talking like a damn book, you’ve got nothing to teach me. She would say this with a mixture of anger, sadness and disgust. I, on the other hand, would hear her remarks as compliments: finally I belonged to the world where people read books.

Rather than saving us, books were what kept us down. A book by Hemingway was much more violent, from our point of view, than a photo of Trump in his enormous gold-covered living room. The photo would have left us dreaming of gold and riches: my mother spent hours looking at pictures of huge houses on the glossy pages of magazines. The Hemingway, by contrast, gave us nothing to dream of: it would have left us feeling defeated.

Are books doomed to reproduce such social barriers? There is one counterexample among my memories. A little while after my trip to Japan, I was invited to give a talk in Oslo about an author I loved. I chose Toni Morrison. When I walked into the room I was struck by the large proportion of black women in the audience. I spoke with many of them after the event: some had read Morrison, others hadn’t – but all of them felt welcomed by her books. They knew that novels such as Jazz or Home were addressed to them – not to them only, but above all to them. Today, authors such as Zadie Smith, Ta‑Nehisi Coates and a few others work obstinately to invent a more welcoming and more inclusive literature. I don’t mean to suggest that Morrison publishing a book is sufficient to interrupt the reproduction of the social order or social inequality. But at least literature will have done its job; it remains for the politicians to do theirs.

My books are born out of an absence: I began writing because I could not find the world of my childhood anywhere in books. We had not had the good fortune to find our Morrison. This is the literary revolution that is necessary today. As long as a large proportion of books are addressed only to the privileged elite, as long as literature continues to assault people like my mother or the taxi driver, literature can die. I will watch its death with indifference.

• Translated by Michael Lucey. Édouard Louis is the author of The End of Eddy, published by Harvill Secker. To order a copy for £11.04 (RRP £12.99) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.