I’m going to talk about something that, until now, I have largely kept to myself. It’s odd, as I consider myself a writer of extreme honesty, and I try to carry that over into real life. And yet, even now, I’m hesitating, and I realise to some degree I have procrastinated even more than usual about the thinking, and writing, of this. The committing of a hidden life event to the written word. That’s always a scary act.

I used to wonder if my reluctance was driven by shame, or simply my incredulity at what took place all those years ago. Now, I think that it is those things mostly, but also a hell of a lot more. Over the last few years, particularly in the recent crosswinds of our racial and cultural political climate, this life event bubbled to the surface of my memory, never quite boiling over. I’ve talked about it to a few of my close male friends, but that’s it. I almost never mention it to women.

So, OK. I’ll stop stalling and try to be as straightforward as I can.

A few decades ago, when I was just becoming a published author, I was discussing projects with various companies. In one, I dealt with a white male creative, and, when he left, I was assigned to someone else, a white woman. I was overjoyed to be taken seriously at last, a bit starry-eyed from the blitz of media and publishing parties, both of which I was unused to. My new contact, charming and jovial, was full of great ideas and encouragement. We hit it off, and got to work right away.

I’d travel into the office several times a week, full of excitement. I was young and eager to change the world. We’d sit in a room together and thrash out story outlines. Almost right away, my editor began making personal comments that I found highly unprofessional. She had black female friends, she said, who would “love” me. She said I was cute, and, sometimes when we were sitting at a desk side by side, she would stare into my face when we were meant to be working. It was unnerving, and, while I appreciated the compliments, which would occur every time we worked together, I began to feel a little uncomfortable in her presence.

Then she suffered a small injury. There was a meeting due, and she called me up, insisting that I come to her house. Given what had been going on in the office, I wasn’t that keen, so I asked if we could meet in a public place. She refused. We went back and forth until the conversation ended with her screaming down the phone, swearing at me and insisting I came to her house. I refused. The following day, someone in the company rang me up to inform me I had lost the job.

I tried to fight it, but there was nothing I could do. The whole deal collapsed. I knew what had happened to me was a commonplace occurrence for women, and I’d long felt outraged about that reality, but I quickly saw there was no outrage for me. When I spoke to anyone about what happened, there was a sympathetic shrug and a change of subject. So I responded the same way the majority of people would in this situation. I let it go.

It’s clear to me that this incident is an example of white female privilege being used to dominate a young black man. I was perceived to have no recourse, no agency. I had to submit to being exoticised in accordance with the hypersexualised stereotype that black men are often framed by. When I refused to reciprocate, I was punished. It wasn’t the first time I had seen this happen, and it wouldn’t be the last opportunity I would lose because of something said about me by a white woman. My most recent loss was a university teaching post. The interventions of other students saved my professional reputation, but I lost the job anyway.

Many years after, as the fallout from Harvey Weinstein’s behaviour rippled and spread, and the #MeToo movement rose in its wake, I was reminded of my first experience in the media, of negative experiences involving white women from childhood to the present day and of events that happened to friends, or have occurred within the wider diasporic black community. And, although it’s obvious that none of my personal experiences come anywhere close to the heinous crimes of rape and enforced sexual harassment committed globally by men, I have seen white privilege used by women as an oppressive tool far too many times to believe there should not be the same level of accountability.

I’ll hold my hands up and admit this is a complicated thing to tackle. I know this, and it has in part fuelled my hesitance. So, while I fully support any movement that seeks to address the rampant misogyny and patriarchy driving our society, which of course includes men of colour on too many occasions, I wonder if it’s possible to have a conversation about the role white women play in the continual oppression of black men; to speak about this in a historical context, tracing the direct line from enslavement and colonisation to the present day. To have an honest discussion about the fact that white women, who obviously face a cis, white patriarchal system of oppression, also use that patriarchal system to oppress those perceived as lower on the racial and social hierarchy?

I believe we must.

Of course, I’m not writing to generalise a whole race and gender. Many white women do not use their privilege adversely. Many are allies, instrumental in standing beside us, even speaking on subjects such as this. They exist. We see them and acknowledge their presence. That much should be obvious, although I feel it must be stated here to avoid the very real chance of being misconstrued. Still, the fact remains that black men’s relationship with white women is fraught with complexity and rarely addressed, except, sometimes, as an examination into the lengths some men go to mine sexual otherness, or exoticism: black men’s appreciation sex clubs almost exclusively visited by white women, mock plantation orgies, “beach boy” holidays in Africa and the Caribbean. These examinations are usually from a feminine perspective. What’s missing is any deep analysis of black male psychology. The mental displacement needed to attend those parties and become a “bull” for the night, or be paraded on the arm of a white woman in Hastings, Barbados. Is sex work less morally demeaning if a man is the sex worker and a woman the client? Why is this seen as less mentally destructive, or nuanced?

White women have said I’m not good-looking enough to be a successful writer

Social media is a slippery ally, on occasion doing as much to hinder as help the cause of social justice, but it has been exemplary at charting the numerous cases of white women caught in “acts of oppression”. The Memphis property manager’s insistence that a black man shouldn’t wear socks in a swimming pool (she called the police when he refused to leave), for instance. Or the woman who rang 911 after seeing a group of black people barbecuing in a park in Oakland, California. Or the employees who called the police on black men “loitering” in a Philadelphia Starbucks while waiting for a friend. And the woman who threatened to report an eight-year-old black girl selling water in San Francisco – and even a Hispanic woman sheltering from the rain in New York.

At the same time, I have often seen social media posts where black men are accused of having very little to complain about, other than whether white women “like” us; if a little thing such as our likability and racism in general were cleared up, black men would declare the problem solved. It’s also disappointing how often these declarations are written, and intensely felt, by black women. It seems an odd conflict; on the one hand, social media proves that contact with certain types of white women can ruin your day, if not your life. On the other lies the argument that all black men have to fear is whether we’re deemed sexually viable by said women.

All I can say at this point in time, as a solo writer putting one word after another, is a feeling: intense isolation, vulnerability, the wariness that comes from needing to trust in order to continue with our lives, yet having that trust broken time and again. The fear of being in close proximity with people who may become colleagues, family, lovers, assailants, accusers, abusers or harassers. The danger of loving someone who might possibly racially abuse you in the furious heat of a domestic argument. The confusing seesaw desire of wanting to be an ally for someone’s struggle while not having your struggle recognised in return.

Among my lived experiences, I have had white women punch me for getting on the bus before them, say that I’m not good-looking enough to be a successful writer, rub against me on a packed tube train as if I wasn’t even there, and, as I’ve said above, be instrumental in my losing two jobs that I know of, possibly more that I don’t. After that second incident, I was left in freefall, jobless, with a child to raise and a mortgage to pay. Being liked wasn’t the issue; my survival was. Something – dumb luck or the spirits of my long-deceased grandmothers – came through for me. I prefer to believe the latter. Throughout it all, and every incident before or since, I have tried to walk as good as I can muster, and live. That’s all any black man wants really, to walk in a straight line and to live. Maybe one day we will.

Extracted from Safe: On Black British Men Reclaiming Space, edited by Derek Owusu, published by Orion on 7 March.

• This article was amended on 28 February 2019. The woman in New York who was reported while sheltering from the rain is Hispanic, and not black, as we originally said.