Over the centuries Europeans of various strains have tried to fulfil their fantasies in Africa. I should know because I'm one of them. Not that I have ever nursed urges to convert and conquer, trade and enslave, or paternalise, dominate and discriminate. But when I set off to Angola at the end of the summer of 1998 I was just one of many who had hoped to contribute to a socialist project on the continent.

I was convinced that, at its core, the ruling Popular Movement for the Liberation of Angola (MPLA) was a radical socialist movement that epitomised the heroism of African liberation. I had been inspired by the writings of Basil Davidson and other British Marxists who left me in no doubt about the integrity of the MPLA under Agostinho Neto, the first president of independent Angola. Unlike its CIA-backed rivals – the National Front for the Liberation of Angola, and the National Union for the Total Independence of Angola (Unita), the latter having allied with South Africa's white minority regime for the best part of two decades – I believed that the MPLA had fought for the freedom of all Angolan people regardless of their ethnic origin, place of birth or skin colour.

That said, I also knew that after the fall of the Berlin Wall the MPLA had made a political U-turn. Abandoning Marxism and Leninism, it had adopted a market-driven economics that morphed rapidly into crony capitalism. The power of the one-party state, which had endured since 1975 until flawed elections in 1992, was now concentrated in President José Eduardo dos Santos. Nevertheless, like many on the left my loathing was focused so intensely on Unita that it was easy to view the MPLA as little more than a cold war victim of US foreign policy.

When I arrived in Luanda, the MPLA had long been – and still is – a member of the Socialist International, an organisation that claims to pursue "progressive politics for a fairer world". I remember my pleasure on hearing politicians and other members of the urban elite calling each other camarada (comrade). Even the party rhetoric sounded remarkably similar to that of the revolutionary years of the 1970s. But a few months into my new job, when the country's "fourth war" finally erupted, I could no longer hide from the blindingly obvious: if revolutionary politicians were what I was after, I was at least 20 years too late.

In fact, this was also wrong. I began to discover that the idea of a 1970s MPLA heyday was just as misguided. An Angolan colleague told me about 27 May 1977, the day an MPLA faction rose up against the leadership, and the honeymoon of revolution crashed to a halt. Some called it an attempted coup, but my colleague insisted it was a demonstration that was met with a brutal overreaction.

Whichever story you believe, six senior members of the MPLA were killed that day by supporters of the uprising. In response, President Neto, the politburo and the state media made many highly inflammatory statements that incited extraordinary revenge. In the weeks and months that followed, thousands of people – possibly tens of thousands – were killed. Some of the executions were overseen by Cuban troops sent to Angola by Fidel Castro to repel a South African invasion.

I found this knowledge profoundly challenging. It turned everything I thought I knew on its head, especially when I began to understand that the 1977 purge cemented a culture of fear that has shaped a generation. How, I asked myself, had this appalling event remained so little known outside Angola?

The question began to obsess me. Back in London several years later, I started searching through my book collection for references to what Angolans refer to as the vinte e sete (the 27th). I found the odd sentence here and there; in one book, a few paragraphs. But what rattled me was that Angola-watchers on the left – intellectuals whom I admired – all seemed to have turned a blind eye to the thousands of killings. It was as if their commitment to the party was so deep that, in the end, they heard only the voices of its leaders and fell deaf to the calls from below.

Often I felt torn between my socialist beliefs and the search for truth. At one stage I became so disillusioned with the politics of revolution that I came close to giving up – on everything. But the words of an old friend, a man who was imprisoned and tortured by the MPLA in the 1970s, kept me going. "We cannot be afraid," he said. "We must write what we see and what we feel. Don't worry about what people will say … just get it down."

The dilemma of whether to tell the truth or keep stumm is hardly new. The European left has a history of toeing the party line – it is called discipline and unity – in the pursuit of freedom, equality and justice. The Spanish civil war is an obvious reference here, exemplified by George Orwell's account of his personal experiences with the Spanish communists in Homage to Catalonia. Let's not forget that Victor Gollancz, Orwell's publisher, refused to print it, "believing, as did many people on the left, that everything should be sacrificed in order to preserve a common front against the rise of fascism". Over the course of the past century the "sacrificed" range from the millions of victims of Stalin's brutality, via Cuban writers tiring of dictatorship, to female comrades in the Socialist Workers party seeking justice for alleged sexual abuse.

I know there will be some people who will insist, as the French philosopher Gilles Deleuze has, that "arguments from one's own privileged experience are bad and reactionary arguments". But in considering the case of Angola and the MPLA's record of brutality, it seems to me that privileging ideological theory over people's lived experiences, which are almost always contradictory, complicated and fuzzy, is far more dangerous. What many of us on the left have failed to see in Angola is that, for the majority, politics has always been about far more than merely a battle with the right.