Rhodes Must Fall has failed. Rhodes Must Fall has succeeded. The statue high up on the wall of a college building on the High Street in Oxford will not be removed, instead receding into its former pigeon-spattered obscurity. But the student protest movement has sparked a valuable debate about how Britain deals with its colonial past. I think both these results are good ones.

It was a brilliant stroke of student activism to identify that obscure statue as the target. Every newspaper could print photographs of the honeystone facade in which it stands, looking Brideshead Revisited-cliché Oxford. Dave Spart biffs Evelyn Waugh.

Daily Telegraph readers would predictably chunter and international media pick up the story. The statue was just big enough to command attention and just small enough for there to be a sporting chance of something being done. In the event, Oriel College first said it was going to have a big debate about it and then, reportedly under pressure from donors, abruptly declared the statue would not be taken down – thus giving the Rhodes Must Fall activists an even better story. I foresee a bright political future for these guys.

Unlike the spreading student practice of “no-platforming” in the name of “safe spaces”, I don’t think there is much of a free speech issue here. As the Oxford historian David Priestland pointed out in a panel discussion I organised at the university this week, no one claimed free speech was being infringed when statues of Lenin were taken down across eastern Europe. Rather, this is a perfectly legitimate debate about the politics of memory.

But the Lenin comparison also shows up a difference. Lenin was a prominent symbol of a recent oppression of people in the country where the statue stood. So was a large statue of Cecil Rhodes that stood in front of Cape Town University in South Africa, until the original Rhodes Must Fall movement got it moved last spring.

But this Oxford Rhodes statue was neither genuinely prominent (I have lived in Oxford for years and never even knew it was there), nor a symbol of the recent, brutal oppression of most of those who live here. It is more like an obscure statue of Lenin somewhere in Russia today: a relic and a question to the former imperialists.

The debate about symbols is entirely legitimate, but the arguments for removing this particular symbol from this particular place are not strong enough. There should be a presumption in favour of the continuity of an ensemble of historic buildings in the centre of an old town.

More pertinent and practical might be to demand more Rhodes scholarships for African students, given that the money originally came from Africa. This is something that past and present Rhodes scholars from more privileged parts of the world should support. One of the student activists who spoke up at our debate was a Rhodes scholar from Zimbabwe (formerly Rhodesia). She said she regarded her Rhodes scholarship as a kind of atonement or restitution.

Like so many student movements, this one is both made and marred by its hyperbole. The current list of demands from Oxford’s Rhodes Must Fall movement starts with exhorting the university to “acknowledge and confront its role in ongoing physical and ideological violence of empire”. There is a huge amount that can be said about Oxford’s historic involvement with the British empire, including Rhodes, but implicated in current physical violence of empire? Where? How?

The truly liberal reaction is not to get distracted by this hyperbole but to listen carefully and engage with what the protesters are saying, while resisting anything that would make the university less open, free and pluralist. And they raise some important issues: the representation of people of colour among both faculty and students; the often subtle ways in which students of colour feel not wholly accepted in a university, even when there is no outright discrimination or racism.

A statue of Cecil Rhodes is removed at Cape Town University after weeks of protest by students. Photograph: Schalk van Zuydam/AP

Addressing these effectively, not just rhetorically, is quite complicated. There are people at Oxford who spend a lot of time trying to do so, but there is certainly more that we, and other British universities, can do.

The demand that touches me most personally is for “decolonisation of curriculum” and, more broadly, for Britain to face up to its colonial past. My grandfather was a member of the Indian civil service, the small band of men who governed India under the British empire. I have spent much time studying the way countries such as Germany face up to difficult pasts, whether fascist or communist. Only recently did I start wondering whether there was not a little facing up to be done in my own family.

Obviously, I was aware that bad things were done by British imperialists. But I think it is true that one can study history in Britain, and live as a politically conscious citizen here, without being pressingly confronted with this legacy. The British memory of empire is, I think, quite woolly – and that also means soft on ourselves. Unlike in Germany, there is little agonising about what your grandfather might have done. In a very British way, we just don’t talk about it.

This feeling is reinforced by visiting an exhibition at Tate Britain, Artist and Empire. Although its subtitle is Facing Britain’s Imperial Past, the facing in question seems more like peering into something remote, exotic and half-forgotten than confronting something morally difficult. Inevitably, most of what is on display is seen through the eyes of those who were on top, not those who suffered underneath.

A gentle foreword to the catalogue by Paul Gilroy explains why it’s all so complicated. A last room shows some images by the oppressed, or descendants of the oppressed, including an ironical reworking of a statue of the slave trader Edward Colston that still stands prominently in Bristol.

A painting by the black British artist Sonia Boyce is wonderfully entitled Lay Back, Keep Quiet and Think Of What Made Britain So Great. Exactly so. That’s us. It’s not that we can’t find this out if we want to. There are plenty of good books by historians, including some at Oxford. It’s that on the whole we don’t feel urgently compelled to enquire. Or perhaps I had better just speak for myself: I haven’t, until recently.

So thank you, Rhodes Must Fall, for violating my safe space.