I spent my Saturday in a south London lecture room festooned with St George's flags, at an event attended by vocal factions of English Democrats and Campaign for an English Parliament. Billed as a festival of Englishness, titled England, My England, the day involved fish and chips and a demonstration of broadsword fencing, and was rounded off with a glass of English ale.

I wasn't the only one to feel a bit queasy about all of this. In fact, the day was not quite what it seemed. It was organised by the left-leaning Institute for Public Policy Research and British Future, whose director, Sunder Katwala, is former head of the Fabians. Its starting point was a piece of IPPR research that confirmed the increase in English residents identifying themselves as English rather than British, and found that those who do are more likely to be anti-European, resentful of the Scots and inclined towards Ukip. Although not framed this way, both the organising groups clearly fear that if the left doesn't address, indeed celebrate, Englishness, it will leave an open goal for the right. Hence the day not being "a conference about" but "a festival of".

Not unsuitably, there was a tone of gentle, self-deprecating irony about much of the proceedings. Several people on the culture panel (two novelists, a poet, Vanessa Whitburn of the Archers and me) challenged the premise of the day. On the comedy panel, it was revealed that, for the English, the top three most characteristic comedy characters – Del Boy, Basil Fawlty and Hyacinth Bucket – are either socially insecure or bad at their job, or both. The sport session applauded the heroic failure of Eddie the Eagle, while suggesting that winning the 1966 World Cup set British football back 20 years. As in Ed Miliband's 2012 speech about Englishness, a general desire to address the topic ("we have been too nervous to talk of English pride and English character") was matched with few policy proposals. The most frequently suggested were a St George's Day bank holiday, an English national anthem, and the exclusion of Scottish and Welsh MPs from voting on matters devolved from Westminster.

There is, of course, a reason for keeping it vague. If you exclude those constitutional and institutional arrangements that apply across the UK (which give us democracy, the rule of law and our hard-won freedoms), and then knock out those heartwarming characteristics that we like to claim as our exclusive possession (as if the Canadians were intolerant, the Finns not fair-minded, the Peruvians indecent), then you are left not with the question of whether there is a meaningful definition of English culture, but the problem of which definition to choose.

Is it the "deep England" of the countryside, the Anglican church or, indeed, the Archers? Or the England of the freeborn radical, the Levellers, Chartists, Tolpuddle martyrs and suffragettes? Is it urban, working-class England in its various, and sometimes contradictory, manifestations, from pubs and the music hall to northern non-conformity (and comedy)? Or love of the language and its literature (and, if so, is that devalued by a more internationalist attitude to art forms that don't bump up against the English incapacity to speak foreign languages)? And is celebrating England as a place that is open and welcoming to outsiders – admirable and often true – really a definition of what is particular to its character?

The truth is that most Englishnesses – including those listed above – are records not of consensus but contest, then and now. Today's working-class culture is what survived a largely English government's assault on its workplaces, its institutions and its communities. The Countryside Alliance was not set up to challenge European or Scottish prejudice against killing small animals, but against urban England. Fans don't complain about the Archers becoming too much like Australian or Brazilian soaps, but too much like EastEnders or Emmerdale. And, of course, the rights of the freeborn English were not won from the French or German governments but from other English people who devoted the full resources of the British state to denying them.

So the problem with the question at the heart of the debate – the ringing, historically charged "Who speaks for England?" – isn't just a natural suspicion of any question to which the answer might be Boris Johnson. It's a matter of which England is being talked about, and whether any of those possible Englands have sufficient coherence to trump both what separates some English residents from others, and the growing divisions within England itself.

Of course, One Nation Englishness could prove a flag of convenience for a campaign against the yawning disparaties of wealth, the hollowing out of local democratic institutions and the demonisation of the poor. Clearly, there have been success stories in reclaiming the union flag and the flag of St George from the far right. But the queasiness about those symbols remains. As was said on Saturday, for many black and Asian people, both flags evoke memories of National Front marches and assaults in the 1970s, and EDL demonstrations today. The idea that, deep down, still, there ain't no black in the union jack is reiterated every time a Conservative leader evokes Enoch Powell's grim dystopia – from Margaret Thatcher's 1978 "swamping" statement via William Hague's 2001 "journey to a foreign land" speech to David Cameron's 2011 warning against the "discomfort and disjointedness" created by immigrants into settled neighbourhoods. And that's before the government started texting suspected overstayers and sending vans round London telling foreigners to go home.

There are bits of Englishness I'm fond of, bits I can leave alone. I love the coastline, but not the cuisine. Just because I live in the West Midlands, that doesn't mean I have to prefer the pre-Raphaelites to the Impressionists, or Elgar to Beethoven. I'm delighted that both technology and migration have given most people a much wider cultural choice. Now let's get down to building a fairer, more just society, not because it's English but because it's right.