It was a throwaway line in a TV review. Writing about Channel 4’s Benefits Street, I wrote: “But look at the community spirit, you don’t get that in Chipping frigging Norton, do you?”

It probably didn’t help that it was picked out for the headline. But anyway, the complaints came firing in. “You actually do get a lot of community spirit among ordinary folk of Chipping Norton,” tweeted @SueFJones, who later added: “don’t tar us all with the awful Chipping Norton Set.” From @ Beanylfc: “You obviously have no idea what it’s like to live in Chipping Norton.” “Rude!!!!” tweeted @BeccaJMarriott. “Complete balderdash!” added @ ExperienceCN. And so on.

Bit chippy, in Chippy, it seems. To be fair, so was the criticism. I was dumping the lot of them in with the Chipping Norton set, unfairly. I just wanted a place that was, to my mind’s eye, the antithesis of Stockton-on-Tees (where the current Benefits Street comes from), in terms of wealth and politics and community spirit; a place where if anyone is speeding dangerously through the streets, it’s not hoodied youths on stolen mopeds, but more likely to be Jeremy Clarkson in a Ferrari; or David Cameron on a police horse loaned to him by Rebekah and Charlie Brooks.

David Cameron is a wonderful person Tahirul Hasan

Taxi driver and town councillor Tahirul Hasan with David Cameron in Chipping Norton. Photograph: PR

Not only do I have no idea what it’s like to live in Chipping Norton, I’ve never even been there. So when @ Rupert_Parsons tweeted me: “You are hereby cordially invited to visit our community of Chipping Norton”, cc-ing @ExperienceCN and @ChippyNews, I felt I had little choice but to accept. Rupert also cordially offered to take an afternoon away from making award-winning Womersley fruit vinegars, and mending clocks, to show me around.

First stop: the lido. Trustee Claire Jarvis talks me through the pool’s history, a story full to bursting with community spirit. In the 1960s, local people raised the money to build it. When, in 2002, a leisure centre was built in town and the district council decided to close the old lido, the community stepped in again, campaigning to save it. Now it’s a registered charity, run by the community, for the community.

“All generations come here and mix, spend the whole afternoon, play on the grass and dip in and out – across all ages and every demographic,” says Claire. “It’s our day at the seaside.” (Chipping Norton is about as far as you can get from the actual seaside.) The local MP – also the PM – has been. The local Gypsy community frequent the place. True, Clarkson did once drive a Rolls-Royce into the pool for some TV show he used to be in, but they let him, because he also does the charity auction. On a chilly weekday there’s no one at all in the pool, but I’m getting a picture, on a hot summer day, of a kind of My Big Fat Classless Picnic Paradise, where everyone’s the same in a cossie.

The wealth – and the set – is all outside the town, in the villages, Rupert and Claire explain. “People don’t appreciate the levels of deprivation,” says Claire. She tells me, proudly, that in the 1970s they had the highest proportion of social housing of any borough in western Europe. Does it make her cross when people get the wrong idea? “I nearly choked on my cornflakes,” she says about what I wrote. A Guardian reader, in Chipping frigging Norton! And a Labour voter – she’s still got the posters in her car. Rupert voted Green. In fact, two of Chipping Norton’s three regional ward councillors are Labour, a tiny island of red in an ocean of blue.

Middle Row, Chipping Norton. Photograph: Alamy

In the independent bookshop at one end of the pretty town square, I meet Jo Graves, who taught at the primary school for 20 years and now sits on the town council on an independent ticket. “I truly choked on my muesli,” she says. Oh gawd, not just a Guardian reader but an actual muesli-eating one, in Chipping frigging Norton. It was the ignorance of it that really annoyed her. Sure it might not be the working town it used to be, it has become gentrified (you only need to look at the cars and the prices in the estate agent windows), but there’s another side to it. And she, too, talks of the deprivation and poverty, the high proportion of council houses, the food bank, and – of course, the community. “I think that David Cameron got his ideas of the Big Society from Chipping Norton, because there is so much done in the community. I don’t think I know about any other society where, proportionally, so many people give up so much time voluntarily.” Jo describes the prime minister as “a very pleasant man and a very good constituency MP”. I’m finding it hard to find a bad word about him, even from people who might not agree with him politically. Bloody hell, they’re generous to everyone round here. Even the alpacas at Fairytale Farm, a sensory learning attraction for all the family, especially those with kids with disabilities, are friendly.

To the theatre, where Anne Gill and Vanessa Managhan talk about the brilliant and extensive community (obviously) and the educational work they do, and Rupert tells me how a youth programme there made a such a big difference to the confidence of his son, now grown up and working for Thames Valley Police. At the Chipping Norton Community (!) Hospital, Jenny Nolan tells me about the Lawrence Home Nursing Team, which provides care for local residents who want to die in their own homes. It’s very well supported by the local community, naturally; just the other day, they collected £636.88 in a single morning. The PM, who was in town, contributed generously, of course. To both buckets.

Time for a pint at the Red Lion where Experience Chipping Norton, a local organisation that promotes trade, tourism and – of course – community, is having a get-together to find out how they can help each other. It may not be the same kind of community spirit as you find on the Tilery Estate in Stockton-on-Tees, where single mum-of-six Julie helps out a neighbour who’s been turned down for a crisis loan, but this is a different kind of community. What’s become clear is that Chipping Norton doesn’t just have it, community spirit is Chipping Norton’s middle name. Chipping frigging community spirit Norton. I was wrong, I’m sorry.

“It’s a very good community, very friendly,” says Tahirul Hasan, my taxi driver, on the way back to Kingham Station. Yeah, I know; I’m bored now – I’ve said sorry enough. And Hasan is not just a taxi driver, he’s a town councillor too, and a Tory. Ah, a true blue, at last. And, he says, the only non-white councillor in all Oxfordshire. He is also “very close to Mr David Cameron. He is a wonderful person.” And he shows me a picture on his phone of the two of them together, to prove it. And another with Jeremy Clarkson. Maybe I have finally met one of the Chipping Norton set – even if he is their least known and least likely member: a taxi driver in a people carrier from Bangladesh.