It’s not just pollsters, the left, or the Labour and Liberal Democrat parties that have serious reflection to do after the “surprise” election result. We in the media do, too. The Conservative victory came as a surprise only to those who focused too much on the deluge of numbers and data, the TV knockabout of he-said-she-said arguments, and the manufactured campaign moments – and not on the real reactions of people in their actual communities, far from the Westminster village.

Travelling the country with John Harris for the Guardian’s Anywhere but Westminster series over the past 12 weeks, we only occasionally intersected with the campaign as seen on national TV, usually to satirise it. Going in to the final weeks, it increasingly felt like the Labour party were in trouble on the street – yet in the national narrative it was increasingly “too close to call”.

We found people were genuinely gripped by the idea that an SNP-led Scotland would be holding the country to ransom

We went into our final video from Nuneaton, days before the campaign ended, dead on our feet and with very little idea as to what we would film. Our video was based on little more than speaking to as many people on the street as we could in a 48-hour period, and reflecting that truthfully in an edit. We found a fairly broad sense that this was not a “change moment”, and that people were genuinely gripped by the idea that an SNP-led Scotland would be holding the country to ransom under Labour.

Our series began six years ago, when John Harris and I were sent to cover the political party conferences; we soon realised that a more interesting and revealing film could be made outside “the bubble” of the party and its entourage, than within.

The key principles of Anywhere but Westminster have always been:

• Go Anywhere but Westminster

• Don’t follow the media herd

• Prioritise speaking – and listening – to members of the public

• Don’t script out your story before you even start filming

• Visually and narratively break away from the conventions of TV news

Too often, TV news plays along with, and therefore encourages, the manufactured scenes of an election campaign. Why send your key correspondents and camera crews to film David Cameron doing handprints with eight year-olds, or Ed Miliband carting his lectern to a field for a carefully choreographed Q&A? Why not pull the camera out for a wide-shot that shows the event for what it really is?

We have a broadcast news media that is largely bound by conventions, and these are as much about style as they are about content: the “noddies”, “pieces to camera”, cutaways and set-piece interviews. The public, when they do appear, are used as soundbites, chosen to fit what feels like a pre-existing script. This is compounded by adherence to Ofcom impartiality rules that result in carefully balanced presentation of views, even if that is not how the story feels on the ground. I treasure the importance of fairness in news, but too often truth is sacrificed at the altar of a strictly formal objectivity.

The result is that news “packages” rarely feel like voyages of discovery, but have the very same sterility as the political campaigns. Add to this an over-reliance on polling data and electoral arithmetic, and you end up with a media narrative that is as distant from the people as we accuse the politicians of being.

This election has weakened my faith in opinion polls and affirmed my belief in the power of the humble vox pop

This election has weakened my faith in opinion polls and news packages, and affirmed my belief in the power of the humble vox pop. Vox populi, “voice of the people” – the phrase has become a bit of a dirty word in journalism, the job you send the junior reporter to go and do. But we need to reclaim it, and rediscover the art of speaking to the public – this applies to both our politicians and our media. Politics is, after all, a social science. It cannot rely on quantitative findings, but needs meaningful, direct encounters with people to understand what is going on.

Asking people straightforwardly who they will vote for, or what they think of Ed Miliband or a particular policy, tells you little on its own. This is especially true if you are a marketing company on the phone, or a focus group with a carefully balanced audience brought together in a hotel room. People will often answer according to their perceived role.

But go to their town, ask them how the place is doing, or what they think 70 years of voting for the Labour party has done for where they live, or why they still vote for a certain party – and you start to hear recurring themes that have a ring of truth. Seeing and hearing how they answer is as important as what they say. And if you don’t have a script, your story will be led by what people tell you, and not vice versa.

Our final film felt very gloomy about the prospects of the Labour party, and wasn’t necessarily what we wanted to report. It didn’t match what the numbers were saying, and it wasn’t a polished piece of work by TV standards. But it’s a good place to start if you want to understand why the Tories won.