On a recent pre-season visit to Boreham Wood FC, my eye was drawn to the programme notes detailing the match mascots. Each one supported a Premier League club, with a secondary mention of their favourite Woods player shoehorned in. This may seem a trivial observation but imagine a local Salford lad stumbling out on to the Old Trafford pitch, with an outsize shirt flapping round his knees, being introduced as a Barcelona fan. Learning that your matchday mascot supports a different team took me back to a past life following Kidderminster Harriers.

There, a similar story was frequently played out in the Conference. Although a predominantly semi-professional league in those days, it still featured strong sides capable of pulling in four-figure crowds.

But in all my years following the club – and this may be the output of a selective memory – the only mascot I recall pledging a primary allegiance to the Harriers was me. Even my friend and co-mascot on the day (who, as the excitable PA announcer, Captain Crazy, remarked, "comes all the way from Ludlow!") kept a safe distance, declaring himself to be an Aston Villa fan.

It was a curiosity that manifested way beyond the then-corrugated stands of Aggborough. Supporting a non-league team in a school obsessed with Liverpool and Manchester United, the stock response when you said who you supported was usually either "who are they?" or, even worse, "but who is your real club?" It's unclear why revealing primary allegiances to non-league teams elicits so much incredulity.

Maybe it's the dated image of beer bellied forwards hoofing the ball into neighbouring gardens, or the stigma of following "park football", but sometimes you imagine confessing to such activities will lead to a cross being painted on your door. Nowadays, as a regular follower of Wingate & Finchley of the Isthmian League, I find myself offering an almost apologetic justification when asked who I support, followed by a pre-emptive "you probably wouldn't have heard of them". Usually I'm right.

Declaring these loyalties to smaller clubs is like a high-stakes card game. Supporters are often faced with a deal or bust decision: stick to your guns and hope for the best or choose a big side with tenuous connections and suppress feelings of desertion and treachery. Non-league devotees often lead a duplicitous life, where who they say they support and who they actually support can be two vastly different things.

To cite an example, a Stourbridge-supporting acquaintance of mine wrestled with this conundrum on starting a new job in Birmingham. Having never had an affiliation to a "big" team, he mulled over replying "West Brom" or "Aston Villa" if anyone asked, as he would "feel a bit silly saying Stourbridge". Further afield, a seasoned follower of Halifax Town I know resorted to saying "Leeds United" when asked the question abroad, to avoid the inevitable looks of confusion.

On the other hand, there are distinct advantages to following clubs at this level of the pyramid. Non-league paraphernalia are like magic amulets. I was challenged on the Tube once about the claret and blue scarf round my neck. Once I had revealed that it was emblazoned with "Colwyn Bay FC", the demeanour of my would-be assailant changed and he threw in a congratulatory "good for you", as if addressing someone recovering from a critical disease.

Inevitably, most non-league clubs and their followers accept all this as an occupational hazard. But an inverse snobbery is on the increase among fans of lower-league or non-league sides. In a commercially driven age where the British football supporter appears increasingly homogenised and brand-driven, it is sometimes the fan of the top-flight club being asked who their "real" team is.

Of course, you could forgo the need to acquiesce to external pressures or one-upmanship and replace conventional wisdom with simple pragmatism. As one Telford United fan I met resolutely states: "When asked 'but who else do you support?' I say 'Witham Town'. This usually ends the conversation."

• This is an article from our Guardian Sport Network

• This article first appeared on When Saturday Comes

• Follow When Saturday Comes on Twitter