The reputation of football fans appears to be on the up again. Campaigns calling for a fairer deal on away tickets and the introduction of Safe Standing have shown how supporters can change attitudes when they act together. But scratch the surface and there remains hostility to a group despised by their peers and mocked by broadcasters. They are the fans who leave games early.

This sub-genre falls into two camps. The first will head off no matter what the score and deserve all the scorn that gets thrown at them. It wouldn't matter if their team were grimly holding on to a lead or laying siege to the opposition's goal looking to grab a late draw, at the appointed hour they will gather their belongings, make their excuses and head for the exit.

To attend a game and then leave when the result is still not certain just defies common sense. But there is a second group of early leavers who deserve to be treated with a bit more respect and I am one of them.

Saturday 16 January 1999 was a typically cold winter's day. I was in my usual seat in the Bobby Moore Stand at Upton Park as West Ham took on Sheffield Wednesday. Like many games at the time, West Ham were pretty abysmal and, with just under 20 minutes left, Benito Carbone scored a penalty to put the Owls 4-0 up.

I then received a text from my brother who was sitting with my dad in the East Stand. If I wanted my lift home I would need to meet them in five minutes. They'd had enough. There was no feeling of guilt as I left. The team weren't trying for me so why would I hang around to watch them? Especially as it would mean having to travel home by train rather than car. Yet for some reason fans that choose to walk out of games when their team is getting hammered are treated with the same disdain as the clockwatchers.

Just to be clear, these are not the sort of people who left the 2005 Champions League final at half time. They are the ones who can see that with 20 minutes to go both the current score and performance of the team mean a comeback is highly improbable. Leaving a ground at the end of a match can be a traumatic experience – there are queues everywhere.

You queue to get from your seat back into the concourse, then to get out the stadium and that's before you've even thought about the queue for the train or on the roads. If you're already having a miserable day because of your team, there is no point prolonging it. Inside the ground there is rarely any animosity to the fan who makes this choice. Everyone is thinking it, but some fans, for whatever misguided reason, choose to always stay until the end.

When I have mentioned that I'm quite happy to leave a game if my team are getting thrashed I've been accused of disloyalty. In my mind it's actually the people who stay who are doing more damage. Walking out is a form of protest. You are saying to the players that they do not deserve your support.

Those confused souls who choose to stay are actually telling the team that however badly they perform, however much they choose not to care, they are still valued. These fans also like to say that when the final whistle goes they can vent their frustration by booing.

A far better way of getting the message over would be if, when the final whistle went, your team walked off to stony silence as everyone had gone home. In recent years broadcasters have started honing in on supporters who head off early, a picture which is usually accompanied by some snide remark from a co-commentator. Inevitably this is then followed by criticism on social media.

But there's no reason to give any weight to these comments, especially when they are from people whose biggest effort of the night has been to reach the remote control to change channels. When you've paid up to £70 to watch a game of football, the decision to leave should never be taken lightly. But if you're having a miserable time the best thing to do is cut your losses. Staying until the bitter end has never made anyone feel better – and then you still have to get home.

• This is an article from our Guardian Sport Network

• This article first appeared on When Saturday Comes

• Follow When Saturday Comes on Twitter