Every autumn, to my great delight, The X Factor returns to our TV screens. I am the sort of fan who watches the show religiously, weeps along with the contestants and shouts obscenities when the judges take against my favourites. I love it despite myself.

It's self-evident that there's always been a Cinderella element to The X Factor: on one hand it's a singing competition, and on the other it's a rags-to-riches tale of ordinary people trying to make it in this cruel world; but never has that narrative been more dominant than the in the last couple of years, where it has virtually eclipsed the singing altogether. Last year's shows featured montages of contestants working in Asda (Jahmene Douglas) and living in a bedsit (James Arthur), each one consciously positing The X Factor as their only route out of the poverty and mundaneness of modern working-class life.

This shows no sign of abating: one of this year's contestants Relley Clarke has repeatedly referred to herself as "just a housekeeper" and done vox pops about how The X Factor is the only way she can achieve some kind of self-improvement. Another contestant, Hannah Barrett, began her second round of auditions by referring to her troubled past and vowing "I'm telling you now I'm not going back there" with a determination it was hard not to feel in your bones.

It's strange that something so obviously reflective of class politics is presented as totally apolitical. I've always thought that The X Factor represents a kind of rightwing wet dream; elevating neoliberal values such as individuality, competition and self-improvement over the socialist ideals of collectivism, community and common good. Indeed, the show straightforwardly appropriates working-class traditions which promote togetherness, such as working men's clubs and karaoke; and turns them into a competition for prosperity arbitrated by judges who are members of the super-rich.

Politicians frequently profess their love for The X Factor as a way of showcasing their common touch, but the only instance I can remember of one delving into the show's politics was in 2011, when Iain Duncan Smith – with trademark insight – blamed the "get rich quick" culture of the programme for the riots that summer. His musings led me to conclude that he is not a fan of the show; for if he were, he'd know there's nothing "get rich quick" about it. The X Factor is saturated with scenes of the judges eliciting promises of hard work and dedication from contestants. The overarching message seems to be a familiar one: if you just keep your head down, work hard and be very grateful, some glamorous millionaires might choose you above all others to join them in the lap of luxury.

Last year, the show featured its most telling Cinderella telling. A homeless man called Robbie Hance made it through the first auditions, partly because he could turn out a tune and partly because the judges decided they wanted to "give him a chance". In the second round of auditions, Hance forgot his lyrics and walked out, which led to a load of lamentations from self-confessed rightwinger Gary Barlow that he had thrown away the opportunity to "turn his life around". No consideration was given to the complications that come with homelessness (Hance later claimed he left because the producers wouldn't give him food. The subtext was of an establishment that had been kind enough to throw a poor boy a lifeline, and because he'd lacked self-discipline, he deserved the poverty he would return to. On the other hand the contestants who offered diligence and genuflection were universally praised.Yet for all its flaws, The X Factor is one of the few mainstream television programmes to actually show the lives of working-class people. One of the reasons I love it is the fact that, in the contestants' vulnerability, ordinariness and unrelenting optimism, The X Factor offers a realness that most other programmes do not – despite its deeply contrived format. Perhaps fans enjoy The X Factor because it allows us to invest in a Cinderella story without having to risk personal failure. In fact, with the public vote, we get the final say in how deserving the contestants are of their happy ending – something the producers are taking to new lengths this year by developing an app which allows viewers to play the part of the "fifth judge". As the show goes on, the judges retreat into the role of audience as the viewers begin to arbitrate who is deserving and undeserving – and often the most grateful, conscientious and obedient contestants do better than those who are more talented but also controversial (remember Misha B in 2011?). It often makes me wonder what sort of values we internalise when we watch The X Factor, and how many of those values remain when the show has finished, while the human stories of its participants fade from memory.

• This article was amended at 12:03pm GMT on 11 October 2013. An earlier draft of it had accidently been published and has now been replaced with the final draft.