Downton Abbey is back for another series. It was a huge hit in 2010, gaining millions of viewers in Britain and worldwide, and is one of a series of 'master and servant' television productions in the past decade or so, including The 1900 House (1999), The Edwardian Country House (2003), Servants (2003), and What the Butler Saw (2004), and a relaunch of Upstairs, Downstairs (2010). The cinema has also seen its share with the enormously successful The Remains of the Day (1993) and Gosford Park (2003).

Why are audiences so fascinated by the servant-keeping setting? What kind of psychic comfort is there in watching people labour, or enjoy the benefits of that labour?

Downton Abbey specialises in dramatic twists and love affairs at all social levels. The world of domestic service provides an ideal backdrop for thwarted passions and sexual machinations of all sorts. This idea is not new – the initial publicity for Upstairs, Downstairs in 1969 stressed that ‘with a preponderance of frustrated females below stairs, jealousy and emotional scenes were the order of the day and love affairs of every sort were unrestrained’. Perhaps ironically, Upstairs, Downstairs had been conceived as a part of the 'history from below' movement. Originator and actress Jean Marsh recalled that 'we wanted to show that, for the servants, it was freezing in their attic bedrooms and boiling hot in the kitchen’. But while class disparities, and ill-fated love, feature in all these series, the actual physical labour of servants is often curiously absent.

Though Downton Abbey is scrupulously historically accurate, it nonetheless fails to convey much of the texture of domestic service, the chapped hands and rough knees ('like a nutmeg grater', the Victorian general servant Hannah Cullwick noted of her knees) that real servants suffered. The things which made service a much disliked job are rarely observed; the long hours, that might stretch from 5.30am to 11pm, the monotonous labour, and the absence of time-off, are obscured by the dramatic events that this fast-paced TV drama offers its viewers.

The erotic charge of the domestic service TV histories may seem contrived, but there is some historical justification for the relatively glamorous life that Downton Abbey's servants lead. Service to upper-class families, in town houses or country estates, was a relatively good job, with prospects for saving money, promotion, and a comfortable retirement. The upper servants might be proud of their skills, and enjoy aspects of their work.

The 'memory work' of popular culture, however, has paid far less attention to the much more prevalent work of the single-handed general servant, 'slavey' or 'maid-of-all-work', who worked in lower-middle-class suburban villas, or served in shops and pubs. Some were residential, but as the 20th century unfolded, many lived out. The relationships of domestic service were not always the distant formality of Downton Abbey, but might be intensely intimate. One general maid, Mrs Lilley, for example, left a description of her work as a servant when she entered an Age Concern competition in 1961, with an essay on the theme of 'My First Job'. She called her Edwardian employers ‘Mrs’ and ‘Mr’. They were probably of a similar background to her, and she recalled that they took a daily bottle of stout at 11am: ‘I well remember they used to leave me a drop in the bottom of one of their glasses.’ Their generosity amused rather than disgusted her: ‘I still have some of the presents they gave me. They always kept writing to me.’

Servants like Mrs Lilley did sometimes find their work intolerable – laborious, demeaning and dead-end. But we must be wary of making generalisations: some found friendship and enjoyed their work. One Essex-based servant, Mrs Bainbridge, commented: ‘Steps? Well they were my pride. I took a real pleasure in doing them and was very often complimented on them. Sometimes now I pass that house and see those steps all green and as if they’re never cleaned and it makes my heart ache and wish I could get down to them again.’

The new series of Downton Abbey has the appeal of being set during the years of the First World War –widely seen as an era of transition and flux. This period lends itself well to a sense of deep social change and urgent rethinking of social relationships. In particular, the war has sometimes been understood as the swansong of 'high society', and upper-class, landed lifestyles.

Was it the end of servant-keeping too? In reality, there was little to suggest that servant-keeping was not a permanent part of British society. After a brief dip in numbers during the First World War, the economic depression of the 1920s and 1930s forced more women back into service, aided by hostile government benefits policies which regarded all women as potential servants and, whatever their training and employment history, refused to pay benefits to those who rejected domestic work. The later 1930s also saw the start of a refugee crisis that would supply German and Austrian servants to British households. Housewife magazine brutally described the rise of European fascism as ‘Your Opportunity’ to its readers in a 1939 article making ‘The Case for the Foreign Maid’.

While the post Second World War period did see fewer young women working as servants, there were more women employed in new categories – married women took jobs as cleaners, foreign women arrived as au pairs – and these workers also need to be integrated into the story of 20th-century domestic service. 'Mrs Mopp' was a comic figure of mid-20th-century radio and humour magazines, but we seem much less fascinated by the lives of cleaners and chars from our 21st-century perspective, despite employing them in ever-increasing numbers since the 1980s. Will television ever tell their stories?

Much of our understanding of the servant-keeping world is based on nostalgia, a dream of projecting ourselves into a past that is terminated and resolved. We don't particularly want to draw the links to our own world of inequalities and global migration which has kept domestic service alive. Downton Abbey is a pleasurable world of fantasy, and great entertainment. But the real stories of Mrs Lilley and Mrs Bainbridge are just as compelling.

Lucy Delap is a Fellow and Director of Studies in History at St Catharine’s College, Cambridge. Her book Knowing Their Place: Domestic Service in Twentieth-Century Britain was published by Oxford University Press in 2011.