One blustery spring morning, Lucy Worsley, the chief curator of the Historic Royal Palaces and TV historian, marched me round Hampton Court Palace in search of Tudor pee. Traces were, she said, to be found on one of her favourite artefacts – a pisspot excavated in Henry VIII’s privy garden. “It doesn’t look anything at all,” she said. “But when you realise it’s got 16th-century urine in it, it’s fantastic.”

Despite some enjoyable rootling around in the palace’s storerooms, we never found the pisspot. But it would be hard to imagine a better example of Britain’s, or rather England’s, obsession with its 16th-century past – which revolves around the bodies and appetites of its rich and famous – than searching for traces of what may, or may not, be royal Tudor urine.

Right now, the Tudors hold the nation in an especially fearsome grip. In the British history section of my local Waterstones, in a not terribly royalist part of London, I measured four shelves of Tudoralia, covering the period from Henry VII’s snatching of the crown on Bosworth field in 1485 to the death of Elizabeth I in 1603. (The next most popular period was the 19th century, with two-and-a-half shelves.) The Tudor titles were mostly variations on a very limited set of themes: The Six Wives of Henry VIII; Henry VIII: King and Court; Elizabeth the Queen; The Elizabethans; Young Henry: the Rise of Henry VIII; Six Wives: the Queens of Henry VIII; Elizabeth’s Women.

The Tudors, it seems, are everywhere, neatly calibrated to appeal to every register of age, taste and education. The Six Wives of Henry VIII et al are the serious-to-popular history books; but there are also avalanches of fictional Tudors, on screen and in novels. There is a whole subgenre of racy romantic fiction about Anne Boleyn, the most obsessed-over of Henry’s queens, with such titles as The Kiss of the Concubine and Between Two Kings. There is The Tudors, a rumpy-pumpy four-season series made for an American cable channel and shown on the BBC, in which Jonathan Rhys Meyers’s Henry VIII never got fat, perhaps as a result of the sheer amount of shagging he did. The show was, said Worsley, “risible – but good for business”. (For her, the most characteristic moment was when Catherine Howard tried out the block on which she was to be beheaded – an act that is historically attested. For the purposes of The Tudors, however, she did it naked.)

Wolf Hall and Bring Up the Bodies, Hilary Mantel’s bestselling, Man Booker prize-winning novels, have been the most potent recent manifestation of Tudormania. They have also stoked the national appetite for more and more Tudor things. Martin Davidson, who commissions factual history for the BBC, told me that the TV adaptation of the novels, which leads the field in Sunday’s television Baftas, had “unlocked hidden reservoirs of interest in the Tudors that aren’t yet exhausted”. And so he has commissioned a new blockbuster BBC1 history show from Worsley, to be filmed this month at the National Trust’s Barrington Court, which was also used as a location in Wolf Hall. The programme is called Henry VIII’s Six Wives, and is not to be confused with the current Channel 5 series, presented by historians Suzannah Lipscomb and Dan Jones, titled Henry VIII and his Six Wives.

Jonathan Rhys Meyers as Henry VIII and Natalie Dormer as Anne Boleyn in The Tudors. Photograph: 2008 CPT Holdings, Inc. All Rights Reserved. (Sony Pictures Television International)

Tudormania spawns more Tudormania: so it is with the great houses and palaces that dot the English landscape. Hampton Court, for example, has become, essentially, a shrine to Henry VIII and his tragic wives, despite having been built by Henry’s lord chancellor Cardinal Wolsey and inhabited by a range of monarchs until well into the 18th century. The appeal of the Tower of London has become less about its Norman builders than its Tudor victims: visitors can see costumed interpreters in the guise of Anne Boleyn or Lady Jane Grey, and over the bank holiday weekend it hosted the Tudors at the Tower Family Festival. “It is 1533,” announced the website, “and you are invited to join the magnificent celebrations for Anne Boleyn’s coronation as Queen of England. Immerse yourself in a living Tudor world … Have a go with a crossbow!” Tracy Borman, the co-chief curator of the Historic Royal Palaces, told me about the time they tried to promote the Georgians at Hampton Court, in 2014, to mark the tercentenary of George I’s accession. “As a theme it just didn’t work,” she said. “Now we are more consciously giving visitors the Tudors. Commercially, the Tudors are the strongest sellers.” The Tudors are not just for domestic consumption. They are Britain’s most lucrative royal export, Worsley told me, after the current incumbents.

When speaking to historians, novelists and curators of the period, I found that they would, at some point in the conversation, reach for a pop-cultural analogy. The Tudors are like the Kardashians, said one. They are like the Caesars, or the Kennedys, said another. They are like Game of Thrones, said one. They are like House of Cards, said another. They are like Dallas, they are like Dynasty, said another.

The past is not fixed, but an encounter with the present. Our relationship with our history tells us about ourselves as much as about the past. It is, at least in part, an act of creation. What is it about the Tudors that has drawn us to them? And what does the way we have refashioned them tell us about the present moment?

The execution of Anne Boleyn in Henry VIII and his six Wives. Photograph: Channel 5

There is no more eloquent demonstration of the Tudors’ hold on our imaginations than Mantel’s books about Thomas Cromwell, the man who rose to be the most powerful man at Henry VIII’s court. Before Wolf Hall and Bring Up the Bodies, Mantel was respected and loved by a modest number of readers. After publishing those two novels, she became a bestselling, double Man Booker winner with stage and screen adaptations to her name; the BBC2 drama was the most popular on the channel since modern ratings began 14 years ago.

But the subtle, gloriously rich Wolf Hall was not Mantel’s first historical novel. A Place of Greater Safety, set during the French Revolution, was published in 1992 – 13 years after she had first sent it to publishers, when, Mantel told me, she “couldn’t even get anyone to read it”. People don’t want to read about revolutionaries, she said, “these sneaky villains with daggers and a guillotine in the middle distance”. She added, “I didn’t feel I’d had to move my ground with Wolf Hall: the readers came to me. But they wouldn’t do that with the French Revolution and they probably never will … The lure of the [Tudor] period is something that worked to my advantage. In that people are willing to give it a break. You get a category of reader that wouldn’t come to your other fiction because they think, ‘I know that story, and I want more of it.’”

Every nation has a changing and sometimes fraught relationship with its past. Different periods of history swim into view at different times, bringing parallels and visions of lost glory or past blunders. For the Americans, aside from the origin story represented by the founding fathers, perhaps the most often revisited period of domestic history is the civil war – except that, in contrast with the Tudors, it is still live as an issue, providing a lightly encoded way of talking about present anxieties about race. Italy, an even younger country, is certainly invested in the past, but that is more about the glories of regional visual cultures than a fascination with a single family. The French have enough national mythology to go around, but it is invested more in ideas and values than with people.

For the English, by contrast, nostalgia is both a national disease and a profit-making enterprise, selling everything from Keep Calm and Carry On mugs to Henry VIII outfits for little boys from the Hampton Court shop. The present moment of Tudormania does not come from nowhere: it can exist only because of our sense of familiarity with them. We have been raised on the Tudors. When I was young I adored a book called A Traveller in Time by Alison Uttley, about Catholic recusants under Elizabeth I in rural Derbyshire. Other kids had Annie Lennox or Morrissey on their bedroom walls: I had a poster of Elizabeth I, a reproduction of the Darnley portrait, in which she looks especially porcelain-skinned and icy, but also tough and clever. Like many other middle-class children, I was taken to see some of the great Tudor houses of England. For me, it was the grand Hardwick Hall, and the half-timbered beauties Ford Green Hall and Little Moreton Hall. I still associate the Tudors with long galleries lined with linenfold panelling, the scent of National Trust lavender in bowls, and a wandering imagination.

The Darnley portrait of Elizabeth I. Photograph: Dea Picture Library/De Agostini/Getty Images

The Tudors are the first people in British history into whose eyes we feel we can gaze. We slip into their world – or what we imagine to be their world – with ease. They were the first people to have lived in recognisable houses, rather than in the draughty great halls and militaristic castles of their medieval forebears. There is something comfortingly domestic and ineffably English about these dwellings, which is surely why mock-Tudor has been, since the late 19th century, such a persistently revived vernacular architecture. Some of the inhabitants of these houses had even been women, whose characters were a little more than cyphers (such as the redoubtable matriarch Bess of Hardwick who, with her four husbands, rivalled Henry VIII in marital energy if not murderous tendency). Hans Holbein’s portraits of Henry’s court carry an air of realism verging on the photographic; even those stiff, flat-as-a-pancake Elizabethan portraits of country squires and their be-ruffed wives have something bewitching about them. Unlike their forebears, they spoke what we can recognise as our language, the language crystallised by England’s most towering and inescapable literary figure.

For those raised in the 1990s and beyond, early immersion in the 16th century has much to do with the wildly popular, enjoyably gory Horrible Histories books, also televised by CBBC, in which the Terrible Tudors loom large. The English national curriculum requires the Tudors to be studied between ages seven and nine and, according to a state-sector history teacher I spoke to, the period becomes so familiar that pupils “turn up to secondary school feeling like they are already experts” – knowing, for example, the old mnemonic about Henry’s wives, “divorced beheaded died; divorced beheaded survived”. Jessie Childs, a popular historian who has written about the period, said: “The Tudors are very clearly defined for children: you have Henry VII, the battlefield king; then Henry VIII, the tyrant who marries six times; then Edward VI, the boy king; then Lady Jane Grey, the nine days’ queen; then [the Catholic] Mary I who burns 300 people; and then Elizabeth I, the virgin queen. They are like a boyband: each has an identity. If you look at the Plantagenets, on the other hand – people don’t know which is which.”

The same tidiness of identity works for the wives of Henry VIII, if you want it to. Worsley’s forthcoming BBC1 programme takes as its premise, she told me, that they represent “archetypes of femininity”, though, when she explained, they sounded more like stereotypes harvested from the pages of the Daily Mail. “We have Katherine of Aragon, the angry older dumped woman, Anne Boleyn, the Angelina Jolie character; Jane Seymour, the good woman who stays at home; Anne of Cleves, the one defined by her looks; Catherine Howard, the sexy teenager; and Katherine Parr, the career girl who gets through the marriage, survives, marries for love, then dies in childbirth because she’s left it too late.” If the most characteristic cultural achievement of our time is branding, then it would seem that we have done a very effective job on our Tudor past.

Damian Lewis as Henry VIII and Mark Rylance as Thomas Cromwell in Wolf Hall. Photograph: Giles Keyte/BBC/Company Productions Ltd

The Tudors have come to represent a certain kind of English exceptionalism: an idea of an England that stands apart from, and above, other nations. This is no coincidence – it is precisely how the Tudor monarchs framed themselves. When Henry VIII, the original Brexiteer, wrenched himself away from Rome and Catholic Europe, a new rhetoric and a new history were required. They were even more urgently after Mary I, the successor to the zealously Protestant Edward VI, briefly reversed the nation into Catholicism.

This exceptionalism began under Elizabeth I, when new kinds of writers – including chroniclers and antiquarians – embarked on what historian John Burrow once called “a love affair with England and Englishness”. The story of English history, he wrote, was “refurbished in a way that was Protestant, patriotic and providential”. Holinshed’s Chronicles, for example, first published in 1577, was an important source of material for Shakespeare, whose versions of English history have proved far more persistent and formative than more sober and factual accounts of, say, the Wars of the Roses. John Foxe’s multi-edition Acts and Monuments, more often known as Foxe’s Book of Martyrs, was an entire history of the church from its birth to the Elizabethan age, framing Roman Catholicism as a corrupt aberration from the true path and pointing to the English as God’s chosen people. Like so many radical departures, Protestantism needed to be backfilled with the trappings of tradition, however spurious. Elizabethan England created our sense of itself; to a large extent, it also created our sense of its own medieval past – as a bloody, violent pit out of which the nation had clambered en route to a new golden age.

The English no longer believe that they are an elect nation. But the underlying sense of superiority still lingers: you can hear it when Boris Johnson speaks out against a “half-Kenyan” US president interfering in British democracy, or bewails the removal of an Epstein bust of Churchill from the Oval office. You can hear it in the arguments over whether to retain Trident, over whether Britain is, or is not, in a state of self-delusion when it claims to be a world power. We cleave to this ancestral sense most strongly when the nation feels threatened. At such moments, Elizabeth I’s speech at Tilbury, rallying troops against the invading Spanish, echoes round the national unconscious: “I know I have the body but of a weak and feeble woman; but I have the heart and stomach of a king, and a king of England too, and think foul scorn that Parma or Spain, or any prince of Europe, should dare to invade the borders of my realm.” When extracts from Churchill’s “we shall fight on the beaches” speech were broadcast on the BBC in 1940, Vita Sackville-West wrote to her husband of being stirred by his “Elizabethan phrases”.

The different ways in which the Tudor period was used by past generations can throw light on the way we see them now. Take the narrative of Hampton Court, which was opened to the public by Queen Victoria. The 19th-century tourist clutching the standard guidebook, published in 1841, would have received a completely different view of the palace from that conveyed today. The palace had “a striking lesson” to teach “on the mutability of human greatness”. Its hero, however, was not Henry VIII or Anne Boleyn, but Cardinal Wolsey. Henry gets short shrift – a single brief chapter, as opposed to Wolsey’s six. The Victorian story was much more invested in the lad who pulled himself up by his bootstraps, attained the chancellorship of England – and “fell a sacrifice to his own ambition”. The message was that worldly desires should be tempered by “humility and piety” – a lesson quite absent from our present dealings with the 16th century, whose protagonists provide spectacle rather than moral instruction.

Why are we obsessed with the seductions and the violence, the spying and the intrigue, the corridors and the whispering and the catastrophic falls from grace? Why above all, are we obsessed by the sexy powerplay of the Tudor court? Academic history, popular portrayals, fiction and television have become entangled, but one historian’s work stands out as especially significant.

David Starkey’s PhD thesis, submitted in 1973, took as its subject the king’s privy chamber – which, for the post-’68 generation of young leftwing Cambridge historians, was a fabulously unfashionable thing to do. Starkey argued that the minutiae of court life were inextricably intertwined with the life of the nation. The people who were physically close to Henry – including those with the job of attending to the most basic of his bodily needs, such as the “groom of the stool” – were, rather than menial nobodies, politically important figures.

The political became the personal, and the personal made for compelling storytelling. Starkey has promulgated this approach in numerous later books, and, in the early 2000s, on Channel 4, where he out-BBCed the BBC with his air of establishment-sanctioned authority, his navy blazers and neat cufflinks, his high Toryism. (He has since made programmes about the monarchy as an institution, and about the Duke and Duchess of Cambridge’s wedding – the politics of the enterprise are clear.) There is a straight line from Starkey’s groom of the stool to Worsley’s urine-stained Tudor pisspot.

Historian David Starkey launching English Heritage’s Royal Heritage programme in 2002. Photograph: Frank Baron/The Guardian

“We still have this Merrie England shtick. It’s very parochial and it’s very English, it’s the Ukip version of England before it got some German aristos to run it,” Jerry Brotton told me. The professor of renaissance studies at Queen Mary, University of London is fed up with the Tudors – or at least the manner in which they are popularly portrayed. “Not to mention that they were actually Welsh. It’s the rose-tinted version of history that says, ‘That was when we were powerful.’ And we weren’t! What nobody wants to write about is that the whole period after Henry VIII’s break with Rome in the 1530s is the time when England was completely isolated, a pariah state within the rest of Catholic Europe. We were like Romania under the Soviet Union.”

Brotton is the author of This Orient Isle, which charts England’s links with the Islamic world in the Elizabethan age – links made necessary by the Protestant Tudors’ self-extraction from the political, diplomatic and trade networks of Catholic Europe. The version of the Tudors that the wider culture has embraced – the narrowing of the historical frame to England’s nobs, aristos, courtiers and monarchs – is, Brotton and other scholars would argue, both limiting and self-deceiving.

Dr John Gallagher, an early-modern specialist based in Cambridge, told me that he, too, was interested in restoring historical realism to our picture of the 16th century. Gallagher’s work focuses on “remarginalising” England as a small country perched on the north-western edge of Europe, a Europe that often saw England as unimportant. From our purview in 2016, it can be hard to reimagine the Tudors without the picture being clogged by the leavings of a British imperial dominance that in fact came centuries later. During the Tudor period, the English language, for example, was years away from being any kind of lingua franca. (Gallagher sent me a quote from the linguist and translator John Florio, who wrote in 1578 that English was “a language that wyl do you good in England, but passe Dover, it is woorth nothing”.)

The current cycle of popular retellings of the Tudors, said Brotton, is frequently “just more thin, secondary-source-material crap. It’s seriously bad faith – it’s just giving to a public cynical rubbish”. Other historians simply regard the Tudors as a period that has nothing more to tell us – nothing of use, anyway. “There is some great writing about the Tudors,” the historian Matthew Parker told me, “but for every new book about them, there’s one less about far more interesting and relevant threads in our history – the lives and opinions of ordinary people, economics, empire, race – history that directly shapes our lives today, rather than the soap-opera titillation of Henry’s bloody wives.”

The stories we choose not to tell about the Tudors are as revealing as those we do. Our present interest has coalesced around the two mighty figures of Henry VIII and Elizabeth I. Henry VII, though the subject of a recent and respected study by Thomas Penn, has not reached these heights. Nor has Edward VI, nor has Mary I. The editor of the magazine History Today, Paul Lay, has a theory why: aside from the fact that their reigns were short, the latter two monarchs are inescapably bound up with questions of religion: they are the radical, iconoclastic Protestant and the queen who briefly reinstated Catholicism. It was Henry VIII, of course, who broke with the Roman church, but the story is told, by and large, in the prurient terms of his desire for Anne Boleyn rather than through the lens of international politics or theology.

“The middlebrow accounts,” Lay told me, “and even Mantel is a bit guilty of this, tend to secularise Tudor history. That is just about possible with Henry, but Edward and Mary’s reigns cannot be told in secular terms.” Similarly, with Elizabeth: her early life offers an exciting adventure story of jeopardy and danger and high stakes – will she, won’t she make it to the throne? But “the later part of her reign, the more interesting part, is about the growth of English puritanism and the rise of ideas about providence, the increasing sense of England as an elect nation, singled out for greatness by God. All of this reaches its crisis point in the following century and the civil war … But unfortunately, there’s not a lot of sex, very little violence, and a lot of talk about ideas and fundamentalism.” He added, “These are ideas that are really difficult to explain,” and then he stopped himself. “Actually they’re not. It’s a lack of ambition among writers.”

Mary I of England (1516-1558). Photograph: Rex/Shutterstock

This secularisation, as well as perhaps a discomfort with the idea of revolution, explains how relatively little thought we give, as a culture, to the following century – the century of the English civil war. Martin Davidson told me that for him and his colleagues at the BBC, historical periods have certain textures; they serve as “story engines” for different kinds of narratives. “This place, this time that we call Tudorland, Tudorville has irresistible ingredients,” he said. “It is sex becomes death, via intrigue … I wish we could do more on the civil war, to be honest. But it’s difficult to find attractive characters, and it doesn’t help that they have zero sex lives,” he said. And, besides, the 17th century is too complex. “In a 60-minute programme, you spend 57 minutes explaining the rump parliament, ship money, the Irish issue.” Had we more confidence in the Westminster parliament now, perhaps, like the Whig historians of the 19th century, we would have more fondness for its 17th-century champions.

What we want, it seems – and what the Tudors seem so abundantly to provide, if viewed in a very partial light – is a way of shutting out disturbing considerations of religious fundamentalism, revolution, people from other countries, the role of parliament, relationships with Ireland and Scotland. And yet these are issues that, right now, we should look in the eye. “It’s very tempting, when you ask yourself why a story has such a wide appeal,” said Mantel, “to try to dredge up pieties about the shaping of the English nation: but really, it’s because it’s about sex and violence, let’s face it.” It turns out that what compels us is lust and death; the passions and appetites of the royal, rich and famous; drama set firmly within the family. At this moment, what we want from our relationship with a national past is this: to draw the curtains, shut out the noise of the world, and put on a box set. Thinking about Tudor history is, at least in its laziest manifestations, an excellent way of not thinking about history.

The Tudors of our imaginations some time ago transcended the realm of real lived lives and entered the zone of myth and fairytale. Henry VIII has become Bluebeard – the monstrous character who kills his wives in his castle. The story of the Tudors, said Mantel, “has the power and the resonance of every story that is not made up, that doesn’t have an author, but comes jumping straight out of the collective unconscious”. It cuts to the “verities of the human condition”, she said. “Who to love, who to marry, will we have children, what children will we have.”

It turns out that what compels us is lust and death; the passions and appetites of the royal, rich and famous

Why have we transformed the Tudors into the creatures of myth, and what purpose do these myths serve? It has happened partly because the discipline of history is so closely allied with other genres, especially various forms of fiction. That goes back to the beginning: the first historian, Herodotus, may have refuted Homer’s poetic version of the past, but that does not stop him shaping his stories about the past as just that: stories. The lives of the Tudors, over centuries of retelling, have become malleable, moulded into archetypal patterns. These patterns come to us from the earliest forms of drama and tragedy, straight from the Greeks: the rise and fall of the great; sad stories of the death of kings.

The tale of her Cromwell, said Mantel, is that of “how far you can go in making your fate. It is a story of a man who is climbing and climbing, and a story of what happens when he actually gets to the top. And we see this operating with all sorts of figures: Anne Boleyn, Wolsey.” The wheel of fortune revolves. The great rise, and then they fall. It could be Aeschylus’s Agamemnon, who wins the Trojan war only to be slaughtered in the bath by his wife. It could be Oedipus. Mantel said: “For these people [of the Tudor court] the stakes are so high. It makes modern life seem so feeble, when a minister, for example, messes things up and he gets to resign and spend more time with his family. In Henry’s day they just cut your head off: so you get the sense, all the time, of the characters living on a knife-edge.”

Lucy Worsley, the TV historian and chief curator of the Historic Royal Palaces, at Hampton Court. Photograph: Ben Turner

These people’s stories resonate with us, because their suffering is like ours – terrible reversals, bad marriages, forbidden love – but writ large. They climb higher and they plunge further. In their presence we frame our own tragedies. Every community needs its mythology, and, as the religion the Tudors invented ebbs, it is the Tudors themselves to whom we cling as figures of a collective imagination. They fit us like a glove. If the story of Henry’s court is about a total and arbitrary power quixotically exercised, then it is the perfect story now for a people whose confidence in traditional forms of politics has drained away.

One of Mantel’s great achievements was to bring these figures of fairytale home: to create a story in which the past was both familiar and foreign, small with homely detail and large with myth. Now you recognise them, now you don’t. Peter Kosminsky, the director of the TV adaptations of Mantel’s novels, did something analogous. He told me about how he had tried to direct the series as if it were not a costume drama: here we are in the characters’ homes, to them unremarkable and familiar; therefore there are no protracted, sweeping shots of gorgeous architecture. What he did made perfect sense of our relationship with these creatures of the 16th century, who vibrate between feeling familiar and foreign. Though strange and sometimes frightening, they march through the corridors of our imaginations with an almost insolent ease.

Davidson told me that he thought he might give the Tudors a rest after Worsley’s programme has aired: he is also turning his attention to the Russian revolution for when the anniversary comes round next year, and thinking about how to make history programmes about the 1990s. Perhaps our interest in the Tudors will wane a little, but there is no sign of that yet. Caroline Sanderson, who covers non-fiction releases for the trade magazine the Bookseller, told me she is still seeing at least a book a month on the Tudors. Lucy Worsley, the day she took me round Hampton Court, pressed into my hands her new children’s novel, Eliza Rose, based on a fictitious cousin of Catherine Howard’s. Her colleague at the Historic Royal Palaces, Tracy Borman, has a new book coming out later this year, titled The Private Lives of the Tudors. Mantel is still working on the final book in the Thomas Cromwell trilogy. She told me it was going well, but could not say when it will be finished, or published. When it is, I, and doubtless millions of other readers, will devour it, and doubtless another burst of enthusiasm for the period and its court will follow. On the Tudors will go, our troubling companions from the past, haunting us into the future. It is certainly the past we desire, and it is probably the past we deserve. Whether it is the past we need is another question.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.