When Hunter S Thompson took Welsh artist Ralph Steadman to the Kentucky Derby in 1970, he tried to prepare him for the chaos into which they were descending. “Just pretend you’re visiting a huge outdoor loony bin,” Thompson said. He added, because he was rarely out of character, “If the inmates get out of control we’ll soak them down with Mace.”

“Huge outdoor loony bin” is not the most precise description of the Palio di Siena, the thunderous, lawless, bareback, medieval-style horse race held twice each summer in front of tens of thousands of spectators on a track of packed clay laid down in the heart of Siena, in Tuscany. But it will do for the moment.

This is a race in which jockeys – they ride for various contrade, or neighbourhoods – feel free to bribe one another, out in the open, before the contest begins. Betrayal is common. Guile is prized. There are no rules but one: a rider may not interfere with the reins of another horse.

Jockeys whip their horses, and each other, with crops made from cured distended bull’s penises. If a jockey is thwacked off his mount, his riderless horse can still win on its own, like Garfunkel arriving without Simon.

The jockey who finishes second is held in more contempt than the one who comes in last. After the race, the victors celebrate by sucking on dummies or drinking cheap wine from baby bottles to symbolise rebirth. Siena comes to resemble a playpen in which many of the toddlers have hairy legs and 5 o’clock shadows.

A few years ago, when the Contrada Pantera (the Panther) was beaten by its long-established enemy, the Contrada dell’Aquila (the Eagle), a loudspeaker mounted on the Eagle’s church tower reportedly boomed out a motto mocking the Panther 24 hours a day for more than a month.

If you have thousands to spend you can view the race from a variety of windows and balconies (Getty)

I know these things – they barely scratch the surface of this festival’s part-circus, part-theatre ambience – because my friend, Valentina Rice, has been attending the Palio each summer since she was a child. She tells good stories. I know a bit about the race, too, because I’ve seen Cosima Spender’s fascinating and highly recommended 2015 documentary, Palio.

Last August, I finally witnessed this spectacle, the world’s greatest horse race, for myself. Rice, whose family has long owned a house in the Tuscan hillside nearby, invited me along.

On a blistering August afternoon, 33C in the shade, we were not in the shade. We were standing in the centre of Siena’s main square, the Piazza del Campo, waiting amid a boiling sea of spectators for the race to begin.

We’d been there for hours, having staked out a plum spot on high ground. I’d forgotten my cap. In the strong sunshine, one side of my potato head, I fear, had gone from pink to a gruesome tomato-and-bacon sort of hue.

It’s possible to purchase seats in the stands for the Palio, but they are expensive – as much as several hundred pounds – and look a bit rickety. If you have the right connections, or thousands to spend, you can also view the race from a variety of windows and balconies that ring the piazza and function like opera boxes.

In Rice’s family the tradition is to be in the centre of the piazza, in the scrum. It’s a bit of an endurance test.

There have been dozens of serious injuries over the years and videos of spills are all over YouTube (Getty)

One woman fainted. There was a good deal of jostling for position. Men and women who’d had hard weekdays were not going to put up with being pushed around on this particular weekend, especially by tourists. A man near me had stripped down to tiny red briefs in the heat and was pouring ice water down his grizzled chest.

Behind me groups of men broke into ribald ballads that honour their contrade. Many women attend the Palio and take part in various aspects, but it is as strenuously male and macho as professional wrestling. Yet the scene is mostly peaceful and upbeat.

As the sun moved lower on the horizon, a solemn and highly choreographed two-hour pageant, the Corteo Storico, began. More than 600 people in historical costume, many on horseback, began moving slowly around the piazza. Many of the faces were nearly medieval in their El Greco thinness. There were skilled flag tossers, severe-looking military-style drummers, oxen-pulling chariots, floats of ancient design.

The race dates back to the 13th century (Getty)

The pageant and the race are an eyeful. They feature in the opening chase scene of the James Bond movie Quantum of Solace. At least a few viewers of that film had hoped that Daniel Craig would go all Indiana Jones on a bad guy with a cured bull’s penis, but this sadly was not to be.

Watching this pageant, I realised my cheeks were wet. The parade and the sea of packed-in spectators and the old city centre combined to produce a tableau so stirring it brought my heart up under my ears. I’d always found it hard not to tear up when participating in ritual, from weddings and funeral processions to the singing of anthems and old songs. There is more ritual at the Palio that can be easily comprehended.

The race, which occurs twice a year, on 2 July and 16 August, dates back to the 13th century, and probably began as Roman military training. The earliest races were on buffaloes, and then on donkeys. The word palio itself means banner in Italian, and that’s all that the winning contrada receives. This banner bears the image of the Virgin Mary, in whose honour these wild races are held.

There are 17 fiercely rivalrous contrade that ring Siena. These tend to be named after animals: snails, porcupines, she-wolves. Each contrada has its own museum and church and public square and fountain and traditions and banner. Once there were more than three times as many contrade. One of the most moving portions of the pre-race procession is watching the banners of past contrade rush by, ghosts of earlier contests.

There may be 17 contrade, but there is room in each race for only 10 horses. A form of musical chairs must occur, and the seven contrade that cannot fit in one race are included in the next. The contrade are allowed to choose their jockeys but not their horses, all of which are mixed breed and chosen in part for their ability not to be easily spooked among the crowds and chaos. These are arranged marriages: each contrada meets its horse for the first time just four days before the race.

There’s no official betting at the Palio but allegiances are purchased for tens of thousands of pounds (Getty)

We are coming to the point where, in the explaining of the Palio, things began to grow a bit surreal. This is truly a human endeavour about which the more you know, the less you understand. The contrade pay their jockeys handsomely to ride for them, yet these jockeys are hired guns and fundamentally unfaithful. Everyone is a potential double agent.

There’s no official betting at the Palio but allegiances are purchased for tens of thousands of pounds. Secret negotiations abound. Did your contrada’s jockey miss his opportunity to get ahead at that turn, or was he paid to fall back? There’s no knowing.

This sort of existential criminality, in nearly any other country, would lead to madness among horse people and spectators. Yet in Siena, no one wishes to change a thing about the Palio.

There’s been a good deal of analysis over the years about how the race illustrates the Italian soul. The Italians admire people, it’s often said, who make good via the wily bending of rules and conventions. Witness Silvio Berlusconi, the former prime minister, who has confidently brushed off sex and corruption scandals. Benito Mussolini is said to have adored the Palio.

In response to animal rights concerns, organisers have increased the padding on some turns (Getty)

Sometimes more than guile and payoffs are used to secure a Palio victory. In past years, horses have reportedly been drugged and jockeys kidnapped. Writing in Condé Nast Traveler, Steve King reported a bit of Palio skulduggery that involved inflaming a stallion’s lust: “By the time the race began the poor beast didn’t stand a chance – indeed, could hardly stand at all and barely managed to stagger his knock-kneed way around the course.”

During the race, jockeys take their lives into their hands. The race involves three clockwise laps around the third-of-a-mile track and there are tight turns. There have been dozens of serious injuries; videos of spills are all over YouTube. Horses are more vulnerable. More than 50 have died in these races since 1970; animal rights protesters have staged repeated protests. In response, Palio administrators have increased the padding on some turns and instituted other safety controls. Critics say these measures are not enough.

The race takes just 90 seconds, but seems even shorter (Getty)

The parade ended and a booming cannon-like shot scattered every bird within two miles. The crowd grew quiet as the horses and their riders entered the piazza. Nine of the 10 racers took up their assigned positions at the starting rope. The 10th rider decided when the race started, when he made a go for it.

While this was happening, the riders conversed, swapped taunts and offered bribes. Impatient horses jostled and reared off the crowded line and were ridden back. The 10th horse made multiple exploratory false starts. This to-ing and fro-ing took more than 10 minutes.

And then they were off. The race was a clattering blur, whipping around us. It took less than 90 seconds but seemed even shorter. Several riders fell from their horses but none were seriously injured.

The winner was La Contrada dell’Onda (the Wave), its colours aquamarine. It was this contrada’s first win since 2013 and its jockey, Carlo Sanna, known as Brigante, was an instant hero, hoisted upon shoulders.

He and his horse, the 9-year-old Porto Alabe, were whisked off to receive the winning banner and be blessed at Siena Cathedral. This event was not hard to find. Hundreds, if not thousands, of people poured through the streets to make their way there, as the carabinieri kept close watch.

Out came the dummies and baby bottles. The winners wept with happiness. Meals commenced at huge tables set up in the streets. The festivities ran all night.