I’ve been procrastinating. On my dining room table I have lined up three hot peppers: one habanero, flame-orange and lantern-shaped; one skinny little Thai bird’s eye chilli; and one relatively innocuous jalapeño, looking by comparison like a big green zeppelin. My mission, should I choose to accept, is to eat them.

In ordinary life, I’m at least moderately fond of hot peppers. My fridge has three kinds of salsa, a bottle of sriracha, and a jar of Szechuan hot bean paste, all of which I use regularly. But I’m not extreme: I pick the whole peppers out of my Thai curries and set them aside uneaten. And I’m a habanero virgin. Its reputation as the hottest pepper you can easily find in the grocery store has me a bit spooked, so I’ve never cooked with one, let alone eaten it neat. Still, if I’m going to write about hot peppers, I ought to have firsthand experience at the high end of the range. Plus, I’m curious, in a vaguely spectator-at-my-own-car-crash way.

When people talk about flavour, they usually focus on taste and smell. But there’s a third major flavour sense, as well, one that’s often overlooked: the physical sensations of touch, temperature and pain. The burn of chilli peppers is the most familiar example here, but there are others. Wine mavens speak of a wine’s “mouthfeel”, a concept that includes the puckery astringency of tannins – something tea drinkers also notice – and the fullness of texture that gives body to a wine. Gum chewers and peppermint fans recognise the feeling of minty coolness they get from their confections. And everyone knows the fizzy bite of carbonated drinks.

None of these sensations is a matter of smell or taste. In fact, our third primary flavour sense flies so far under our radar that even flavour wonks haven’t agreed on a single name for it. Sensory scientists are apt to refer to it as “chemesthesis”, “somatosensation”, or “trigeminal sense”, each of which covers a slightly different subset of the sense, and none of which mean much at all to the rest of the world. The common theme, though, is that all of these sensations are really manifestations of our sense of touch, and they’re surprisingly vital to our experience of flavour. Taste, smell, touch – the flavour trinity.

Sensory scientists have known for decades that chilli burn is something different from taste and smell – something more like pain. But the real breakthrough in understanding chilli burn came in 1997, when pharmacologist David Julius and his colleagues at the University of California, San Francisco, finally identified the receptor for capsaicin, the active ingredient in chilli heat. The task demanded a lot of patience: Julius and his team took every gene active in sensory nerve cells, which respond to capsaicin, and swapped them into cultured kidney cells, which don’t. Eventually, they found a gene capable of making the kidney cells respond. The gene turned out to encode a receptor – eventually named TRPV1, and pronounced “trip-vee-one” – that is activated not just by capsaicin but also by dangerously hot temperatures. In other words, when you call a chilli pepper “hot”, that’s not just an analogy – as far as your brain can tell, your mouth really is being burned. That’s a feel, not a smell or taste, and it passes to the brain through nerves that handle the sense of touch.

Like other touch receptors, TRPV1 receptors are found all over the inner layer of your skin, where they warn you of burn risk from midsummer asphalt, baking dishes straight from the oven, and the like. But they can only pick up pepper burn where the protective outer skin is thin enough to let capsaicin enter – that is, in the mouth, eyes, and a few other places. This explains the old Hungarian saying that “good paprika burns twice”.

Further tests showed that TRPV1 responds not just to heat and capsaicin but to a variety of other “hot” foods, including black pepper and ginger. More recently, several more TRP receptors have turned up that give other food-related somatosensations. TRPA1, which Julius calls the “wasabi receptor”, causes the sensation of heat from wasabi, horseradish and mustards, as well as onions, garlic and cinnamon. TRPA1 is also responsible for the back-of throat burn that aficionados value in their extra-virgin olive oil. A good oil delivers enough of a burn to cause a catch in your throat and often a cough. In fact, olive oil tasters rate oils as “one-cough” or “two-cough” oils, with the latter getting a higher rating. (One reason wasabi feels so different from olive oil is that the sulfur-containing chemicals in wasabi are volatile, so they deliver wasabi’s characteristic “nose hit”, while non-volatile olive oil merely burns the throat. Olive oil may also trigger TRPV1 receptors to some extent.) Curiously, TRPA1 is also the heat receptor that rattlesnakes use to detect their prey on a dark night.

Chilli aficionados get pretty passionate about their pods, choosing just the right kind of chilli for each application from the dozens available. The difference among chilli varieties is partly a matter of smell and taste: some are sweeter, some are fruitier, some have a dusky depth to their flavour. But there are differences in the way they feel in your mouth, too.

One difference is obvious: heat level. Chilli experts measure a chilli’s level of burn in Scoville heat units, a scale first derived by Wilbur Scoville, a pharmacist and pharmaceutical researcher, in 1912. Working in Detroit, Scoville had the bright idea that he could measure a pepper’s hotness by diluting its extract until tasters could no longer detect the burn. The hotter the pepper was originally, the more you’d have to dilute it to wash out the burn. Pepper extract that had to be diluted just tenfold to quench the heat scores 10 Scoville heat units; a much hotter one that has to be diluted one hundred thousandfold scores 100,000 Scovilles.

Nowadays, researchers usually avoid the need for expensive panels of tasters by measuring the chilli’s capsaicin content directly in the lab and converting that to Scoville units. The more capsaicin, the hotter the chilli.

However you measure it, chillies differ widely in their heat level. Anaheims and poblanos are quite mild, tipping the scale at about 500 and 1,000 Scovilles, respectively. Jalapeños come in around 5,000, serranos about 15,000, cayennes about 40,000, Thai bird’s eye chillies near 100,000, and the habanero on my table somewhere between 100,000 and 300,000 Scovilles. From there, intrepid souls can venture into the truly hot, topping out with the Carolina Reaper at a staggering 2.2 million Scovilles, which approaches the potency of police-grade pepper spray.

Many chilli heads claim that a pepper’s heat is defined by more than just intensity. If anyone would know about this it would probably be Paul Bosland, the director of the Chile Pepper Institute at New Mexico State University. As a plant breeder by trade, he has a keen professional interest in all the tiny details of how chilli heat differs from one pod to the next.

Bosland says he and his colleagues distinguish four other components to chilli heat in addition to heat level. The first is how fast the heat starts. “Most people, when they bite the habanero, it maybe takes 20 to 30 seconds before they feel the heat, whereas an Asian chilli is immediate,” he says. Chillies also differ in how long the burn lasts. Some, like jalapeños and many of the Asian varieties, fade relatively quickly; others, like habaneros, may linger for hours. Where the chilli hits you also varies. “Usually, with a jalapeño, it’s the tip of your tongue and lips, with New Mexico pod types it’s in the middle of the mouth, and with a habanero it’s at the back,” says Bosland. And fourth, Bosland and his crew distinguish between “sharp” and “flat” qualities of burn. “Sharp is like pins sticking in your mouth, while flat is like a paintbrush,” he says. New Mexico chillies tend to be flat while Asian ones tend to be sharp.

For women, there’s no social status to being able to eat the hottest chilli pepper, while for men there is Professor John Hayes, University of Pennsylvania

It’s time to take the plunge. First up, the jalapeño. As you’d expect from its relatively wimpy ranking in the hot pepper standings, it gives only a mild burn, which builds gently and mostly at the front of the mouth. Confronted with such a tame burn, I have plenty of attention left to focus on its thick, crisp flesh and sweet, almost bell-peppery flavour. The Thai bird’s-eye chilli, second on my list, is much smaller, and its flesh proves to be much thinner and tougher. Despite that, though, it almost immediately lets loose a blast of heat that explodes to fill my mouth from front to back, making me gasp for breath. No gradual build to this one – it’s a sledgehammer blow. If I think hard, I might imagine that the chilli heat is a little bit sharper, pricklier, than the jalapeño. But I could just be fooling myself.

Finally, the one I’ve been dreading, the habanero. I cut a tiny slice and start chewing. The first thing that strikes me is how different the flavour is. Instead of a vegetal, bell pepper flavour, the habanero gives me a much sweeter, fruitier impression that’s surprisingly pleasant. For about 15 or 20 seconds, anyway – and then, slowly but inexorably, the heat builds. And builds. And builds, long after I’ve swallowed the slice of pepper itself, until I can’t think of much else besides the fire that fills my mouth. It definitely hits farther back in the mouth than the Thai chilli, though there’s a late-breaking flare-up on my tongue as well. The whole experience lasts five or 10 minutes, and even a good half hour later it’s as though coals are gently banked in my mouth.

Having set my mouth afire, I’d now like to quench the burn. Surprisingly, scientists can’t offer a whole lot of help in this regard. A cold drink certainly helps, because the coolness calms the heat-sensing TRPV1 receptors that capsaicin excites. The only problem – as you’ve no doubt noticed if you’ve tried to cope with a chilli burn this way – is that the effect goes away in just a few seconds, as your mouth returns to normal body temperature. You’ve probably heard, too, that sugar and fat help douse the fire, but the researchers themselves aren’t entirely convinced.

“The best thing out there is probably cold, whole milk,” says John Hayes of the department of food science at the University of Pennsylvania. “The cold is going to help mask the burn, the viscosity is going to mask the burn, and the fat is going to pull the capsaicin off the receptor.” When pressed, though, he notes that there’s not a lot of data to back that up.

Making a food more viscous has been shown to damp down taste – probably just because it provides a competing sensation to distract our attention, Hayes notes, but he can’t think of anyone who’s tested whether it also reduces chilli burn. And he’s not entirely sure that sugar really helps, either. “I’m not convinced that it actually knocks the heat down, or whether it just makes it more pleasant,” he says. Even the value of fats or oils – which sounds like they ought to help wash capsaicin, which is fat soluble, off the receptors – is in dispute. If you’re feeling the burn, says Bruce Bryant of the Monell Chemical Senses Center in Philadelphia, the capsaicin has already penetrated your tissue, so a superficial rinse of whole milk or olive oil isn’t going to help much.

Millions of people actively seek out the pain of hot chillies as a form of pleasure. The burn features prominently in more than a few of the world’s great cuisines, with more than a quarter of the world’s population eating hot peppers daily. Britain spends £20m annually on hot sauce.

We don’t take pleasure in eating food that’s still searingly hot from the oven, even though that delivers exactly the same sensation we get from chillies: same receptors, same nerves. We don’t choose to chemically burn our tongues with strong acids. So why do we happily, even eagerly, inflict pain by chillies? Whatever the secret is, it seems to be unique to humans. No other mammal on the planet has a similar taste for chillies. (Birds eat them enthusiastically, but only because they lack receptors that respond to capsaicin. To a parakeet, the hottest habanero is as bland as a bell pepper.)

One possible explanation is that chilli lovers simply don’t feel the pain as intensely as those who shun hot peppers. In the lab, it’s certainly true that people who are repeatedly exposed to capsaicin become less sensitive to it. Genetics may play some part, too. Studies of identical twins (who share all their genes) and fraternal twins (who share only half) suggest that genes account for 18-58% of our liking for chilli peppers. Some people may have more sensitive TRPV1 receptors, for example – though Hayes, who’s looking into that now, says: “The jury is really still out on whether there is meaningful TRPV1 variation.”

It’s abundantly clear, though, that chilli lovers aren’t immune to the pain. Just ask one. “I like it so all my pores open up and tears are rolling down my face,” says Hayes. “But with two young kids in the house, I don’t get that very often.” For now, Hayes makes do with a handy bottle of sriracha hot sauce. “My kids refer to it as Daddy’s ketchup,” he says.

It’s clear from listening to Hayes that he – and probably most other chilli eaters – actively enjoys the pain. That paradox has drawn the attention of psychologists for several decades now. Back in the 1980s, psychologist and pioneering chilli researcher Paul Rozin of the University of Pennsylvania proposed that chilli eating is a form of “benign masochism”, like watching a scary movie or riding a roller coaster. After all, most forms of pain are warnings of imminent harm. That baked potato still steaming from the oven is hot enough to kill the cells lining your mouth, potentially causing permanent damage. But chilli burn – except at its uppermost, million-Scoville extreme – is a false alarm: a way to get the thrill of living on the edge without the risk of exposing yourself to real danger.

A few decades later, Hayes and his student Nadia Byrnes (perhaps the best name ever for a hot pepper researcher) took Rozin’s ball and ran with it. If chilli heads are looking for thrills, Byrnes and Hayes reasoned, you’d expect them to have sensation-seeking personalities. And, sure enough, when they went to the vast arsenal of tests that psychologists have developed to measure facets of personality, they found several measures of sensation seeking, of which the latest and best was the Arnett Inventory of Sensation Seeking. Then they set out to see whether chilli lovers really do crave excitement.

When Byrnes and Hayes tested nearly 250 volunteers, they found that chilli lovers were indeed more likely to be sensation seekers than people who avoided chillies. And it’s not just that sensation seekers approach all of life with more gusto – the effect was specific to chillies. When it came to more boring foods like candy floss, hot dogs or skimmed milk, the sensation seekers were no more likely to partake than their more timid confreres.

Chilli eaters also tended to score higher on another aspect of personality called sensitivity to reward, which measures how drawn we are to praise, attention and other external reinforcement. And when the researchers looked more closely, an interesting pattern emerged: sensation seeking was the best predictor of chilli eating in women, while in men, sensitivity to reward was the better predictor.

Hayes thinks that’s because machismo plays a role in the chilli eating of men, but not women. “For women, there’s no social status to being able to eat the hottest chilli pepper, while for men there is,” he speculates. Without the heavy hand of machismo on the scales, women’s chilli eating is more strongly governed by their internal drive for excitement.

Incidentally, while chilli lovers laud the rush they get from a spicy dish, and sometimes claim the peppers “wake up” their palate to other flavours, you’ll often hear chilli-averse people complain that the burn keeps them from savouring other flavours in their meal. Which is it? The matter has received surprisingly little scientific study, but the bottom line seems to be that if capsaicin blocks other flavours, the effect is small. Most likely, when people complain that they “can’t taste as well” after a spicy mouthful, it’s largely because they’re paying so much attention to the unfamiliar burn that the other flavours fly under the radar. In other words, it’s not “hot” but “too hot” that interferes with the enjoyment of flavour – and the threshold where hot becomes too hot is a very personal one.

Extracted from Flavour: A User’s Guide to Our Most Neglected Sense by Bob Holmes (Ebury Press, £20). To order a copy for £17, go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min. p&p of £1.99.