Read: The financial benefits of being beautiful

The problem starts with brain chemistry. “When you see an attractive person, the left ventral tegmental area of the brain becomes active and will pump out dopamine,” says Helen Fisher, a biological anthropologist who studies attraction at the Kinsey Institute. “Dopamine is a stimulant to the brain, so some people might react with surprise or awkwardness.” That feeling is the weak-kneed giddiness that very attractive people can inspire, which can leave you fumbling for words and feeling off balance, even though a dopamine rush is a fundamentally pleasurable experience.

Based on Fisher’s research, which used fMRI scans to observe the brain lighting up in response to stimuli, the left ventral tegmental area (commonly referred to as the left VTA) is responsible for pleasurable reactions to beauty. Meanwhile, the right VTA provides the dopamine that fuels romantic love; the two responses are similar but neurologically distinct, which means that what people feel when they see a random pretty face isn’t necessarily a desire for romance or even sex. “The same thing probably happens when you look at a good painting,” says Fisher. “It can pump out the dopamine and perhaps make you slightly giddy.”

The left VTA appraises and appreciates what you see, but lighting up that part of the brain doesn’t necessarily make you want to interact with the person whose appearance gives you pleasure, which is why most people don’t try to ask out every hot person they see. The stress I felt wasn’t the same as a fear of rejection; my hot surgeon wasn’t even my type. Instead, I panicked because of a key difference between gazing at a painting and a hottie: A painting doesn’t judge you back.

That’s where a second, potentially more nefarious brain chemical comes in: cortisol. That’s the stress hormone that gets blamed for everything from weight gain to road rage, and Fisher thinks a cortisol spike is probably what I experienced when surprised by my extraordinarily attractive doctor. “Some people may see someone beautiful and feel very inadequate. Then cortisol would go up,” she says. A spike in the hormone can trigger a fight-or-flight response, which could be why my brain hurtled toward intense irritation and embarrassment at beautiful strangers in situations where I was at a disadvantage: when I was sick, in the middle of moving, or watching The Real Housewives of Atlanta inside my own apartment.

Read: The charisma effect

“It’s the context of who you are, how you feel about yourself, if you enjoy surprises—lots of things,” Fisher says. It doesn’t help that American culture tends to code physical beauty as an indicator of overall superiority, which can make the sense of inadequacy in these interactions particularly stressful.

While people’s brains certainly enjoy beauty, our appreciation is often not that straightforward, because our perceptions are also influenced by everything else about a particular interaction. Indeed, researchers have found that the adrenaline rush created by fear can make other people seem more attractive in the immediate aftermath. And if you’re already feeling good, Fisher says, suddenly encountering an attractive person can make you feel even better by triggering a dip in cortisol levels. In hindsight, that happens to me even more frequently than the panic I had with my surgeon, but humans tend to have better recall for negative memories than positive ones.