You know what they say about people with big brains … Or, actually, maybe you don't.

Despite being a major concept underlying of the neurobiology of intelligence for the last 150 years or so, the connection between brain size and smarts isn't well-understood by Joe and Jane Average. Does it mean smaller people—including women—are less intelligent? What about animals, like elephants, that have much larger brains than ours? Are our academic destinies really written in our hat size?

It's complicated. We know that brain size and intelligence are correlated, but that simple fact is only a starting point for a much more intricate story—one that science is only beginning to understand.

First off, yes, bigger brains really do seem to be smarter brains. That correlation has been pretty solidly proven, experts say, and the connection gets stronger when you calculate total brain volume via MRI technology or post-mortem analysis, rather than simply running a tape measure around somebody's head. Basically, the more accurate and precise the brain measurement, the more size and smarts are connected.

How connected varies a bit, depending on the methodology, but an analysis of previous research, published in 2005 in the journal Intelligence, found a .33 correlation at the population level. Which means, if you look at humans as a whole, a little more than 10% of the difference in intelligence from person to person can be accounted for by brain size.

That's statistically significant. But it also means overall brain size isn't the only thing affecting intelligence. Case in point: Gender.

"It is true that women have smaller brains than men," said Sandra Witelson, Ph.D, professor of psychiatry and behavioral neurosciences at the Michael G. DeGroote School of Medicine at McMaster University in Canada. "But numerous studies have shown that women aren't any less intelligent overall than men."

That works because male and female brains are built differently, according to Witelson and other researchers.

"If you look at the brain areas related to intelligence in men, they're different than the brain areas associated with intelligence in women. It implies at least two different brain architectures that lead to the same level of intelligence," said Richard Haier, Ph.D., a neuroscience consultant and professor emeritus at the University of California, Irvine.

Something about the way women's brains are built allows us to do the same thinking in a smaller space.

This difference helped tip researchers off to other factors that we now know are also correlated with variations in overall intelligence—and with variations in particular types of intelligence, such as verbal and spatial. Small differences in the placement and size of patches of grey matter (the stuff that does the thinking) and white matter (the stuff that helps the thinking get done more efficiently) can make a big difference in IQ.

Take Einstein. In 1999, Witelson's laboratory studied the great thinker's preserved brain.

" His brain was smack within normal brain size for his age," she said. "But he had a region in the parietal lobe that is crucial for visual imagery and mathematical thinking that was exceptionally large in his case. We suggest that it was the expansion of that region that gave him this extraordinary ability."

This also might help explain why some animals with larger brains are less intelligent than animals with smaller brains—the inner architecture and wiring of their brains are different.

Basically, "bigger brain = smarter" is a good rule of thumb, but it comes with a lot of "buts". Brain size can give you a general idea, but to make a really accurate prediction of any individual's intelligence, you'd need to look at multiple factors—from whether the person was right- or left-handed, to their gender, to their grey matter. Some answers are there, researchers say, but they don't fit easily into a sound byte.

Lars Chittka, Ph.D., professor of sensory and behavioral ecology at Queen Mary, University of London, and Jeremy Gray, Ph.D., associate professor in the department of psychology at Yale University, were also interviewed for this story. Their help was invaluable in piecing together the big picture of brain size and intelligence.