DAVID DOUBILET/National Geographic Creative

The octopus is weird: eerily malleable body, sucker-studded arms, skin that can transform into a convincing facsimile of seaweed—or sand—in a flash. It can solve mazes, open jars, use tools. It even has what seems to be a sophisticated inner life. What's confusing about all this is that the octopus has a brain unlike that of almost any creature we might think of as intelligent. In fact, the octopus brain is so different from ours—from most of the animals we're accustomed to studying—that it holds a rare promise: If we can figure out how the octopus manages its complex feats of cognition, we might be closer to discovering some of the fundamental elements of thought—and to developing new ideas about how mental capacity evolved.

"Part of the problem in working out what's essential to intelligence in the brain is working out which are the features that, if you took them away, you would no longer have an intelligent system," says Peter Godfrey-Smith, a philosopher at CUNY who studies animal minds. "What's essential as opposed to an accident of history?" Think about it: Chimpanzees are, like us humans, primates. Dolphins are mammals. Even clever crows and ravens are at least vertebrates. But our last common ancestor with the octopus was probably some kind of wormlike creature with eye spots that lived as many as 750 million years ago; the octopus has a sophisticated intelligence that emerged from an almost entirely different genetic foundation. If you want to study an alien intelligence, Godfrey-Smith says, "octopuses are the closest thing we have."

If you were to measure octopus smarts by the number of neurons the creatures have (500 million to our almost 100 billion), they'd come up pretty dull. But forget that metric. The octopus's neurons aren't even concentrated in its head; about two-thirds of its "brains" are distributed in its arms, dedicated to the fine operation of these limbs and each of their hundreds of suckers. The rest of the neurons are split between a central brain—surrounding the esophagus—and large optic lobes behind the eyes. Like we said: alien.

But somehow octopuses do things that suggest they're brainier than plenty of animals with backbones and more familiar nervous systems. Here's an easy one: Lots of octopods have learned to twist off standard jar lids. But in 2003, biologists at the Seattle Aquarium challenged Billye, a female Enteroctopus dofleini—a giant Pacific octopus—with a childproof bottle, the kind that can baffle even the smartest Homo sapiens. Billye figured out the push-and-twist trick in a little less than an hour. And in subsequent attempts, she popped those tricky tops in a mere five minutes.

Is the vertebrate brain optimized for intelligence? Ask the octopus. | Robert Eikelpoth/Corbis

This is just the beginning of their abilities. Octopuses in the wild may be using tools—a feat that, not so long ago, was considered the exclusive domain of humans (though now we know it's the province of other species too, like dolphins and some birds). Researchers have observed octopuses off the coast of Indonesia collecting—and awkwardly carrying—coconut shell halves along the sandy seafloor. For a shelter on the go, they whip out the two pieces of shell, swoop inside, and snap the pair shut. "That's a spectacular example, because it really does suggest foresight," says Jennifer Mather, who studies animal behavior at Canada's University of Lethbridge. "In terms of cognition, that's pretty good."

The octopus displays sophisticated (some might say even irreverent) behavior in the lab too. Just ask Jean Boal, a behavioral researcher at Millersville University. On the way to feed her octopus subjects one day, she suspected they might not like what was on offer: They preferred the very freshest of frozen squid, but the stuff she bore was a bit stale. She doled it out anyway, walking down the line of tanks, dropping a subpar serving into each one. When she finished, she walked back to the first octopus to see if it had gone for the meal. The food was nowhere to be seen, but the cephalopod was waiting for Boal—waiting and watching. This octopus locked eyes with her and moved slowly sideways to the drain in the front right corner of its tank. Pausing above the outflow, it shot the stale squid out of its arms and down the drain, continuing its stare (or was it a glare?) at Boal, who got the message. Two, actually: This octopus was not going to tolerate crummy food—and maybe it even wanted Boal to understand that.

These behaviors are especially impressive because octopuses are solitary creatures—you can't argue that they learn skills like tool use from their parents. There isn't an octopus culture; you can't posit that their apparent ability to communicate with us stems from group behavior in the wild. (Actually, they tend to be cannibalistic. Even octopuses know that octopuses are delicious.) What you can argue, though—and it's something Boal and other researchers have suggested—is that the octopus got smart because the octopus got soft (or vice versa). It has no bones, no shell, no scary spikes. So if the octopus wanted to go hunting in an ocean full of fish that are also hunting (sometimes for octopus), it had to become fiendishly clever. Not unlike a certain shell-less, clawless, furless primate we could mention.

Just how smart octopuses are, however, has been difficult to determine. They are tough to study—not just conceptually but physically. You need a backup set of everything, because your stretchy-armed and curious subjects will inevitably pull stuff into their tanks for examination. Besides stealing lab equipment, they will yank up tank drain plugs, and they can and will escape through any opening larger than their small beaks. They are also temperamental. As Boal and colleagues put it in a research paper, "A chief roadblock in investigations of octopus learning abilities has been their relative intractability as experimental subjects." Try to test their memory and spatial navigation skills in a maze, as you might with a rat, and lab octopuses often refuse to budge. So while it's easy to make casual observations about their behavior, it's difficult to run the creatures through the kinds of task-based tests (like mazes and object-discrimination trials) that scientists rely on to prove things in vertebrate subjects.

With its distributed brain, the octopus is like the Internet, whereas we humans are stuck with individual CPUs.

Furthermore, their combination of dexterity and smooshiness makes it nearly impossible to use traditional monitoring technology. Octopuses will take off or rip out external or implanted wires, and they lack hard structures on which to affix devices. Michael Kuba, who has studied octopus intelligence at the Hebrew University of Jerusalem, talks with fond laughter about his original postdoc plan to record the neural patterns of octo-subjects as they were learning new tasks. "It was a miserable failure," he says. "They just pulled the wire off." Even with new, smaller wireless data loggers, "it's going to be some time before we get anything like exact neural recordings of an octopus brain," Kuba says.

Just about any animal can pick up a behavior through training—even a snail can "learn" not to open its breathing hole if it's poked enough. But most animals don't seem to be able to master complicated puzzles, make tools, or communicate individual preferences.

Scientists think one reason we humans can do those things is that we have an unusually well-developed capacity to learn and remember. Our short- and long-term memories are lodged in multiple parts of the brain—an arrangement underscored by people who have suffered, say, a prefrontal lobe injury. These people might have difficulty with short-term memory, but their long-term memories remain intact. Simpler organisms like sea slugs, however, use a single spot—and even the same synapses—for both forms of recall. And human long-term memories are made by a process known as long-term potentiation, which strengthens nerve synapse activity, allowing more data to be stored. Most vertebrate lab animals have brains that work the same way, so we've only been able to guess whether ours is the optimal—or the only—solution for achieving as much as we have, cognitively speaking.

The octopus may hold the answer to this question. Despite our last common ancestor being that worm, the octopus has evolved a similar setup for recording and storing memories. This is striking, because "not that many brains are organized to acquire a lot of memories," says Binyamin Hochner, also of the Hebrew University. Hochner and his colleagues have discovered that the octopus depends on long-term potentiation to learn and create long-term memories. The presence of these familiar structures and dynamics in the animal, which has quite a different set of genes than vertebrates, suggests that LTP might be one of those rare, crucial elements of intelligence that Godfrey-Smith alluded to. With that tantalizing suggestion, researchers can now focus more intently on the workings of octopus learning and memory as an alien but useful foil for our own.

The octopus might even present new kinds of intelligence. Most of what we do all day—scratch an itch or sing along to Miley—is controlled by our outsize noggins. Our brains are big and powerful and have a highly developed frontal lobe, which handles the executive functions that make possible amazing things, such as studying octopuses. But it turns out that this sort of centralized encephalization is not the only evolutionary solution for developing substantial intelligence. The octopus's unusual neuronal layout allows its eight individual, flexible arms to act and carry out instructions on their own—and in coordination with one another. That means the central brain doesn't have to be bothered with small, continuous signals from and directions to each of the suckers. They're operating on their own volition, a fascinating alternative to our own jointed, head-directed limbs. And it's not just brain researchers who are learning from octopuses; one scientist has advised the military about ways of replicating this capability for troop and command structures, and roboticists are trying to figure out how to instill this sort of "embodied intelligence" into their bots. As one researcher puts it, the octopus is like the Internet, whereas we are stuck with individual CPUs.

That's the deeper suggestion behind the mind of the octopus—that it's time to get out of our own heads. To really expand and accelerate our understanding of intelligence, we need a model that is sophisticated but so utterly alien that it will keep us reexamining our ways of studying—and thinking about—the brain. The mind-bending model of the mind that octopuses offer us right here on Earth forces us to look beyond the standard lines of study we have set up around ourselves. We need the octopus. Even if it's a challenging, mystifying, and occasionally saucy subject. Or perhaps precisely because it is.

*Katherine Harmon Courage (@KHCourage) is the author of *Octopus! The Most Mysterious Creature in the Sea, to be published in October by Current/Penguin.