But many researchers wanted to move past using markers at all, and they wanted to track more than seven points on their animals. So by combining insights gained from previous work, both on animals and humans, multiple labs have created easy-to-use systems that are now seeing widespread application.

The first of these systems came online last year. DeepLabCut was developed by the Harvard neuroscientists Mackenzie Mathis and Alexander Mathis, who repurposed a neural network that was already trained to classify thousands of objects. Other methods followed in rapid succession: LEAP (Leap Estimates Animal Pose), developed by Pereira and others in the labs of Murthy and Shaevitz; SLEAP, the same team’s forthcoming software for tracking the body-part locations of multiple interacting animals at once; and the Couzin group’s DeepPoseKit, published a few months ago.

“It can learn really fast,” Murthy said of LEAP. “Within 10 or 15 minutes, it can be trained to run automatically on all of your videos.” Other groups are working on modeling poses in three dimensions rather than two, by calibrating similar models using multiple cameras.

“Under the hood, these technologies can be incredibly sophisticated,” Couzin said, “but now they’re actually amazingly easy to apply to a very broad range of problems, from how a mouse’s whiskers move to ant behavior to fish schooling.”

Whitlock has found that in the mice he studies, particular movements and positions are encoded throughout regions of the cortex involved in coordinated movement — and perhaps more widely. “These parts of the brain really care a lot about how the animal is holding its head,” he said. “This is an aspect of cortical processing that we just simply haven’t appreciated before” because researchers hadn’t been able to track freely moving animals.

By delineating posture, the algorithms open a window into a deeper understanding of behavior. Essentially, all measurable behaviors are “changes in posture through time,” Whitlock said. “And we’ve got posture. We’ve nailed that.”

Because pose-tracking software has simplified data collection, “now we can think about other problems,” said Benjamin de Bivort, a behavioral biologist at Harvard University. Starting with: How do we define the building blocks of behavior, and how do we interpret them?

A Hidden Language

Attempts to answer these questions have long relied on the observer’s intuition — “immaculate perception,” as ethologists (animal behaviorists) jokingly call it. But intuition is hobbled by inherent biases, a lack of reproducibility, and difficulty in generalizing.

The zoologist Ilan Golani at Tel Aviv University has spent much of the past six decades in search of a less arbitrary way to describe and analyze behavior — one involving a fundamental unit of behavior akin to the atom in chemistry. He didn’t want behaviors to be tagged simply as courting or feeding. He wanted those characterizations to arise “naturally,” from a common set of rules grounded in an animal’s anatomy. Golani has his own model of what those units and rules should look like, but he thinks the field is still far from arriving at a consensus about it.

Other researchers take the opposite position, that machine learning and deep learning could bring the field to a consensus sooner. But while DeepLabCut, LEAP and the other cutting-edge pose-tracking algorithms rely on supervised learning — they’re trained to infer the locations of body parts from hand-labeled data — scientists hope to find and analyze the building blocks of behavior with unsupervised learning techniques. An unsupervised approach holds the promise of revealing the hidden structure of behaviors on its own, without humans dictating every step and introducing biases.

An intriguing example of this appeared in 2008, when researchers identified four building blocks of worm movement that could be added together to capture almost all the motions in the animal’s repertoire. Dubbed the “eigenworm,” this compact representation offered a quantitative way to think about behavioral dynamics.

Datta took this approach to a whole new level with his Xbox Kinect hack in 2013, and he was quickly rewarded for it. When he and his colleagues looked at the data describing the movements of the mice, they were surprised to immediately see an overarching structure within it. The dynamics of the animals’ three-dimensional behavior seemed to segment naturally into small chunks that lasted for 300 milliseconds on average. “This is just in the data. I’m showing you raw data,” Datta said. “It’s just a fundamental feature of the mouse’s behavior.”

Those chunks, he thought, looked an awful lot like what you might expect a unit of behavior to look like — like syllables, strung together through a set of rules, or grammar. He and his team built a deep neural network that identified those syllables by dividing up the animal’s activity in a way that led to the best predictions of future behavior. The algorithm, called Motion Sequencing (MoSeq), spat out syllables that the researchers would later name “run forward” or “down and dart” or “get out!” In a typical experiment, a mouse would use 40 to 50 of them, only some of which corresponded to behaviors for which humans have names.

“Their algorithms can pull out behaviors that we don’t have words for,” Whitlock said.

Now researchers are trying to determine the biological or ecological significance of these previously overlooked behaviors. They’re studying how the behaviors vary between individuals or sexes or species, how behavior breaks down with age or disease, and how it develops during learning or in the course of evolution. They’re using these automatic classifications to discern the behavioral effects of different gene mutations and medical treatments, and to characterize social interactions.

And they’re starting to make the first connections to the brain and its internal states.

Predicting Brain States and Behaviors

Datta and his colleagues discovered that in the striatum, a brain region responsible for motor planning and other functions, different sets of neurons fire to represent the different syllables identified by MoSeq. So “we know that this grammar is directly regulated by the brain,” Datta said. “It’s not just an epiphenomenon, it’s an actual thing the brain controls.”

Intriguingly, the neural representation of a given syllable wasn’t always the same. It instead changed to reflect the sequence in which the syllable was embedded. By looking at the activity of the neurons, for instance, Datta could tell whether a certain syllable was part of a very fixed or very variable sequence. “At the highest level,” he said, “what that tells you is that the striatum isn’t just encoding what behavior gets expressed. It’s also telling you something about the context in which it’s expressed.”

He supported this hypothesis further by testing what happened when the striatum no longer worked properly. The syllables themselves remained intact, but the grammar became scrambled, the sequences of actions seemingly more random and less adaptive.

Other researchers are looking at what’s going on in the brain on longer timescales. Gordon Berman, a theoretical biophysicist at Emory University, uses an unsupervised analysis technique called Motion Mapper to model behavior. The model, which places behaviors within a hierarchy, can predict hierarchical neural activity in the brain, as demonstrated in a paper published by a team of researchers at the University of Vienna two weeks ago. (Berman says that “an aspirational goal” would be to someday use Motion Mapper to predict social interactions among animals as well.)

And then there’s Murthy and her team, and their search for hidden internal states. They had previously created a model that used measurements of the flies’ movements to predict when, how and what the male fly would sing. They discovered, for example, that as the distance between the male and female flies decreased, the male was likelier to produce a particular type of song.

In the work recently published in Nature Neuroscience, the scientists extended this model to include potential hidden internal states in the male flies that might improve predictions about which songs the flies would produce. The team uncovered three states, which they dubbed “Close,” “Chasing” and “Whatever.” By activating various neurons and examining the results with their model, they discovered that a set of neurons that had been thought to control song production instead controlled the fly’s state. “It’s a different interpretation of what the neuron is doing in the service of the fly’s behavior,” Murthy said.

They’re now building on these findings with SLEAP. “It’ll be really exciting to see what kind of hidden states this type of model is able to tease out when we incorporate higher-resolution pose tracking,” Pereira said.

The scientists are careful to note that these techniques should enhance and complement traditional behavioral studies, not replace them. They also agree that much work needs to be done before core universal principles of behavior will start to emerge. Additional machine learning models will be needed, for example, to correlate the behavioral data with other complex types of information.

“This is very much a first step in terms of thinking about this problem,” Datta said. He has no doubt that “some kid is going to come up with a much better way of doing this.” Still, “what’s nice about this is that we’re getting away from the place where ethologists were, where people were arguing with each other and yelling at each other over whether my description is better than yours. Now we have a yardstick.”

“We are getting to a point where the methods are keeping up with our questions,” Murthy said. “That roadblock has just been lifted. So I think that the sky’s the limit. People can do what they want.”

Editor’s note: The work by Bob Datta, Jonathan Pillow and Adam Calhoun is funded in part by the Simons Foundation, which also funds this editorially independent magazine.

Animated pose-model of a walking fly courtesy of Pierre Karashchuk, Tuthill/Brunton labs, University of Washington; anipose.org