Humans engage in a lot of complex behaviors, but many of them are learned. Genetics gives us a nervous system that's flexible enough to incorporate new behaviors, and we pick them up socially. But many animals display highly complex behavior that appears to be instinctual. Which raises the question of how these sorts of behaviors can be programmed into the nervous system genetically.

Pretty simply, if a new study of mice is to be believed. The work compared the architecture of burrows built by two closely related species. The researchers find the design of the animals' burrows is modular, and the two modules are largely controlled by a handful of genes—perhaps as few as four in total.

The work relied on two closely related species. One is the deer mouse, which is widespread in North America. This species makes very simple burrows, with a short entry passage leading to a nest. In the southeastern US, however, there's a closely related species, the oldfield mouse, that builds a far more complex home. This includes a much longer entrance passage and a secondary "escape tunnel" from the back of the nest that allows it a safe route out should a predator come in the front door.

Various information, such as the species' geographic ranges and their close similarity, suggest the oldfield mice are offshoots of the more widely ranging deer mouse. If accurate, this would suggest the oldfield's burrowing behavior evolved since its separation from the parent species.

To confirm the burrowing behavior was under genetic control, the authors brought some of each species into the lab and raised offspring in cages that lacked any material for them to dig burrows. Then, when the mice were mature, they were set loose into a larger area filled with a sandy soil. Despite never having seen a parent's burrow, the mice quickly dug one that was similar to those that their fellow species members dig in the wild. You can see an example of how the authors managed to study the burrow's properties in the video below.

With the importance of genetics established, the authors set about studying it. Because the two species are so closely related, they were able to get the mice to mate, creating an offspring that was a 50/50 mix of the DNA of the two species. They tested these hybrids for behavior, and in all cases, they acted like the oldfield mice, making longer burrows with an escape tunnel. That, in a classic Mendelian way, suggests the genes that control these behaviors are dominant.

At first glance, it also suggested the suite of behaviors might all be inherited together. But the authors did further crosses to test that by mating the hybrids with regular deer mice (which build simple burrows). The burrows made by the animals that resulted tended to be a bit shorter, on average, than those of the oldfield mice, and only some of these offspring made escape tunnels. The presence of an escape tunnel did not correlate with the length of the burrow, suggesting these two traits are unlinked.

To understand the inheritance better, the authors turned to DNA. Although the two species were still closely enough related that they could mate, they have been separated for long enough to pick up differences in their DNA that could be tracked using molecular tools. So the authors performed a type of analysis (termed quantitative trait locus mapping) to see which parts of the genome were associated with burrow length and the presence of an escape route.

They found three different regions in the genome that, combined, accounted for over half the genetic variation in the length of the burrow. Just one area of the genome seemed to be linked to the presence or absence of an escape route. Even the authors termed the number of sites involved "surprisingly small" (although each site could have more than one gene contributing to its impact).

They also note what appears to be a very complex series of differences is actually the product of both modular contributions (the contributions of genes to length and escape route) and additive (several sites contribute to generate the longer burrow length).

The obvious next step is to try to identify the actual genes involved. This won't necessarily be easy, in that the difference may involve changes in the regulation of a gene rather than the properties of the protein it encodes for, and these can be difficult to identify. Still, once that's done, we can start looking at what these genes actually do and where in the brain they're active. And that could give us the clearest picture yet of how complex instincts get wired into the brain.

Nature, 2013. DOI: 10.1038/nature11816 (About DOIs).