The reconstructed brains from four NT crania are shown in Fig. 1. The brains were reconstructed using the population-averaged modern human brain (upper panel) or from one representative modern human brain (middle panel). The reconstructed brains with the neuroanatomical labels were also presented (lower panel). Identifying cortical features such as imprints of sulci and gyri on the endocranial surface (and placing landmarks) is actually very difficult since such imprints are very subtle on human and Neanderthal fossil crania. However, here we assumed that modern human brain maps are the best available and most parsimonious proxy of the brain morphology of the last common ancestor of humans and Neanderthals, and the location of brain regions was predicted from the NT and EH endocasts. Therefore, using the present reconstruction method, the position and shape of the sulci and gyri can be well estimated, allowing more detailed, unbiased investigations of the brain morphology.

Figure 1 Reconstructed Neanderthal brains. (a) Population-average. (b) Representative modern human subject. (c) The reconstructed brains with the neuroanatomical labels. Full size image

The cerebral and cerebellar volumes of the reconstructed NT and EH brains were 1304 and 182 cc for Amud 1; 1159 and 140 cc for La Chapelle-aux-Saints 1; 1268 and 166 cc for La Ferrassie 1; 912 and 106 cc for Forbes’ Quarry 1; 1075 and 147 cc for Qafzeh 9; 1053 and 146 cc for Skhul 5; 1205 and 165 cc for Mladeč 1; and 1208 and 156 for Cro-Magnon 1, respectively (See Method). The mean (±standard deviation) cerebral and cerebellar volumes of NT, EH and MH were 1161 ± 177 cc and 149 ± 33 cc, 1135 ± 83 cc and 153 ± 9 cc, and 1097 ± 115 cc and 149 ± 15 cc, respectively. No statistically significant between-group difference was detected in the total brain volume. However, the mean ratios of cerebellum to cerebrum in NT, EH and MH were 0.127 ± 0.010, 0.135 ± 0.004 and 0.136 ± 0.005, respectively. One-way ANOVA with multiple comparisons (Ryan’s method) indicated that NT had significantly smaller relative cerebellar volume than EH and MH (F 2,1190 = 8.53, p < 0.001, NT vs. EH t 1190 = 2.47, p < 0.05, NT vs. MH t 1190 = 4.09, p < 0.05, respectively).

The surface morphology between NT, EH and MH was compared using a surface displacement-based morphometry (Fig. 2) (See Method). There were significant morphological differences in the cerebellar, parietal, occipital and medial temporal regions, but no differences in the frontal regions between NT and MH (Fig. 2(a) NT vs MH). Between NT and EH, there were significant differences in the cerebellar and part of the right medial temporal and right somato-motor regions (Fig. 2(a) NT vs EH). Virtually no difference was observed between MH and EH, except for small part of the right somato-motor region (Fig. 2(a) MH vs EH). Significant difference was also noted in the basilar region, but this occurred possibly due to complex morphology of the sphenoid bone. Therefore, the largest morphological difference between NT and the EH-MH lineage was observed in the cerebellar hemisphere, which was significantly more inferiorly projected in EH and MH than in NT (Fig. 2).

Figure 2 Comparisons of the brain surface morphology among Neanderthal, early Homo sapiens and modern Homo sapiens. (a) Surface statistical map shows the surface area where the differences are statistically significant (p < 0.05 with family-wise error (FWE) correction). See Extended Data Fig. 1 for more details. (b) Surface displacement maps show the morphological difference in the direction perpendicular to the tangential surface. The displacement maps were calculated by subtracting modern Homo sapiens (MH) from Neanderthal (NT), early Homo sapiens (EH) from NT, and MH from EH, respectively. See Extended Data Fig. 2 for more details. Full size image

The volume of each parcellated region between the groups was also compared (Fig. 3) after adjustment for total intracranial volume (ICV) was performed by including ICV as a covariate in a linear model (ANCOVA) and regressing it out31. This size adjustment is necessary to correct for large interindividual variability in the geometrical size of the specimens used in the present study. The results of analysis of variance showed that the size-adjusted volume differences among three groups were found in the superior and inferior region of parietal lobe, occipital regions, and cerebellum (Fig. 3). As we conducted post hoc test between 3 groups, only the cerebellum has a significant difference both between NT and EH (t 1190 = 3.41, p < 0.05 corrected for multiple comparisons with Ryan’s method, for cerebellar vermis, and t 1190 = 2.33, p < 0.05 for posterior cerebellar hemispheres) and NT and MH (t 1190 = 3.64, p < 0.05, and t 1190 = 3.64, p < 0.05, respectively). Namely, the size-adjusted volume of the cerebellum (vermis and posterior hemispheres) was significantly larger in the EH–MH lineage.

Figure 3 Comparisons of the relative volumes of the parcellated brain regions among NT, EH and MH. (a) Each parcellated volume was normalized to the mean MH volume to calculate a ratio (i.e. relative volume unit). The regionally specific volume differences were evaluated after removing the effects of ICV by analysis of variance (ANOVA) across 13 parcellated regions. We employed Bonferroni correction for multiple comparisons, so that the threshold of p < 0.003 (=0.05/13) is set to statistically significant. The relative volume differences were found in Pa SI (F 2,1190 = 9.31, p = 0.0001), Oc SM (F 2,1190 = 8.15, p = 0.0003), Ce V (F 2,1190 = 7.34, p = 0.0007), and Ce P (F 2,1190 = 6.70, p = 0.0013) (Extended Data Table 3). The mean (±standard deviation) MH volumes of the parcellated brain regions are 161.61 ± 5.22 cc for Fr SM, 41.57 ± 1.00 cc for Fr I, 65.46 ± 3.45 cc for Fr O, 96.93 ± 2.82 cc for Sm, 88.30 ± 1.98 cc for Pa SI, 37.38 ± 1.41 cc for Pa TP, 91.27 ± 3.92 cc for Te SM, 82.50 ± 2.60 cc for Te I, 93.70 ± 4.17 cc for Oc SM, 39.99 ± 1.32 cc for Oc I, 12.38 ± 0.23 cc for Ce V, 13.86 ± 0.21 cc for Ce A, and 114.41 ± 3.76 cc for Ce P. (b) As the ANOVA results indicated a significant group-by-laterality difference in the size-adjusted volume of the cerebellar hemisphere (F 2,1190 = 14.28, p < 0.001 for Ce A, F 2,1190 = 12.73, p < 0.001 for Ce P), we tested if there is a significant difference between the size-adjusted volume of the left and right cerebellar regions within each group and between groups based on the symmetrized volume analysis. Fr, frontal lobe; Pa, parietal lobe; Te, temporal lobe; Oc, occipital lobe; Ce, cerebellum; Sm, sensorimotor cortex; SM, superior and middle region; I, inferior region; O, orbitofrontal region; SI, superior and inferior region; TP, temporo-parietal junction; A, anterior region; P, posterior region; V, vermis. *p < 0.05 corrected for multiple comparisons. Data are means ± s.d. See Extended Data Table 1 for correspondence between the automated anatomical labelling (AAL) atlas and the parcellated brain regions. Full size image

Our results of the surface displacement-based morphometry and the comparison of the size-adjusted volume of each parcellated region indicated that the cerebellum has the most prominent morphological and volumetric difference between NT and the EH-MH lineage. Previously, Weaver suggested that the cerebellum was relatively larger in MH than in terminal Pleistocene humans, and the enlargement of the cerebellum in the MH lineage started to occur sometime after 28,000 years ago14. However, this study clearly indicated that the cerebellum started to enlarge in the EH-MH lineage far before the time when NT disappeared because the relative cerebellar volume was much larger than NT not only in Mladeč 1 and Cro-Magnon 1, but also in Qafzeh 9 and Skhul 5.

There is now strong evidence that the cerebellar hemispheres are important for both motor-related function and higher cognition including language, working memory, social abilities and even thought32,33,34. Further, whole cerebellar size is correlated with cognitive abilities, especially in the verbal and working memory domain35. Thus, we examined the relationship between cerebellar volumes and various cognitive task performances using a large data set from the human connectome project (see Methods). Multiple regression analyses revealed that attention and inhibition task score was most strongly correlated with size-adjusted whole cerebellar volumes (t 1090 = 4.27, p < 0.001), followed by cognitive flexibility task score (t 1090 = 3.24, p = 0.001). There was also a significant correlation of size-adjusted cerebellar volumes with speech comprehension (t 1090 = 3.33, p = 0.001), speech production (t 1090 = 2.86, p = 0.004), working memory (t 1090 = 2.92, p = 0.004), episodic memory (t 1090 = 2.84, p = 0.005) task scores, but not with processing speed task score (t 1090 = 1.29, p = 0.199). Note that the functions such as attention, inhibition, cognitive flexibility, working memory, are thought to be main components of executive functions36. These results indicate that the cerebellar hemispheres are involved in the abilities of executive functions, language processing, and episodic memory function.

Unlike complex neuronal networks in the cerebrum, the cerebellar neural circuit (module or microcomplex) is anatomically simple and uniform32. As the cerebellar hemisphere contains many of these modules37, a larger cerebellar volume is directly correlated with larger number of the modules, and therefore with higher language processing and larger working memory capacity. Language processing refers to the ability to produce and comprehend sounds and signs, which enables shared communication between individuals38. Working memory is a temporary memory storage and executive information processing system used for cognitive abilities such as learning and reasoning39. In addition, these functional modules can encode essential properties of mental representation in the cerebrum for various cognitive activities40, possibly leading to the correlation between the size-adjusted cerebellar volume and the ability of executive functions. Thus, Homo sapiens with relatively larger cerebellar hemispheres may possess higher cognitive and social functions.

Furthermore, we noticed that there seems to exist a possible evidence for bilateral volumetric asymmetry in the NT cerebellum but not in the cerebellum of the EH-MH lineage. To examine the volumetric laterality of the cerebellum within a group, we recalculated the volume of each cerebellar region using a symmetrized automated anatomical labelling (AAL) atlas as well as a mirror of this symmetrized atlas (See Methods). We found that the right side of anterior and posterior cerebellum was significantly smaller than that of the left in NT (simple main effect F 1,1190 = 34.85, p < 0.001, and F 1,1190 = 26.44, p < 0.001, respectively) but no statistical differences were detected in EH (F 1,1190 = 2.24, p = 0.13, and F 1,1190 = 3.84, p = 0.05, respectively) and MH (F 1,1190 = 0.77, p = 0.38 and F 1,1190 = 0.99, p = 0.32, respectively). We also found that the relative volume of the right cerebellar hemisphere was significantly smaller in NT compared with that in EH (t 2380 = 2.41, p < 0.05 corrected for multiple comparisons with Ryan’s method for anterior cerebellar hemisphere, and t 2380 = 3.70, p < 0.05 for posterior cerebellar hemisphere) and MH (t 2380 = 2.77, p < 0.05 and t 2380 = 4.74, p < 0.05, respectively), with no differences between EH and MH and no differences in the left hemisphere among the three groups.

The functions of the cerebellar hemispheres differ according to location, as different parts of the cerebellum are anatomically and functionally connected to different regions of the cerebrum41. In particular, the lateral parts of the cerebellar hemisphere are anatomically connected to the opposite side of the association cortices in the cerebrum42. Our finding of laterality in terms of the relatively small right cerebellar hemisphere of NT indicates minimal connection to the left prefrontal regions, which has one of the major role in language processing38, potentially causing disparity of language ability between NT and Homo sapiens. However, the preservation in the cerebellar region of the fossils is certainly not perfect and there might be asymmetry related to taphonomy in addition to the innate morphological asymmetry in the region. Therefore, morphological laterality of the Neanderthal cerebellum needs to be confirmed in future studies with a large number of cases.

In the present study, the MH had relatively larger parietal regions than the NT with significant difference, particularly in the superior medial and lateral areas (Figs 2 and 3), as suggested by Bruner et al.11. However, there were no differences in the relative size of the parietal region between NT and EH. The superior medial part of the parietal lobule (the precuneus) plays important roles in highly integrated tasks, including visuo-spatial imagery, episodic memory and self-related mental representations43, whereas the superior lateral region is involved in integration and coordination between the self and the external space, generation of body image and sense of agency44. In addition, the parietal regions have strong connections to the cerebellar hemispheres and the frontal cortex41. These findings indicate that enlargement of this region in MH may have improved cognitive function in harmony with the cerebellar hemispheres and the frontal region.

Previous studies have reported significant differences in the occipital and medial temporal regions between NT and Homo sapiens. Pearce et al. estimated that NT had larger visual cortices than EH based on the orbit size of fossil crania45. In support, the occipital region was significantly larger in NT than in EH in the present study (Figs 2 and 3). There are also reported differences in the basicranial morphology between NT and EH, with NT having a relatively narrower orbitofrontal cortex, smaller olfactory bulbs and less increased and forward-projecting temporal lobe poles12. We also observed clear differences in basicranial morphology between the two species. The anterior part of the medial temporal region was more inferiorly projected in NT than in EH and MH (Fig. 2), which is consistent with data from Bastir et al.12 showing a relatively low temporal pole position in NT.

In the present study, we used the average human brain to reconstruct NT and EH brains. Therefore, the variation within NT or EH was basically estimated based on four fossil brains. However, to account for possible larger variation within NT or EH, we also reconstructed 4 × 1185 NT and 4 × 1185 EH brains, assuming that the variation within NT and EH was equivalent to that of MH. The intra- and inter-specific variation in each parcellated brain region was analysed and evaluated based on the Cohen’s d effect size and statistical test. We confirmed that our results are not affected by the use of the averaged brain for our reconstructions (See Extended Data Fig. 4).

In conclusion, we found that NT had significantly relatively smaller cerebellar hemispheres than Homo sapiens, particularly on the right side. Larger cerebellar hemispheres were related to higher cognitive and social functions including executive functions, language processing and episodic and working memory capacity. Based on archaeological records, Wynn and Coolidge suggested that NT had a smaller capacity of working memory46, which is also related to the capacity for cognitive fluidity proposed by Mithen47. Moreover, such differences in the capacity for cognitive fluidity were hypothesized to mainly originate from language processing ability48. Thus, the differences in neuroanatomical organization of the cerebellum may have resulted in a critical difference in cognitive and social ability between the two species. Consequently, ability to adapt to changing environment by creating innovation may have been limited in NT and this difference possibly affected their chance of survival and drove the replacement process.