For some other units, the association was established the other way around, i.e., a neuron initially responding to a landmark changed its firing to the associated person after learning. Figure 3 shows a multi-unit in the parahippocampal cortex that, in Task 1 (before learning), originally fired to an image of the White House (mean = 17.8 spikes/s, SD = 7.2, median = 15) and not to American beach volleyball player Kerri Walsh (mean = 5.0 spikes/s, SD = 3.6, median = 3.3). After the patient learned the association between these two concepts (trial 1 in Task 2, see Experimental Procedures for learning criterion), there was an increase in the firing of the neuron to the picture of Kerri Walsh (mean = 13.8 spikes/s, SD = 9.2, median = 14.2), which was statistically significant (p < 0.05, Wilcoxon rank-sum test between baseline and response periods). This increase in the neuron’s response to Kerry Walsh (NP stimulus) after learning was observed in all tasks: a mean of 12.9 spikes/s in Task 2 (post-learning trials only), 16.7 spikes/s in Task 3, and 9.4 spikes/s in Task 5. The response to the preferred stimulus (the White House) increased slightly after learning to 25.6 spikes/s (SD = 8.9), but this difference was not significant (Wilcoxon rank-sum test). Additional examples are shown in Figure S1 and Movie S1

Conventions are the same as in Figure 2 . A multi-unit in the parahippocampal cortex of participant 3 fired at a rate of 17.8 spikes/s (SD = 7.2) to the picture of the White House (preferred stimulus) from a baseline of 4.4 spikes/s (SD = 4.0). This cell only fired at a rate of 5.0 spikes/s (SD = 3.6) to the picture of the American volleyball player Kerri Walsh before learning (Task 1). After learning (trial 1 in Task 2), the cell selectively increased (by 246%) its response to the pair associate (mean response: 13.8 spikes/s, SD = 9.1, p < 0.05).

In 14 patients, who participated in 25 experimental sessions (and only 22 for Task 5), we recorded the activity of multiple single neurons using electrodes implanted in the MTL for clinical reasons. Figure 2 shows a neuron in the hippocampus that responded strongly to the picture of a member of the patient’s family (with a mean firing rate of 13.1 spikes/s, SD = 3.9, median = 12.5) but not to the Eiffel tower (3.6 spikes/s on average, SD = 3.4, median = 3.3. The firing to the Eiffel tower during the response period did not differ significantly from the one during baseline (3.9 spikes/s on average, SD = 2.0, median = 4.2), according to a Wilcoxon rank-sum test (p = 0.84, W = 40.5, n= n= 6). With our experimental design, we aimed to establish whether MTL neurons will widen their tuning to encode the formed association by selectively increasing their firing to the associated stimulus. After a single exposure of the composite picture, the subject learned the association (i.e., family member at the Eiffel tower) and the firing rate in response to the Eiffel tower increased to 7.6 spikes/s on average (SD = 5.1, median = 8.3), a 230% increase compared to the presentations of the Eiffel Tower before learning took place (Task 1). This difference was significant (p = 0.002, W = 563, n= n= 27, Wilcoxon rank-sum test between baseline and response periods, see Experimental Procedures ). In contrast, the response to the preferred stimulus (family member) did not change significantly after learning the association (9.4 spikes/s, SD = 4.5, median = 10.8) and it was similar to the response to the composite image of “family member at the Eiffel tower” (7.8 spikes/s, median = 8.3; p = 0.96, W = 325, n= 27, n= 15, Wilcoxon rank-sum between the response to the Eiffel tower and the composite image). In order to verify that the increase in firing after learning was specific to the associated stimulus pair (NP) and not common to other stimuli used in the experiment, for example, due to an increase in familiarity, we also examined the response to the other stimuli. For each neuron X with a preferred stimulus Px and a non-preferred stimulus NPx, we defined the non-associated (NA) stimuli for neuron X to be all the other pictures used in the association experiment corresponding to the same category of the NPx stimulus (person or landmark). The bottom-right plot of Figure 2 shows the average response to all the NA stimuli, which decreased from a mean of 5.3 spikes/s (SD = 5.6) to 3.8 spikes/s (SD = 4.9) after learning.

A unit in the left hippocampus of participant 14 was activated with a response of 13.1 spikes/s when the image of the patient’s family was presented (preferred stimulus, black squares have been added for privacy reasons). The same cell was not responsive (response: 3.3 spikes/s) to the image of the Eiffel tower before learning (Task 1). For each task the corresponding raster plots (ordered from top to bottom) of each picture are given. Blue rasters represent pre-learning (Task 1) or incorrect trials. Red rasters represent correct or post-learning (Task 5) trials. The spike density function for trials before (BL) and after (AL) learning in response to the non-preferred (left), preferred (middle), and to the mean of the non-associated stimuli (average over 7 pictures) are shown at the bottom panels. Crosses indicate that the stimulus was not shown during a given task. After single-trial learning (Tasks 2, 3, and 4), the unit fired strongly to the picture of the patient’s family (mean: 10.8 spikes/s, left), to the composite picture (7.8 spikes/s, right) and to the picture of the Eiffel tower (7.6 spikes/s). There was a 230% increase in firing to the non-preferred stimulus. The response to the non-associated stimuli slightly decreased from 5.3 spikes/s before learning to 3.6 spikes/s after learning.

Population Responses

Higuchi and Miyashita, 1996 Higuchi S.

Miyashita Y. Formation of mnemonic neuronal responses to visual paired associates in inferotemporal cortex is impaired by perirhinal and entorhinal lesions. Naya et al., 2003 Naya Y.

Yoshida M.

Miyashita Y. Forward processing of long-term associative memory in monkey inferotemporal cortex. 1 = n 2 = 42, p = 0.007, Kolmogorov-Smirnov test, see Figure 4 Population: Visually Responsive Units Show full caption (A) Response changes for all visually responsive units. Each row represents one cell and each column represents one stimulus. The rows were sorted by the strength of the change in the NP stimulus and the columns were unsorted. Blank squares represent stimuli that were not shown during the corresponding session. The mean values across all cells are shown in the middle panel (in colors) and in the bottom panel including SEMs. Ex1, Ex2 correspond to the exemplary units shown in Figures 2 and 3 Higuchi and Miyashita, 1996 Higuchi S.

Miyashita Y. Formation of mnemonic neuronal responses to visual paired associates in inferotemporal cortex is impaired by perirhinal and entorhinal lesions. (B) Cumulative frequency histograms of the correlation coefficient (defined as in) for units before learning (BL) and after learning (AL). Correlation coefficients were significantly higher after learning than before learning (p = 0.007, Kolmogorov-Smirnov test). (C) Average differential activity index DAI = (P r − NP r / P r + NP r ) for all tasks. Lower values of DAI denote more similar responses. Responses to the preferred and non-preferred stimuli become more similar after learning for all tasks (p < 0.001, average decrease by a factor of 1.6, range: 1.4–1.8). (D) Average normalized spike density function (SDF) for 51 visually responsive units to the P, NP, and NA before and after learning. There was a significant increase in the response strength to the NP stimuli after learning (p < 0.05, Wilcoxon rank-sum test). We recorded from a total of 613 units (438 multi-units and 175 single units) from the hippocampus (138 units), entorhinal cortex (117 units), amygdala (194 units), and parahippocampal cortex (164 units). We first identified visually responsive units, defined as those that, before learning, showed a significant difference in the response to at least one stimulus using a Wilcoxon rank-sum test between baseline and response (see Experimental Procedures ). Altogether, we found 51 visually responsive units (31 single units and 20 multi-units) that significantly increased their firing rate in response to the preferred stimulus (P), with P being one individual (27 units) or landmark (24 units). Figure 4 shows the population results for all visually responsive units. Figure 4 A shows the increase in response strength (comparing before and after learning) for each of the 51 visually responsive units and for all stimuli. The population averages are shown at the bottom of Figure 4 A for all types of stimuli, where we observe a larger increase in firing after learning for the NP compared to the other stimuli. The change in firing rate after learning (see “Visually Responsive Units” in Experimental Procedures ) was significantly different for the different stimuli according to a one-way ANOVA F(11,492) = 3.15, MSE = 0.46, p = 0.0001 (n = 42 cells with at least 12 stimuli—9 units that corresponded to sessions where less than 12 stimuli were presented were excluded from this analysis to avoid unbalanced data). This significant difference was largely due to the change in the NP stimuli and not any other non-associated stimulus. In fact, the difference was still significant when excluding the P stimuli (p = 0.01) but not when also excluding the NP stimuli (p = 0.76). Moreover, the only two stimuli that showed a median significantly different from zero were the preferred stimulus (decrease, p = 0.001; see below for interpretation in terms of repetition suppression) and the NP stimulus (increase, p = 0.005). Furthermore, paired t tests showed that the increases in the NP responses were significantly larger than the ones to any other stimulus (all p values between 0.0008 and 0.03). To further validate these results, we performed a permutation test, adjusted for multiple comparisons, by shuffling the labels of the stimuli and taking as test statistic the smallest difference between the activity to the NP stimulus and the one to any other stimuli. We ran 5,000 permutations and found the p value of the NP stimulus to be statistically significant (p = 0.012, see Supplemental Experimental Procedures for details). Of all the 613 units that we recorded from, 51 were visually responsive and 562 were non-responsive (i.e., did not have a significant response compared to baseline before learning). Of the 562 non-responsive units, 12 (2.1%) exhibited a significant increase to at least one image (mean = 4.1 images, SD = 1.5) after learning took place, according to a Wilcoxon rank-sum test between the baseline and response periods. This number is within what could be expected by chance (n = 28) with a false positive rate of 0.05. Only three of the non-responsive units had a change in response to an association pair (P and NP) that was larger than the one to the other pictures (Wilcoxon rank-sum test, p < 0.05). To further quantify the responses of all visually responsive neurons (to all of the presented stimuli), we calculated a pair-coding index (PCI), a correlation coefficient for each neuron between the mean response to each stimulus and its paired associate (as defined in). This statistic has been used to assess how neurons acquire stimulus selectivity through associative learning and is expected to approach zero for a large number of neurons firing with a pattern independent of the stimulus pairs (). Across the population of visually responsive units, we found that the pair-coding indices after learning (median = 0.35) were significantly higher (median = −0.03, D = 0.36, n= n= 42, p = 0.007, Kolmogorov-Smirnov test, see Figure 4 B), thus showing the formation of an association between the P and NP stimulus pairs.

r − NP r / P r + NP r ), where P r , NP r denote the mean activity in the response interval (see r > NP r , and it quantifies the difference in the response to the preferred and non-preferred stimuli. As expected, the largest DAI values were obtained for Task 1 ( To assess the changes that occurred in different tasks, we calculated, for the whole population of visually responsive units, an average differential activity index DAI = (P− NP/ P+ NP), where P, NPdenote the mean activity in the response interval (see Experimental Procedures ). The DAI is expected to be positive, since P> NP, and it quantifies the difference in the response to the preferred and non-preferred stimuli. As expected, the largest DAI values were obtained for Task 1 ( Figure 4 C) before learning took place, indicating a large difference in the response to the P and the NP stimuli. For the following tasks, DAI values were significantly smaller (p < 0.001, see Experimental Procedures ).

To study the time course of the responses, we separated the normalized population response for all visually responsive neurons according to the type of stimuli (P, NP, and NA) and condition (before and after learning). After learning ( Figure 4 D), we found a 172% increase in the response strength to the NP stimuli compared to the pre-learning value. This increase was statistically significant (p = 0.05, n = 51, Wilcoxon rank-sum test between the mean response before versus after learning). In contrast, the mean response to the preferred stimuli decreased to 87% of its pre-learning value (p = 0.3, n = 51, Wilcoxon rank-sum test), while the mean response to the non-associated stimuli did almost not change (101% of the pre-learning value, p = 0.9, n = 51, Wilcoxon rank-sum test).

−13), according to a binomial test with a chance level of 0.05 (see Given these population results, we next evaluated how many of the visually responsive neurons encoded the enforced associations. For this, we defined “pair-coding neurons” as the ones that: (1) showed a significant response to the NP stimulus after learning, using a Wilcoxon rank-sum test comparing baseline and response periods (with p < 0.05), and (2) the distribution of increases of single-trial responses to the NP stimulus after learning was larger than the distribution of increases of single-trial responses to all the other pictures (excluding P) after learning (see “Pair-Coding Units” under Experimental Procedures ). Of the 51 visually responsive units, 21 (41%) were “pair-coding neurons” and selectively increased their response to the NP stimuli after learning. As expected by construction—since based on the screening sessions we chose the NP stimuli to be one that the neuron originally did not fire to—these units showed no significant response to the NP stimuli before learning took place (Wilcoxon rank-sum test). The number of neurons encoding the association (pair-coding neurons, n = 21/51) far exceeded the number expected by chance (p < 10), according to a binomial test with a chance level of 0.05 (see Supplemental Experimental Procedures ). We also verified that the observed distribution of p values was significantly lower than the one generated by neurons with a Poisson firing probability and the same mean firing rates as the responsive units (p < 0.004; see “Proportion of Pair-Coding Units” under Supplemental Experimental Procedures ).