Figure 3 shows for the INTRA and INTER-manual conditions the average responses to each target orientation in trials without conflict, which do not differ appreciably between the two experimental conditions: statistical analyses on the aligning errors showed no significant effect of the experimental condition (ANOVA F(1,15) = 2.13, p = 0.17) or interactions between condition and target orientation (ANOVA F(6,90) = 0.92, p = 0.49). On the other hand, clear differences can be observed in Figure 4 for the responses in the trials with conflict: in the INTRA condition the responses were not consistently deviated by the conflict, in the INTER condition conflict caused the responses to all target orientations to be deviated in the same direction. These latter observations were confirmed by the statistical analysis of the global deviation of the responses. Figure 5A shows that in the INTER-manual condition responses were significantly deviated by the inclination of the visual surround (one-tailed t-test with respect to 0 : t(15) = 2.27, p = 0.02), whilst no statistical difference from the null deviation could be detected in the INTRA-manual condition (one-tailed t-test with respect to 0 : t(15) = −0.37, p 0.25). The differing effects between the two experimental conditions was confirmed by a significant difference between the INTRA- and INTER-manual conditions (ANOVA F(1,15) = 8.81, p = 0.009). The comparison of the response variability between the two experimental conditions, reported in Figure 5B, shows that subjects were more precise when they had to reproduce an orientation felt with the same hand than with the other hand (ANOVA F(1,15) = 10.38, p = 0.005).

The responses for the trials with and without conflict are represented separately. Thick lines are the average responses (transparent areas’ width represent the standard error) of all subjects, combining trials with left and right head tilt as explained in the methods. Green arrows represent the measured responses’ deviations due to the tilt of the visual scene.

A. Experimental results for the deviation of responses induced by the imperceptible tilt of the visual scene and the variability of responses in the INTRA- and INTER-manual conditions. Results are expressed as a percentage of the theoretical deviation expected if subjects aligned the response with respect to the visual scene. B. Average within-subject variability (standard deviation) of the response without conflict. In both panels vertical whiskers represent the 0.95 confidence intervals. Stars represent the significance of the main effects in the ANOVA and the results of the t-test comparison with the nominal 0 value. (** ; * ).

Theoretical Modeling

Given the clear empirical observations shown above, we then undertook a mathematical analysis to better understand the implications of these results for recent theories of sensorimotor integration. To represent the performance of this task, we considered the two candidate models shown in Fig. 2, both of which assume that the brain performs concurrent target-hand comparisons in different sensory modalities and then optimally combines the result of these comparisons to create the movement vector ( ). For model A, . For model B (for details about the notations, see the caption to Figure 2). Both models also assume that the brain can reconstruct missing information from available signals through recurrent neural networks (for review see [18]). To make predictions using these two models, we hypothesized that the weighting ( ) of each individual comparison would be determined by the maximum likelihood principle, which states that the two or three quantities computed in each case will be combined according to the relative variance of each signal (see Methods). We assumed that sensorimotor transformations add variability [19]–[21], such that reconstructed signals will have greater variance than the source signal. Note that this is a slight simplification. A transformation might literally add stochastic noise if the transformation involves the integration of the primary signal with other, noisy sensory inputs. But transformations might also amplify the variability of the primary signal if they included distortions, perceptual biases or other non-linearities. We assume here, however, that on a target-by-target basis the effects of such distortions will be small, such that the effects of a sensorimotor transformation on the variability of the transformed output can be adequately represented by the simple addition of variance and that the MLP equations can be applied directly to the sum, as was done in other sensori-motor integration models [4], [14]. We hence computed relative weights between directly sensed and reconstructed comparisons based on the assumption that transformations add variance [6].

With few exceptions [23], all studies in Neuroscience to date have, to our knowledge, applied the maximum likelihood equations as if the original and reconstructed signals are independent (i.e. uncorrelated). To demonstrate the limitations of that approach, and to gain insight into how covariance might affect the results, we first computed the model predictions without taking into account the covariance between the original sensory inputs and internal representations that are computed through sensorimotor transformations of those signals. Applying Eqs. 4 and 5 from the Methods for model A we have: (A1) (A2)where and are the variance associated to the kinesthetic and visual comparison respectively. For model B, applying Eqs 6–8 from the Methods gives: (B1) (B2) (B3)where , , and are the variances associated to the left-hand kinesthetic, right-hand kinesthetic and visual comparison respectively. The net variance of each of the comparisons, , , and , depends on the variance of the input signals (e.g. , the variance of target orientation as measured via kinesthesia) and the variance added by any sensorimotor transformations required to reconstruct an internal representation (e.g. , representing the variance added when transforming target information from the kinesthetic to visual domains, including the variability of the sensory information that allows one to measure the orientation of the head, such as the visual scene, vestibular signals and neck proprioception). For the detailed equations defining the variance of each comparison, depending on the experimental conditions and hypotheses, see Figure 6. In order to reduce the number of parameters of the model and allow a meaningful fitting to the experimental data, the following assumptions were made:

PPT PowerPoint slide

PowerPoint slide PNG larger image

larger image TIFF original image Download: Figure 6. Model predictions when MLP is applied while ignoring the covariance between direct and reconstructed sensory signals. On the right, graphical representations of the information flow for model A and B and for the INTRA and INTER conditions, together with the equations of the variance corresponding to each concurrent target-hand comparison. The best-fit Model A predicts equal weight given to visual versus kinesthetic information between the INTRA and INTER tasks, and the same response variability in both cases. Model B predicts a greater weighting of visual information and higher response variability for the INTER task, qualitatively similar to the empirical data, but does not correctly predict a zero weighting of visual information in the INTRA-manual task. https://doi.org/10.1371/journal.pone.0068438.g006

The variability of the kinesthetic information from the left and right arm was similar [24]:

The variance associated to the inter-limb transformation of kinesthetic information about the target and about the response was the same (

The variance associated to the reconstruction of visual information from kinesthesia from either the right or left arm was the same (

Note that asymmetries in errors when matching kinesthetic or visual targets with the right or left hand [25] mean that the last of these three assumptions may not be entirely true. We nevertheless adopted these assumptions so as to reduce the number of free parameters to three (variance of the kinesthetic signals , variance due to inter-limbs transformations and variance due to visuo-kinesthetic transformations ) whose values could be determined to best fit the experimentally observed values of response deviation and response variability that are reported in Figure 5.

Figure 6 shows the model predictions based on equations A1–A2 and B1–B3, i.e. when co-variation between sensory signals and reconstructed internal representations is ignored. To the left we show a 2×2 grid, corresponding to the predictions of each of the two models (A and B) for each of the two experimental conditions (INTRA and INTER). Within each model we show the sources of variance associated with each individual comparison necessary for equations A1–A2 and B1–B3, including the noise associated with input signals that are present and variance added by any sensory transformations that may be required. Sensory inputs that are not available directly are grayed out in the schematic diagrams. To the right we superimpose on the experimental data the predictions of each of the two models with weights calculated from equations A1–A2 and B1–B3 that best fit all of the data (deviations and variability). The estimated variability associated with the kinesthetic information and each transformation as a result of the best-fit procedure are shown in Table 1.

From this analysis one can eliminate Model A as an explanation of the empirical data. According to this model, there is no difference between comparing the posture of the right hand to the left hand versus comparing the right hand’s posture with itself. Thus, this model predicts precisely the same weight given to the visual comparison, and thus precisely the same deviations of the response due to tilt of the visual scene between the INTER and INTRA tasks. Model A also predicts that the variability of responses will be equal between the INTER and INTRA tasks. We observed, however, a statistically-significant difference in both deviation and variability between the INTER and INTRA conditions, inconsistent with these predictions (two-tailed t-test between Model A’s prediction and the experimentally observed difference for response deviations between the INTER and INTRA conditions: t(15) = 3.0, p 0.01).

Model B does a better job of capturing the qualitative features of the measured data. According to this model, comparing the posture of the left hand to the right hand requires inter-limb sensory transformations. Thus, the kinesthetic comparisons will be more variable in the INTER task compared to the INTRA task. This allows the model to predict a different weighting between visual and kinesthetic information for the INTER vs. INTRA tasks, meaning that the deviations induced by rotations of the visual scene and the overall variability of responses are expected to be greater in the INTER task than in the INTRA task. Nevertheless, Model B, with weights computed to best fit the data, predicts a smaller differential of the response deviations between the INTRA and INTER tasks than what we actually observed (two-tailed t-test between Model B's prediction and the experimentally observed difference in response deviations between the INTER and INTRA conditions: t(15) = 2.2, p 0.05). Indeed, using reasonable assumptions about the relative amounts of variability in each sensory signal and in each sensory reconstruction, one cannot expect to see zero weight given to a visual comparison in the INTRA condition using Model B, even though that is what we observed experimentally.

This inability to reproduce the deviation data with Model B was insensitive to the precise values of response variability used to perform the fitting. The only way that one could expect to see the observed difference in deviations between INTER and INTRA with Model B would be if the difference in response variability between INTRA and INTER was much greater than what was observed. But to achieve the better than 7∶1 ratio that would be required to reproduce the deviation data with Model B, either the overall response variability would have to be much smaller ( ) in the INTRA condition or much larger ( ) in the INTER condition than the actually observed values ( 1.9 for INTRA and for INTER).

To adequately fit both the deviation and variability data we needed to take into account the covariance between a reconstructed signal and its source when computing the MLP weights. As explained in the Methods section, this means that the relative weighting of the parallel comparisons is based on the variance of the independent components of each comparison, neglecting the variance of the common components. In Figure 7 we therefore report the variance associated to each possible comparison with the component of the variance common to all branches grayed out.

PPT PowerPoint slide

PowerPoint slide PNG larger image

larger image TIFF original image Download: Figure 7. Model predictions when covariance is taken into account in the calculation of the optimal weights for each individual comparison. On the right, graphical representations of the information flow for Model A’ and B’ and for the INTRA and INTER conditions, together with the equations of the variance corresponding to each concurrent target-hand comparison. Components of variance common to all branches, and hence not used to define the relative weights for , are grayed out. Model A’ predicts that there will be no reconstruction of the task in visual space for either INTRA or INTRA, because there is no variance in the comparison that is not included in the comparison. Deviation of the response with visual scene tilt should be 0 and the variance of the response should be the same for both conditions. Model B’ predicts that the task will be carried out only in kinesthetic space for the INTRA condition, but that both kinesthetic and visual comparisons will be made in the INTER condition. Only Model B’ can accurately fit the data. https://doi.org/10.1371/journal.pone.0068438.g007

For model A we note that there is no component of variance in kinesthetic comparison that is not also included in the visual comparison . Applying Eqs. 16–17 of the Methods, the predicted weight is zero for the visual comparison and one for the kinesthetic comparison for both INTER and INTRA conditions: (A'1) (A'2)

Because both conditions rely on only, one should also observe equal variance of responses between the two conditions. The statistically significant differences of deviations and variability between INTER and INTRA, and the statistically non-zero weight given in to visual information in the INTER condition mean that Model A should be rejected even if covariance is taken into account (two-tailed t-test between the predictions of A’1–A’2 and the experimentally observed difference for response deviations between the INTER and INTRA conditions: t(15) = 3.0, p 0.01).

On the other hand applying the same concepts for Model B does predict different results between the INTER and INTRA conditions. As shown in Figure 7, for the INTRA situation, there is no variance associated with the direct intra-manual comparison that is not also included in and . Applying Eqs. 18–20 to this situation gives the following weights for the three concurrent comparisons: (B'1) (B'2) (B'3)

No weight will be given to either or and thus, the deviation of the responses in the INTRA condition is predicted by Model B to be strictly zero. On the contrary, Figure 6 shows that in the INTER condition each of the comparisons , and contains a component of variance that is not included in the two others, because each comparison requires at least one transformation, the transformations are all different, and each of these transformation adds variability that is independent from the others. In this case applying Eqs. 18–20 predicts that some non-zero weight will be given to each of the three comparisons to determine the optimal outcome. (B'4) (B'5) (B'6)

The estimated variances associated with the kinesthetic information and sensory transformations as a result of the best-fit procedure are shown in Table 1. The non-zero weight given to means that Model B does predict a deviation of responses due to tilt of the visual scene. Unlike for the predictions of Eqs. B1–B3, predictions made by taking into account the covariance when applying MLP (Eqs. B’1–B’6) are not statistically different from the empirically estimated difference in weights between the INTER and INTRA conditions (two-tailed t-test between the predictions of Model B’ and the experimentally observed difference for response deviations between the INTER and INTRA conditions: t(15) = 0.4, p 0.70). The variability added by the different transformations also means that the final variance of responses is expected to be greater for the INTER condition, compared to the INTRA condition, which is what we observed. Equations B’1–B’6 are therefore able to fit the salient features of the empirical data (deviation of responses in INTER but not INTRA, different variability between INTER and INTRA).