Brain-computer interfaces (BCIs) using signals acquired with intracortical implants have achieved successful high-dimensional robotic device control useful for completing daily tasks. However, the substantial amount of medical and surgical expertise required to correctly implant and operate these systems greatly limits their use beyond a few clinical cases. A noninvasive counterpart requiring less intervention that can provide high-quality control would profoundly improve the integration of BCIs into the clinical and home setting. Here, we present and validate a noninvasive framework using electroencephalography (EEG) to achieve the neural control of a robotic device for continuous random target tracking. This framework addresses and improves upon both the “brain” and “computer” components by increasing, respectively, user engagement through a continuous pursuit task and associated training paradigm and the spatial resolution of noninvasive neural data through EEG source imaging. In all, our unique framework enhanced BCI learning by nearly 60% for traditional center-out tasks and by more than 500% in the more realistic continuous pursuit task. We further demonstrated an additional enhancement in BCI control of almost 10% by using online noninvasive neuroimaging. Last, this framework was deployed in a physical task, demonstrating a near-seamless transition from the control of an unconstrained virtual cursor to the real-time control of a robotic arm. Such combined advances in the quality of neural decoding and the practical utility of noninvasive robotic arm control will have major implications for the eventual development and implementation of neurorobotics by means of noninvasive BCI.

In all, the framework presented here demonstrates a systematic approach to achieving continuous robotic arm control through the targeted improvement of both the user learning (“brain” component) and the machine learning (“computer” component) elements of a BCI. Specifically, using a CP task training paradigm increased BCI learning by nearly 60% for traditional DT tasks and by more than 500% in the more realistic CP task. The utility of real-time ESI further introduced a significant 10% improvement in CP BCI control for users experienced in classical sensor-based BCI. Through the integration of these improvements, we demonstrated the continuous control of a robotic arm (movies S4 to S7) at almost identical levels to that of virtual cursor control, highlighting the potential of noninvasive BCI to translate to real-world devices for practical tasks and eventual clinical applications.

Poor signal quality can further complicate the ability to decode neural events, especially when using noninvasive signals such as EEG ( 21 ). Spatial filtering has long been used to denoise noninvasive BCI signals ( 22 , 23 ) and has recently offered promise in detecting increasingly diverse realistic commands ( 10 , 24 , 25 ). Electrical source imaging (ESI) is one such approach that uses the electrical properties and geometry of the head to mitigate the effects of volume conduction and estimate cortical activity ( 26 ). Dramatic improvements in offline neural decoding have been observed when using ESI compared with traditional sensor techniques ( 24 , 27 ); however, these approaches have yet to be validated online. By developing a real-time ESI platform, we were able to isolate and evaluate neural decoding in both the sensor and the source domains without introducing the confounding online processing steps that often accompany other spatial filtering techniques (different classifiers, time windows, etc.).

BCI learning rates can vary among individuals, but it is generally thought that a user’s motivation and cognitive arousal play significant roles in the process of skill acquisition and eventual task performance ( 15 , 16 ). Although levels of internal motivation vary across populations and time ( 17 ), engaging users and maintaining attention via stimulating task paradigms may diminish these differences. Current BCI task paradigms overwhelmingly involve simple cued center-out tasks defined by discrete trials (DTs) of neural control ( 18 ). These tasks provide robust test beds for new decoding algorithms but do not account for the random perturbations that invariably occur in daily life. Continuous analogs, in which users are not bound by time-limited objectives, enable control strategies that facilitate the extension of BCI toward the realistic control of physical devices in the home and clinic ( 8 ). Here, to produce robust robotic arm control that would be useful for daily life, we developed a continuous pursuit (CP) task, in which users performed motor imagination (MI) to chase a randomly moving target (movies S1 to S3) ( 18 , 19 ). We found that CP task training produced stronger behavioral and physiological learning effects than traditional DT task training, an effect that can be credited to the Yerkes-Dodson law ( 20 ).

Detecting mental intent and controlling external devices through brain-computer interface (BCI) technology has opened doors to improving the lives of patients suffering from various neurological disorders, including amyotrophic lateral sclerosis and spinal cord injury ( 1 – 5 ). These realizations have enabled patients to communicate with attending clinicians and researchers in the laboratory by simply imagining actions of different body parts ( 6 , 7 ). Although achievable task complexity varies between invasive and noninvasive systems, BCIs in both domains have restored once-lost bodily functions that include independent ambulation ( 8 ), functional manipulations of the hands ( 3 , 4 ), and linguistic communication ( 9 , 10 ). Hence, clinical interest is rapidly building for systems that allow patients to interact with their environment through autonomous neural control ( 2 , 8 , 11 ). Nevertheless, although technology targeting the restoration or augmentation of arm and hand control is of the highest priority in the intended patient populations, electroencephalography (EEG)–based BCIs targeting such restorative interventions are some of the least effective ( 12 , 13 ). With exemplary clinical applications focusing on robotic- or orthosis-assisted hand control ( 4 ), it is paramount to improve coordinated navigation of a robotic arm, because precise positioning will be vital for the success of downstream actions ( 14 ). To meet this need, we present here a unified noninvasive framework for the continuous EEG-based two-dimensional (2D) control of a physical robotic arm.

When users were directly controlling the robotic arm, the behavior of a hidden virtual cursor was also recorded to ensure proper mapping of the arm position in physical space. Across all sessions and individuals, median squared tracking correlation values reached ρ hor 2 = 0.13 ( IQR = 0.04 − 0.32 ) and ρ ver 2 = 0.09 ( IQR = 0.03 − 0.28 ) in the horizontal and vertical dimensions, respectively, for 2D control. In transitioning between virtual cursor and robotic arm control, we observed similar MSE values among the three tracking conditions—virtual cursor, hidden cursor, and robotic arm (F 2,40 = 2.62, P = 0.086; Fig. 6D )—indicating a smooth transition from the control of a virtual object to a real-world device. This likeness in control quality was further revealed through a lack of significant difference in the squared tracking correlation (ρ 2 ) for both the horizontal (F 2,40 = 0.13, P = 0.88; Fig. 6E ) and vertical (F 2,40 = 0.77, P = 0.47; Fig. 6E ) dimensions. Tracking performance was significantly greater than the chance levels determined for all control conditions and dimensions (Mann-Whitney U test with Bonferroni correction, all P < 0.05). Overall, the notable similarity between virtual cursor control and robotic arm control highlights the possibility of integrating virtual cursor exposure into future clinical training paradigms, where patients have limited access to robotic arm training time.

( A ) Robotic arm CP BCI setup. Users controlled the 2D continuous movement of a 7–degree of freedom robotic arm to track a randomly moving target on a computer screen. ( B ) Depiction of the CP edge repulsion feature (in contrast to the edge wrapping feature; Fig. 2A ) used to accommodate the physical limitations of the robotic arm. ( C ) Group-level feature maps for the horizontal (top row) and vertical (bottom row) control dimensions projected onto a template brain. ( D ) Group-level 2D MSE for the various control conditions. Bars indicate mean + SEM. ( E ) Box-and-whisker plots for the group-level squared tracking correlation (ρ 2 ) values for the horizontal (left) and vertical (right) dimensions during 2D CP control for the various control conditions. Blue lines indicate the medians, tops and bottoms of the boxes indicate the 25th and 75 percentiles, and the top and bottom whiskers indicate the respective minimum and maximum values. Control conditions include virtual cursor (white), hidden cursor (gray), and robotic arm (black). The red dotted line indicates chance level. Statistical analysis using a repeated-measures two-way ANOVA (n = 6 per condition) with main effects of time and control condition.

Having robustly validated our proposed BCI framework in a controlled environment, we completed our study by transitioning to the applied physical source control of a robotic arm ( Fig. 6A ). Although the cursor and target wrapping allowed for more complicated control strategies and scenarios, such a feature could not exist in a real-world setting. Therefore, we implemented a modified form of the CP task in a robotic arm control paradigm, where the edge-wrapping feature was replaced with an edge repulsing feature ( Fig. 6B and movies S4 to S7). Six experienced users (8.3 ± 2.9 hours of previous BCI training) participated in five source CP BCI sessions containing both virtual cursor and robotic arm control, block-randomized across individuals and sessions. Because no paradigm was implemented to determine performance values before and after training in the modified task, participants were screened for experience and skill level beforehand (see Materials and Methods). Physiological support for user skill level was additionally observed in the group-level feature maps ( Fig. 6C ) that displayed comparable characteristics with those of other experienced users participating in this study ( Fig. 5G ).

When looking at the feature maps ( Fig. 5G ), an important dichotomy can be observed between naïve (weak, sporadic clusters) and experienced (strong, focal clusters) users for both control dimensions that parallels the trends previously observed in the modulation index topographies before (low, sporadic modulation) and after (high, focal modulation) training ( Fig. 3, B and D ). To quantify the focality/diffuseness of these features, we computed the spread of the group-level feature maps ( Fig. 5H ), defined as the average weighted distance between the feature location and the hand knob (source space) or C3/C4 electrode (sensor space) (see Materials and Methods). We observed both significant or near-significant reductions in the feature spread for experienced users, compared with naïve users, in both the horizontal (Mann-Whitney U test with Bonferroni correction: source, P < 0.005; sensor, P < 0.05) and the vertical (Mann-Whitney U test with Bonferroni correction: source, P < 0.005; sensor, P = 0.22) control dimensions. This physiological difference between naïve and experienced users is in line with their performance difference (MSE) and further supports the contrast in BCI proficiency among the two groups and the overarching effect of source-based control depending on user skill level.

Naïve users also demonstrated an overall improvement in online performance with source control, although this improvement did not reach significance for 2D control (F 1,12 = 3.02, P = 0.11; Fig. 5C ). Nevertheless, the effect size for the performance difference was notably similar to that of experienced users ( Fig. 5, A and C , and table S1), indicating an improvement of similar magnitude. As expected, the sensor and source control MSE values for the naïve users were comparable with those of the CP training group before training (baseline, also naïve). This consistency, independent of skill level, highlights a robust positive influence of source control on online performance. Furthermore, the squared error histograms ( Fig. 5D ) and extent threshold measures for naïve users ( Fig. 5E ) displayed analogous trends to those of experienced users; however, these did not reach significance (F 1,12 = 2.02, P = 0.18; Fig. 5F ).

( A and B ) Experienced user performance (n = 16). (A) Group-level MSE for source and sensor 2D CP cursor control. Light and dark gray blocks represent performance for the CP training group (n = 11; Fig. 2D ) before (naïve) and after training (experienced). The effect size, ∣r∣, is indicated under the pair of bars. (B) Group-level squared-error histograms for 2D CP sensor and source cursor control. ( C and D ) Naïve user performance (n = 13). Same as (A) and (B) for naïve user data. ( E ) Scale drawing of the CP paradigm workspace displaying the spatial threshold derived from experienced (yellow) and naïve (green) user data (fig. S6). ( F ) Cursor dwell time within the spatial threshold for experienced (left) and naïve (right) users. ( G ) Group-level feature maps for horizontal (top) and vertical (bottom) cursor control for naïve (right) and experienced (left) users. User-specific features were projected onto a template brain for group averaging. ( H ) Feature spread analysis between experienced and naïve users for source (left) and sensor (right) features for horizontal (top) and vertical (bottom) control. Bars indicate mean + SEM. Statistical analysis using a one- (C and D) or two-way repeated-measures (A and B) ANOVA with main effects of decoding domain, and time and decoding domain, respectively. Main effect of decoding domain: ### P < 0.005 (A, C, and F), gray bar; P < 0.05 uncorrected, red bar; P < 0.05, FDR corrected (B and D). Mann-Whitney U test with Bonferroni correction for multiple comparisons (H): + P < 0.05, +++ P < 0.005.

For experienced users, source control improved performance over that of conventional sensor control, producing a significant reduction in the 2D MSE (F 1,69 = 9.83, P < 0.01; Fig. 5A ). Unsurprisingly, the sensor and source MSE values clustered near those of the CP training group after training (evaluation), reinforcing their skilled state. The spatial extent of the observed improvement in the CP task was characterized through squared error histograms ( Fig. 5B ), with source values shifting toward smaller errors and sensor values shifting toward larger errors. By fitting gamma functions to these histograms, we derived a quantitative threshold, independent of cursor/target size, for statistically testing the spatial extent of the performance difference (fig. S6). Experienced users dwelt within this defined region, a disc with a diameter of 16.67% of the workspace width centered on the target ( Fig. 5E ), for significantly more time during source control than sensor control (F 1,69 = 20.96, P < 0.005; Fig. 5F ).

To thoroughly investigate the effects of source control (real-time ESI) on CP task performance (and potential future benefits for robotic arm control), we performed within-session comparisons of source and sensor virtual cursor control on users in stable skill states. The CP task was chosen for further analysis because it is more applicable to robotic arm control than the DT task and displayed both increased difficulty and skill acquisition. Our investigation included both extremes of the BCI skill spectrum; experienced users (12.8 ± 8.9 hours of previous BCI training, n = 16) participated in up to three sessions, and naïve users (no previous BCI training, n = 13) participated in a single session (to avoid confounding effects of early learning in more than one session). User- and session-specific inverse models were also used for these participants.

Feature selection in the source domain identified distinct cortical clusters, optimized through anatomical and functional constraints, for online control and was performed on a session-by-session basis (see Materials and Methods). As expected, sCP training feature maps highlighted hand cortical regions for both control dimensions throughout training ( Fig. 4D ). It should be noted that the baseline and evaluation sessions for the sCP group were completed in the sensor domain to maintain consistent conditions with the other training groups. Although training duration was fixed at eight sessions with no intermediary testing, further investigation at different stages of learning may help pinpoint when source-based decoding may benefit BCI skill acquisition.

( A and B ) 2D BCI performance for the CP (A) and DT (B) task at baseline and evaluation for the CP and source CP (sCP) training groups. The red dotted line indicates chance level. The effect size, ∣r∣, is indicated under each pair of bars. ( C ) Task learning for the CP (left) and DT (right) tasks. Bars indicate mean + SEM. Statistical analysis using a one- (C) or two-way repeated-measures (A and B) ANOVA (n = 11 per group) with main effects of training decoding domain, and time and training decoding domain, respectively. Main effect of time: # P < 0.05, ### P < 0.005. Tukey’s HSD post hoc test: *P < 0.05, ***P < 0.005. n.s., not significant. ( D ) Group-level training feature maps for the training groups for horizontal (top) and vertical (bottom) cursor control. User-specific features were projected onto a template brain for group averaging.

Whereas the CP task allowed us to target user learning and progress toward the robust online control of a robotic arm, we additionally wanted to address the machine learning element. To evaluate whether real-time ESI-based decoding improved performance throughout training, we recruited an additional group of BCI naïve individuals (n = 11) for CP training using source neurofeedback (source control, sCP). This sCP group was baseline-matched to the previous CP (and DT) group (sensor control) (fig. S5A). For source control, we implemented user- and session-specific inverse models into the online decoding pipeline for the CP task. Similar to the CP group, the sCP group significantly improved in both the 2D CP (Tukey’s HSD post hoc test, P < 0.05; Fig. 4A , right bars) and the 2D DT tasks (Tukey’s HSD post hoc test, P < 0.05; Fig. 4B , right bars) after training. Accordingly, very similar learning effects were observed for both tasks in the CP and sCP groups ( Fig. 4C ). The final performance and learning rates were consistent between the two training groups (CP and sCP), supporting the groups’ shared familiar and unfamiliar task proficiency.

( A and B ) Left versus right MI task analysis. (A) Maximum sensorimotor R 2 value for the CP and DT training groups for horizontal control task. The effect size, ∣r∣, is indicated under each pair of bars. (B) R 2 topographies at baseline (top row) and evaluation (bottom row) for the CP and DT training groups for horizontal control tasks. ( C and D ) Both hands versus rest MI task analysis. Same as (A) and (B) for vertical control tasks. ( E and F ) Statistical topographies indicating electrodes that displayed a significant increase in R 2 values for the horizontal (E) and vertical (F) control tasks. The electrode map in the middle provides a reference for the electrodes shown. Bar graphs below each topography provide a count for the number of electrodes meeting the various significance thresholds. Bars indicate mean + SEM. Statistical analysis using a one- (E and F) or two-way repeated-measures (A and C) ANOVA (n = 11 per group) with main effects of time [blue, P < 0.05; green, P < 0.01; yellow, P < 0.005; red outline, P < 0.05, false discovery rate (FDR) corrected] and time ( # P < 0.05, ### P < 0.005) and training task, respectively. Tukey’s HSD post hoc test: *P < 0.05.

Whereas BCI feedback plays a significant role in facilitating sensorimotor rhythm modulation ( 33 ), MI without feedback can provide a measure of a user’s natural ability to produce the associated discriminative EEG patterns. Left-hand MI versus right-hand MI (left versus right) and both hands MI versus rest (up versus down) runs were analyzed individually. An index of modulation between any two mental states is represented as the regression output (R 2 ) between the EEG alpha power and the task labels (see Materials and Methods). Only the 57 sensorimotor electrodes used for online control were included in this analysis. Although sensorimotor modulation significantly increased for both task pairs from baseline to evaluation (horizontal: F 1,20 = 4.70, P < 0.05; vertical: F 1,20 = 21.01, P < 0.005; Fig. 3, A and C ), the spatial distribution of these improvements is more meaningful in evaluating the effectiveness of BCI training. Except for mild baseline modulation in the DT group, no strong patterns were apparent for either task pair before training. For the horizontal dimension at the evaluation session, the CP group produced highly focal bilateral modulation patterns, whereas more global modulation was observed for the DT group ( Fig. 3B ). Evaluation topographies were more consistent between the two training groups for the vertical dimension ( Fig. 3D ). Electrodes displaying a significant improvement in modulation were far more numerous for the CP group than for the DT group for both horizontal (CP, 12; DT, 3; Fig. 3E ) and vertical (CP, 37; DT, 13; Fig. 3F ) tasks. Furthermore, these significant electrodes clustered far closer to scalp regions covering the approximate hand cortical regions (e.g., C3-4 and CP3-4) in the CP group. These localized changes provide compelling evidence that the enhanced behavioral improvement seen in the CP training group was accompanied by consistent physiological changes in sensorimotor modulation (R 2 values) ( 34 ).

To delineate the underlying physiology of these training differences, we investigated user engagement during both tasks by quantifying eye blink activity. Decreased blink activity has been implicated in heightened attentional processes and cognitive arousal during various tasks ( 32 ). These mental states can dramatically influence task training and performance; whereas stimulating tasks can facilitate skill acquisition, boring or frustrating tasks can inhibit performance ( 20 ). The eye blink component of the EEG was extracted during the baseline and evaluation sessions using independent component analysis (ICA) ( Fig. 2G and fig. S4). Across all participants, blink activity was strongly dampened at the baseline (F 1,63 = 9.84, P < 0.005; Fig. 2G ), suggesting heightened attention that was likely due to the novelty of BCI in general. Increased blink activity at the evaluation supports user skill acquisition, because less attention was required for improved performance. The large reduction in blink activity observed during the CP task, compared with the DT task (F 1,63 = 3.51, P = 0.066; Fig. 2G ), suggests that the CP task elicited heightened user engagement during active control, a feature that may explain the more dramatic positive training effects.

Because the two tasks varied greatly in control dynamics, it was difficult to draw comparisons between these differences. Therefore, in addition to statistical testing, we also examined the effect size (point biserial correlation, see Materials and Methods), a measure, unconfounded by sample size, of the magnitude of the difference within each performance metric between baseline and evaluation sessions. Compared with the DT group, the effect sizes were far larger for the CP group for both tasks ( Fig. 2, D and E ), displaying a 500% learning improvement in the CP task and a nearly 60% learning improvement in the DT task ( Fig. 2F ).

DT task performance was measured in terms of percent valid correct (PVC), computed as the number of hit trials divided by the total number of trials, in which a final decision was made (valid trials). The corresponding CP task performance metric was mean squared error (MSE), i.e., the average normalized squared error between the target and cursor location over the course of a single run. Across these 22 participants, the results of a repeated-measures two-way analysis of variance (ANOVA) revealed a significant main effect of time for both the CP MSE (F 1,20 = 7.39, P < 0.05; Fig. 2D ) and DT PVC (F 1,20 = 19.80, P < 0.005; Fig. 2E ) metrics. To examine skill generalizability, we specifically considered the effects of training on the performance of familiar and unfamiliar tasks. Individuals trained with the CP task significantly improved in the same task after training [Tukey’s honestly significant difference (HSD) post hoc test, P < 0.05; Fig. 2D , left bars], whereas those trained with the DT task did not (Tukey’s HSD post hoc test, P = 0.14; Fig. 2E , right bars). Previous work has indicated that DT task training can lead to strong learning effects ( 31 ); however, some users have required nearly 70 training sessions to do so ( 18 ). When considering unfamiliar tasks, the DT training group only modestly improved in the CP task after training (Tukey’s HSD post hoc test, P = 0.96; Fig. 2D , right bars), whereas the CP training group displayed a significant improvement in the DT task (Tukey’s HSD post hoc test, P < 0.005; Fig. 2E , left bars).

Electrodes used for online control were optimized on a session-by-session basis (see Materials and Methods), chosen from a set of 57 sensors covering the sensorimotor regions. Electrodes were identified for the horizontal and vertical control dimensions independently using the corresponding right-hand MI versus left-hand MI and both hands MI versus rest datasets. Throughout training, the two groups derived nearly identical feature (electrode) maps in the sensor domain containing focal bilateral scalp clusters overlying the cortical hand regions ( Fig. 2C ). These clusters were located and weighted in accordance with the underlying event-related (de)synchronization generated during the corresponding MI tasks ( 19 ) and are similar to those used in other noninvasive cursor control studies, identified through either data-driven ( 30 ) or manual ( 28 ) selection processes.

We investigated the utility of using the CP task for BCI skill acquisition in a pre-post study design by comparing BCI performance between populations trained by either CP or DT task. Twenty-two individuals participated in a baseline session, eight training sessions, and an evaluation session. Baseline and evaluation sessions contained both DT and CP tasks (and MI without feedback), whereas training sessions contained only one task type, consistent throughout training according to each user’s assigned group (DT or CP, n = 11 per group; see Materials and Methods). All sessions for both groups used scalp sensor information. 1D horizontal DT performance was used to baseline match the two groups (fig. S3A).

( A ) Depiction of the CP edge wrapping feature. ( B ) Tracking trajectory during an example 2D CP trial. ( C ) Training feature maps for the DT and CP training groups for horizontal (top) and vertical (bottom) cursor control. ρ 2 , squared correlation coefficient. ( D and E ) 2D BCI performance for the CP (D) and DT (E) task at baseline and evaluation for the CP and DT training groups. The red dotted line indicates chance level. The effect size, ∣r∣, is indicated under each pair of bars. ( F ) Task learning for the CP (top) and DT (bottom) tasks. ( G ) Eye blink EEG component scalp topography (top) and activity (bottom left) at baseline and evaluation, and activity during each task (CP versus DT) (bottom right). Bars indicate mean + SEM. Statistical analysis using a one- (F) or two-way repeated-measures (D, E, and G) ANOVA (n = 11 per group) with main effects of task, and time and task, respectively. Main effect of time: # P < 0.05, ### P < 0.005. Tukey’s HSD post hoc test: *P < 0.05, ***P < 0.005.

Throughout all experimental sessions, users were instructed to control the trajectory of a virtual cursor using MI tasks; left- and right-hand MI for the corresponding left and right movement, and both hands MI and rest for up and down movement, respectively. These tasks were chosen on the basis of previous cursor control ( 28 ) and neurophysiological ( 19 ) exploration. Horizontal and vertical cursor movements were controlled independently. CP trials lasted 60 s each and required users to track a randomly moving target within a square workspace ( Fig. 2, A and B ; fig. S1, A and B; and movies S1 to S3). Previous implementations of similar tasks used technician-controlled (manual) target trajectories, which can introduce inconsistencies and biases during tracking ( 29 ). To avoid such scenarios, we governed target trajectories in the current work by a Gaussian random process (see Materials and Methods). Nevertheless, it is possible for such a random process to drive the target toward stagnation at an edge/corner, which could synthetically distort performance. Therefore, to better estimate the difference between DT and CP task training, and contrary to previous work ( 18 , 29 ), our initial CP task allowed the cursor and target to fluidly wrap from one side of the workspace to the other (top to bottom, left to right, and vice versa) upon crossing an edge ( Fig. 2, A and B ; fig. S1, A and B; and movies S1 to S3). Trajectories from experienced users were unwrapped (fig. S1C) to reveal squared tracking correlations of ρ hor 2 = 0.48 ± 0.20 and ρ ver 2 = 0.47 ± 0.19 (fig. S2).

The proposed framework addressed both user and machine learning aspects of BCI technology before being implemented in the control of a realistic robotic device. User learning was addressed by investigating the behavioral and physiological effects of BCI training using sensor-level neurofeedback with a traditional DT center-out task (n = 11) and a more realistic CP task (n = 11) (top left). The effects of BCI training were further tested in the CP task using source-level neurofeedback (n = 11) obtained through online ESI with user-specific anatomical models (center). This design allowed us to determine both the optimal task and neurofeedback domain for BCI skill acquisition. The machine learning aspect was further examined across the skill spectrum by testing the effects of source-level neurofeedback, compared with sensor-level neurofeedback, in naïve (n = 13) and experienced (n = 16) users in a randomized single-blinded design (top right). The user and machine learning components of the proposed framework were then combined to achieve real-time continuous source-based control of a robotic arm (n = 6) (bottom). Comparing BCI performance of robotic arm and virtual cursor control demonstrated the ease of translating neural control of a virtual object to a realistic assistive device useful for clinical applications.

Before addressing whether our online ESI-based decoding strategy could be used for the continuous control of a robotic arm, the CP task and source signal approaches needed to be thoroughly validated as useful training and control strategies, respectively ( Fig. 1 ). Thirty-three individuals naïve to BCI participated in a virtual cursor BCI learning phase. The training length was set at 10 sessions to facilitate practical data acquisition and to establish a threshold for future training applications. These 33 users were split into three groups: sensor domain CP training (CP), sensor domain DT training (DT), and source domain CP training (using real-time ESI; sCP). This design allowed us to answer (i) which training task (CP versus DT) and (ii) which neurofeedback domain (source versus sensor control) led to more effective BCI skill acquisition (see Materials and Methods for details on participant demographics and baseline group metrics). The within-session effects of source versus sensor control (virtual cursor) on CP BCI performance were tested on 29 individuals, 16 with previous BCI experience (sensor control) and 13 naïve to BCI. Furthermore, six individuals with BCI experience (sensor DT cursor control) participated in experiments designed to compare performances between virtual cursor and robotic arm control in a physically constrained variation of the CP task.

DISCUSSION

The research presented here describes an encompassing approach aimed at driving noninvasive neural control toward the realistic daily use of a robotic device. We have demonstrated that the CP BCI paradigm not only can be used to successfully gauge a user’s BCI proficiency but also can serve as a more effective training tool than traditional center-out DT tasks, accelerating the acquisition of neural cursor control and driving the associated physiological changes. Contrary to users trained with the DT task, those trained with the CP task displayed significant performance improvements in familiar and unfamiliar tasks (Fig. 2, D to F), demonstrating highly flexible skill acquisition. These results were further supported in a third group that also trained with the CP task (Fig. 4, A to C). Participants in this group (sCP) displayed nearly identical learning effects as the original sensor CP group while training with source control, providing confidence for the reproducibility of the effects of CP task training.

As training progressed, it became apparent that the strategies developed by users differed significantly, depending on the training task. For example, various individuals in the DT training group reported using strategies involving selectively attending to their hand(s) through peripheral vision without necessarily focusing on the cursor position. Although such strategies were effective for DT tasks, users using them often struggled with the CP tasks in the evaluation session, because the moving target and cursor required constant visual attention and adjustment of motor-related mental intent. In this sense, many of these users somewhat ignored the feedback when training with the DT task and treated it similarly to the MI without feedback, reducing its effectiveness (33). The lower success of such strategies manifested within the MI EEG of the DT group as sporadic patterns of modulation after training (Fig. 3D), which are also consistent with the lower levels of cognitive arousal observed during the traditional DT task compared with the CP task (Fig. 2G). We believe that the target dynamics and screen wrapping feature of the CP task (Fig. 2A) likely perturb fluid target tracking and require heightened attention during cursor control. These conclusions support the overarching concept of integrating human factors, such as virtual reality techniques (34, 35), into cognitive-based training tools for improving both user engagement and task performance (20, 36–38) and should be considered in future generations of BCIs.

Seminal works implementing similar continuous tracking tasks using invasively acquired signals reported comparable squared tracking correlation values over a decade ago (29). Although the field of invasive neural decoding has surpassed these benchmark results to include high degree-of-freedom and anthropomorphically functional tasks (3, 39, 40), qualitative similarities can be seen between these two modalities. In accordance with invasive reports, users in our study struggled to keep the cursor in a single location, often exhibiting oscillatory tracking behavior around the target (Fig. 2B and fig. S1, A and B). Although these actions demonstrate directed cursor trajectories toward the target and highlight the ability of our system to accurately capture the users’ dynamic mental intent, the tracking correlation is effectively reduced and may benefit from more advanced decoding methods.

It has been argued that motor neurons encode cursor velocity during neural cursor control (41), with numerous decoding algorithms using such properties to drastically improve user performance over classical techniques (38). In particular, modeling neuronal behavior as a dynamical system has recently yielded significantly improved online decoding results (42) and may provide even more complex and efficient device control in upcoming invasive and noninvasive work. This decoding strategy would be particularly attractive to neural control in the CP task presented here, given the clear analog of our control output to under-dampened control dynamics. Although this information would be valuable to reduce or eliminate the previously described cursor oscillations, it has yet to be observed whether these details can be detected via scalp recordings. Nevertheless, noninvasive neural signals have recently been shown to contain information encoded on the spatial scale of cortical columns (submillimeter), indicating the ability to decode neural activity with very fine spatial-temporal resolution from outside the skull (43).

Over the past few decades, the reconstruction of cortical activity through ESI has exemplified the push to increase the spatial specificity of noninvasive recordings and has been shown to provide superior neural decoding when compared with scalp sensor information (23, 24, 27). Similar to these previous works, we found that, in general, source features were more correlated with cued motor-related mental states than sensor features (fig. S9) (44, 45). Furthermore, in closed-loop CP BCI control, we found that the inclusion of online ESI improved performance in naïve and experienced users, consistent with offline enhancements (Fig. 5 and figs. S7 to S9). The increased task-specific source modulation indicates a higher sensitivity for detecting changes in a user’s motor-related mental state and is likely a product of the principles of ESI and its use in modeling and counteracting volume conduction. CP cursor control requires highly dynamic cognitive processes to recognize and correct for the random and sudden changes in the target’s trajectory during tracking. We therefore hypothesize that the fast, real-time control required during the CP paradigm takes advantage of the heightened sensitivity of ESI modulation, allowing for quicker responses that more accurately resemble the dynamics involved in the CP task. This phenomenon was apparent during the within-session comparisons of source and sensor control (Fig. 5 and figs. S7 and S8); however, it is possible that, with sufficient training, the feedback domain becomes less important for skill acquisition (Fig. 4).

We feel that it is necessary to acknowledge the decline in performance that occurred between the original CP task and the modified CP task, which we believe to be strongly attributed to the task modifications made for the physical constraints of the robotic arm. The presence of the physical robotic device inherently creates a more distracting environment for neural control compared with that of a virtual cursor. We found that, with the robotic arm mounted on the right side of the user (Figs. 1, bottom, and 6A), visual obstruction of the target was common when the arm was directed to reach across the user to the left side of the screen, often perturbing target tracking. In addition, although participants here displayed previous BCI proficiency, they had less experience than those participating in the original CP task validation. We believe that this combination of reduced user experience and enhanced sensory loading caused by the more complex human-device interaction involving the robotic arm led to a reduction in performance compared with the highly controlled virtual cursor control environment.

The results presented here demonstrate that CP control provides a unique opportunity for the complex control of a virtual cursor and robotic device (14, 39), without requiring discretized, prolonged task sequences (46) that can make even simple task completion long and frustrating. Users were able to smoothly transition between virtual cursor and robotic arm control with minimal changes in performance (Fig. 6, D and E), indicating the potential ease of integrating such a noninvasive assistive tool into clinical applications for autonomous use in daily life. Invasive systems have already demonstrated a level of control similar to such a noninvasive hypothetical; however, although such invasive approaches may offer much-needed help to a restricted number of patients with severe physical dysfunctions, most impaired persons will likely not qualify for participation due to both medical and financial limitations. Additionally, previous work has suggested that accessing sufficiently large patient populations for concrete and statistically significant conclusions may be difficult to obtain (1–5, 8, 11, 29, 39). Therefore, there is a strong need to further develop noninvasive BCI technology so that it can benefit most patients and even the general population in the future. The effective training paradigm and additional ESI-based performance improvement demonstrated here, as well as the integration of such targeted enhancements toward robotic arm control, offer increasing confidence that noninvasive BCIs may be able to expand to widespread clinical investigation. We observed that, for robotic arm control, generic head models, rather than those derived from user-specific magnetic resonance imaging (MRI), were sufficient for high-quality performance (see Materials and Methods). Therefore, in all, the work presented in this paper is necessary for current EEG-based BCI paradigms to achieve useful and effective noninvasive robotic device control, and its results are pertinent in directing both ongoing and future studies.