Experimental Setup

Subjects

18 healthy human subjects were recruited for the present study. Five of them dropped within the first three sessions due to their scheduling conflicts; the remaining 13 subjects (seven women; mean age, 27.3 y; range, 18–54) participated in 8–15 experiment sessions within an average of 81 ± 34 days. Each session took place on a different day, with one day to one week between sessions except for a few extreme cases. The length of duration was mainly due to scheduling need instead of the duration to learn the BCI skills. Data for these 13 subjects were analyzed for this study. All procedures and protocols were approved by the Institutional Review Board of the University of Minnesota. Informed consents were obtained from all of the subjects before they participated in the experiment. One of the subjects was left handed and all others were right handed. Eligibility and screening form is used to recruit healthy subjects within the age of 18–64 without traumatic brain injury or brain lesion and without any history of neurological deficit or neurodegenerative disorder in this study. The exclusion criteria were established before any subject was recruited to participate in the experiment. This study is registered with ClinicalTrials.gov, NCT02069938 (additional information about the clinical trial is available at https://clinicaltrials.gov/ct2/show/NCT02069938). All methods and research activities were performed in accordance with the guidelines and regulations.

Data Acquisition

EEG data were recorded by a 64-channel Neuroscan cap with SynAmps RT headbox and SynAmps RT amplifier (Neuroscan Inc, Charlotte, NC). The reference electrode was located on the vertex and the ground electrode was on the forehead. During the recording, the participants were seated in a comfortable chair and rested their hands on armrests. Each subject sat in front of a computer monitor at a distance of 90 cm. The robotic arm was mounted 50 cm to the right of the subject. All electrode impedances were maintained below 5 kΩ. The EEG signals were sampled at a rate of 1,000 Hz and bandpass-filtered in the range of 0.5–200 Hz. A notch filter of 60 Hz was applied to the raw EEG signals. A JACO arm (Kinova Robotics, Montreal, Canada), a seven DOF human-like robotic arm with three fingers, served as the BCI actuator and means of visual feedback for the subjects during the experiments. A Microsoft Kinect Motion Sensor was used to locate and send the position of a target to the computer. For the session of stage five, blocks were moved from table in front of the subjects to a three layer shelf of 5 inches × 19 inches × 19 inches.

Task design for Brain-control Task

In the first stage, virtual cursor only, subjects were asked to complete only virtual cursor tasks for initial learning purposes. There were one to four sessions in this stage, depending on each subject’s performance. Each session consisted of four or five runs of one dimensional (1D) left vs. right cursor movement (1D,LR), four or five runs of 1D up vs. down (1D,UD) and four runs of two dimensional (2D) voluntary cursor movement in a plane if their 1D performance of (1D,LR) and (1D,UD) exceeded 80% accuracy on average in three consecutive runs for one session. Each run contained 25 trials of motor imagery tasks. The exact number of sessions and runs depended on the subject’s individual ability and availability. The subjects were instructed to imagine their left hand moving, their right hand moving, both hands moving or both hand relaxing to control left, right, up and down cursor movement, respectively (Fig. 1b).

In the second stage, four-target grasp, the subjects performed robotic arm control while a cursor was simultaneously moving on a computer monitor. Four foam blocks (size 4.5 cm × 5 cm × 10 cm) were placed in fixed positions on a flat table corresponding to the four target positions on the monitor. The hand of the robotic arm was controlled in end-effector velocity space. We defined a square workspace (size 32 cm × 32 cm) on the table to restrict where the blocks could be placed. The arm was also confined within this boundary to avoid collisions with the tabletop and participant. Each participant was required to complete two sessions of the four-target grasp on separate days. There were three runs of the 1D, LR control task, followed by three runs of the 1D,UD target grasping task and four runs of the four-target grasp task in each session. For the robotic arm to grasp an object on the table, a two-step task sequence of reach-and-grasp was employed to facilitate the participants’ ability to reach and grasp a block in 3D space. In the first step, a target object location was indicated to the subject on the monitor display. At this point, the subject would attempt to move the robotic arm within a horizontal plane to approach the center of the block at 17 cm above the block by performing the same 2D motor imagination tasks as for virtual cursor control. During the movement of the robotic arm, there was simultaneous cursor movement on the monitor. This was to represent the robotic arm’s hand position on the screen to make it clear that the physical target was approached and reached successfully; that is, the virtual cursor hit the target and changed colors from pink to yellow if the arm moved to the indicated block. A “hover area” was defined by a virtual cylindrical region centered above the target block with a radius of 3 cm. If the robotic arm maintained its position within the hover area for 2 seconds the trial was considered a successful hit and the task progressed to the second step of the sequence. In this step the subject was presented with a 1D UD robotic arm task to reach and grasp the block. Similar to the first step, if the arm was lowered down to the target and remained there within 2 cm of the center of the block for 2 seconds, the hand of the robotic arm would automatically grasp the target. Each subject participated in two sessions of this paradigm. The operators stood by and monitored the participant. The operators also placed or replaced the blocks as needed. The physical workspace containing the blocks was an area of 32 cm from left to right, 45 cm in depth, and 32 cm from front to back. The fingertips of the robotic arm returned to the center of the workspace at the beginning of each run.

In the third stage, five-target grasp, an additional block was placed in the center of the workspace, and was surrounded by the other four targets. This stage included the same reach-and-grasp sequence as stage 2. This stage was repeated for three sessions for each subject. Sessions were composed of one run of the 1D,LR cursor control task and one run of the 1D,UD cursor control task, followed by two runs of the 2D cursor control task and six runs of the five-target grasp task which is similar to four-target grasp task but with five blocks to grasp.

In the fourth stage, random-target grasp, the block was randomly placed in the pre-defined workspace by the operator. The participant was instructed to pick up the randomly positioned block using the same reach-and-grasp sequence as stages 2 and 3. The position of the randomly placed block was equally distributed among the four quadrants of the square workspace. The order of quadrant selection for block placement was randomly assigned. Once the subject successfully completed the reach-and-grasp task for the block, it would then be placed at another random location by the operator. If hovering in any place other than the target region for 2 seconds it would not proceed to the next step until reaching the maximum feedback duration (12 s) of the trial. For this stage, each subject was required to perform three sessions on separate days and sessions contained one run of both the 1D,LR and 1D,UD cursor control task, followed by three runs of the 2D cursor control task and five runs of the random-target grasp task.

In the fifth stage, shelf-target grasp, three blocks were placed on the table with fixed positions. Those positions were changed in each run. The participant was instructed to pick up the identified block and place it at a designated position within a three-layered shelf. The robotic arm started movement from the center of the cubic workspace and first move across the horizontal plane parallel to the table to select the block which was to be grasped. After the arm hovered above the specified block within the hover area for 2 seconds, the arm locked on the target and was then to move downward to grasp the target in the next step. These two steps were similar to the previous reach-and-grasp sequence. If the robotic arm grasped the target successfully it returned to the center, otherwise a new block location was selected and the procedure repeated until the subject successfully grasped a block from the table. When the subject successfully grasped a block on the table, the robotic arm moved back to the center and prepared for the vertical movement across the vertical plane parallel to the shelf. If the subject hovered over the specified position of the shelf for 2 seconds, the subject would be able to move forward and drop the block in the following step. This procedure also repeated until the subject successfully chose a position on the shelf and moved forward to drop the block at the specific position of the shelf. These two steps were named reach-and-release. In order to move a block successfully, the participants had to finish each of the four sequential steps correctly; otherwise they had to start from the beginning of the sequence of reach-and-grasp or reach-and-release. If the subject successfully placed a block onto the shelf, the empty space was filled with a new block (Paradigm ➎ in Fig. 1c). The positions for the targets in each run were fixed but varied across runs (see Supplementary Figure 6 for details). This paradigm was repeated during three sessions on separate days. Sessions contained one run of both 1D,LR and 1D,UD cursor control tasks, followed by three runs of the 2D cursor control task and five runs of the shelf-target grasp task.

Besides these five experiment stages, six of the subjects performed four extra sessions of the shelf-target task with decreased time periods between the different steps of the sequence. This additional stage was termed the fast-shelf-target task. The robotic arm was set to move with a constant speed of 8 cm/s for all of experiments in the above five stages. To test whether the subjects could operate the robotic arm with a higher speed by BCI control, the robotic arm was allowed to move with a maximum speed of 20 cm/s in the additional session of fast-shelf-target grasp task. In the fast-shelf-target grasp task session, the same task as in the shelf-target grasp task was repeated with a different parameter setting, where the prefeedback, feedback, postfeedback, and inter-trial interval duration were all decreased to about two thirds of the previous settings.

Finally, to further test the applicability of moving a block solely by the robotic arm in the absence of a cursor on the monitor, we repeated the four-target and five-target grasp tasks by removing the visual feedback of cursor movement from the subjects’ view. The subjects only received feedback from the movement of the robotic arm. These six subjects performed three sessions of the four-target and five-target grasp tasks in absence of the virtual cursor. Each of these sessions contained four runs of the four-target and two runs of five-target grasp tasks.

For each trial, there was an inter-trial interval that consisted of a black screen. This was followed by a “prefeedback” period indicating which target/block should be picked up, identified by a rectangular yellow bar on the monitor. After this was the “feedback” period in which the robotic arm moved according to the subject’s motor imagination toward the center of the specified block. The block was selected if the robotic arm remained in the hover area for 2 seconds (1 second for the fast-shelf-target grasp task). There was a maximum feedback duration (12 s) to let each trial end properly if the subject could neither hit or miss. Finally, if the target was selected, the hand of the robotic arm automatically opened or closed its fingers during the “postfeedback” period and was prepared for the next step’s grasping or releasing (Fig. 1d). During the fast-shelf-target task, the aforementioned segments of each trial were shortened to increase the speed of task progression and overall grasp sequence completion.

Software and algorithm

BCI200036 was used to control the movement of a virtual cursor and also to display the targets that indicated where the cursor should be moved to and which block should be selected. A custom C++ based program was used to control the movement of the robotic arm to track the position of the cursor. We acquired 62 channels of EEG signals; EEG channels C3 and C4 and surrounding channels over left and right motor cortex are utilized for online control. EEG activity from the controlling channels were spatially filtered by a small Laplacian filter37 and then fed into an autoregressive (AR) model to extract the power spectra features. The power activity in the upper mu frequency band over the left and right hemispheres were linearly mapped to the velocity of the virtual cursor or position of the robotic arm.

Statistical analysis

Nonparametric statistical test, i.e. Wilcoxon signed-rank test is applied throughout the analysis of the results. Because the sample size in this study is relatively small, nonparametric statistical test is more appropriate. All of the significance tests are two sided and reported with the significance level of α = 0.05.

Brain rhythm online calculation

The subjects learned to modulate their sensorimotor rhythm amplitude in the upper mu (10–14 Hz) frequency band over the left and right sensorimotor cortex to move the cursor and the robotic arm in one or two dimensions. An autoregressive (AR) model, as shown in Eq. (1), was used to estimate the amplitudes of sensorimotor rhythm:

where y t is the estimated signal at time t, w i is the weight coefficient and is the error of estimation. In the current study we applied the 16th order AR model with a window length of 400 ms to calculate the online amplitude of mu rhythmic activity. The weight coefficients of which were estimated by the least-squares criteria.

ERD/ERS quantification

The event related desynchronization (ERD) and event related synchronization (ERS) are brain oscillatory activity in diverse frequency bands. In this study we focus on the mu rhythmic activity which is modulated during participants’ motor imagination. The mu brain oscillatory activity during the experiment was externally paced by the appearance and disappearance of targets, or in other words, was time locked to the trial events. There are several methods to calculate the ERD/ERS time courses. In this paper, we used a bootstrap-based method38 to show a time-frequency map with significant changes of ERD or ERS for specific electrodes. In general, the calculation of ERD/ERS is performed by bandpass filtering the EEG signals, segmenting individual trials, detrending the trials, squaring the samples and subsequently averaging over trials and sample points. The procedures can be expressed as the following steps:

where N is the total number of trials, is the jth sample of the ith trial of the bandpass filtered EEG signals and is the mean of the j th sample averaged over all trials. R is the average power in the reference period [r 0 , r 0 + K], r 0 is the starting time point of the reference period and K is the number of samples in the baseline reference period.

The above calculation provides the ERD/ERS values for each time point and each frequency bin of interest. In order to show those significant changes of ERD/ERS activity, we utilized the bootstrap resampling technique. This procedure is realized in the Biosig toolbox39.

Data availability

All relevant data within the paper which are de-identified is available online at http://dx.doi.org/10.5061/dryad.nh109.