Research ArticleHUMAN-ROBOT INTERACTION

Noninvasive neuroimaging enhances continuous neural tracking for robotic device control

See allHide authors and affiliations

Science Robotics  19 Jun 2019:
Vol. 4, Issue 31, eaaw6844
DOI: 10.1126/scirobotics.aaw6844

Abstract

Brain-computer interfaces (BCIs) using signals acquired with intracortical implants have achieved successful high-dimensional robotic device control useful for completing daily tasks. However, the substantial amount of medical and surgical expertise required to correctly implant and operate these systems greatly limits their use beyond a few clinical cases. A noninvasive counterpart requiring less intervention that can provide high-quality control would profoundly improve the integration of BCIs into the clinical and home setting. Here, we present and validate a noninvasive framework using electroencephalography (EEG) to achieve the neural control of a robotic device for continuous random target tracking. This framework addresses and improves upon both the “brain” and “computer” components by increasing, respectively, user engagement through a continuous pursuit task and associated training paradigm and the spatial resolution of noninvasive neural data through EEG source imaging. In all, our unique framework enhanced BCI learning by nearly 60% for traditional center-out tasks and by more than 500% in the more realistic continuous pursuit task. We further demonstrated an additional enhancement in BCI control of almost 10% by using online noninvasive neuroimaging. Last, this framework was deployed in a physical task, demonstrating a near-seamless transition from the control of an unconstrained virtual cursor to the real-time control of a robotic arm. Such combined advances in the quality of neural decoding and the practical utility of noninvasive robotic arm control will have major implications for the eventual development and implementation of neurorobotics by means of noninvasive BCI.

INTRODUCTION

Detecting mental intent and controlling external devices through brain-computer interface (BCI) technology has opened doors to improving the lives of patients suffering from various neurological disorders, including amyotrophic lateral sclerosis and spinal cord injury (15). These realizations have enabled patients to communicate with attending clinicians and researchers in the laboratory by simply imagining actions of different body parts (6, 7). Although achievable task complexity varies between invasive and noninvasive systems, BCIs in both domains have restored once-lost bodily functions that include independent ambulation (8), functional manipulations of the hands (3, 4), and linguistic communication (9, 10). Hence, clinical interest is rapidly building for systems that allow patients to interact with their environment through autonomous neural control (2, 8, 11). Nevertheless, although technology targeting the restoration or augmentation of arm and hand control is of the highest priority in the intended patient populations, electroencephalography (EEG)–based BCIs targeting such restorative interventions are some of the least effective (12, 13). With exemplary clinical applications focusing on robotic- or orthosis-assisted hand control (4), it is paramount to improve coordinated navigation of a robotic arm, because precise positioning will be vital for the success of downstream actions (14). To meet this need, we present here a unified noninvasive framework for the continuous EEG-based two-dimensional (2D) control of a physical robotic arm.

BCI learning rates can vary among individuals, but it is generally thought that a user’s motivation and cognitive arousal play significant roles in the process of skill acquisition and eventual task performance (15, 16). Although levels of internal motivation vary across populations and time (17), engaging users and maintaining attention via stimulating task paradigms may diminish these differences. Current BCI task paradigms overwhelmingly involve simple cued center-out tasks defined by discrete trials (DTs) of neural control (18). These tasks provide robust test beds for new decoding algorithms but do not account for the random perturbations that invariably occur in daily life. Continuous analogs, in which users are not bound by time-limited objectives, enable control strategies that facilitate the extension of BCI toward the realistic control of physical devices in the home and clinic (8). Here, to produce robust robotic arm control that would be useful for daily life, we developed a continuous pursuit (CP) task, in which users performed motor imagination (MI) to chase a randomly moving target (movies S1 to S3) (18, 19). We found that CP task training produced stronger behavioral and physiological learning effects than traditional DT task training, an effect that can be credited to the Yerkes-Dodson law (20).

Poor signal quality can further complicate the ability to decode neural events, especially when using noninvasive signals such as EEG (21). Spatial filtering has long been used to denoise noninvasive BCI signals (22, 23) and has recently offered promise in detecting increasingly diverse realistic commands (10, 24, 25). Electrical source imaging (ESI) is one such approach that uses the electrical properties and geometry of the head to mitigate the effects of volume conduction and estimate cortical activity (26). Dramatic improvements in offline neural decoding have been observed when using ESI compared with traditional sensor techniques (24, 27); however, these approaches have yet to be validated online. By developing a real-time ESI platform, we were able to isolate and evaluate neural decoding in both the sensor and the source domains without introducing the confounding online processing steps that often accompany other spatial filtering techniques (different classifiers, time windows, etc.).

In all, the framework presented here demonstrates a systematic approach to achieving continuous robotic arm control through the targeted improvement of both the user learning (“brain” component) and the machine learning (“computer” component) elements of a BCI. Specifically, using a CP task training paradigm increased BCI learning by nearly 60% for traditional DT tasks and by more than 500% in the more realistic CP task. The utility of real-time ESI further introduced a significant 10% improvement in CP BCI control for users experienced in classical sensor-based BCI. Through the integration of these improvements, we demonstrated the continuous control of a robotic arm (movies S4 to S7) at almost identical levels to that of virtual cursor control, highlighting the potential of noninvasive BCI to translate to real-world devices for practical tasks and eventual clinical applications.

RESULTS

Before addressing whether our online ESI-based decoding strategy could be used for the continuous control of a robotic arm, the CP task and source signal approaches needed to be thoroughly validated as useful training and control strategies, respectively (Fig. 1). Thirty-three individuals naïve to BCI participated in a virtual cursor BCI learning phase. The training length was set at 10 sessions to facilitate practical data acquisition and to establish a threshold for future training applications. These 33 users were split into three groups: sensor domain CP training (CP), sensor domain DT training (DT), and source domain CP training (using real-time ESI; sCP). This design allowed us to answer (i) which training task (CP versus DT) and (ii) which neurofeedback domain (source versus sensor control) led to more effective BCI skill acquisition (see Materials and Methods for details on participant demographics and baseline group metrics). The within-session effects of source versus sensor control (virtual cursor) on CP BCI performance were tested on 29 individuals, 16 with previous BCI experience (sensor control) and 13 naïve to BCI. Furthermore, six individuals with BCI experience (sensor DT cursor control) participated in experiments designed to compare performances between virtual cursor and robotic arm control in a physically constrained variation of the CP task.

Fig. 1 Source-based CP BCI robotic arm framework.

The proposed framework addressed both user and machine learning aspects of BCI technology before being implemented in the control of a realistic robotic device. User learning was addressed by investigating the behavioral and physiological effects of BCI training using sensor-level neurofeedback with a traditional DT center-out task (n = 11) and a more realistic CP task (n = 11) (top left). The effects of BCI training were further tested in the CP task using source-level neurofeedback (n = 11) obtained through online ESI with user-specific anatomical models (center). This design allowed us to determine both the optimal task and neurofeedback domain for BCI skill acquisition. The machine learning aspect was further examined across the skill spectrum by testing the effects of source-level neurofeedback, compared with sensor-level neurofeedback, in naïve (n = 13) and experienced (n = 16) users in a randomized single-blinded design (top right). The user and machine learning components of the proposed framework were then combined to achieve real-time continuous source-based control of a robotic arm (n = 6) (bottom). Comparing BCI performance of robotic arm and virtual cursor control demonstrated the ease of translating neural control of a virtual object to a realistic assistive device useful for clinical applications.

Noninvasive continuous virtual target tracking via motor intent

Throughout all experimental sessions, users were instructed to control the trajectory of a virtual cursor using MI tasks; left- and right-hand MI for the corresponding left and right movement, and both hands MI and rest for up and down movement, respectively. These tasks were chosen on the basis of previous cursor control (28) and neurophysiological (19) exploration. Horizontal and vertical cursor movements were controlled independently. CP trials lasted 60 s each and required users to track a randomly moving target within a square workspace (Fig. 2, A and B; fig. S1, A and B; and movies S1 to S3). Previous implementations of similar tasks used technician-controlled (manual) target trajectories, which can introduce inconsistencies and biases during tracking (29). To avoid such scenarios, we governed target trajectories in the current work by a Gaussian random process (see Materials and Methods). Nevertheless, it is possible for such a random process to drive the target toward stagnation at an edge/corner, which could synthetically distort performance. Therefore, to better estimate the difference between DT and CP task training, and contrary to previous work (18, 29), our initial CP task allowed the cursor and target to fluidly wrap from one side of the workspace to the other (top to bottom, left to right, and vice versa) upon crossing an edge (Fig. 2, A and B; fig. S1, A and B; and movies S1 to S3). Trajectories from experienced users were unwrapped (fig. S1C) to reveal squared tracking correlations of ρhor2=0.48±0.20 and ρver2=0.47±0.19 (fig. S2).

Fig. 2 BCI performance and user engagement.

(A) Depiction of the CP edge wrapping feature. (B) Tracking trajectory during an example 2D CP trial. (C) Training feature maps for the DT and CP training groups for horizontal (top) and vertical (bottom) cursor control. ρ2, squared correlation coefficient. (D and E) 2D BCI performance for the CP (D) and DT (E) task at baseline and evaluation for the CP and DT training groups. The red dotted line indicates chance level. The effect size, ∣r∣, is indicated under each pair of bars. (F) Task learning for the CP (top) and DT (bottom) tasks. (G) Eye blink EEG component scalp topography (top) and activity (bottom left) at baseline and evaluation, and activity during each task (CP versus DT) (bottom right). Bars indicate mean + SEM. Statistical analysis using a one- (F) or two-way repeated-measures (D, E, and G) ANOVA (n = 11 per group) with main effects of task, and time and task, respectively. Main effect of time: #P < 0.05, ###P < 0.005. Tukey’s HSD post hoc test: *P < 0.05, ***P < 0.005.

BCI skill acquisition and user engagement

We investigated the utility of using the CP task for BCI skill acquisition in a pre-post study design by comparing BCI performance between populations trained by either CP or DT task. Twenty-two individuals participated in a baseline session, eight training sessions, and an evaluation session. Baseline and evaluation sessions contained both DT and CP tasks (and MI without feedback), whereas training sessions contained only one task type, consistent throughout training according to each user’s assigned group (DT or CP, n = 11 per group; see Materials and Methods). All sessions for both groups used scalp sensor information. 1D horizontal DT performance was used to baseline match the two groups (fig. S3A).

Electrodes used for online control were optimized on a session-by-session basis (see Materials and Methods), chosen from a set of 57 sensors covering the sensorimotor regions. Electrodes were identified for the horizontal and vertical control dimensions independently using the corresponding right-hand MI versus left-hand MI and both hands MI versus rest datasets. Throughout training, the two groups derived nearly identical feature (electrode) maps in the sensor domain containing focal bilateral scalp clusters overlying the cortical hand regions (Fig. 2C). These clusters were located and weighted in accordance with the underlying event-related (de)synchronization generated during the corresponding MI tasks (19) and are similar to those used in other noninvasive cursor control studies, identified through either data-driven (30) or manual (28) selection processes.

DT task performance was measured in terms of percent valid correct (PVC), computed as the number of hit trials divided by the total number of trials, in which a final decision was made (valid trials). The corresponding CP task performance metric was mean squared error (MSE), i.e., the average normalized squared error between the target and cursor location over the course of a single run. Across these 22 participants, the results of a repeated-measures two-way analysis of variance (ANOVA) revealed a significant main effect of time for both the CP MSE (F1,20 = 7.39, P < 0.05; Fig. 2D) and DT PVC (F1,20 = 19.80, P < 0.005; Fig. 2E) metrics. To examine skill generalizability, we specifically considered the effects of training on the performance of familiar and unfamiliar tasks. Individuals trained with the CP task significantly improved in the same task after training [Tukey’s honestly significant difference (HSD) post hoc test, P < 0.05; Fig. 2D, left bars], whereas those trained with the DT task did not (Tukey’s HSD post hoc test, P = 0.14; Fig. 2E, right bars). Previous work has indicated that DT task training can lead to strong learning effects (31); however, some users have required nearly 70 training sessions to do so (18). When considering unfamiliar tasks, the DT training group only modestly improved in the CP task after training (Tukey’s HSD post hoc test, P = 0.96; Fig. 2D, right bars), whereas the CP training group displayed a significant improvement in the DT task (Tukey’s HSD post hoc test, P < 0.005; Fig. 2E, left bars).

Because the two tasks varied greatly in control dynamics, it was difficult to draw comparisons between these differences. Therefore, in addition to statistical testing, we also examined the effect size (point biserial correlation, see Materials and Methods), a measure, unconfounded by sample size, of the magnitude of the difference within each performance metric between baseline and evaluation sessions. Compared with the DT group, the effect sizes were far larger for the CP group for both tasks (Fig. 2, D and E), displaying a 500% learning improvement in the CP task and a nearly 60% learning improvement in the DT task (Fig. 2F).

To delineate the underlying physiology of these training differences, we investigated user engagement during both tasks by quantifying eye blink activity. Decreased blink activity has been implicated in heightened attentional processes and cognitive arousal during various tasks (32). These mental states can dramatically influence task training and performance; whereas stimulating tasks can facilitate skill acquisition, boring or frustrating tasks can inhibit performance (20). The eye blink component of the EEG was extracted during the baseline and evaluation sessions using independent component analysis (ICA) (Fig. 2G and fig. S4). Across all participants, blink activity was strongly dampened at the baseline (F1,63 = 9.84, P < 0.005; Fig. 2G), suggesting heightened attention that was likely due to the novelty of BCI in general. Increased blink activity at the evaluation supports user skill acquisition, because less attention was required for improved performance. The large reduction in blink activity observed during the CP task, compared with the DT task (F1,63 = 3.51, P = 0.066; Fig. 2G), suggests that the CP task elicited heightened user engagement during active control, a feature that may explain the more dramatic positive training effects.

Learning to modulate sensorimotor rhythms

Whereas BCI feedback plays a significant role in facilitating sensorimotor rhythm modulation (33), MI without feedback can provide a measure of a user’s natural ability to produce the associated discriminative EEG patterns. Left-hand MI versus right-hand MI (left versus right) and both hands MI versus rest (up versus down) runs were analyzed individually. An index of modulation between any two mental states is represented as the regression output (R2) between the EEG alpha power and the task labels (see Materials and Methods). Only the 57 sensorimotor electrodes used for online control were included in this analysis. Although sensorimotor modulation significantly increased for both task pairs from baseline to evaluation (horizontal: F1,20 = 4.70, P < 0.05; vertical: F1,20 = 21.01, P < 0.005; Fig. 3, A and C), the spatial distribution of these improvements is more meaningful in evaluating the effectiveness of BCI training. Except for mild baseline modulation in the DT group, no strong patterns were apparent for either task pair before training. For the horizontal dimension at the evaluation session, the CP group produced highly focal bilateral modulation patterns, whereas more global modulation was observed for the DT group (Fig. 3B). Evaluation topographies were more consistent between the two training groups for the vertical dimension (Fig. 3D). Electrodes displaying a significant improvement in modulation were far more numerous for the CP group than for the DT group for both horizontal (CP, 12; DT, 3; Fig. 3E) and vertical (CP, 37; DT, 13; Fig. 3F) tasks. Furthermore, these significant electrodes clustered far closer to scalp regions covering the approximate hand cortical regions (e.g., C3-4 and CP3-4) in the CP group. These localized changes provide compelling evidence that the enhanced behavioral improvement seen in the CP training group was accompanied by consistent physiological changes in sensorimotor modulation (R2 values) (34).

Fig. 3 Electrophysiological learning effects.

(A and B) Left versus right MI task analysis. (A) Maximum sensorimotor R2 value for the CP and DT training groups for horizontal control task. The effect size, ∣r∣, is indicated under each pair of bars. (B) R2 topographies at baseline (top row) and evaluation (bottom row) for the CP and DT training groups for horizontal control tasks. (C and D) Both hands versus rest MI task analysis. Same as (A) and (B) for vertical control tasks. (E and F) Statistical topographies indicating electrodes that displayed a significant increase in R2 values for the horizontal (E) and vertical (F) control tasks. The electrode map in the middle provides a reference for the electrodes shown. Bar graphs below each topography provide a count for the number of electrodes meeting the various significance thresholds. Bars indicate mean + SEM. Statistical analysis using a one- (E and F) or two-way repeated-measures (A and C) ANOVA (n = 11 per group) with main effects of time [blue, P < 0.05; green, P < 0.01; yellow, P < 0.005; red outline, P < 0.05, false discovery rate (FDR) corrected] and time (#P < 0.05, ###P < 0.005) and training task, respectively. Tukey’s HSD post hoc test: *P < 0.05.

Source neurofeedback does not further facilitate CP BCI learning

Whereas the CP task allowed us to target user learning and progress toward the robust online control of a robotic arm, we additionally wanted to address the machine learning element. To evaluate whether real-time ESI-based decoding improved performance throughout training, we recruited an additional group of BCI naïve individuals (n = 11) for CP training using source neurofeedback (source control, sCP). This sCP group was baseline-matched to the previous CP (and DT) group (sensor control) (fig. S5A). For source control, we implemented user- and session-specific inverse models into the online decoding pipeline for the CP task. Similar to the CP group, the sCP group significantly improved in both the 2D CP (Tukey’s HSD post hoc test, P < 0.05; Fig. 4A, right bars) and the 2D DT tasks (Tukey’s HSD post hoc test, P < 0.05; Fig. 4B, right bars) after training. Accordingly, very similar learning effects were observed for both tasks in the CP and sCP groups (Fig. 4C). The final performance and learning rates were consistent between the two training groups (CP and sCP), supporting the groups’ shared familiar and unfamiliar task proficiency.

Fig. 4 Source-level neurofeedback.

(A and B) 2D BCI performance for the CP (A) and DT (B) task at baseline and evaluation for the CP and source CP (sCP) training groups. The red dotted line indicates chance level. The effect size, ∣r∣, is indicated under each pair of bars. (C) Task learning for the CP (left) and DT (right) tasks. Bars indicate mean + SEM. Statistical analysis using a one- (C) or two-way repeated-measures (A and B) ANOVA (n = 11 per group) with main effects of training decoding domain, and time and training decoding domain, respectively. Main effect of time: #P < 0.05, ###P < 0.005. Tukey’s HSD post hoc test: *P < 0.05, ***P < 0.005. n.s., not significant. (D) Group-level training feature maps for the training groups for horizontal (top) and vertical (bottom) cursor control. User-specific features were projected onto a template brain for group averaging.

Feature selection in the source domain identified distinct cortical clusters, optimized through anatomical and functional constraints, for online control and was performed on a session-by-session basis (see Materials and Methods). As expected, sCP training feature maps highlighted hand cortical regions for both control dimensions throughout training (Fig. 4D). It should be noted that the baseline and evaluation sessions for the sCP group were completed in the sensor domain to maintain consistent conditions with the other training groups. Although training duration was fixed at eight sessions with no intermediary testing, further investigation at different stages of learning may help pinpoint when source-based decoding may benefit BCI skill acquisition.

EEG source imaging enhances neural control in defined skill states

To thoroughly investigate the effects of source control (real-time ESI) on CP task performance (and potential future benefits for robotic arm control), we performed within-session comparisons of source and sensor virtual cursor control on users in stable skill states. The CP task was chosen for further analysis because it is more applicable to robotic arm control than the DT task and displayed both increased difficulty and skill acquisition. Our investigation included both extremes of the BCI skill spectrum; experienced users (12.8 ± 8.9 hours of previous BCI training, n = 16) participated in up to three sessions, and naïve users (no previous BCI training, n = 13) participated in a single session (to avoid confounding effects of early learning in more than one session). User- and session-specific inverse models were also used for these participants.

For experienced users, source control improved performance over that of conventional sensor control, producing a significant reduction in the 2D MSE (F1,69 = 9.83, P < 0.01; Fig. 5A). Unsurprisingly, the sensor and source MSE values clustered near those of the CP training group after training (evaluation), reinforcing their skilled state. The spatial extent of the observed improvement in the CP task was characterized through squared error histograms (Fig. 5B), with source values shifting toward smaller errors and sensor values shifting toward larger errors. By fitting gamma functions to these histograms, we derived a quantitative threshold, independent of cursor/target size, for statistically testing the spatial extent of the performance difference (fig. S6). Experienced users dwelt within this defined region, a disc with a diameter of 16.67% of the workspace width centered on the target (Fig. 5E), for significantly more time during source control than sensor control (F1,69 = 20.96, P < 0.005; Fig. 5F).

Fig. 5 Online 2D CP source versus sensor BCI performance.

(A and B) Experienced user performance (n = 16). (A) Group-level MSE for source and sensor 2D CP cursor control. Light and dark gray blocks represent performance for the CP training group (n = 11; Fig. 2D) before (naïve) and after training (experienced). The effect size, ∣r∣, is indicated under the pair of bars. (B) Group-level squared-error histograms for 2D CP sensor and source cursor control. (C and D) Naïve user performance (n = 13). Same as (A) and (B) for naïve user data. (E) Scale drawing of the CP paradigm workspace displaying the spatial threshold derived from experienced (yellow) and naïve (green) user data (fig. S6). (F) Cursor dwell time within the spatial threshold for experienced (left) and naïve (right) users. (G) Group-level feature maps for horizontal (top) and vertical (bottom) cursor control for naïve (right) and experienced (left) users. User-specific features were projected onto a template brain for group averaging. (H) Feature spread analysis between experienced and naïve users for source (left) and sensor (right) features for horizontal (top) and vertical (bottom) control. Bars indicate mean + SEM. Statistical analysis using a one- (C and D) or two-way repeated-measures (A and B) ANOVA with main effects of decoding domain, and time and decoding domain, respectively. Main effect of decoding domain: ###P < 0.005 (A, C, and F), gray bar; P < 0.05 uncorrected, red bar; P < 0.05, FDR corrected (B and D). Mann-Whitney U test with Bonferroni correction for multiple comparisons (H): +P < 0.05, +++P < 0.005.

Naïve users also demonstrated an overall improvement in online performance with source control, although this improvement did not reach significance for 2D control (F1,12 = 3.02, P = 0.11; Fig. 5C). Nevertheless, the effect size for the performance difference was notably similar to that of experienced users (Fig. 5, A and C, and table S1), indicating an improvement of similar magnitude. As expected, the sensor and source control MSE values for the naïve users were comparable with those of the CP training group before training (baseline, also naïve). This consistency, independent of skill level, highlights a robust positive influence of source control on online performance. Furthermore, the squared error histograms (Fig. 5D) and extent threshold measures for naïve users (Fig. 5E) displayed analogous trends to those of experienced users; however, these did not reach significance (F1,12 = 2.02, P = 0.18; Fig. 5F).

When looking at the feature maps (Fig. 5G), an important dichotomy can be observed between naïve (weak, sporadic clusters) and experienced (strong, focal clusters) users for both control dimensions that parallels the trends previously observed in the modulation index topographies before (low, sporadic modulation) and after (high, focal modulation) training (Fig. 3, B and D). To quantify the focality/diffuseness of these features, we computed the spread of the group-level feature maps (Fig. 5H), defined as the average weighted distance between the feature location and the hand knob (source space) or C3/C4 electrode (sensor space) (see Materials and Methods). We observed both significant or near-significant reductions in the feature spread for experienced users, compared with naïve users, in both the horizontal (Mann-Whitney U test with Bonferroni correction: source, P < 0.005; sensor, P < 0.05) and the vertical (Mann-Whitney U test with Bonferroni correction: source, P < 0.005; sensor, P = 0.22) control dimensions. This physiological difference between naïve and experienced users is in line with their performance difference (MSE) and further supports the contrast in BCI proficiency among the two groups and the overarching effect of source-based control depending on user skill level.

Source-based CP BCI control of a robotic arm

Having robustly validated our proposed BCI framework in a controlled environment, we completed our study by transitioning to the applied physical source control of a robotic arm (Fig. 6A). Although the cursor and target wrapping allowed for more complicated control strategies and scenarios, such a feature could not exist in a real-world setting. Therefore, we implemented a modified form of the CP task in a robotic arm control paradigm, where the edge-wrapping feature was replaced with an edge repulsing feature (Fig. 6B and movies S4 to S7). Six experienced users (8.3 ± 2.9 hours of previous BCI training) participated in five source CP BCI sessions containing both virtual cursor and robotic arm control, block-randomized across individuals and sessions. Because no paradigm was implemented to determine performance values before and after training in the modified task, participants were screened for experience and skill level beforehand (see Materials and Methods). Physiological support for user skill level was additionally observed in the group-level feature maps (Fig. 6C) that displayed comparable characteristics with those of other experienced users participating in this study (Fig. 5G).

Fig. 6 Source-based CP BCI robotic arm control.

(A) Robotic arm CP BCI setup. Users controlled the 2D continuous movement of a 7–degree of freedom robotic arm to track a randomly moving target on a computer screen. (B) Depiction of the CP edge repulsion feature (in contrast to the edge wrapping feature; Fig. 2A) used to accommodate the physical limitations of the robotic arm. (C) Group-level feature maps for the horizontal (top row) and vertical (bottom row) control dimensions projected onto a template brain. (D) Group-level 2D MSE for the various control conditions. Bars indicate mean + SEM. (E) Box-and-whisker plots for the group-level squared tracking correlation (ρ2) values for the horizontal (left) and vertical (right) dimensions during 2D CP control for the various control conditions. Blue lines indicate the medians, tops and bottoms of the boxes indicate the 25th and 75 percentiles, and the top and bottom whiskers indicate the respective minimum and maximum values. Control conditions include virtual cursor (white), hidden cursor (gray), and robotic arm (black). The red dotted line indicates chance level. Statistical analysis using a repeated-measures two-way ANOVA (n = 6 per condition) with main effects of time and control condition.

When users were directly controlling the robotic arm, the behavior of a hidden virtual cursor was also recorded to ensure proper mapping of the arm position in physical space. Across all sessions and individuals, median squared tracking correlation values reached ρhor2=0.13 (IQR=0.040.32) and ρver2=0.09 (IQR=0.030.28) in the horizontal and vertical dimensions, respectively, for 2D control. In transitioning between virtual cursor and robotic arm control, we observed similar MSE values among the three tracking conditions—virtual cursor, hidden cursor, and robotic arm (F2,40 = 2.62, P = 0.086; Fig. 6D)—indicating a smooth transition from the control of a virtual object to a real-world device. This likeness in control quality was further revealed through a lack of significant difference in the squared tracking correlation (ρ2) for both the horizontal (F2,40 = 0.13, P = 0.88; Fig. 6E) and vertical (F2,40 = 0.77, P = 0.47; Fig. 6E) dimensions. Tracking performance was significantly greater than the chance levels determined for all control conditions and dimensions (Mann-Whitney U test with Bonferroni correction, all P < 0.05). Overall, the notable similarity between virtual cursor control and robotic arm control highlights the possibility of integrating virtual cursor exposure into future clinical training paradigms, where patients have limited access to robotic arm training time.

DISCUSSION

The research presented here describes an encompassing approach aimed at driving noninvasive neural control toward the realistic daily use of a robotic device. We have demonstrated that the CP BCI paradigm not only can be used to successfully gauge a user’s BCI proficiency but also can serve as a more effective training tool than traditional center-out DT tasks, accelerating the acquisition of neural cursor control and driving the associated physiological changes. Contrary to users trained with the DT task, those trained with the CP task displayed significant performance improvements in familiar and unfamiliar tasks (Fig. 2, D to F), demonstrating highly flexible skill acquisition. These results were further supported in a third group that also trained with the CP task (Fig. 4, A to C). Participants in this group (sCP) displayed nearly identical learning effects as the original sensor CP group while training with source control, providing confidence for the reproducibility of the effects of CP task training.

As training progressed, it became apparent that the strategies developed by users differed significantly, depending on the training task. For example, various individuals in the DT training group reported using strategies involving selectively attending to their hand(s) through peripheral vision without necessarily focusing on the cursor position. Although such strategies were effective for DT tasks, users using them often struggled with the CP tasks in the evaluation session, because the moving target and cursor required constant visual attention and adjustment of motor-related mental intent. In this sense, many of these users somewhat ignored the feedback when training with the DT task and treated it similarly to the MI without feedback, reducing its effectiveness (33). The lower success of such strategies manifested within the MI EEG of the DT group as sporadic patterns of modulation after training (Fig. 3D), which are also consistent with the lower levels of cognitive arousal observed during the traditional DT task compared with the CP task (Fig. 2G). We believe that the target dynamics and screen wrapping feature of the CP task (Fig. 2A) likely perturb fluid target tracking and require heightened attention during cursor control. These conclusions support the overarching concept of integrating human factors, such as virtual reality techniques (34, 35), into cognitive-based training tools for improving both user engagement and task performance (20, 3638) and should be considered in future generations of BCIs.

Seminal works implementing similar continuous tracking tasks using invasively acquired signals reported comparable squared tracking correlation values over a decade ago (29). Although the field of invasive neural decoding has surpassed these benchmark results to include high degree-of-freedom and anthropomorphically functional tasks (3, 39, 40), qualitative similarities can be seen between these two modalities. In accordance with invasive reports, users in our study struggled to keep the cursor in a single location, often exhibiting oscillatory tracking behavior around the target (Fig. 2B and fig. S1, A and B). Although these actions demonstrate directed cursor trajectories toward the target and highlight the ability of our system to accurately capture the users’ dynamic mental intent, the tracking correlation is effectively reduced and may benefit from more advanced decoding methods.

It has been argued that motor neurons encode cursor velocity during neural cursor control (41), with numerous decoding algorithms using such properties to drastically improve user performance over classical techniques (38). In particular, modeling neuronal behavior as a dynamical system has recently yielded significantly improved online decoding results (42) and may provide even more complex and efficient device control in upcoming invasive and noninvasive work. This decoding strategy would be particularly attractive to neural control in the CP task presented here, given the clear analog of our control output to under-dampened control dynamics. Although this information would be valuable to reduce or eliminate the previously described cursor oscillations, it has yet to be observed whether these details can be detected via scalp recordings. Nevertheless, noninvasive neural signals have recently been shown to contain information encoded on the spatial scale of cortical columns (submillimeter), indicating the ability to decode neural activity with very fine spatial-temporal resolution from outside the skull (43).

Over the past few decades, the reconstruction of cortical activity through ESI has exemplified the push to increase the spatial specificity of noninvasive recordings and has been shown to provide superior neural decoding when compared with scalp sensor information (23, 24, 27). Similar to these previous works, we found that, in general, source features were more correlated with cued motor-related mental states than sensor features (fig. S9) (44, 45). Furthermore, in closed-loop CP BCI control, we found that the inclusion of online ESI improved performance in naïve and experienced users, consistent with offline enhancements (Fig. 5 and figs. S7 to S9). The increased task-specific source modulation indicates a higher sensitivity for detecting changes in a user’s motor-related mental state and is likely a product of the principles of ESI and its use in modeling and counteracting volume conduction. CP cursor control requires highly dynamic cognitive processes to recognize and correct for the random and sudden changes in the target’s trajectory during tracking. We therefore hypothesize that the fast, real-time control required during the CP paradigm takes advantage of the heightened sensitivity of ESI modulation, allowing for quicker responses that more accurately resemble the dynamics involved in the CP task. This phenomenon was apparent during the within-session comparisons of source and sensor control (Fig. 5 and figs. S7 and S8); however, it is possible that, with sufficient training, the feedback domain becomes less important for skill acquisition (Fig. 4).

We feel that it is necessary to acknowledge the decline in performance that occurred between the original CP task and the modified CP task, which we believe to be strongly attributed to the task modifications made for the physical constraints of the robotic arm. The presence of the physical robotic device inherently creates a more distracting environment for neural control compared with that of a virtual cursor. We found that, with the robotic arm mounted on the right side of the user (Figs. 1, bottom, and 6A), visual obstruction of the target was common when the arm was directed to reach across the user to the left side of the screen, often perturbing target tracking. In addition, although participants here displayed previous BCI proficiency, they had less experience than those participating in the original CP task validation. We believe that this combination of reduced user experience and enhanced sensory loading caused by the more complex human-device interaction involving the robotic arm led to a reduction in performance compared with the highly controlled virtual cursor control environment.

The results presented here demonstrate that CP control provides a unique opportunity for the complex control of a virtual cursor and robotic device (14, 39), without requiring discretized, prolonged task sequences (46) that can make even simple task completion long and frustrating. Users were able to smoothly transition between virtual cursor and robotic arm control with minimal changes in performance (Fig. 6, D and E), indicating the potential ease of integrating such a noninvasive assistive tool into clinical applications for autonomous use in daily life. Invasive systems have already demonstrated a level of control similar to such a noninvasive hypothetical; however, although such invasive approaches may offer much-needed help to a restricted number of patients with severe physical dysfunctions, most impaired persons will likely not qualify for participation due to both medical and financial limitations. Additionally, previous work has suggested that accessing sufficiently large patient populations for concrete and statistically significant conclusions may be difficult to obtain (15, 8, 11, 29, 39). Therefore, there is a strong need to further develop noninvasive BCI technology so that it can benefit most patients and even the general population in the future. The effective training paradigm and additional ESI-based performance improvement demonstrated here, as well as the integration of such targeted enhancements toward robotic arm control, offer increasing confidence that noninvasive BCIs may be able to expand to widespread clinical investigation. We observed that, for robotic arm control, generic head models, rather than those derived from user-specific magnetic resonance imaging (MRI), were sufficient for high-quality performance (see Materials and Methods). Therefore, in all, the work presented in this paper is necessary for current EEG-based BCI paradigms to achieve useful and effective noninvasive robotic device control, and its results are pertinent in directing both ongoing and future studies.

MATERIALS AND METHODS

BCI tasks

Motor imagery without feedback

EEG data during motor imagination (MI) without feedback were collected at the beginning of each session, one run for left-hand MI versus right-hand MI and one for the both hands MI versus rest. Each run consisted of 10 randomly presented trials per task. Each trial consisted of 3 s of rest, followed by 4 s of a visually cued MI task.

DT task

The DT paradigm was composed of fixed target locations and center-out intended cursor trajectories. This paradigm consisted of 21 trials, with targets presented in a random order. Each trial began with a 3-s rest period, followed by a 2-s preparation period, in which the target was presented to the user. Users were then given up to 6 s to move the cursor to hit the target. A 1-s intertrial interval bridged two adjacent trials. Feedback (cursor movement) was not provided during the first trial to calibrate the normalizer as described in the “Online signal processing” section.

During baseline and evaluation sessions, trials ended either upon a collision with a target or after 6 s with no collision. During training sessions, each trial lasted a full 6 s, requiring users to maintain their cursor over the target location for as long as possible within a boundary-constrained workspace. In this sense, during training, each DT run contained 120 s of online BCI control, consistent with the 120-s CP runs.

CP task

The CP stimulus paradigm was implemented using custom Python scripts in the BCPy2000 application module of BCI2000 (47). This paradigm involved the continuous tracking of a target; each run was composed of two 60-s trials separated by a 1-s intertrial interval. To produce smoothly varying random target movement, the position of the target was updated in each frame using a simple kinematic model. Random motion was obtained by applying a randomly generated 1D or 2D external force Fext, as in Eq. 1, drawn from a zero-mean fixed-variance normal distribution.FextN2(0,σ2)(1)

To effectively limit maximum target velocity, we also applied a friction force Ff and a drag force Fd. The friction and drag forces are represented in Eqs. 2 and 3, respectively, where μ indicates the coefficient of friction, δ indicates the drag, and v(t) indicates the velocity of the cursor at time step t. Here, ‖‖ denotes the Euclidian normFf=μv(t)v(t)2(2)Fv=δv(t)v(t)2(3)

When divided by the arbitrary target mass m, the combination of these forces represents the total instantaneous acceleration of the target. Integrating with respect to time, as noted in Eq. 4, produces the updated target velocity v(t+1) at the new time pointv(t+1)=v(t)+Fext+Ff+Fdmdt(4)

For the training and source versus sensor experiments described in subsequent sections, the cursor and target were allowed to wrap from one side of the workspace to the other (left to right, top to bottom, and vice versa). Contrary to this, for the robotic arm versus virtual cursor experiments, the target was repelled by the edges of the workspace to make the task more realistic and accommodate the physical limitations of the robotic arm. Repulsion was accomplished by inverting all applied forces that would push the target continuously into a wall while still randomly generating magnitudes and directions for irrelevant forces. Unlike the target dynamics, the cursor and robotic arm could press against the edge of the bounded region, given the appropriate force vector.

Noise/chance performance estimation

Chance performance in the CP paradigm was estimated by collecting 15 (standard) or 70 (physically constrained) datasets each for 1D horizontal (LR), 1D vertical (UD), and 2D control tasks with the electrode sets plugged in but not connected to a human scalp. Chance performance in the DT paradigm was determined by dividing 100% by the number of targets in each control dimension. This is valid because trials that time out are typically excluded when calculating performance for the DT task.

Experimental design

Sixty-eight healthy humans were informed and participated in different phases of this study after providing written consent to a protocol approved by the relevant Institutional Review Board at the University of Minnesota or Carnegie Mellon University.

Training

Thirty-three individuals (average age: 24.8 ± 10.6 years, 30 right handed, 18 male) naïve to BCI participated in longitudinal BCI training over the course of 10 experimental sessions that included one baseline session, eight training sessions, and one evaluation session. Participants were tested on all tasks at the baseline and evaluation time points to assess training effectiveness, completing one block of DT tasks and one block of CP tasks, block-wise randomized across individuals. The blocks for each paradigm were composed of two runs of 1D LR, 1D UD, and 2D control. Participants were divided into three training groups using the 1D LR DT performance as the balancing metric (figs. S3A and S5A). Naïve participants obtaining PVC values of >80% for both runs of any of the three DT dimensions were excluded from the training cohort because these users are often considered proficient (n = 5/38) (14, 28). Participants underwent eight training sessions at 12 runs per session, with only their specified task paradigm—DT sensor, CP sensor, or CP source. These eight training sessions were broken into 2 × 1D LR, 2 × 1D UD, and 4 × 2D control to progress toward more difficult tasks near the end of training. The evaluation session was identical to the baseline session, again with the task block order randomized across individuals. Baseline and evaluation sessions were all completed using sensor control for consistency across groups. Participants underwent two to three sessions per week with an average intersession interval of 3.69 ± 2.99 days.

Source versus sensor

Twenty-nine individuals participated in experiments testing the within-session effects of source versus sensor control on the CP BCI task. Sixteen users (average age: 22.67 ± 8.1 years, 15 right handed, 6 male) with an average of 12.8 ± 8.9 hours of previous BCI experience and 13 users (average age: 21.8 ± 5.0 years, 12 right handed, 8 male) naïve to BCI participated in this portion of the study. Experienced users participated in up to three BCI sessions, and the naïve users participated in a single session to avoid the confounding effects of learning. There were no exclusion criteria in this phase of the study because participants were in well-defined naïve or experienced states. A user-specific anatomical MRI was collected for each individual according to the MRI acquisition section. In each BCI session, participants completed 12 runs of CP BCI (4 × 1D LR, 4 × 1D UD, and 4 × 2D), with the decoding strategy (sensor or source) being randomized and balanced across the population.

Robotic arm versus virtual cursor

Six individuals (average age: 25.2 ± 6.5 years, five right handed, three male, 8.3 ± 2.9 hours of previous BCI training) participated in experiments comparing virtual cursor and robotic arm control. Participants for this phase were screened using sensor-based 1D and 2D DT tasks using the BCI2000 AR alpha (8 to 13 Hz) power estimation of C3 and C4, spatially filtered with the local pseudo-Laplacian using a Neuroscan Synamps2 (Compumedics Ltd., Victoria, Australia) 64-channel system. Participants were excluded on the basis of a two-stage performance evaluation: (i) failure to achieve >70% 1D PVC (sessions 1 and 2) or >40% 2D PVC (session 2) in two sequential runs and (ii) failure to achieve >90% 1D PVC and >70% 2D PVC (sessions 3 to 5) in two sequential runs. Six of 19 recruited participants passed these criteria.

All robotic arm experiments were conducted on a Samsung 43-inch 4K television, allowing large, practical workspaces for both the robotic arm and the virtual cursor. Each user participated in five source CP BCI sessions containing 12 runs (60 s) (sessions 1 and 2, 6 × 1D LR and 6 × 1D UD; sessions 3 to 5, 3 × 1D LR, 3 × 1D UD, and 6 × 2D) of both virtual cursor and robotic arm control in block-randomized order across users. Some users were asked to return for a sixth session to record video of continuous robotic arm and virtual cursor control. Robotic arm endpoint locations were mapped 1:1 to cursor positions on the screen, with inverse kinematics used to solve for optimal joint angles and arm trajectories. The robotic arm workspace was square with a 0.48-m side length. All robotic arm control was conducted using the Kinova JACO Assistive Robotic Arm with a three-finger attached gripper.

For all BCI sessions, participants were seated in a padded chair about 90 cm from a computer screen. Unless otherwise stated, users were fitted with a 128-channel Biosemi (Biosemi, Amsterdam, The Netherlands) EEG headcap of appropriate size and positioned according to the international 10-20 system. EEG was recorded at 1024 Hz using an ActiveTwo amplifier with active electrodes (Biosemi, Amsterdam, The Netherlands).

MRI acquisition

User-specific anatomical MRI images were acquired on a 3T MRI machine (Siemens Prisma, Erlangen, Germany) using a 32-channel head coil. High-resolution (1-mm isotropic) anatomical images were acquired for each participant using a T1-weighted magnetization prepared rapid acquisition gradient echo sequence (repetition time/echo time, 2350 ms/3.65 ms; flip angle, 7°; acquisition time, 05:06 min; R = 2 acceleration; matrix size, 256 by 256; field of view, 256 mm by 256 mm).

Frequency-domain ESI

For the source versus sensor experiments, the anatomical MRI from each user was segmented in FreeSurfer and uploaded into the MATLAB-based Brainstorm toolbox. For the robotic arm versus virtual cursor experiments, the Colin27 template brain was used for all users. The cortex was downsampled to a tessellated mesh of ~15,000 surface vertices and broken into 12 bilateral regions based on the Destrieux atlas. A central region of interest (ROI), composed of various sensorimotor areas (table S2), was used for feature extraction and online source control.

At the beginning of each BCI session in the source versus sensor experiments, EEG electrode locations were recorded using a FASTRAK digitizer (Polhemus, Colchester, Vermont) using the Brainstorm toolbox. Electrode locations were co-registered with the user’s MRI using the nasion and left and right preauricular landmarks. A three-shell realistic geometry head model with a conductivity ratio of 1:1/20:1 was generated using the boundary element method implemented in the OpenMEEG toolbox.

The inverse operator for each session was generated according to the following theory. Equation 5 depicts the linear system relating scalp and cortical activity, where ϕ(t) represents the scalp-recorded EEG at time t, L represents the user- and session-specific leadfield, and J(t) represents the cortical current density at time tϕ(t)=LJ(t)(5)

Linear programming techniques can help stabilize the often ill-conditioned nature of the leadfield to find optimal estimates of the source distribution. In the current work, we used Tikhonov regularization (Eq. 6). This optimization suggests a solution J(t) that depends on various known parameters that include the sensor covariance matrix C, source covariance matrix R, regularization parameter λ, leadfield, and scalp EEGminJC12(ϕ(t)LJ(t))22+λ2R12J(t)22,where λ2=tr(LRLT)tr(C)SNR2(6)

The closed-form solution to Eq. 6, solving for an optimal source distribution Ĵ(t), is shown in Eq. 7 in the time domain and belongs to the family of minimum-norm estimates. Here, 20 s of resting-state EEG collected at the beginning of each session was used to compute a diagonal sensor covariance matrix C. The source covariance matrix was also a diagonal matrix, with nonzero elements containing a depth-weighted reciprocal of source location power. This modification to the source covariance matrix forms the weighted minimum-norm estimate.Ĵ(t)=RLT (LRLT+λ2C)1ϕ(t)(7)

This solution can be applied in the frequency domain by solving for both the real and imaginary frequency-specific cortical activity independently (45) and subsequently taking the magnitude at each cortical location (Eq. 8)ĴRe(f)=RLT (LRLT+λ2CRe)1ϕRe(f)ĴIm(f)=RLT (LRLT+λ2CIm)1ϕIm(f)(8)

To use the spatial filtering properties of inverse imaging and extract task-related activity, we subjected the reconstructed cortical activity to both anatomical and functional constraints. The anatomical constraint is represented by limiting cortical activity to the central sensorimotor ROI previously described. The functional constraint is based on the data-driven parcellation of the ROI into discretized, functionally coherent cortical clusters. Parcellation is particularly attractive for real-time applications because it improves the condition of the EEG inverse problem and reduces computation time (48). Parcellation was performed using the multivariate source prelocalization algorithm using the MI without feedback data collected at the beginning of each session (48). Solving for the activity in each of these cortical clusters extends Eqs. 8 to 9, where the subscript k represents the number of cortical parcelsĴk, Re(f)=RkLkT ((kLkRkLkT)+λRe2CRe)1ϕRe(f)Ĵk, Im(f)=RkLkT ((kLkRkLkT)+λIm2CIm)1ϕIm(f)(9)

Channel-frequency optimization

Each of the MI without feedback runs was analyzed individually to identify features used to control cursor movement in the two dimensions. For the sensor domain, the alpha power (8 to 13 Hz) at each electrode was extracted at a 1-Hz resolution using a Morlet wavelet technique previously described in (24). A stepwise linear regression approach similar to (30) was used with a forward inclusion step (P < 0.01) and backward removal step (P < 0.01) to find the electrodes and weights that best separated the two tasks used for each control dimension. This procedure was applied to frequency-specific R2 montages in the order of descending maximum values until at least one electrode survived the statistic thresholding. The weight of each selected electrode was set to −1 or +1 based on the sign of the regression beta coefficient. A weight of 0 was applied to all other electrodes not selected. If no electrodes were selected for any frequency, then a default setup assigned −1 and +1 to the C3 and C4 electrodes, respectively, for horizontal control and −1 and −1 to both electrodes for vertical control (28).

For feature selection in the source domain, the MI EEG was first mapped to the cortical model according to Eq. 9. The stepwise linear regression procedure was applied to all ROI parcels, and weights were assigned accordingly. If no parcels were selected, then the default source setup was defined by assigning a weight of −1 to those parcels containing the left motor cortex hand knob, +1 to those containing the right motor cortex hand knob for horizontal control, and −1 to bilateral hand knob parcels for vertical control. These parcels were identified on the basis of seed points assigned to the hand knobs [similar to (44)] by the operators before the experimental session.

Feature spread was calculated as the average Euclidean distance between the feature location and the lateral hand knob (source space) or C3/C4 electrode (sensor space). The hand knob location was defined as the average location of the previously mentioned seed points. The distance was also weighted by the magnitude of the feature weight to account for its strength. Distances were calculated for the left and right sides of the head individually and pooled together for each dimension.

Online signal processing

All online processing was performed using custom MATLAB (MathWorks Inc., MA, USA) scripts that communicated with BCI2000 using the FieldTrip buffer signal processing module. Fifty-seven electrodes covering the motor-parietal region of the scalp were used for online processing. The EEG was downsampled to 256 Hz and bandpass-filtered between 8 and 13 Hz using a fourth-order Butterworth filter before common average referencing. The most recent 250 ms of data was analyzed and used to update the cursor velocity every 100 ms. The instantaneous control signal was computed as the weighted sum of the alpha power in the selected electrodes. If ϕt(f) represents the magnitude of the alpha power across the entire EEG montage at time window t and xh and xv are vectors containing the electrode weights (1s, −1s, and 0s) assigned during the optimization process, then the instantaneous control signal for each dimension can be represented asCh,t=xhTϕt(f) Cv,t=xvTϕt(f)(10)

The velocity of the cursor in each dimension was then derived by normalizing these values to zero mean and unit variance based on the values stored from the previous 30 s of online control in the respective dimensionVh,t=Ch,tC¯h,tn:t1σCh,tn:t1 Vv,t=Cv,tC¯v,tn:t1σCv,tn:t1(11)

The same procedure was performed for source control using the reconstructed cortical frequency information Ĵt(f) and the corresponding cortical cluster weights. Robotic arm positions were controlled via a custom C++ script, which read and translated cursor positions into optimal joint angles.

Offline data analysis

CP data files contained cursor and target positions. These values were normalized to the screen size and used to obtain an error, defined as the Euclidean distance between the cursor and target, at each time point. The tracking correlation was computed as the Pearson correlation coefficient (ρ) between the target and cursor position time series. The MSE value was computed as the average of the error time series between these same two position vectors. The choice to use ρ2 (squared tracking correlation) was based on the concept of user control; signed values of ρ much less than 0 are superior to small positive values (e.g., –1 versus +0.01) because they suggest high-quality control that is inverted and that the simple inversion of weights can lead to high tracking performance. Furthermore, very few tracking correlation values were negative for both the original and physically constrained CP tasks. DT data files contained target and result codes for each trial used to compute PVC values. Artifactual trials for both DT and CP runs were identified during online BCI control or by offline visual inspection of the EEG and removed from subsequent analysis.

MI without feedback data files contained the 128-channel EEG and MI task labels. Nonstationary high-variance signals were initially removed from the raw EEG using the artifact subspace reconstruction EEGlab plugin. Bad channels were spherically interpolated. The clean EEG was downsampled to 128 Hz, filtered between 5 and 30 Hz using a fourth-order Butterworth filter, and re-referenced to the common average. The alpha (8 to 13 Hz) power was extracted from each channel using a Morlet wavelet for the time periods of 0.5 to 4.0 s after each stimulus presentation; a 0.5-s delay was included to account for user reaction time (after the visual cue). The alpha power in each channel and each frequency was regressed against the task labels. For the source domain, cortical alpha power was computed according to the frequency-domain ESI section and regressed against the task labels.

Eye activity was extracted using ICA. Clean EEG data for all tasks in the baseline and evaluation sessions were concatenated into separate datasets and decomposed using the extended infomax algorithm. The dimensionality of the data was first reduced using principal components analysis. The vertical and horizontal eye activity components (for fig. S4 analysis) were identified as those containing high delta (1 to 4 Hz) activity and strong monopolar and bipolar frontal electrode projections, respectively (49). Not all sessions contained both distinct components meeting these criteria. Blink activity was computed as the variance of the (vertical/blink) independent component (IC) activation sequence during DT and CP control separately. To determine the influence of eye activity on BCI performance, we performed a regression analysis between the vertical or horizontal eye activity IC activation sequence and target location in the corresponding dimension.

Statistical analysis

Statistical analysis was performed using custom R and MATLAB scripts. Effect sizes are reported throughout the article as the point biserial correlation, ∣r∣, to highlight within-group (e.g., training) and across-condition (e.g., source versus sensor and robotic arm versus virtual cursor) differences. The point biserial correlation was computed according to Eq. 12, where M1 and M2 are the means of the two distributions being compared and is the pooled SD (d is also known as Cohen’s d)r=dd2+4,d=M1M2SDpooled(12)

Unless otherwise stated, two-way repeated-measures ANOVAs were used with main effects of time and training task (DT versus CP), decoding domain (source versus sensor), or control method (robotic arm versus virtual cursor). All behavioral and electrophysiological metrics were first evaluated with the Shapiro-Wilk test to test for the normality of the residuals of a standard ANOVA. If the P value of the majority of all multiple comparisons was less than 0.05, then a rank-transformed ANOVA was used. Otherwise, a standard ANOVA was used. If less than 10 multiple comparisons were made, then a Tukey’s post hoc test was used to correct for multiple comparisons, and if greater than 10 comparisons were made, then FDR correction (P < 0.05) was used. A Mann-Whitney U test with Bonferroni correction for multiple comparisons was used for specific cases: comparing squared tracking correlation values (ρ2) of neural control with noise in the constrained CP task and comparing the feature spread in the source and sensor domains in naïve and experienced users.

SUPPLEMENTARY MATERIALS

robotics.sciencemag.org/cgi/content/full/4/31/eaaw6844/DC1

Fig. S1. Example CP trajectories.

Fig. S2. Squared tracking correlation histograms.

Fig. S3. CP versus DT BCI learning.

Fig. S4. Influence of eye activity on BCI control.

Fig. S5. Source versus sensor BCI learning.

Fig. S6. 2D CP source versus sensor spatial threshold.

Fig. S7. Online 1D horizontal CP source versus sensor BCI performance.

Fig. S8. Online 1D vertical CP source versus sensor BCI performance.

Fig. S9. Offline source versus sensor sensorimotor modulation.

Table S1. Absolute effect sizes for source-based versus sensor-based CP control.

Table S2. Source-level sensorimotor ROI anatomical structures.

Movie S1. 1D horizontal CP (unconstrained) BCI virtual cursor control example trial.

Movie S2. 1D vertical CP (unconstrained) BCI virtual cursor control example trial.

Movie S3. 2D CP (unconstrained) BCI virtual cursor control example trial.

Movie S4. 1D horizontal CP (physically constrained) BCI robotic arm control example trial.

Movie S5. 1D vertical CP (physically constrained) BCI robotic arm control example trial.

Movie S6. 2D CP (physically constrained) BCI robotic arm control example trial.

Movie S7. 2D CP (physically constrained) BCI virtual cursor control example trial.

REFERENCES AND NOTES

Acknowledgments: We thank A. Sohrabpour for helpful discussions and S. Sherman for help in participant recruitment. Funding: This work was supported, in part, by NIH AT009263, MH114233, EB021027, NS096761, and NSF CBET-1264782. B.J.E. was supported, in part, by NIH/NINDS Fellowship F31NS096964. Author contributions: B.J.E., J.M., D.S., B.S.B., and B.H. designed experiments. B.J.E., J.M., D.S., C.Z., and E.N. collected the data. B.J.E., J.M., and D.S. analyzed the data. B.J.E., J.M., D.S., and B.H. wrote the manuscript. B.J.E. created the real-time processing software. C.C.C. designed the CP task. J.M. designed the robotic arm interface. B.H. supervised the project involving all aspects of the study design and data collection and analysis. Competing interests: The authors declare that they have no competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper or the Supplementary Materials. BCI and EEG data used to create Figs. 2, 3, and 5 will be available at doi:10.5061/dryad.v46p2jh. Additional materials can be requested from B.H. under a material transfer agreement with Carnegie Mellon University.
View Abstract

Stay Connected to Science Robotics

Navigate This Article