Monkey With Machine Gun Video
Gail Elfert
Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.
The wheelchair remains the main device to assist navigation in people with motor disabilities, particularly those suffering from severe cases of body paralysis1. Up to date, noninvasive BMI approaches, mostly based on electroencephalography (EEG), have been dominant in assistive devices that enable direct brain control over navigation in a powered wheelchair2,3,4,5. Although intracranial BMIs hold promise to offer a superior performance over noninvasive systems6,7 and approximately 70% of paralyzed patients are willing to accept surgically implanted electrodes in their brains to gain control over their assistive devices8,9, only a few studies have previously attempted to apply intracranial BMIs to wheelchair control10,11.
One design for an intracranial BMI that controls wheelchair movements is based on joystick movement. This design was previously introduced by our laboratory10 and by Xu et al.11. In our experiments, rhesus monkeys were housed in a home cage equipped with a joystick. They used the joystick to continuously control the movements of an external wheelchair. This control was also performed in BMI mode, where the wheelchair coordinates were extracted from cortical neuronal ensembles10. In some BMI experiments the monkeys continued to assist movement with the joystick, in the others the joystick was removed and the monkeys controlled the wheelchair without producing overt arm movements. Recently, Xu et al. described how one monkey seated in a wheelchair11 was trained to move a joystick in one of four directions to set the direction of wheelchair movement. Under BMI control, this monkey continued to move the joystick, but the control signal was derived from a discrete classifier that extracted joystick position from cortical activity.
These BMI designs are similar to those previously used to reproduce arm movements, only replacing the end actuator with a robotic wheelchair. However, such BMIs cannot be used by severely paralyzed quadriplegic patients, who cannot produce arm movements. Additionally, neural control of arm movements may be an unnecessary intermediary to enable wheelchair navigation. Here we hypothesized that kinematics of whole-body movements could be extracted directly from sensorimotor cortical ensembles, and utilized to control a BMI for wheelchair control. As a first step towards the development of a clinically relevant device of this type, we utilized large scale recordings from multiple cortical areas7 obtained by our recently developed multichannel wireless recording system10 to enable BMI control over whole-body navigation in a robotic wheelchair.
The study was conducted in two monkeys (K and M) chronically implanted with multielectrode arrays in multiple cortical areas in both hemispheres (Fig. 1). Neuronal ensemble activity was sampled using our 512-channel wireless recording system10. In Monkey K, 79 neurons were recorded bilaterally in the primary motor cortex (M1), 35 in the right primary somatosensory cortex (S1), and 26 in the right dorsal premotor cortex (PMd). In Monkey M, 72 neurons were recorded in bilateral M1, and 72 in bilateral S1.
(A) Both monkeys show significant improvement in the traveling time and distance as they learn. The circles represent the median and the error bars show the interquartile range of the medians. (B) Increased correlations between decoders trained in earlier sessions and the last session. (left, Monkey K; right, Monkey M (C) Both monkeys demonstrate impaired performance once their decoded movement commands were inverted (forward now becomes backward, and right turn becomes left turn). The bar graph shows the median and the error bars indicate the interquartile range of the medians.
The video tracking of monkey head position showed two basic head orientations: (1) toward the food reward, and (2) straight relative to the chair. The first orientation was evident from the distribution of monkey head orientation relative to the feeder as the function of trial time (Fig. S6). While the monkeys turned their heads in different directions, they looked at the grape feeder more often than at different room locations. This tendency was especially prominent in Monkey K who clearly looked at the feeder throughout the trial. In both monkeys, head orientation at the feeder became especially prominent when the wheelchair arrived at the feeder location. The second type of head orientation, i.e. straight relative to the chair was clear from the distribution of head angle relative to the chair (Fig. S7). The tendency to look straight was especially strong in Monkey M and less strong in Monkey K who, as shown in the previous analysis, often looked at the food location.
The present study has demonstrated that intracranial wireless cortical ensemble recordings can be harnessed to control whole-body navigation in a mobile device such as a robotic wheelchair. Heretofore, intracranial BMI research in primates has focused primarily in reconstructing isolated upper limb movements12. A few years ago, our laboratory introduced the use of BMIs to investigate the possibility of restoring more complex movements, such as bipedal locomotion17 and then bimanual movements16. Yet, the investigation of cortical representation of whole-body translations, and the likelihood of using intracranial BMIs for wheelchair control, has been almost completely neglected. A few primate studies on spatial representation of the environment by hippocampal neurons18,19,20 have been performed, but these results are not directly applicable to the BMI for wheelchair control. Neuronal mechanisms of spatial encoding in rodents have received much more attention21,22,23 compared to primate studies, but the prospects of BMI navigation based on these neuronal properties was not explored.
Schwartz et al.10 demonstrated a joystick-based BMI paradigm where monkeys controlled movements of a wheelchair by their cortical modulations. The monkeys were not seated in the wheelchair in this study. More recently, Xu et al. trained one monkey to steer while seated in a wheelchair using a hand-held joystick to generate discrete but not continuous navigation commands11. These authors also demonstrated a BMI version of this joystick control, where hand movements were decoded from M1 ensemble activity to produce steering commands11. This group, however, did not attempt to translate cortical activity directly into whole-body navigation, without using hand movements as an intermediary. In another study, monkeys navigated in a virtual environment using a joystick while their bodies remained stationary24. Under these conditions, neuronal firing modulations observed in the medial superior temporal cortex depended on whether monkeys actively steered with the joystick or passively observed the visual flow of the scene.
In contrast to these studies, we trained our BMI decoder using a passive wheelchair navigation paradigm. A somewhat similar approach was explored in several BMI studies, where subjects passively observed the movements of an external device and/or imagined voluntarily controlling those movements while the BMI decoder was trained16,25,26. This BMI training approach has obvious clinical significance since severely paralyzed subjects cannot produce overt body movements to train a BMI decoder. We observed that, after passive navigation was employed to set up the decoder parameters, animals significantly improved their navigation performance through learning, likely mediated by widespread cortical plasticity16.
Irrespective of its origin, we learned that the distance to reward could be decoded offline using the cortical neuronal ensemble activity recorded in our experiments, yielding very reasonable and accurate predictions. Previously, we had decoded target location for an arm reaching task, where target location was also not part of the BMI control variables37. The presence of such novel cortical representation suggests that, in the future, potentially new control parameters may be extracted from cortical neuronal ensembles, in addition to the variables normally employed to control a BMI. Indeed, such emergent properties could add a new level of versatility to BMIs. Recently Marsh et al.28 proposed that cortical representation of reward should be incorporated in BMI design to make it more autonomous. Our study adds to these results by describing the existence of neuronal signals which modulate not just the presence or absence of reward, but a continuous representation of reward location; a tuning of space that coexisted with the representation of chair velocity. In this context, future studies should explore the possibility that cortical neuronal ensembles engaged in navigation could trigger an incorporation of the navigated space (not necessarily related with reward location only) into novel cortical representations.
In the wheelchair domain, numerous noninvasive EEG-based BMIs have been described. These applications employed motor imagery46 and P300 potentials47 as the source of brain-derived motor commands. Although these systems perform with an acceptable (80%) success rate in tasks that involve predefined paths and target locations, and can be improved to cope with real environments48, they are clearly limited and cannot be considered as the final solution for routine clinical use in the future. One approach to improve these systems is to apply shared control schemes where some commands are delegated to the robotic system49,50. With regard to our current study we plan to integrate navigation in complex environments in future experiments.
The objective of this study was to demonstrate whole-body navigation with control signals derived from neuronal ensemble recordings in multiple cortical areas. Notably, our design did not require the subjects to map overt movements to the navigational direction, which makes this paradigm applicable to the needs of severely paralyzed patients who cannot move their limbs but desire to restore whole-body mobility.
c80f0f1006