Research ArticlePROSTHETICS

Restoring tactile sensations via neural interfaces for real-time force-and-slippage closed-loop control of bionic hands

See allHide authors and affiliations

Science Robotics  20 Feb 2019:
Vol. 4, Issue 27, eaau9924
DOI: 10.1126/scirobotics.aau9924

Abstract

Despite previous studies on the restoration of tactile sensation to the fingers and the hand, there are no examples of use of the routed sensory information to finely control a prosthestic hand in complex grasp and manipulation tasks. Here, it is shown that force and slippage sensations can be elicited in an amputee by means of biologically inspired slippage detection and encoding algorithms, supported by a stick-slip model of the performed grasp. A combination of cuff and intraneural electrodes was implanted for 11 weeks in a young woman with hand amputation and was shown to provide close-to-natural force and slippage sensations, paramount for substantially improving manipulative skills with the prosthesis. Evidence is provided about the improvement of the participant’s grasping and manipulation capabilities over time resulting from neural feedback. The elicited tactile sensations enabled the successful fulfillment of fine grasp and manipulation tasks with increasing complexity. Grasp performance was quantitatively assessed by means of instrumented objects and a purposely developed metrics. Closed-loop control capabilities enabled by the neural feedback were compared with those achieved without feedback. Further, the work demonstrates that the described amelioration of motor performance in dexterous tasks had as central neurophysiological correlates changes in motor cortical plasticity and that such changes were not of purely motor origin, but were the effect of a strong and persistent drive of the sensory feedback.

INTRODUCTION

Human dexterity and manipulation capabilities are enabled by the hand’s complex biomechanics, a sophisticated sensory system, and a sensorimotor control loop based on a bidirectional communication with the brain. Sensory contribution is so important that, in peripheral nerves, sensory fibers largely outnumber motor axons; this is extremely pronounced in forearm and hand nerves (1). This is why, in case of hand loss, any attempt to restore a physiological motor control of hand prostheses should go primarily through the restoration of sensory information allowing a proficient sensorimotor integration. The impossibility of current prostheses to provide the user with a pleasant and meaningful sensory feedback is one of the main reasons for the high percentage of prosthesis use abandonment (>30%) (2).

Up to now, neuroprosthetics research has shown the following: (i) The most robust and accurate way to extract upper limb prosthetic user’s intention is by decoding muscle electrical activity (3, 4). (ii) It is possible to deliver close-to-natural sensations to the human brain—such as pressure, object stiffness and shape, and texture (510)—by electrically stimulating peripheral nerves (6, 11, 12). (iii) Stability over time of the evoked sensations has been demonstrated for up to 24 months for cuff electrode and flat interface nerve electrode (12, 13). (iv) User satisfaction resulting from the receipt of sensory feedback and alleviation of phantom limb pain is significant (11). (v) Sensory feedback promotes a sense of ownership (i.e., embodiment) of the robotic limb (3, 14).

In affected individuals, amputation distorts cortical areas devoted to the control of the limb (15, 16). A consistent reversion of the amputation-induced aberrant cortical plasticity has been reported after the use of bionic prostheses, enabling some kind of sensory feedback (11, 14, 17). A robotic hand controlled through intraneural electrodes improved motor cortical representation of the lost hand (11), electroencephalography activation pattern during movement of the phantom hand (18), functional interhemispheric interaction (19), and the cortico-cortical functional connectivity (20, 21) and made them more physiological.

Invasive stimulation of peripheral nerves, exploiting the natural pathways of communication between the hand and the brain, may represent a promising strategy to achieve close-to-natural feedback (6, 1113). Nevertheless, closed-loop control of complex grasp and fine manipulation through prosthetic hands is still a core challenge.

This work shows that it is possible to recover sensorimotor integration through neural electrodes and to enable real-time closed-loop control of bionic hands in tasks of fine grasp and manipulation, by using routed sensory information. Neural electrodes implanted in a young woman with hand amputation were shown to provide close-to-natural force and slippage sensations, paramount for significantly improving the participant’s manipulative skills with the prosthesis. As it happens with physiological fast adapting units, which are sensitive to high-frequency vibrations (22), slippage has been detected through vibrations induced in the force signals recorded by sensors embedded in the prosthetic hand; hence, slippage information was packaged into spatiotemporally discrete patterns, which integrate signals across skin area and time (23), and delivered to the person by means of nerve electrical stimulation. A stick-slip model was used for deducing the slippage stimulation strategy. The model showed that slippage generates movement of the grasped object relative to the skin of the fingers that are involved in grasping. Hence, electrical stimulation was delivered sequentially to two adjacent fingers participating in grasping to produce a sensation that integrated signals across the skin area. Force and slippage information was translated into electrical stimuli; this allowed the amputee participant to actively control the grasp stability, modulate the force level, and hinder the object’s fall with a myoelectric control of the prosthesis.

Modifications over time of the stimulation parameters and the evoked sensations went in parallel with improvement of the participant’s grasping capabilities in tasks of increasing complexity up to dexterous manipulation, quantitatively assessed by means of instrumented objects and a purposely developed metrics. Closed-loop control capabilities enabled by the neural feedback were compared with those achieved without feedback. Force-and-slippage closed-loop control was replicated with both a research robotic hand prototype and a commercial prosthesis, showing that performance was independent of the adopted prosthesis.

Moreover, in parallel with manipulative skills, a major role in driving brain reshaping accompanying the use of bionic prosthesis is the achievement of a functional sensorimotor closed loop through an effective bidirectional communication in the human-machine interfacing (16). Hitherto, the reported changes in mapping and connectivity in favor of an amelioration of cortical hand processing were static pictures and were limited to either the motor or the sensory domain. Here, to extract a more dynamic index able to predict amputees’ ability to recover, we evaluated the propensity of the brain to undergo plastic changes induced by the prosthetic training through the response to plasticity-inducing protocols and explored the relative weights of pure motor versus sensorimotor induction of plasticity in one amputee controlling bidirectionally interfaced bionic prostheses.

RESULTS

A stick-slip model for validating the proposed slippage detection and encoding approach

The biological plausibility of the encoding algorithm adopted to elicit slippage sensations was investigated by developing a stick-slip model (23) of multifingered grasps (Fig. 1). The main purpose of the model was to analyze in depth the slippage mechanism in healthy participants and then deduce the slippage stimulation strategy for the amputee participant. The model (described in detail in Materials and Methods) replicates realistic conditions of multifingered grasps of an object under the effect of gravity and accounts for the load force due to the object mass (Fp) and the elastic force due to skin elasticity (Fe). Fs is the external disturbance that causes slippage. The input-output relationship allows determination of the object displacement induced by the external force Fs (given the applied normal force) or alternatively the applied normal force (given the object displacement). Slippage causes a relative movement of the object on the fingers’ skin. The estimation of the object displacement by means of the model and the measurements on the healthy participants allow evaluating the size of the skin area involved by slippage and, subsequently, defining the locations of the hand to be electrically stimulated.

Fig. 1 Stick-slip model of a multifingered grasp.

(A) The model describes the mechanism of stick-slip during grasps involving from two to five fingers. A tridigital grasp of an object is shown. Fn1 is the force applied by the thumb, whereas Fn is the resultant of the normal forces applied by all the fingers (i.e., Fn = Fn1 + Fn2+ Fn3). Fs is the external disturbance that causes slippage. When an external disturbance force Fs is applied to the spring, it will store elastic energy and an increasing force will be exerted on the object that is opposed by the frictional force Ft = Ft1 + Ft2 + Ft3. When FtFpFe + Fs, the object sticks; conversely, when Ft < FpFe + Fs, the object slips. (B) Slip occurrence and corresponding force variation. (C) The object displacement caused by disturbance Fs and computed by the stick-slip model, and the displacement measured by the sensors on the object. The difference between the measured object displacement and the computed object displacement is not statistically significant (P = 0.84). The red horizontal lines show the medians, box limits indicate the 25th and 75th percentiles, and the whiskers extend to the most extreme data points (i.e., maximum and minimum).

The reliability of the model to predict the object displacement caused by perturbation Fs was proven by an on-purpose study on 10 healthy participants performing power and precision grasps (see the Supplementary Materials). Normal forces and object displacement were measured to validate the model. Hence, given the measured normal force, the object displacement was also computed by means of Eq. 1. The study showed (Fig. 1) that the object displacement computed by the model as 26.41 ± 10.22 mm was comparable with the measured one, given as 28.12 ± 8.95 mm (P = 0.84, Wilcoxon signed-rank test). Moreover, all the participants referred that, during slippage, they felt the object flowing along the index and middle fingers because of the object displacement. This was also supported by the data because the measured object displacement was comparable with the sum of the two finger sizes in the slippage direction (i.e., 32.20 ± 4.76 mm). This achievement represented the key element of the adopted slippage stimulation strategy: i.e., slippage sensation was delivered with an electrical stimulation that flowed along the index and middle fingers in contact with the object.

Hence, for the amputee participant, slippage was detected through the induced vibrations that were found in the normal force signal (Fn) measured by the force sensors embedded in the prosthetic fingers, in a way similar to the afferent response in the natural hand (22). An on/off signal was then generated by means of the online processing algorithm of the force signal described in (24, 25). Then, the slippage information was encoded as trains of cathodic rectangular biphasic electrical current pulses with fixed parameters sequentially injected on two adjacent fingers (i.e., index and middle fingers) to deliver a sensation that integrated signals across the skin area of the fingers in contact with the object.

Figure 2 shows the total force applied by the amputee participant in power and precision grasps performed with neural feedback; it was comparable with the total force obtained by the model adapted to the amputee participant. In both cases, slippage was hindered and the object was stably grasped. The displacement of the object on the prosthesis fingers induced by the slippage was given by 26.36 ± 7.43 mm.

Fig. 2 Grasp force from the model versus the measured force from the amputee.

Closed-loop control with neural feedback in power (A) and precision (B) grasps. Sensors embedded in the object measured the normal component of the force (violet) and, after online processing, provided the slippage signal (red). The participant modulated the level of force, after feeling slippage through neural stimulation. Therefore, a stable grasp was achieved up to the end of the trial and the release of the object. The normal component of the force extracted from the model under the same perturbation condition is shown in light blue. All traces were normalized with respect to the maximum forces exerted by the hand (i.e., 7.33 N for power grasp and 3.96 N for precision grasp) and to maximum time duration (i.e., 26.90 s for power grasp and 19.35 s for precision grasp).

Real-time force-and-slippage closed-loop control

The possibility was investigated to elicit in the participant close-to-natural sensations of grasping force and slippage through invasive nerve electrical stimulation (as described above) and to use them to perform a real-time closed-loop control of force and slippage in tasks with increasing complexity (Fig. 3). The blindfolded and acoustically shielded participant was asked to grasp objects placed close to the fingers of the prosthesis under two conditions: with neural feedback and without any feedback.

Fig. 3 Real-time force-and-slippage closed-loop control of hand prosthesis with neural feedback.

The sensory output produced by a biomechatronic hand embedding force sensors was routed back through neural stimulation to evoke close-to-natural force and slippage feedback. (A) Participant’s intention was decoded by the muscular activity through sEMG sensors in the socket and a pattern recognition algorithm that classified the gesture and the force level. (B) Position and force control was implemented on a biomechatronic prosthetic hand for performing the task. (C) The hand fingers with force-sensing resistors read the applied forces and detected slippage. (D) The measured force applied to the grasped object and the detected slippage event are encoded in force and slippage stimulation patterns. (E) Force and slippage sensations are delivered to the participant by means of cuff and intraneural electrodes. (F) Photograph of the surgical intervention for implanting cuff and intraneural electrodes in ulnar and median nerves.

Four categories of tasks, ordered by increasing complexity, were tested as follows: (i) lateral grasp of large and small objects, (ii) pick and place of large objects with a power grasp, (iii) pick and place of small objects with a precision grasp, and (iv) manipulation tasks of pouring water from a bottle to a cup and shape sorter with small cylinders and discs. Complexity of the grasp was considered as it related to the extension of the contact area between the object and the hand. Lower contact area, typical of precision and manipulation tasks, requires higher manipulative skills and more efficient control of grasp stability.

Force-sensitive resistor sensors embedded in the prosthetic fingers measured the applied forces and provided a binary slippage signal through the developed slippage detection algorithm presented in detail in (24, 25). Force and slippage feedback was provided to contact numbers 10, 12, and 16 of the intraneural electrode in the median nerve that the participant referred to map on the thumb, index, and middle fingers.

The two cases of use of neural feedback and absence of any kind of feedback are shown in Fig. 4 and fig. S7. When the object is touched, force feedback is provided. In the case of neural feedback, the participant actively controls hand closing with the desired level of force by producing a variation in the electromyography (EMG) signal related to the perceived force and slippage sensations. Instead, in the case of manipulation task without feedback, the object can fall because of slippage and forces vanish accordingly, but the hand is still closed because of the absence of sensation.

Fig. 4 Real-time force-and-slippage control of a manipulation task with neural feedback.

(A) With neural feedback. The participant performed a manipulation task of shape sorter of a small cylindrical object: The pinch gesture was selected by the EMG classifier, and thumb and index fingers started moving. Once the object was touched, force feedback was provided. The slippage event was felt by the participant, who closed the hand and actively tuned the level of force by producing a variation in the EMG signal. Grasp stability was reached up to the end of the trial. Hence, the open hand gesture was classified and the hand reopened. (B) Without feedback. The participant performed a manipulation task of shape sorter of a small cylindrical object: The pinch gesture was selected by the EMG classifier, and thumb and index fingers started moving. Once the object was touched, the applied force was measured, and slippage was detected by the sensors. There was no stimulation. The amputee participant was not able to feel the detected slippage event and, consequently, the object fell. The forces vanished accordingly. At the end of the trial, the open hand gesture was classified, and the hand reopened.

Performance of closed-loop grasp control and improvement over time

The participant’s ability to grasp and manipulate objects with neural feedback and without feedback and the improvement over time were monitored after the first week of training with the closed-loop control (i.e., week 4, named T0), in the middle of the training period (i.e., week 7, named T1), and at the end of the experimental study (i.e., week 10, named T2). The participant was asked to perform 24 repetitions for each of the four aforementioned categories of tasks at each time point. The total number of trials was 96 for each observation period. Manipulative skills were assessed through the performance index named weighted success (defined in Materials and Methods). One of the key points of this work is to show the participant’s capability to stably handle the object by actively managing forces and slippage, enabled by neural feedback. This means that slip events can occur, and the task is completely unsuccessful only when the object falls. Otherwise, the success measure is decreased by the slip occurrence.

The comparative analysis of the performance achieved with neural feedback and without any feedback showed that the difference was statistically significant for lateral (P = 0.0062) and power (P = 0.015) categories at T0 and for precision (P = 0.0045) and manipulation (P = 0.0009) categories at T2 (Fig. 5). In absence of feedback, the participant showed a relevant increase in performance with learning, probably because she had not used a myoelectric prosthesis before this study. She started from very low performance at T0 and improved over time. The difference was statistically significant (i) for lateral grasp between T0 and T1 (P = 0.002) and between T0 and T2 (P = 0.002), (ii) for power grasp between T0 and T2 (P = 0.015), and (iii) for manipulation tasks between T0 and T2 (P = 0.0078).

Fig. 5 Temporal evolution of grasp performance without feedback and with neural feedback.

The participant’s grasp performance was measured through the weighted success and monitored over time. Four categories of tasks (lateral, power, precision, and manipulation) were performed at T0, T1, and T2. Mean value and SD of the weighted success index are shown for each time point. Statistical significance for the three time points is indicated with *P < 0.016 (Friedman nonparametric tests, Wilcoxon post hoc test, Bonferroni correction). Statistical significance between neural feedback (fb) and no feedback (NO fb) is indicated with +P < 0.05 (Wilcoxon signed-rank test).

In case of real-time force-and-slippage closed-loop control with neural feedback, the participant achieved high performance already from the first time point, except for manipulation (which was more complex than the other tasks). For the manipulation category, performance improved over time up to a value of 0.89 ± 0.08 at T2, which was significantly different with respect to the value achieved at T0 (P = 0.015).

Dexterity of closed-loop control via neural feedback

It was also investigated in depth the advantage of using neural feedback to improve dexterity at time T2. The following additional performance indicators were considered: (i) the force index, which provides a measure of the capability to apply the appropriate level of forces to prevent the object from falling, and (ii) the execution time, which provides the speed of task accomplishment. A comparative analysis with the case of no feedback was performed in the four categories of tasks. Twenty-four repetitions for each task category were performed. Furthermore, to assess the interoperability of the developed closed-loop control and the robustness of the achieved results, we adopted both a research prototype and a commercial prosthesis.

The participant achieved globally comparable performances with the two prosthetic hands (Fig. 6). The weighted success in absence of feedback was always lower than the success rate achieved with neural feedback (Fig. 6). For precision and manipulation tasks, the difference was statistically significant (P = 0.0045 and P = 0.0009) for the research prototype; the difference was statistically significant for manipulation tasks (P = 0.015) with the commercial hand. Similarly, the total applied force with no feedback was on average lower than with neural feedback (Fig. 6). For the research prototype, the difference became significant for manipulation tasks (P = 0.0023), whereas for the commercial prosthesis, the difference was significant for lateral grasp (P = 0.002), precision grasp (P = 0.02), and manipulation (P = 0.002). Moreover, the participant, in general, took more time to complete the tasks when no feedback was provided (Fig. 6). The difference was statistically significant in manipulation tasks (research prototype, P = 0.006; commercial hand, P = 0.015) and lateral grasp performed with the commercial hand (P = 0.019).

Fig. 6 Grasp and dexterity assessment without feedback and with neural feedback and two different prosthetic hands.

The participant’s grasp performance and dexterity were measured through the weighted success, the force index, and the execution time for the two cases of no feedback and neural feedback and two different prosthetic hands (a research prototype and a commercial hand). Statistical significance between neural feedback and no feedback is indicated with *P <0.05 (Wilcoxon signed-rank test). (A) Weighted success. (B) Force index. (C) Execution time. (D) Statistically significant differences between no feedback and neural feedback for the three indices and the two prosthetic hands. A significant improvement of grasp performance and dexterity was achieved in manipulation tasks, resulting from neural feedback, independently of the adopted prosthetic hand.

A user satisfaction questionnaire was administered to the participant at the end of the experimental study. She reported that the sensations elicited by the nerve electrical stimulation were evocative of the object slippage, very well distinguished from force sensation, and provided a quasi-realistic representation of the change of the contact area with the fingers involved in grasping (movie S1).

Mapping of elicited sensations and change over time

Three electrodes elicited sensations in the amputee participant as shown in Fig. 7. Up to time T0, the participant reported that most stimulations evoked a sense of movement (tables S1 to S4). However, after T0 and particularly when the participant began to extensively use the closed-loop control with neural feedback, the reported quality of sensations changed (Fig. 7B and tables S1 to S4). Up to time T0, 3 of 16 contacts of the intraneural electrode in the median nerve and all the contacts of the cuff electrode evoked EMG responses. No EMG responses were obtained from all the other contacts. Contacts not evoking muscle twitch changed the induced sensation from movement to touch and were used for the real-time closed-loop control.

Fig. 7 Sensation locations and quality over time.

(A) The three electrodes elicited sensations in 13 different locations of the hand on anterior and posterior parts of the hand. Red areas refer to sensations evoked by stimulation of the ulnar nerve, and yellow represents territories of sensations elicited by stimulation on the median nerve. C1 indicates the region of sensations elicited with cuff electrode of median nerve, C2 refers to the cuff on ulnar nerve, and I indicates the intraneural electrode in the median nerve. (B) Modification of the elicited sensations for the intraneural electrode on the median nerve. Up to time T0 (i.e., pre), most of the elicited sensations evoked movement (brown); after T0 (i.e., post), most of the elicited sensation evoked touch (blue). In separate series, histograms represent the cumulative percentage of stimulated contacts, considering all contacts, contacts evoking EMG activity (twitch+) and contacts evoking no EMG activity (twitch−).

Tests of sensorimotor integration

At T0, the transcranial magnetic stimulation (TMS) test of short-latency somatosensory afferent inhibition (SAI) showed a pronounced inhibition in a muscle involved in the amputation (i.e., flexor carpi ulnaris, 47%), while the amount of inhibition was closer to a physiological value in a more proximal muscle not involved in the amputation (i.e., biceps brachialis, 32%). The training period with sensory feedback induced a reduction of 16% in the overexpressed flexor carpi ulnaris SAI (from 47 to 31%) but left biceps brachialis SAI (from 32 to 34%) mostly unchanged.

Tests of sensorimotor-induced cortical plasticity

Repetitive TMS (rTMS) was used to test sensorimotor associative plasticity induced by inhibitory and facilitatory paired associative stimulation (PAS) protocols (PAS− and PAS+, respectively), measured before T0 (baseline) and after T2 (after training). The training with the sensorimotor closed-loop bionic prosthesis induced a strong disinhibition of sensorimotor cortices, demonstrated by a consistent reduction of the effect of PAS− (from −47 to 0%) and by a still ineffective facilitatory PAS+ (from 3 to 3%; Fig. 8).

Fig. 8 rTMS protocols.

Effects of rTMS protocols inducing changes in motor cortical excitability based on intracortical mechanisms (iTBS and cTBS; red bars) and on sensorimotor integration (PAS− and PAS+; blue bars), tested before T0 (light bars) and after T2 (dark bars). Values represent the percentage of changes from baseline after each rTMS protocol. Dotted lines represent changes obtained with the same rTMS protocols in control participants [data from (26)].

Tests of intramotor cortical plasticity

rTMS was used to test intramotor cortical plasticity induced by facilitatory [intermittent theta-burst stimulation (iTBS)] and inhibitory [continuous TBS (cTBS)] protocols, applied before T0 (baseline) and after T2 (after training). At baseline, cTBS increased motor evoked potential (MEP) amplitude by 66%, and iTBS reduced MEP amplitude by 34%, showing opposite effects compared with unimpaired participants (Fig. 8) (26). After training, cTBS produced only a slight facilitation (+13%), and iTBS was no longer inhibitory but produced a slight facilitation (+4%; Fig. 8).

DISCUSSION

Peripheral nerve electrodes implanted in the median and ulnar nerves were shown to recover sensorimotor integration in an amputee participant using the provided force and slippage sensations and enabled manipulative skills through real-time closed-loop control of a bionic hand.

In the first phase of this study, the stick-slip model extended to the multifingered grasp was used to predict object displacement during slippage. The results demonstrated that slippage causes a change in the contact area of the object with the fingers and that the object displacement involves two fingers, the index and middle fingers, in contact with the object. This laid the foundations of the stimulation strategy adopted for eliciting slippage sensations.

Afterward, we verified whether the elicited tactile sensations could enable a physiological control of force and slippage during grasping and manipulation tasks. The participant actively controlled position and force of the prosthesis through voluntary muscle contractions. Once deciding the type of task and position-controlling the prosthesis to accomplish the desired grasp, she was able to define the grasp force and autonomously increase or decrease it because of the elicited sensations of force and slippage and the online pattern recognition algorithm. The prosthetic hand was position-controlled during hand preshaping; it was force-controlled during grasping and manipulation. When force sensors on the prosthesis detected a slip event, a train of electrical stimuli on the fingers was delivered for the duration of the slippage. As a reaction, the muscular activity produced a new pattern aimed at increasing the applied force level and preventing the object from falling. The time spent to measure the applied force and detect slippage via the algorithm in (24, 25) was below 50 ms. The hand controller acquired a new class from the classifier, including corrective actions due to slippage, every 100 ms. This delay is fully compatible with the time delays in human sensorimotor control loops engaged in corrective actions (~100 ms) (22, 27, 28). The overall closed-loop time for the sensorimotor control observed in the amputee participant was around 500 ms—that is, the time needed to decode the intended gesture, control the hand, provide the sensory feedback, and apply a corrective action to avoid slippage.

On the other hand, when the participant was not provided with feedback, slippage was not felt, no muscular reaction was observed, and the object fell when not stably grasped. In particular, when the contact area between the hand and the object decreased (as in precision and manipulation tasks), grasp stability was more difficult to ensure, and the role of sensations became paramount (Fig. 5). Hence, in the absence of feedback, the probability of object fall increased, and consequently, grasp performance decreased. All the tests were performed in the absence of visual and auditory feedback to avoid compensatory mechanisms due to other feedback modalities. It is expected that, in a real context of everyday life with visual and auditory feedback, the improvement of the participant’s grasp and manipulation capabilities enabled by the delivered force and slippage sensory feedback could be even more evident.

A weighted success index was introduced to assess the participant’s capability to stably handle the object, to compare the neural feedback with no feedback, and to study the temporal evolution of manipulative skills. The results corroborate that sensory feedback needs time to be mastered. Figure 6 shows that grasp fundamental abilities were stable over time during the 11 weeks of experiments, whereas manipulative skills gradually increased and significantly improved by 25.7% at T2. Performance was shown to improve with continuous usage as the participant learned how to incorporate sensory feedback. Performance with neural feedback was always better than without feedback; the difference became statistically significant at T2 for precision and manipulation tasks. For fundamental grasp categories (i.e., lateral and power grasps), performance improved over time also without feedback, probably because the participant had never used a myoelectric hand before the study.

Dexterity enabled by neural feedback was investigated in depth by means of other two performance indicators: the force index and the execution time. The comparative analysis at T2 showed that neural feedback allowed achieving good manipulative skills. Performance achieved with neural feedback was always higher than that with no feedback. The participant performed all task categories with higher forces and shorter execution times when neural feedback was provided. The difference became significant for manipulation tasks. This can be newly justified by the paramount role played by sensory feedback in more complex tasks. In the power grasp, the object stability can be easily achieved also by applying lower grasp force because of the wide contact area between the prosthetic hand and the object. This explains why power and lateral grasps can also be successfully performed when no feedback is provided. Precision and manipulation tasks are characterized by a reduction of the contact area between the object and the prosthetic hand. This entails a mastered control of the applied forces to ensure stability. It seems that the neural stimulation provides a rapid and effective sensory feedback, which allows finely tuning the applied forces, in line with the spared mechanotransduction time, shortness of pathways, and lower cognitive load. The execution time is reduced accordingly.

All the results were confirmed by the tests performed also with a commercial hand. The participant was able to perform the four categories of tasks with similar performance and dexterity, thus proving interoperability of the system and robustness of the achieved results.

The sensorimotor closed-loop training performed by the participant induced a normalization of the quality of afferent stimuli, evolving from movement sensations to tactile sensations. This could be caused by a reeducation of central processing of the stimuli. In favor of a sensory-driven amelioration of sensorimotor central processing, there was also the very strong and consistent effect seen in the participant for sensorimotor-induced plasticity, in line with the sensorimotor closed-loop abilities of our system. To establish plasticity and relative weights of motor-motor drive or sensorimotor drive, we exploited specific rTMS neuromodulatory interventions, mostly relying on different neurophysiologic mechanisms: TBS for the former and PAS for the latter (2932). Plastic changes induced by rTMS protocols in the motor cortex contralateral to amputation were investigated before T0 and after T2. The training induced a reduction of facilitation of pure motor intervention (cTBS) and a reduction in the inhibition induced by sensorimotor-driven plasticity (PAS−). Coherently, the level of cortical afferent inhibition measured with SAI testing was also reduced.

Overall, the observed effects indicated that intramotor cortical plasticity became closer to the level observed in normal participants (8, 16). However, the training did not induce a hyperexpressed intramotor cortical plasticity, decreasing the likelihood of a pure motor origin of the achieved improvement of performance in our participant. On the contrary, inhibitory cortical phenomena related to integration of afferent information, expressed by either cortical afferent inhibition or cortical associative plasticity, are strongly reduced with sensory training. With the obvious caution that our findings come from the analysis of a single case, the above data strongly support that the increased efficacy of afferent information is the main factor responsible for favoring motor learning during the training.

MATERIALS AND METHODS

Study design

To stimulate deep and surface nerve fibers, we implanted a combination of cuff and intraneural electrodes in the median and ulnar nerves of an amputee’s residuum for 11 weeks. The sensory output produced by a biomechatronic hand during grasping and manipulation tasks was relayed through neural stimulation to the participant to evoke real-time close-to-natural force and slippage feedback related to her missing hand.

To show the biological plausibility of the slippage encoding algorithm, we developed a stick-slip model of multifingered grasps that is described below. The model was used to analyze in depth the slippage mechanism in healthy participants and deduce the slippage stimulation strategy for the amputee participant. Ten healthy participants were recruited to validate the model (see the Supplementary Materials). On the basis of these observations, the strategy for encoding slippage information by means of neural electrical stimulation in an amputee participant was defined.

In the experimental study with the amputee participant, closed-loop force-and-slippage control based on the elicited sensations was carried out in four categories of tasks with increasing complexity: (i) lateral grasp of large and small objects, (ii) pick and place of large objects with a power grasp, (iii) pick and place of small objects with a precision grasp, and (iv) manipulation tasks of pouring water from a bottle to a cup and shape sorter with small cylinders and discs. The four categories involved the following:

1) Different fingers (from two to five) in lateral, precision, and power grasp configurations;

2) Grasping tasks (e.g., pick and place) and manipulation tasks (e.g., pouring, shape, and sorter);

3) Objects of different shapes (e.g., cylinder, parallelepiped, disk, cube, and triangle), volume (2.54 × 103 mm3 ÷ 2.65 × 106 mm3), and weight (18.28 g ÷ 198.65 g).

A comparative analysis with the case of no feedback was carried out. Twenty-four repetitions for each task category were performed under two different conditions: (i) no feedback (n = 96 trials) and (ii) force and slippage feedback through neural stimulation (n = 96 trials). Moreover, improvement of grasping and manipulation capabilities was monitored over time at three time points (i.e., T0, T1, and T2). Grasping performance was measured through three performance indicators.

The same set of trials was repeated at T2 with two different biomechatronic prostheses (n = 384 trials) to demonstrate that the results are general and independent of the prostheses used. During the experiments, the participant was blindfolded and acoustically isolated. In this way, with neither vision nor auditory input, compensatory mechanisms due to other feedback modalities were avoided, and the improvement of sensorimotor performance entailed only by the delivered force and slippage feedback was assessed.

Last, a neurophysiological assessment was carried out. It investigated whether the described amelioration of motor performance in dexterous tasks had as central neurophysiological correlates changes in motor cortical plasticity and whether such changes were more likely of purely motor origin, thus relaying on intramotor cortical activity, or rather the effect of a strong and persistent drive of the sensory feedback.

Participant recruitment

This study was conducted at Campus Bio-Medico University Hospital of Rome in accordance with the Declaration of Helsinki and following amendments and was approved by the local ethics committee and the assigned office of the Italian Ministry of Health. The volunteer participant signed an informed consent form.

The enrolled participant was exposed to an explosion that produced a transradial left upper limb amputation almost 30 years before. She is a right-handed female and 40 years old at the time of the experiment. She demonstrated good intellective abilities and comprehension. The surgical procedure for implanting neural electrodes is described in the Supplementary Materials.

Experimental setup

The bionic system was composed of commercial devices and research prototypes. The following two biomechatronic hands were used: the IH2 Azzurra (Prensilia s.r.l.) and the RoboLimb (Touch Bionics s.r.l.). They were equipped with force-sensing resistors (Interlink Electronics Inc.) for measuring normal forces between fingers and objects and for detecting slippage, owing to the algorithm in (24). This detects the vibrations in the force signal due to sliding movements. A custom-made socket was developed by INAIL (Italian Institute for Insurance against Accidents at Work) Prosthetic Center.

The tactile sensation was restored electrically by stimulating median and ulnar nerves via intraneural (ds-FILEs) and cuff electrodes. The ds-FILE is characterized by 16 active contacts and 2 ground electrodes arranged on both sides of the structure (33). The wrap cuff electrode (Ardiem Medical Inc.) is made of a total of 14 active contacts and 2 ground channels, distributed on four rings. The Multi Channel System STG4008 stimulator was connected to the electrodes, owing to a custom-developed hub facilitating the channel selection.

For decoding muscular activity, six commercial active surface EMG (sEMG) sensors (Ottobock 13E200 = 50; 27 mm by 18 mm by 9.5 mm) were embedded into the socket. The data were sampled at 1-kHz frequency with a 12-bit resolution. The participant was instructed to reproduce with her phantom limb one of five gestures—“rest” (relaxed hand), “power” (hand with all fingers closed), “pinch” (hand precision grasp with two fingers), “open” (hand with all fingers opened), and “lateral” (hand lateral configuration)—and three levels of forces—high, medium, and low. Once the level of force for a given gesture was classified, the corresponding grasping force was applied by the prosthetic hand. Force-sensing resistors embedded in the prosthetic fingers measured the applied force and checked slippage. If slippage was detected, then a slippage sensation was delivered to the participant, who could apply a corrective action. To this purpose, the grasping force was proportionally varied with the EMG signal to promptly oppose slippage and prevent the object from falling. The raw filtered sEMG signals were taken as input features to a pattern recognition algorithm based on a nonlinear logistic regression algorithm (fig. S6), as described in (34). The classifier ran on an embedded system that relied on an ARM4 32-bit NXP microcontroller with a 128-kB flash memory and 100-MHz clock frequency.

Stick-slip model of multifingered grasp

The proposed model is an extension of the stick-slip model in the literature to the more realistic situation of a multifingered grasp of an object under gravity conditions. In Fig. 1, Fn1 is the force applied by the thumb, and Fn is the resultant of the normal forces applied by all the fingers (i.e., Fn = Fn1 + Fn2 + Fn3 in Fig. 1). Force tangential components (Ft1, Ft2, and Ft3) are related to the normal components through the coefficient of kinetic friction μ. Fp accounts for the load force, Fe is the elastic force generated by the skin elasticity, and Fs is the external disturbance that causes slippage; it is modeled as a step function (Fs = Fu(t)). Skin elasticity is modeled with a spring.

When a force Fs is applied to the spring, it will store elastic energy, and an increasing force will be exerted on the object that is opposed by the frictional force Ft = Ft1 + Ft2 + Ft3. When FtFpFe + Fs, the object sticks; on the other hand, when Ft < FpFe + Fs, the object slips. The equilibrium can be written asEmbedded Imagewith m being the mass of the object and Embedded Image the object acceleration. The object displacement induced by the external force Fs can be computed asEmbedded Image(1)whereEmbedded Imageand k is the skin stiffness.

Electrical stimulation for sensory feedback

Force sensation was elicited by means of a train of three cathodic rectangular biphasic pulses with fixed frequency of 50 Hz and a fixed pulse width of 80 μs. The current amplitude was directly proportional to the voltage provided by the force sensors asEmbedded Imagewhere V was the readout of the force-sensing resistor, I was the current amplitude, Ilow and Ihigh were the lowest and the highest current values tolerable by the participant, and Vlow and Vhigh were the corresponding voltage outputs of the sensors (i.e., −4.15 and −3.5 V).

Slippage sensations were elicited through a train of three cathodic rectangular biphasic pulses with fixed current amplitude (150 μA), frequency (50 Hz), and pulse width (80 μs). The slippage information was encoded as sequences of trains of three cathodic rectangular biphasic electrical current pulses with fixed parameters sequentially injected on index and middle fingers for the duration of the slippage event.

Grasp assessment

During the closed-loop control, sensors embedded into the hand fingers were used to measure force and slippage and produce a feedback signal for the amputee participant. On the other hand, objects were instrumented with force-sensing resistors for grasp assessment. Three performance indicators—named weighted success, force index, and execution time—were introduced.

The weighted success is a normalized measure of the task success rate and is expressed as the task success modulated by the number of occurred slippage events and normalized over the maximum number of slip events detected with the same feedback condition (see the Supplementary Materials). It ranges in the interval [0, 1], where 0 is a failed trial and 1 is a successful trial with no slippage.

The force index, expressed in newtons, measures the total force applied by the fingers involved in the grasping or manipulation task. The execution time is the time spent for performing the task.

Neurophysiological assessment

Tests of sensorimotor integration

The effect of sensory afferent stimulation on motor cortical excitability was tested by combining electrical nerve stimulation with TMS of M1. The SAI paradigm was used (35), in which an electrical stimulus of the ulnar nerve, delivered immediately above the elbow, was followed by a TMS pulse over M1 at an interstimulus interval (ISI) exceeding the latency of the cortical somatosensory potential evoked by stimulation of the same nerve by 2, 3, or 4 ms.

For the test condition (M1 TMS alone) and for the three SAI ISIs, 10 MEPs were collected and averaged from both flexor carpi ulnaris and biceps brachialis muscles. For each ISI, SAI was then expressed as the percentage of reduction of the average MEP compared with the average MEP evoked by the test condition.

Tests of motor cortical plasticity

Motor cortical plasticity was assessed by means of rTMS protocols at two time points (before T0 and after T2). Motor cortical excitability was assessed before and immediately after each rTMS protocol by single-pulse TMS, as described above. MEPs to 15 magnetic stimuli of M1 for each condition were collected and averaged from the biceps brachialis muscle.

rTMS was applied to the right primary motor cortex (contralateral to the amputated limb) using a DuoMAG XT-100 magnetic stimulator (Deymed Diagnostic, Czech Republic), producing a biphasic magnetic pulse. Four different rTMS protocols were used on the basis of the two general paradigms of TBS (36) and of PAS (37): (i) the cTBS, in which three pulses of stimulation were given at 50 Hz, repeated every 200 ms, for a total of 600 pulses; (ii) the iTBS, in which three pulses of stimulation were given at 50 Hz, repeated every 200 ms, and with a pause of 8 s every 2 s of stimulation, for a total of 600 pulses; (iii) PAS−, in which 90 pairs were delivered and each pair was made of a peripheral nerve electrical stimulation, followed by TMS of the motor cortex with an ISI of 10 ms shorter than the latency of the cortical somatosensory evoked potential; (iv) PAS+, in which 90 pairs were delivered and each pair was made of a peripheral nerve electrical stimulation, followed by TMS of the motor cortex with an ISI exceeding the latency of the cortical somatosensory evoked potential by 5 ms. Such different rTMS-based protocols were chosen to explore, in a selective way, alternative possible mechanisms that result in cortical plasticity. The changes of cortical excitability that can be seen after TBS protocols are due to the induction of synaptic plasticity in synapses between neurons all located within the motor cortex; thus, TBS-induced changes of MEP can be considered as a proxy of pure motor cortical plasticity. Alternatively, repetitive stimulation of PAS protocols produces changes in cortical excitability due to synaptic plasticity in synapses between sensory and motor neurons; thus, they can be considered a proxy for the part of motor cortical plasticity driven by the sensory system or, that is, sensorimotor associative plasticity.

SUPPLEMENTARY MATERIALS

robotics.sciencemag.org/cgi/content/full/4/27/eaau9924/DC1

Materials and Methods

Fig. S1. Median and ulnar nerve.

Fig. S2. Intraneural electrode sutured to epineurium.

Fig. S3. Cuff electrode.

Fig. S4. Percutaneous cables.

Fig. S5. Threshold charge over 11 weeks in the thumb, index, and middle fingers.

Fig. S6. Classification performance of the EMG pattern recognition algorithm.

Fig. S7. Real-time force-and-slippage closed-loop control of a power grasp.

Table S1. Percept qualities evoked by electrical stimulation of the cuff electrode on median nerve before T0.

Table S2. Percept qualities evoked by electrical stimulation of the cuff electrode on ulnar nerve before T0.

Table S3. Percept qualities evoked by electrical stimulation of the ds-FILE intraneural electrode on median nerve before T0.

Table S4. Percept qualities evoked by electrical stimulation of the ds-FILE intraneural electrode on median nerve after T0.

Movie S1. Restoring tactile sensations.

REFERENCES AND NOTES

Acknowledgments: We are grateful to participant C.P. for special commitment to this study, patience, and dedication to the 11 weeks of experiments. Funding: This work was supported by INAIL with PPR2 project “Control of upper-limb prosthesis with neural invasive interfaces” (CUP:E58C13000990001) and PPR AS 1/3 project “Implantable System for the control of an upper limb prosthesis with invasive wireless neural interfaces” (CUP:E57B16000160005) and also by the European Research Council (ERC) Starting Grant 2015 RESHAPE “REstoring the Self with embodiable Hand ProsthesEs” (ERC-2015-STG, project no. 678908). Author contributions: L.Z. and G.D.P. designed the study, performed the experiments, analyzed the data, and wrote the paper. A.L.C. developed the overall system integration, collaborated during the design of the study, performed the experiments, analyzed the data, and wrote the paper. F.R. collaborated during the enrollment of the participant, performed the experiments, analyzed the data, and wrote the paper. F.C. collaborated during the integration of all the components of the device, performed the experiments, analyzed the data, and wrote the paper. E.N., C.G., R.A.R., and A.D.B. developed the software, collaborated during the integration of all the components of the device, performed the experiments, and analyzed the data. G.V. and L.D.-B. collaborated during the surgery and during the experiments and wrote the paper. S.M. collaborated during the enrollment of the participant and during the experiments and provided occupational therapy support. A.M. collaborated during the experiments. M.B. collaborated during the experiments and provided occupational therapy support. K.-P. H. and A.S. developed the ds-FILE electrodes. L.D. performed the surgery. A.D., R.S., and S.C. collaborated during the experiments. E. Gruppioni developed the pattern recognition software and collaborated during the experiments. S.S. and V.D.L designed the study and collaborated during the experiments. V.D. selected the participant, performed the surgery, supervised the experiments, and wrote the paper. E. Guglielmelli designed the study, supervised the experiments, and wrote the paper. All authors discussed the results and commented on the manuscript. Competing interests: L.Z., R.A.R., and E. Guglielmelli are inventors on the patent entitled “Method for automatic detection of phenomena of mutual sliding between two surfaces” (no. 102016000105302, patent pending). All other authors declare that they have no competing interests. Data and materials availability: All data needed to evaluate the conclusions are in the paper or the Supplementary Materials.
View Abstract

Stay Connected to Science Robotics

Navigate This Article