Research ArticlePROSTHETICS

Restoring tactile sensations via neural interfaces for real-time force-and-slippage closed-loop control of bionic hands

See allHide authors and affiliations

Science Robotics  20 Feb 2019:
Vol. 4, Issue 27, eaau9924
DOI: 10.1126/scirobotics.aau9924
  • Fig. 1 Stick-slip model of a multifingered grasp.

    (A) The model describes the mechanism of stick-slip during grasps involving from two to five fingers. A tridigital grasp of an object is shown. Fn1 is the force applied by the thumb, whereas Fn is the resultant of the normal forces applied by all the fingers (i.e., Fn = Fn1 + Fn2+ Fn3). Fs is the external disturbance that causes slippage. When an external disturbance force Fs is applied to the spring, it will store elastic energy and an increasing force will be exerted on the object that is opposed by the frictional force Ft = Ft1 + Ft2 + Ft3. When FtFpFe + Fs, the object sticks; conversely, when Ft < FpFe + Fs, the object slips. (B) Slip occurrence and corresponding force variation. (C) The object displacement caused by disturbance Fs and computed by the stick-slip model, and the displacement measured by the sensors on the object. The difference between the measured object displacement and the computed object displacement is not statistically significant (P = 0.84). The red horizontal lines show the medians, box limits indicate the 25th and 75th percentiles, and the whiskers extend to the most extreme data points (i.e., maximum and minimum).

  • Fig. 2 Grasp force from the model versus the measured force from the amputee.

    Closed-loop control with neural feedback in power (A) and precision (B) grasps. Sensors embedded in the object measured the normal component of the force (violet) and, after online processing, provided the slippage signal (red). The participant modulated the level of force, after feeling slippage through neural stimulation. Therefore, a stable grasp was achieved up to the end of the trial and the release of the object. The normal component of the force extracted from the model under the same perturbation condition is shown in light blue. All traces were normalized with respect to the maximum forces exerted by the hand (i.e., 7.33 N for power grasp and 3.96 N for precision grasp) and to maximum time duration (i.e., 26.90 s for power grasp and 19.35 s for precision grasp).

  • Fig. 3 Real-time force-and-slippage closed-loop control of hand prosthesis with neural feedback.

    The sensory output produced by a biomechatronic hand embedding force sensors was routed back through neural stimulation to evoke close-to-natural force and slippage feedback. (A) Participant’s intention was decoded by the muscular activity through sEMG sensors in the socket and a pattern recognition algorithm that classified the gesture and the force level. (B) Position and force control was implemented on a biomechatronic prosthetic hand for performing the task. (C) The hand fingers with force-sensing resistors read the applied forces and detected slippage. (D) The measured force applied to the grasped object and the detected slippage event are encoded in force and slippage stimulation patterns. (E) Force and slippage sensations are delivered to the participant by means of cuff and intraneural electrodes. (F) Photograph of the surgical intervention for implanting cuff and intraneural electrodes in ulnar and median nerves.

  • Fig. 4 Real-time force-and-slippage control of a manipulation task with neural feedback.

    (A) With neural feedback. The participant performed a manipulation task of shape sorter of a small cylindrical object: The pinch gesture was selected by the EMG classifier, and thumb and index fingers started moving. Once the object was touched, force feedback was provided. The slippage event was felt by the participant, who closed the hand and actively tuned the level of force by producing a variation in the EMG signal. Grasp stability was reached up to the end of the trial. Hence, the open hand gesture was classified and the hand reopened. (B) Without feedback. The participant performed a manipulation task of shape sorter of a small cylindrical object: The pinch gesture was selected by the EMG classifier, and thumb and index fingers started moving. Once the object was touched, the applied force was measured, and slippage was detected by the sensors. There was no stimulation. The amputee participant was not able to feel the detected slippage event and, consequently, the object fell. The forces vanished accordingly. At the end of the trial, the open hand gesture was classified, and the hand reopened.

  • Fig. 5 Temporal evolution of grasp performance without feedback and with neural feedback.

    The participant’s grasp performance was measured through the weighted success and monitored over time. Four categories of tasks (lateral, power, precision, and manipulation) were performed at T0, T1, and T2. Mean value and SD of the weighted success index are shown for each time point. Statistical significance for the three time points is indicated with *P < 0.016 (Friedman nonparametric tests, Wilcoxon post hoc test, Bonferroni correction). Statistical significance between neural feedback (fb) and no feedback (NO fb) is indicated with +P < 0.05 (Wilcoxon signed-rank test).

  • Fig. 6 Grasp and dexterity assessment without feedback and with neural feedback and two different prosthetic hands.

    The participant’s grasp performance and dexterity were measured through the weighted success, the force index, and the execution time for the two cases of no feedback and neural feedback and two different prosthetic hands (a research prototype and a commercial hand). Statistical significance between neural feedback and no feedback is indicated with *P <0.05 (Wilcoxon signed-rank test). (A) Weighted success. (B) Force index. (C) Execution time. (D) Statistically significant differences between no feedback and neural feedback for the three indices and the two prosthetic hands. A significant improvement of grasp performance and dexterity was achieved in manipulation tasks, resulting from neural feedback, independently of the adopted prosthetic hand.

  • Fig. 7 Sensation locations and quality over time.

    (A) The three electrodes elicited sensations in 13 different locations of the hand on anterior and posterior parts of the hand. Red areas refer to sensations evoked by stimulation of the ulnar nerve, and yellow represents territories of sensations elicited by stimulation on the median nerve. C1 indicates the region of sensations elicited with cuff electrode of median nerve, C2 refers to the cuff on ulnar nerve, and I indicates the intraneural electrode in the median nerve. (B) Modification of the elicited sensations for the intraneural electrode on the median nerve. Up to time T0 (i.e., pre), most of the elicited sensations evoked movement (brown); after T0 (i.e., post), most of the elicited sensation evoked touch (blue). In separate series, histograms represent the cumulative percentage of stimulated contacts, considering all contacts, contacts evoking EMG activity (twitch+) and contacts evoking no EMG activity (twitch−).

  • Fig. 8 rTMS protocols.

    Effects of rTMS protocols inducing changes in motor cortical excitability based on intracortical mechanisms (iTBS and cTBS; red bars) and on sensorimotor integration (PAS− and PAS+; blue bars), tested before T0 (light bars) and after T2 (dark bars). Values represent the percentage of changes from baseline after each rTMS protocol. Dotted lines represent changes obtained with the same rTMS protocols in control participants [data from (26)].

Supplementary Materials

  • robotics.sciencemag.org/cgi/content/full/4/27/eaau9924/DC1

    Materials and Methods

    Fig. S1. Median and ulnar nerve.

    Fig. S2. Intraneural electrode sutured to epineurium.

    Fig. S3. Cuff electrode.

    Fig. S4. Percutaneous cables.

    Fig. S5. Threshold charge over 11 weeks in the thumb, index, and middle fingers.

    Fig. S6. Classification performance of the EMG pattern recognition algorithm.

    Fig. S7. Real-time force-and-slippage closed-loop control of a power grasp.

    Table S1. Percept qualities evoked by electrical stimulation of the cuff electrode on median nerve before T0.

    Table S2. Percept qualities evoked by electrical stimulation of the cuff electrode on ulnar nerve before T0.

    Table S3. Percept qualities evoked by electrical stimulation of the ds-FILE intraneural electrode on median nerve before T0.

    Table S4. Percept qualities evoked by electrical stimulation of the ds-FILE intraneural electrode on median nerve after T0.

    Movie S1. Restoring tactile sensations.

  • Supplementary Materials

    The PDF file includes:

    • Materials and Methods
    • Fig. S1. Median and ulnar nerve.
    • Fig. S2. Intraneural electrode sutured to epineurium.
    • Fig. S3. Cuff electrode.
    • Fig. S4. Percutaneous cables.
    • Fig. S5. Threshold charge over 11 weeks in the thumb, index, and middle fingers.
    • Fig. S6. Classification performance of the EMG pattern recognition algorithm.
    • Fig. S7. Real-time force-and-slippage closed-loop control of a power grasp.
    • Table S1. Percept qualities evoked by electrical stimulation of the cuff electrode on median nerve before T0.
    • Table S2. Percept qualities evoked by electrical stimulation of the cuff electrode on ulnar nerve before T0.
    • Table S3. Percept qualities evoked by electrical stimulation of the ds-FILE intraneural electrode on median nerve before T0.
    • Table S4. Percept qualities evoked by electrical stimulation of the ds-FILE intraneural electrode on median nerve after T0.

    Download PDF

    Other Supplementary Material for this manuscript includes the following:

    • Movie S1 (.mp4 format). Restoring tactile sensations.

    Files in this Data Supplement:

Stay Connected to Science Robotics

Navigate This Article