Research ArticlePROSTHETICS

A closed-loop hand prosthesis with simultaneous intraneural tactile and position feedback

See allHide authors and affiliations

Science Robotics  20 Feb 2019:
Vol. 4, Issue 27, eaau8892
DOI: 10.1126/scirobotics.aau8892
  • Fig. 1 Overview of the multimodal sensory feedback experimental setup.

    (Left) Bottom: The robotic hand is driven using sEMG activity acquired from the participant’s forearm muscles and classified into distinct motor commands. As the robotic hand closes its fingers around an object, both pressure and position are measured in real time. Top: Information about pressure and position is then encoded into stimulation pulses, where stimulation amplitude is directly proportional to finger position or pressure. Pressure perception is restored using a somatotopic approach, where the induced sensation corresponds to the fingers being touched. Position information (remapped proprioception) is restored using sensory substitution, whereas the sensation does not correspond to the natural area. (Right) Top: Both sensory streams are delivered using intraneural stimulation through TIME electrodes implanted in the median and ulnar nerves. Bottom: The TIME implant is inserted transversally through the exposed nerve fascicles.

  • Fig. 2 TDPM and JAR tasks.

    (A) The TDPM is reported for each prosthesis position tested. The median is reported as a dashed line. A histogram of the data, with bin sizes of 3°, is shown on the right. A total of 244 measures were collected with two participants (115 for participant 1 and 129 for participant 2). (B) The robotic fingers’ range of motion and the way the angle are reported. (C) JAR precision during the fixed position reproduction task for four target positions. The reproduced positions are reported as median (full, colored line) and IQR (shaded area). The MAD for the pooled performance on all positions is reported for each participant. (D) Box plots reporting the detailed absolute deviation for each requested position. The median is reported as a blue line, and the box represents the IQR. The whiskers encompass all data samples (no outliers removed). A total of 80 (40 for participant 1 and 40 for participant 2) repetitions were collected for the task. Asterisks indicate conditions found to be statistically different after a Kruskall-Wallis test with multigroup correction. (E) A box plot showing the absolute deviation around the median for randomly switched prosthesis actuation speeds (three speeds). For this task, only participant 2 participated, and a total of 48 repetitions were performed. (F) Scatterplot of the measured error in JAR for each position tested during the JAR task with random and continuous positions. A histogram of the data, with bin sizes of 3°, is shown on the right hand size. A total of 171 measures were collected with two participants (81 for participant 1 and 90 for participant 2).

  • Fig. 3 Identification of object size.

    (A) Schematic representation of the four different objects used during the object size identification task and their labeling (not to scale). (B) Overall performance during the task with remapped proprioception only for both participants in the form of a confusion matrix (left) and performance in identifying each object (right). Median correct identifications and a 95% CI for each object are reported alongside the matrix. Asterisks identify levels that were statistically different from chance level. A total of 160 repetitions (40 for participant 1 and 120 for participant 2) were performed with two amputee participants. (C) Overall performance during the object size recognition task with simultaneous touch and remapped proprioceptive feedback in the form of a confusion matrix. A total of 100 repetitions were performed with participant 2. (D) Representative position traces obtained during the experiments. One example was chosen for each cylinder size to illustrate the difference in measured position obtained in each case. In addition, the stimulation amplitude computed from the position is reported on the second y axis. (E) Overall performance for each tested hand actuation speed during a control trial with changing speeds. A total of 96 repetitions were performed with participant 2. (F) The performance obtained during a control condition where only touch feedback was delivered. In this case, 100 repetitions were performed with participant 2.

  • Fig. 4 Identification of object size and compliance.

    (A) Schematic representation of the four different objects used during the object size and compliance task and how they were labeled (not to scale). (B) Overall task performance with both remapped proprioception and touch, for both participants, reported as a confusion matrix. The combined performance is shown under the image. Median correct identifications and a 95% CI for each object feature (size and compliance) are reported alongside the matrix. Asterisks identify levels that were statistically different from chance level. A total of 220 repetitions were performed with two participants (40 for participant 1 and 180 for participant 2). (C) Performance during the same object size and compliance task when only remapped proprioceptive feedback is provided. A total of 80 repetitions were performed with participant 2. (D) Representative force and position traces, as measured by the robotic hand, for each object type. The full lines represent hand position (0° to 110°), and the dashed lines represent measured force (normalized). The four patterns are not contiguous (illustrated by dashed lines), but the relative duration of each pattern is conserved to allow meaningful comparison of the slopes. (E) Performance during the same task when only touch feedback is provided. A total of 80 repetitions were performed with participant 2. (F) Compliance decoding performance broken down by object, with touch only, remapped proprioception only, or both sensory modalities. Compliance decoding performances above chance level are shown with an asterisk. A total of 380 repetitions were used for this panel [combination of data from (B), (C), and (E)].

  • Fig. 5 Comparison of invasive and superficial sensory substitution on TDPM.

    (A) TDPM values are reported for all tested starting positions. Median TDPM is reported as a dashed line. A histogram of the data, with bin sizes of 1.5°, is shown on the right. A total of 65 trials are shown. (B) A comparison of TDPM task metrics between the two approaches used for sensory substitution (invasive and superficial electrical stimulation). Both median TDPM (shown with IQR) and number of direction detection errors are shown (along with the 95% CI). A total of 65 TDPM trials were performed using noninvasive stimulation, and 129 trials were performed using intraneural stimulation.

  • Fig. 6 Comparison of invasive and superficial sensory substitution during functional tasks.

    (A) Schematic representation of the experimental setup. Top left shows the normal object size task, where all fingers wrap around four objects of different sizes. Bottom left shows the multijoint object size task, where the first three fingers, and the last two, come into contact with different parts of the objects, which can be of different sizes. The right side shows the object size and compliance task. In this case, two noninvasive approaches were used (TSPS and TIPS). In both cases, size (proprioception) was encoded using noninvasive sensory substitution. In the case of TSPS, tactile feedback was also encoded using noninvasive sensory substitution. In the case of TIPS, a hybrid approach was used, where tactile feedback was encoded using intraneural electrical stimulation leading to somatotopic tactile sensations. Also shown are the abbreviations used for the various objects. (B) Overall performance during the object size task performed with noninvasive sensory substitution. Identification performance broken down by object is reported with 95% CIs, and asterisks identify levels that were statistically different from chance level. Overall (median) correct identifications are reported underneath the plots. A total of 80 trials were recorded for this task. (C) Performance of the multijoint object size task displayed using the same data presentation structure as in (B). A total of 80 trials were recorded for this task. (D) The overall performance during the size and compliance task using the TSPS encoding, shown as the percentage of correct identification broken down by object (shown with 95% CIs). Underneath, the overall task performance is reported. A total of 160 trials were recorded for this task. (E) The overall performance during the size and compliance task using the TIPS encoding displayed using the same data presentation structure as in (D). A total of 160 trials were recorded for this task. (F) Comparisons of overall task performances for the object size (left), the multijoint object size (middle), and the object size and compliance (right) tasks using either superficial or intraneural stimulation as a source of sensory substitution.

Supplementary Materials

  • robotics.sciencemag.org/cgi/content/full/4/27/eaau8892/DC1

    Fig. S1. Reported referred sensations and stimulation parameters.

    Fig. S2. TDPM and JAR performance broken down by participant.

    Fig. S3. Object size and compliance recognition broken down by participant.

    Fig. S4. Performance of healthy controls during object identification tasks.

    Fig. S5. Control condition and time progression of object recognition tasks.

    Fig. S6. Multijoint proprioceptive task performance.

    Fig. S7. Functional tasks performed under increased cognitive load.

    Fig. S8. General overview of the two sensory substitution approaches compared in this study.

    Fig. S9. Comparison of invasive and superficial sensory substitution on embodiment.

    Movie S1. Sensing object size (participant 1).

    Movie S2. Sensing object size (participant 2).

  • Supplementary Materials

    The PDF file includes:

    • Fig. S1. Reported referred sensations and stimulation parameters.
    • Fig. S2. TDPM and JAR performance broken down by participant.
    • Fig. S3. Object size and compliance recognition broken down by participant.
    • Fig. S4. Performance of healthy controls during object identification tasks.
    • Fig. S5. Control condition and time progression of object recognition tasks.
    • Fig. S6. Multijoint proprioceptive task performance.
    • Fig. S7. Functional tasks performed under increased cognitive load.
    • Fig. S8. General overview of the two sensory substitution approaches compared in this study.
    • Fig. S9. Comparison of invasive and superficial sensory substitution on embodiment.

    Download PDF

    Other Supplementary Material for this manuscript includes the following:

    • Movie S1 (.mp4 format). Sensing object size (participant 1).
    • Movie S2 (.mp4 format). Sensing object size (participant 2).

    Files in this Data Supplement:

Stay Connected to Science Robotics

Navigate This Article