Research ArticleSENSORS

Prosthesis with neuromorphic multilayered e-dermis perceives touch and pain

See allHide authors and affiliations

Science Robotics  20 Jun 2018:
Vol. 3, Issue 19, eaat3818
DOI: 10.1126/scirobotics.aat3818
  • Fig. 1 Prosthesis system diagram.

    Tactile information from object grasping is transformed into a neuromorphic signal through the prosthesis controller. The neuromorphic signal is used to transcutaneously stimulate peripheral nerves of an amputee to elicit sensory perceptions of touch and pain.

  • Fig. 2 Multilayered e-dermis design and characterization.

    (A) The multilayered e-dermis is made up of conductive and piezoresistive textiles encased in rubber. A dermal layer of two piezoresistive sensing elements is separated from the epidermal layer, which has one piezoresistive sensing element, with a 1-mm layer of silicone rubber. The e-dermis was fabricated to fit over the fingertips of a prosthetic hand. (B) The natural layering of mechanoreceptors in healthy glabrous skin makes use of both RA and SA receptors to encode the complex properties of touch. Free nerve endings (nociceptors) that are primarily responsible for conveying the sensation of pain in the fingertips are also present in the skin. (C) The prosthesis with e-dermis fingertip sensors grasps an object. (D) The epidermal layer of the multilayered e-dermis design is more sensitive and has a larger change in resistance compared with the dermal layer. (E) Differences in sensing layer outputs are captured during object grasping and can be used for adding dimensionality to the tactile signal.

  • Fig. 3 Sensory feedback and perception.

    (A) Median and ulnar nerve sites on the amputee’s residual limb and the corresponding regions of activation in the phantom hand due to TENS. Psychophysical experiments quantified the perception of the nerve stimulation including (B) detection and (C) discrete frequency discrimination thresholds. In both cases, the stimulation amplitude was held at 1.4 mA. (D) The perception of the nerve stimulation was largely a tactile pressure on the activated sites of the phantom hand, although sensations of electrical tingling also occurred. (E) The quantification of pain from nerve stimulation shows that the most noxious sensation is perceived at higher stimulation pulse widths with frequencies in the range of 10 to 20 Hz. (F) Contralateral somatosensory cortex activation during nerve stimulation shows relevant cortical representation of sensory perception in the amputee participant (movie S1).

  • Fig. 4 E-dermis and neuromorphic tactile response from different objects.

    (A) Three different objects, with equal width but varying curvature, were used to elicit tactile responses from the multilayered e-dermis. (B) Pressure heatmap from the fingertip sensor on a prosthetic hand during grasping of each object and (C) corresponding pressure profile for each of the sensing layers. (D) The pressure profiles were converted to the input current, I, for the Izhikevich neuron model for sensory feedback to the amputee user (movie S2). Note the highly localized pressure during the grasping of object 3 and the resulting nociceptor neuromorphic stimulation pattern, which is realized through changes in stimulation pulse width and the neuromorphic model parameters.

  • Fig. 5 Prosthesis grasping and control.

    To demonstrate the ability of the prosthesis to determine safe (innocuous) or unsafe (painful) objects, we performed the PDT. The objects were (A) object 1, (B) object 2, and (C) object 3, each of which is defined by their curvature. In the case of a painful object (object 3), the prosthesis detected the sharp pressure and released its grip through its pain reflex (movie S3).

  • Fig. 6 Tactile features for prosthesis perception.

    To determine which object is being touched during grasping, we implemented LDA to discriminate between the independent classes. As input features into the algorithm, we used (A) sensor pressure values, (B) the rate of change of the pressure signal, and (C) the number of active sensing elements during loading.

  • Fig. 7 Real-time prosthesis pain perception.

    (A) LDA classifier’s accuracy across the various conditions and (B) percentage of trials where the prosthesis perceived pain during the online PDT. Note the high percentage of detected pain during the PDT for object 3. (C) Pain reflex time of the prosthesis, using the rate of change of the pressure signal to determine object contact and release, compared with previously published data of pain reflex time in healthy adults (28).

  • Fig. 8 Innocuous (mechanoreception) and noxious (nociception) prosthesis sensing and discrimination in an amputee.

    (A) The amputee could discriminate which region of his phantom hand was activated, if at all. (B) Perception of pain increases with decreasing radius of curvature (i.e., increase in sharpness) for the objects presented to the prosthetic hand. (C) Discrimination accuracy shows the participant’s ability to reliably identify each object presented to the prosthesis based purely on the sensory feedback from the neuromorphic stimulation. (D) Results from the PDT during user-controlled movements, with pain reflex enabled.

Supplementary Materials

  • robotics.sciencemag.org/cgi/content/full/3/19/eaat3818/DC1

    Fig. S1. Sensory mapping over time.

    Fig. S2. Stimulation thresholds over time.

    Fig. S3. EEG activation.

    Fig. S4. Amputee pressure discrimination.

    Fig. S5. Average fingertip pressures.

    Fig. S6. Custom prosthetic arm.

    Fig. S7. Prosthesis pain reflex.

    Fig. S8. Power law object edge radius of curvature.

    Table S1. Scaled comfort responses.

    Table S2. Amputee survey.

    Movie S1. Dynamic EEG activity during nerve stimulation.

    Movie S2. Neuromorphic transduction during grasping.

    Movie S3. Prosthesis PDT with reflex.

    Movie S4. Amputee PDT with reflex.

  • Supplementary Materials

    Supplementary Material for:

    Prosthesis with neuromorphic multilayered e-dermis perceives touch and pain

    Luke E. Osborn*, Andrei Dragomir, Joseph L. Betthauser, Christopher L. Hunt, Harrison H. Nguyen, Rahul R. Kaliki, Nitish V. Thakor*

    *Corresponding author. Email: losborn{at}jhu.edu (L.E.O.); nitish{at}jhu.edu or eletnv{at}nus.edu.sg (N.V.T.)

    Published 20 June 2018, Sci. Robot. 3, eaat3818 (2018)
    DOI: 10.1126/scirobotics.aat3818

    This PDF file includes:

    • Fig. S1. Sensory mapping over time.
    • Fig. S2. Stimulation thresholds over time.
    • Fig. S3. EEG activation.
    • Fig. S4. Amputee pressure discrimination.
    • Fig. S5. Average fingertip pressures.
    • Fig. S6. Custom prosthetic arm.
    • Fig. S7. Prosthesis pain reflex.
    • Fig. S8. Power law object edge radius of curvature.
    • Table S1. Scaled comfort responses.
    • Table S2. Amputee survey.

    Download PDF

    Other Supplementary Material for this manuscript includes the following:

    • Movie S1 (.mp4 format). Dynamic EEG activity during nerve stimulation.
    • Movie S2 (.mp4 format). Neuromorphic transduction during grasping.
    • Movie S3 (.mp4 format). Prosthesis PDT with reflex.
    • Movie S4 (.mp4 format). Amputee PDT with reflex.

    Files in this Data Supplement:

  • Supplementary Materials

    Supplementary Material for:

    Prosthesis with neuromorphic multilayered e-dermis perceives touch and pain

    Luke E. Osborn*, Andrei Dragomir, Joseph L. Betthauser, Christopher L. Hunt, Harrison H. Nguyen, Rahul R. Kaliki, Nitish V. Thakor*

    *Corresponding author. Email: losborn{at}jhu.edu (L.E.O.); nitish{at}jhu.edu or eletnv{at}nus.edu.sg (N.V.T.)

    Published 20 June 2018, Sci. Robot. 3, eaat3818 (2018)
    DOI: 10.1126/scirobotics.aat3818

    This PDF file includes:

    • Fig. S1. Sensory mapping over time.
    • Fig. S2. Stimulation thresholds over time.
    • Fig. S3. EEG activation.
    • Fig. S4. Amputee pressure discrimination.
    • Fig. S5. Average fingertip pressures.
    • Fig. S6. Custom prosthetic arm.
    • Fig. S7. Prosthesis pain reflex.
    • Fig. S8. Power law object edge radius of curvature.
    • Table S1. Scaled comfort responses.
    • Table S2. Amputee survey.

    Download PDF

    Other Supplementary Material for this manuscript includes the following:

    • Movie S1 (.mp4 format). Dynamic EEG activity during nerve stimulation.
    • Movie S2 (.mp4 format). Neuromorphic transduction during grasping.
    • Movie S3 (.mp4 format). Prosthesis PDT with reflex.
    • Movie S4 (.mp4 format). Amputee PDT with reflex.

    Files in this Data Supplement:

Navigate This Article