Research ArticleSENSORS

Prosthesis with neuromorphic multilayered e-dermis perceives touch and pain

See allHide authors and affiliations

Science Robotics  20 Jun 2018:
Vol. 3, Issue 19, eaat3818
DOI: 10.1126/scirobotics.aat3818

Abstract

The human body is a template for many state-of-the-art prosthetic devices and sensors. Perceptions of touch and pain are fundamental components of our daily lives that convey valuable information about our environment while also providing an element of protection from damage to our bodies. Advances in prosthesis designs and control mechanisms can aid an amputee’s ability to regain lost function but often lack meaningful tactile feedback or perception. Through transcutaneous electrical nerve stimulation (TENS) with an amputee, we discovered and quantified stimulation parameters to elicit innocuous (nonpainful) and noxious (painful) tactile perceptions in the phantom hand. Electroencephalography (EEG) activity in somatosensory regions confirms phantom hand activation during stimulation. We invented a multilayered electronic dermis (e-dermis) with properties based on the behavior of mechanoreceptors and nociceptors to provide neuromorphic tactile information to an amputee. Our biologically inspired e-dermis enables a prosthesis and its user to perceive a continuous spectrum from innocuous to noxious touch through a neuromorphic interface that produces receptor-like spiking neural activity. In a pain detection task (PDT), we show the ability of the prosthesis and amputee to differentiate nonpainful or painful tactile stimuli using sensory feedback and a pain reflex feedback control system. In this work, an amputee can use perceptions of touch and pain to discriminate object curvature, including sharpness. This work demonstrates possibilities for creating a more natural sensation spanning a range of tactile stimuli for prosthetic hands.

INTRODUCTION

One of the primary functions of the somatosensory system is to provide exteroceptive sensations to help us perceive and react to stimuli from outside of our body (1). Our sense of touch is a crucial aspect of the somatosensory system and provides valuable information that enables us to interact with our surrounding environment. Tactile feedback, in conjunction with proprioception, allows us to perform many of our daily tasks that rely on the dexterous manipulation of our hands (2). Mechanoreceptors and free nerve endings in our skin give us the means to perceive tactile sensation (2). The primary mechanoreceptors in the glabrous skin that convey tactile information are Meissner corpuscles, Merkel cells, Ruffini endings, and Pacinian corpuscles. The Merkel cells and Ruffini endings are classified as slowly adapting (SA) and respond to sustained tactile loads. Meissner and Pacinian corpuscles are rapidly adapting (RA) and respond to the onset and offset of tactile stimulation (1, 3). More recently, research has shown the role of fingertips in coding tactile information (4) and extracting tactile features (5).

A vital component of our tactile perception is the sense of pain. Although often undesired, pain provides a protection mechanism when we experience a potentially damaging stimulus. In the event of an injury, increased sensitivity can render even innocuous stimuli as painful (6). Nociceptors are dedicated sensory afferents in both glabrous and nonglabrous skin responsible for conducting tactile stimuli that we perceive as painful (6). Nociceptors, free nerve endings in the epidermal layer of the skin, act as high threshold mechanoreceptors (HTMRs) and respond to noxious stimuli through Aβ, Aδ, and C nerve fibers (1), which enable our perception of tactile pain. It was discovered that Aδ fiber HTMRs respond to both innocuous and noxious mechanical stimuli with an increase in impulse frequency while experiencing the noxious stimuli (7). It is also known that mechanoreceptor activation along with nociceptor activation helps inhibit our perception of pain, and our discomfort increases when only nociceptors are active (8), which helps to explain our ability to perceive a range of innocuous and noxious sensations. Although novel approaches have improved prosthesis motor control (9), comprehensive sensory perceptions are not available in today’s prosthetic hands.

The undoubted importance of our sense of touch, and lack of sensory capabilities in today’s prostheses, has spurred research on artificial tactile sensors and restoring sensory feedback to those with upper limb loss. Novel sensor developments use flexible electronics (1012), self-healing (13, 14) and recyclable materials (15), mechanoreceptor-inspired elements (16, 17), and even optoelectronic strain sensors (18), which will likely affect the future of prosthetic limbs. Local force feedback to a prosthesis is known to improve grasping (19), but in recent years, there has been a major push toward providing sensory feedback to the prosthesis and the amputee. Groundbreaking results show that implanted peripheral nerve electrodes (2023) and noninvasive electrical nerve stimulation methods (24) can successfully elicit sensations of touch in the phantom hand of amputees.

Recent approaches aim to mimic the biological behavior of tactile receptors using advanced skin dynamics (25) and what are known as neuromorphic (26) models of tactile receptors for sensory feedback. A neuromorphic system aims to implement components of a neural system, for example, the representation of touch through spiking activity based on biologically driven models. One reason for using a neuromorphic approach is to create a biologically relevant representation of tactile information using actual mechanoreceptor characteristics. Neuromorphic techniques have been used to convey tactile sensations for differentiating textures using SA-like dynamics for the stimulation paradigm to an amputee through nerve stimulation (26) and for feedback to a prosthesis to enhance grip functionality (27). Although important, methods of sensory feedback have been limited to sensations of pressure (21), proprioception (23), and texture (26), even though our perception of tactile information culminates in a sophisticated, multifaceted sensation that also includes stretch, temperature, and pain.

Current forms of tactile feedback fail to address the potentially harmful mechanical stimulations that could result in damage to cutaneous tissue or, in this context, to the prosthesis itself. We investigated the idea that a sensation of pain could benefit a prosthesis by introducing a sense of self-preservation and the ability to automatically release an object when pain is detected. Specifically, we implemented a pain reflex in prosthesis hardware that mimics the functionality of the polysynaptic pain reflex found in biology (2830). Pain serves multiple purposes in that it allows us to convey useful information about the environment to the amputee user while also preventing damage to the fingertips or cosmesis, a skin-like covering, of a prosthetic hand. It is worth noting that an ideal prosthesis would allow the user to maintain complete control and overrule pain reflexes if desired. However, in this paper, we focus on the ability to detect pain through a neuromorphic interface and initiate an automated pain reflex in the prosthesis.

We postulate that the presence of both innocuous and noxious tactile signals will help in creating more advanced and realistic prosthetic limbs by providing a more complete representation of tactile information. We developed a multilayered electronic dermis (e-dermis) and neuromorphic interface to provide tactile information to enable the perception of touch and pain in an upper limb amputee and prosthesis. We show closed-loop feedback to a transhumeral amputee through transcutaneous electrical nerve stimulation (TENS) to elicit either innocuous or painful sensations in the phantom hand based on the area of activation on a prosthesis (Fig. 1). Furthermore, we identified features of peripheral nerve stimulation, specifically pulse width and frequency, that play key roles in providing both innocuous and noxious tactile feedback. Quantifying the differences in perception of sensory feedback, specifically innocuous and noxious sensations, adds dimensionality and breadth to the type and amount of information that can be transmitted to an upper limb amputee, which aids in object discrimination. Finally, we demonstrate the ability of the prosthesis and the user to differentiate between safe (innocuous) and painful (noxious) tactile sensations during grasping and to react appropriately using a prosthesis reflex, modeled as a polysynaptic withdrawal reflex, to prevent damage and further pain.

Fig. 1 Prosthesis system diagram.

Tactile information from object grasping is transformed into a neuromorphic signal through the prosthesis controller. The neuromorphic signal is used to transcutaneously stimulate peripheral nerves of an amputee to elicit sensory perceptions of touch and pain.

RESULTS

Biologically inspired e-dermis

Mechanoreceptors in the human body are uniquely structured within the dermis and, in the case of Meissner corpuscles (RA1) and Merkel cells (SA1), lie close to the epidermis boundary (1). RA1 receptors are often found in the dermal papillae, which lend to their ability to detect movement across the skin, and SA1 receptors tend to organize at the base of the epidermis. However, in glabrous skin, the HTMR free nerve endings extend into the epidermis (i.e., the outermost layer of skin) (1). We used this natural layering of tactile receptors to guide the multilayered approach of our e-dermis (Fig. 2A) to create sensing elements to capture signals analogous to those detected by mechanoreceptors (dermal) and nociceptors (epidermal) in healthy glabrous skin (Fig. 2B). The sensor was designed using piezoresistive (Eeonyx, Pinole, CA) and conductive fabrics (LessEMF, Latham, NY) to measure applied pressure on the surface of the e-dermis. A 1-mm rubber layer (Dragon Skin 10, Smooth-On, Easton, PA) between the artificial epidermal (top) and dermal (bottom) sensing elements provides skin-like compliance and distributes loads during grasping. There are three tactile pixels, or taxels, with a combined sensing area of about 1.5 cm2 on each fingertip. The sensor layering resulted in variation of the e-dermis output during loading (Fig. 2C). The change in resistance in the tactile sensor was greater for the epidermal layer, enabling higher sensitivity. During grasping of an object, the e-dermis sensing layers, which were calibrated for a range of 0 to 300 kPa, exhibited differences in behavior. These differences can be used for extracting additional tactile information such as pressure distribution and object curvature (Fig. 2, D and E).

Fig. 2 Multilayered e-dermis design and characterization.

(A) The multilayered e-dermis is made up of conductive and piezoresistive textiles encased in rubber. A dermal layer of two piezoresistive sensing elements is separated from the epidermal layer, which has one piezoresistive sensing element, with a 1-mm layer of silicone rubber. The e-dermis was fabricated to fit over the fingertips of a prosthetic hand. (B) The natural layering of mechanoreceptors in healthy glabrous skin makes use of both RA and SA receptors to encode the complex properties of touch. Free nerve endings (nociceptors) that are primarily responsible for conveying the sensation of pain in the fingertips are also present in the skin. (C) The prosthesis with e-dermis fingertip sensors grasps an object. (D) The epidermal layer of the multilayered e-dermis design is more sensitive and has a larger change in resistance compared with the dermal layer. (E) Differences in sensing layer outputs are captured during object grasping and can be used for adding dimensionality to the tactile signal.

Touch and pain perception

To provide sensory feedback, we used targeted TENS to extensively map and understand the perception of a transhumeral amputee’s phantom limb during sensory feedback, a method we previously demonstrated in multiple amputees (24). Although the participant did not undergo any targeted muscle or sensory reinnervation during surgery, there was a natural regrowth of peripheral nerves into the remaining muscles, soft tissue, and skin around the amputation. The median and ulnar nerves were identified on the amputee’s left residual limb and targeted for noninvasive electrical stimulation because these nerves innervated relevant areas of the phantom hand. The participant received more than 25 hours of sensory mapping in addition to over 150 trials of sensory stimulation experiments to quantify the perceptual qualities of the stimulation. Extensive mapping of the residual limb showed localized activation of the amputee’s phantom hand (Fig. 3A).

Fig. 3 Sensory feedback and perception.

(A) Median and ulnar nerve sites on the amputee’s residual limb and the corresponding regions of activation in the phantom hand due to TENS. Psychophysical experiments quantified the perception of the nerve stimulation including (B) detection and (C) discrete frequency discrimination thresholds. In both cases, the stimulation amplitude was held at 1.4 mA. (D) The perception of the nerve stimulation was largely a tactile pressure on the activated sites of the phantom hand, although sensations of electrical tingling also occurred. (E) The quantification of pain from nerve stimulation shows that the most noxious sensation is perceived at higher stimulation pulse widths with frequencies in the range of 10 to 20 Hz. (F) Contralateral somatosensory cortex activation during nerve stimulation shows relevant cortical representation of sensory perception in the amputee participant (movie S1).

The amputee identified multiple unique regions of activation in his phantom hand from the electrical stimulation. The participant did not report any sensory activation, other than the physical presence of the probe, of his residual limb at the stimulation sites. He indicated that the dominating perceived sensation during stimulation occurred in his phantom hand, which is supported by our previous work (24). Cutaneous receptors on the residual limb respond to physical stimuli, whereas the electrical stimulation activates the underlying peripheral nerves to activate the phantom hand. Psychophysical experiments showed the amputee’s perception of changes in stimulation pulse width and frequency on his median and ulnar nerves (Fig. 3, B and C). In general, the stimulation was perceived primarily as pressure with some sensations of electrical tingling (paresthesia) (Fig. 3D). Stability of the participant’s sensory activation (fig. S1) and stimulation perceptual thresholds (fig. S2) were tracked over several months in his thumb and index fingers (median nerve) as well as his pinky finger (ulnar nerve).

Sensory feedback of noxious tactile stimuli was delivered using TENS to an amputee, and the perception was quantified. The results show that changes in both stimulation frequency and pulse width influence the perception of painful tactile sensations in the phantom hand (Fig. 3E). The relative discomfort of the tactile sensation was reported by the user on a modified comfort scale ranging from −1 (pleasant) to 10 (very intense, disabling pain that dominates the senses) (table S1). In this experiment, the highest perceived pain was rated as a 3, which corresponded to uncomfortable but tolerable pain. The most painful sensations were perceived at relatively low frequencies between 10 and 20 Hz. Higher frequency stimulation tends toward more pleasant tactile sensation, which is contrary to what might be expected when increasing stimulation frequency (31). In addition, very low frequencies generally resulted in innocuous activation of the phantom hand, whereas frequencies that were closer to the discrete detection boundary (15 to 30 Hz) resulted in the most noxious sensations in the activated region. We used electroencephalography (EEG) signals to localize and obtain an affirmation of the stimulus-associated perception. The stimulation caused activation in contralateral somatosensory regions of the amputee’s brain, which corresponded to his left hand (Fig. 3F) (32). EEG activation during stimulation is significantly higher (P < 0.05) than baseline activity, confirming the perceived phantom hand activation experienced by the user (fig. S3 and movie S1).

Neuromorphic transduction

As mentioned previously, a neuromorphic system attempts to mimic the behavior found in the nervous system. On the basis of the results from the sensory mapping of the participant, we developed the neuromorphic representation of the tactile signal to enable the sensation of both touch and pain. To enable direct sensory feedback to an amputee through peripheral nerve stimulation, we transformed the e-dermis signal from a pressure signal into a biologically relevant signal using a neuromorphic model. The aim for the neuromorphic model was to capture elements of our actual neural system, in this case, to represent the neural equivalent of a tactile signal for feedback to an amputee. To implement the biological activity from tactile receptors, namely, the spiking response in the peripheral nerves due to a tactile event, we used the Izhikevich model of spiking neurons (33), which provides a neuron modeling framework based on known neural dynamics while maintaining computational efficiency and easily allowing for different neuron behaviors from parameter adjustments. The Izhikevich model has been used in previous work for providing tactile feedback to an amputee through nerve stimulation (26). In our work, mechanoreceptor and nociceptor models produced receptor-specific outputs, in terms of neuron voltage, based on the measured pressure signal on the prosthesis fingertips. The mechanoreceptor model combined characteristics of SA and RA receptors through the regular and fast-spiking Izhikevich neurons, respectively, to convey more pleasant tactile feedback to the amputee. The nociceptor model used fast-spiking Izhikevich neuron dynamics to mimic the behavior of the free nerve endings.

When an object was grasped by the prosthesis, a higher number of active taxels indicated a larger distribution of the pressure on the fingertip, which was conveyed in the neuromorphic transduction as an innocuous (i.e., nonpainful) tactile sensation. Changes in the tactile signal were captured in the neuromorphic transduction by changes in stimulation frequency and pulse width to correspond to the appropriate perceived levels of touch or pain during sensory feedback. On the basis of the results from the psychophysical experiments and the quantification of pain, the perception of noxious tactile feedback was achieved through the nociceptor model (see Materials and Methods).

To demonstrate the neuromorphic representation of a tactile signal, we used three different objects, each of equal width but varying curvature, to elicit different types of tactile perceptions in the prosthesis during grasping (Fig. 4A). The objects follow a power law shape, where the radius of curvature (Rc) was modified using the power law exponent n, which ranges between 0 and 1 and effectively defines the sharpness of the objects (see Materials and Methods). The power law exponents used were 1/4, 1/2, and 1 and correspond to object 1, object 2, and object 3, respectively. The response of the fingertip taxels during object loading captured differences in object curvature based on the relative activation of all sensing elements (Fig. 4, B and C, and movie S2). As expected, the epidermal layer was the most activated taxel during loading and absorbed the largest pressure. The sharp edge of object 3 produced a highly localized pressure source on the epidermal layer of the e-dermis, which triggered the neuromorphic nociceptor model (see Materials and Methods) (Fig. 4D).

Fig. 4 E-dermis and neuromorphic tactile response from different objects.

(A) Three different objects, with equal width but varying curvature, were used to elicit tactile responses from the multilayered e-dermis. (B) Pressure heatmap from the fingertip sensor on a prosthetic hand during grasping of each object and (C) corresponding pressure profile for each of the sensing layers. (D) The pressure profiles were converted to the input current, I, for the Izhikevich neuron model for sensory feedback to the amputee user (movie S2). Note the highly localized pressure during the grasping of object 3 and the resulting nociceptor neuromorphic stimulation pattern, which is realized through changes in stimulation pulse width and the neuromorphic model parameters.

Prosthesis tactile perception and pain reflex

As an extension of the body, a prosthetic hand should exhibit similar behavior and functionality of a healthy hand. The perceptions of innocuous touch and pain are valuable at both the local (i.e., the prosthetic hand) and the global (i.e., the user) levels. At the local level, a reflex behavior from the prosthesis to open when pain is detected can help prevent unintended damage to the hand or cosmesis. It should be noted that, in an ideal prosthesis, this reflex would be modulated by the user based on the perceived pain. To demonstrate a local closed-loop pain reflex, a prosthetic hand with a multilayered e-dermis on the thumb and index finger grasped, held, and released one of the previously described objects (Fig. 5, A to C). The sensor signals were used as feedback to the embedded prosthesis controller to enable differentiation of the various objects and determine pain. We used pressure distribution (Fig. 6A), contact rate (Fig. 6B), and the number of activated sensing elements per finger (Fig. 6C) as input features in a linear discriminant analysis (LDA) algorithm for object detection.

Fig. 5 Prosthesis grasping and control.

To demonstrate the ability of the prosthesis to determine safe (innocuous) or unsafe (painful) objects, we performed the PDT. The objects were (A) object 1, (B) object 2, and (C) object 3, each of which is defined by their curvature. In the case of a painful object (object 3), the prosthesis detected the sharp pressure and released its grip through its pain reflex (movie S3).

Fig. 6 Tactile features for prosthesis perception.

To determine which object is being touched during grasping, we implemented LDA to discriminate between the independent classes. As input features into the algorithm, we used (A) sensor pressure values, (B) the rate of change of the pressure signal, and (C) the number of active sensing elements during loading.

In the online pain detection task (PDT), the prosthesis grabbed, held, and released an object (movie S3). In this work, the curvature of object 3 was assumed to be considered painful during grasping. When pain was detected, a prosthesis pain reflex caused the hand to open, releasing the object. The prosthesis was able to reliably detect which object is being grasped (Fig. 7A). The prosthesis had a high likelihood of perceiving pain while grasping object 3 and a significantly less likelihood of perceiving pain for objects 2 and 1 (P < 0.001) (Fig. 7B). The reaction time for the prosthesis to complete a reflex after perceiving pain was recorded and was similar to reaction times in healthy humans from previously published data (Fig. 7C) (28).

Fig. 7 Real-time prosthesis pain perception.

(A) LDA classifier’s accuracy across the various conditions and (B) percentage of trials where the prosthesis perceived pain during the online PDT. Note the high percentage of detected pain during the PDT for object 3. (C) Pain reflex time of the prosthesis, using the rate of change of the pressure signal to determine object contact and release, compared with previously published data of pain reflex time in healthy adults (28).

User tactile perception

With the added ability to perceive both innocuous and noxious tactile sensations in a single stimulation modality, an amputee user can use more realistic tactile sensations to discriminate between objects with a large or small (sharp) radius of curvature. The participant demonstrated his ability to perceive both innocuous and noxious tactile sensations by performing several discrimination tasks with a prosthetic hand. The neuromorphic tactile signal was passed from the prosthesis controller directly to the stimulator to provide sensory feedback to the amputee. The participant could reliably detect, with perfect accuracy, which of the fingers of the prosthesis were being loaded (Fig. 8A). The participant also received sensory feedback from varying levels of pressure applied to the prosthetic fingers. A light (<100 kPa), medium (<200 kPa), or hard (>200 kPa) touch, as measured by the e-dermis, presented to the prosthesis was translated to the peripheral nerves of the amputee by using the neuromorphic representation of touch (figs. S4 and S5). To demonstrate the ability of the prosthesis and user to perceive differences in object shape through variation in the comfort levels of sensory feedback, we presented each of the three objects to the prosthesis. Sensory feedback to the thumb and index finger regions of the phantom hand enabled the participant to perceive variations in the object curvatures, which were realized through changes in perceived comfort of the sensation. The results show an inversely proportional relationship between the radius of curvature of an object and the perceived discomfort of the tactile feedback (Fig. 8B). In addition to being able to perceive variation in sharpness of the objects, as conveyed by the discomfort in the neuromorphic tactile feedback, the participant could reliably differentiate between the three objects with high accuracy (Fig. 8C). Finally, the participant performed the PDT with his prosthesis (movie S4). The prosthesis pain reflex control was implemented during the grasping task, which resulted in the prosthesis automatically releasing an object when pain was detected (see Materials and Methods). During actual amputee use, the prosthesis pain reflex registered over half of the object 3 movements as painful, significantly more than for the other objects (P < 0.05) (Fig. 8D).

Fig. 8 Innocuous (mechanoreception) and noxious (nociception) prosthesis sensing and discrimination in an amputee.

(A) The amputee could discriminate which region of his phantom hand was activated, if at all. (B) Perception of pain increases with decreasing radius of curvature (i.e., increase in sharpness) for the objects presented to the prosthetic hand. (C) Discrimination accuracy shows the participant’s ability to reliably identify each object presented to the prosthesis based purely on the sensory feedback from the neuromorphic stimulation. (D) Results from the PDT during user-controlled movements, with pain reflex enabled.

Responses from a subjective survey of the perception of the sensory stimulation show that the amputee felt as if the tactile sensations were coming directly from his phantom hand. In addition, the participant stated that he could clearly feel the touch of objects on the prosthetic hand and that it seemed that the objects themselves were the cause of the touch sensations that he was experiencing during the experiments (table S2).

DISCUSSION

Perceiving touch and pain

Being able to quantify the perception of innocuous and noxious stimuli for tactile feedback in amputees is valuable because it enables the replacement of an extremely valuable piece of sensory information: pain. Not only does pain play a role in providing tactile context about the type of object being manipulated, but it also acts as a mechanism for protecting the body. One could argue that this protective mechanism is not necessary in a prosthesis because it is merely an external tool or piece of hardware to an amputee user. We postulate that being able to capture noxious stimuli is actually more valuable to a prosthesis because it does not have the same self-healing characteristics found in healthy human skin, although recent research has shown self-healing materials that could be used for future prosthetic limbs (13, 34). To enable an artificial sense of self-preservation, a noxious tactile signal is useful for the prosthesis to ensure that it does not exceed the limits of a cosmetic covering or the hand itself. As prosthetic limbs become more sophisticated and sensory feedback becomes more ubiquitous, there will be a need to increase not just the resolution of tactile information but also the amount of useful information being passed to the user. We have identified how changing stimulation pulse width and frequencies enables a spectrum of tactile sensation that captures both innocuous and noxious perceptions in a single stimulation modality.

Our extensive phantom hand mapping, psychophysics, and EEG results support the use of TENS for providing relevant sensory information to an amputee. The EEG results are limited in that they do not provide detailed information on how changes in stimulation patterns were perceived, but they do show activation in sensory regions of the brain indicating relevant sensations in the amputee. Furthermore, the results from the user survey (table S2) showed that sensory feedback helped the amputee better perceive his phantom hand and that objects being grabbed by the prosthesis were perceived as being the source of the sensation, which helps support the neuromorphic stimulation as a valid approach for providing relevant sensory feedback.

The results from the PDT showed the ability of the prosthesis to detect pain and reflex to release the object. Object 3 was overwhelmingly detected as painful due to its sharp edge (Fig. 7B). The high success rate for detecting and preventing pain for the benchtop PDT is likely due to the controlled nature of the prosthesis grip. The likelihood of detecting object 3 as painful decreased and the chances of pain being detected for the other objects increased during the PDT with a user-controlled prosthesis (Fig. 8D); however, pain detection and reflex were still significantly more likely for object 3 (P < 0.05). This shift in pain detection is likely due to the amputee’s freedom to pick up the objects with his prosthesis in any way he chose. The variability in grasping orientation and approach for each trial resulted in more instances where object 3 was not perceived as painful by the prosthesis. The ability to handle objects in different positions and orientations raises an interesting point: The amount of pain produced is not an inherent property of an object; rather, it is dependent on the way in which it is grasped. A sharp edge may still be safely manipulated without pain if the pressure on the skin does not exceed the threshold for pain. To reliably encode both touch and pain for prostheses, tactile signals should be analyzed in terms of pressure as opposed to grip force.

The prosthesis pain reflex presented here is autonomous, but one possibility is to use the amputee’s electromyography (EMG) signal as an additional input to the reflex model to enable modulation of the pain sensitivity perceived by the prosthesis. In this work, the pain sensation was not severe enough to generate a reliable EMG reflex signal, so the reflex decision was made by the prosthesis instead of the user. The time for a user to process sensory feedback and produce a voluntary contraction is over 1 s (35), which is why we implemented an autonomous prosthesis pain reflex to achieve a response time closer to what is found in biology (Fig. 7C). Biologically, this autonomous response is equivalent to a fast spinal reflex compared with the slower cortical response for producing a voluntary EMG signal for controlling limb movement.

Another implication of this work is the quantification of perceived noxious and innocuous tactile sensations during TENS of peripheral afferents. One might assume that an increase in discomfort would be associated with an increase in delivered charge; however, we found that the most painful sensations during tactile feedback to an amputee delivered through TENS were primarily dictated by an increase in stimulation pulse width and stimulation frequency. Specifically, frequencies that were near the discrete detection boundary (15 to 30 Hz) were perceived as more painful than higher frequencies. Changes in stimulation frequency seemed to have the largest influence on the perceptions of touch and pain, whereas pulse width affected intensity of the sensation (Fig. 3E). Furthermore, we demonstrated real-time discrimination between object curvature based purely on perceived discomfort in tactile feedback, which was associated with sharpness of the objects by the participant.

Neuromorphic touch

The ability of the participant to discriminate objects, specifically those that cause pain, is rooted in the neuromorphic tactile transduction and corresponding nerve stimulation. The psychophysical results illuminate the stimulation paradigms necessary to elicit tactile sensations that correspond to both mechanoreceptors and nociceptors in the phantom hand of an amputee.

More sophisticated neuron models exist and could be used to capture behavior of individual receptors and transduction (25); however, the limitation of hardware prevents the stimulation of individual afferent nerve fibers. The Izhikevich model is simplistic in its dynamics but still follows basic qualities of integrate-and-fire models with voltage nonlinearity for spike generation and extremely low computational requirements, which allow for the creation of a wide variety of neuron behaviors (33). The advantage of the neuromorphic representation of touch in our work is that we can transform signals from the multilayered e-dermis directly into the appropriate stimulation paradigm needed to elicit the desired sensory percepts in the amputee participant. Specifically, the combination of mechanoreceptor and nociceptor outputs enables additional touch dimensionality while maintaining a single modality of feedback in both physical location and stimulation type. This combination allows the user to better differentiate between objects based on their unique evoked perceptions for each object (Fig. 8, B and C).

The limitation of this work is the small study sample. Although this work is a case study with a single amputee, the extensive psychophysical experiments and stability (figs. S1 and S2) of the results over several months show promise that other amputees would experience a similar type of perception from TENS, a technique we have previously validated for activating relevant phantom hand regions in multiple amputees (24). However, the psychophysics will likely have slight differences based on age and condition of the amputation. The results are promising in that the stimulation parameters used to elicit pain or touch followed the same trend in both median and ulnar nerve sites of the amputee (Fig. 3E). This work implies that both innocuous and noxious touches can be conveyed using the same stimulation modality. In addition, we showed that it is not necessarily a large amount of injected charge into the peripheral nerves that elicits a painful sensation. Rather, a combination of stimulation pulse width and frequency at the discrete detection boundary appears to create the most noxious sensations. Additional amputee participants who are willing to undergo nerve stimulation, sensory mapping, and psychophysical experiments to quantify their perceived pain would be needed to allow us to generalize the clinical significance to a wider amputee population. Our findings have applications not only in prosthetic limb technology but also for robotic devices in general, especially devices that rely on tactile information or interactions with external objects. The overarching idea of capturing meaningful tactile information continues to become a reality, because we can now incorporate both innocuous and noxious information in a single channel of stimulation. Whether it is used for sensory feedback or internal processing in a robot, the senses of touch and pain together enable a more complete perception of the workspace.

This study illustrates, through demonstration in a prosthesis and amputee participant, the ability to quantify and use tactile information that is represented by a neuromorphic interface as both mechanoreceptor and nociceptor signals. Through our demonstration of capturing and conveying a range of tactile signals, prostheses and robots can incorporate more complex components of touch, namely, differentiating innocuous and noxious stimuli, to behave in a more realistic fashion. The sense of touch provides added benefit during manipulation in prostheses and robots, but the sense of pain enhances their capabilities by introducing self-preservation and protection.

MATERIALS AND METHODS

Objectives and study design

Our objectives were to show that (i) a prosthetic hand was capable of perceiving both touch and pain through a multilayered e-dermis and (ii) an amputee was capable of perceiving the sense of both touch and pain through targeted peripheral nerve stimulation using a neuromorphic stimulation model.

Participant recruitment

All experiments were approved by the Johns Hopkins Medicine Institutional Review Board. The amputee participant was recruited from a previous study at Johns Hopkins University in Baltimore, MD. The participant consented to participate in all the experiments and to have images and recordings taken during the experiments used for publication and presentations. At the time of the experiments, the participant was a 29-year-old male with a bilateral amputation 5 years prior, due to tissue necrosis from septicemia. The participant has a transradial amputation of the right arm and a transhumeral amputation of the left arm. The left arm was used for all sensory feedback and controlling the prosthesis in this work. After 2 months of sensory mapping, the experiments were performed on average once every 2 weeks over a period of 3 months with follow-up sessions after 2, 5, and 8 months. EEG data were collected in one session over a period of 2 hours.

Sensory feedback

The sensory feedback experiments consisted of TENS of the median and ulnar nerves using monophasic square-wave pulses. We performed mapping of the phantom hand using a 1-mm beryllium copper (BeCu) probe connected to an isolated current stimulator (DS3, Digitimer Ltd., Hertfordshire, UK). An amplitude of 0.8 mA and frequency of 2 to 4 Hz were used while mapping the phantom hand. Anatomical and ink markers were used, along with photographs of the amputee’s limb, to map the areas of the residual limb to the phantom hand. For all other stimulation experiments, we used a 5-mm disposable Ag-Ag/Cl electrode. A 64-channel EEG cap with Ag-Ag/Cl electrodes (impedance, <10 kΩ) was used for the EEG experiment. The participant was seated, and stimulation electrodes were placed on the median and ulnar nerve sites of his residual limb. Each site was stimulated individually for a period of 2 s, followed by a 4-s delay with 25% jitter before the next stimulation. There was a total of 60 stimulation presentations with varying pulse width (1 to 20 ms) and frequencies (4 to 45 Hz) with an amplitude of 1.6 mA. A time window of 450 ms starting at 400 ms after stimulation was used to average EEG activity across trials and compared with baseline activity using methods similar to those in (36).

Psychophysical experiments

Psychophysical experiments were performed to quantify the perception of TENS on the median, radial, and ulnar nerves of the amputee. Experiments included sensitivity detection (varying pulse width at 20 Hz), detection of discrete versus continuous stimulation (varying frequency with pulse width of 5 ms), and scaled pain discrimination. For the pain discrimination experiment, the participant used a discomfort scale that ranged from pleasant or enjoyable (−1) to no pain (0) to very intense pain (10) (table S1). Stimulation current amplitude was held at 2 mA, whereas frequency and pulse width were modulated to quantify the effect of signal modulation on perception in the participant’s phantom hand. Every electrical stimulation was presented as a 2-s burst with at least 5-s rest before the next stimulation. Experiments were conducted in blocks up to 5 min with a break up to 10 min between each block. Every stimulation condition was presented up to 10 times. Psychometric functions were fit using a sigmoid link function (24).

E-dermis fabrication

The multilayered e-dermis was constructed from piezoresistive transducing fabric (Eeonyx) placed between crossing conductive traces (stretch conductive fabric, LessEMF), similar to the procedure described in previous work (37). The piezoresistive material is pressure-sensitive and decreases in resistance with increased loading. The intersection of the conductive traces created a sensing taxel, a tactile element. Human anatomy expresses a lower density of nociceptors, compared with mechanoreceptors, in the fingertip (38). So, we designed the epidermal layer as a 1 × 1 sensing array, whereas the dermal layer was a 2 × 1 array (Fig. 2A). The size of the prosthesis fingertip and the available inputs to the prosthesis controller limited the number of sensing elements to three per finger. The piezoresistive and conductive fabrics were held in place by a fusible tricot fabric with heat-activated adhesive. A 1-mm layer of silicone rubber (Dragon Skin 10, Smooth-On) was poured between two sensing layers. After the intermediate rubber layer cured, the textile sensors were wrapped into the fingertip shape, and a 2-mm layer of silicone rubber (Dragon Skin 10, Smooth-On) was poured as an outer protection and compliance layer, which is known to benefit prosthesis grasping (19). The e-dermis was placed over the thumb, index, and pinky phalanges of a prosthetic hand (Fig. 1B).

Prosthesis control

A bebionic prosthetic hand (Ottobock, Duderstadt, Germany) was used for the experiments. Prosthesis movement was controlled using a custom control board, with an ARM Cortex-M processor, developed by Infinite Biomedical Technologies (IBT; Baltimore, MD). The board was used for interfacing with the prosthesis, reading in the sensor signals, controlling the stimulator, and implementing the neuromorphic model. During the user-controlled PDT, the amputee used his own prosthesis (fig. S6), a bebionic hand with Motion Control wrist and a Utah Arm 3+ arm with elbow (Motion Control Inc., Salt Lake City, UT). The amputee controlled his prosthesis using an LDA algorithm on an IBT control board for EMG pattern recognition. The electrodes within his socket were bipolar Ag-Ag/Cl EMG electrodes from IBT.

Neuromorphic models

We implemented artificial mechanoreceptor and nociceptor models to emulate natural tactile coding in the peripheral nerve. We tuned the model to match the known characterization of TENS in the amputee to elicit the appropriate sensation. Constant current was applied during stimulation, and both pulse width and spiking frequency were modulated by the model. Higher grip force was linked to increased stimulation pulse width and frequency, which was perceived as increased intensity in the phantom hand. Innocuous tactile stimuli resulted in shorter pulse widths (1 or 5 ms), whereas the noxious stimuli produced a longer pulse width (20 ms), a major contributor to the perception of pain through TENS, as shown by the results. To create the sensation of pain, we varied the parameters of the model in real time based on the output of the e-dermis. We converted the e-dermis output to neural spikes in real time by implementing the Izhikevich neuron framework (33) in the embedded C++ software on the prosthesis control board. The output of the embedded neuromorphic model on the control board was used to control the stimulator for sensory feedback. The neuromorphic mechanoreceptor model was a combination of SA and RA receptors modeled as regular and fast-spiking neurons. The nociceptor model was made up of Aδ neurons, which were modeled as fast-spiking neurons to elicit a painful sensation in the phantom hand. It should be noted that the fast-spiking neuron model was perceived as noxious with an increase in pulse width, which allows us to use the same Izhikevich neuron for both mechanoreceptors and nociceptors. The e-dermis output was used as the input current, I, to the artificial neuron model. The evolution of the membrane potential v and the refractory variable u are described by Eqs. 1 and 2. When the membrane potential reaches the threshold vth, the artificial neuron spikes. The membrane potential was reset to c, and the membrane recovery variable u was increased by a predetermined amount d (Eq. 3). The spiking output was used to directly control the TENS unit for sensory feedback.Embedded Image(1)Embedded Image(2)Embedded Image(3)

Because we are not directly stimulating individual afferents in the peripheral nerves, we tuned the model to represent behavior of a population of neurons. The parameters used for the different receptor types were as follows: A = 0.04/Vs; B = 5/s; C = 140 V/s; Cm = 1 F; R = 1; b = 0.2/s; c = –65 mV; d = 8 mV/s; vth = 30 mV; andEmbedded Image

A, B, and C, are constants of the model, b describes the sensitivity of the recovery variable u, c is the membrane reset voltage, Cm is the membrane capacitance, and R describes the membrane resistance of the neuron. The fast-spiking neurons fire with high frequency with little adaptation, similar to responses from nociceptors during intense, noxious stimuli (7). In the model, fast spiking is represented by a very fast recovery (a). Values for the parameters were taken from (26) and (33).

We limited the spiking frequency of the neuromorphic model to 40 and 20 Hz for the mechanoreceptor and nociceptor models, respectively. The transition of the neuromorphic model from mechanoreceptors to nociceptors relies on the pressure measured at the fingertips of the prosthesis, the number of active sensing elements, and the SD of the pressure signal across the active taxels. The prosthesis fingertip pressure (P) is used to determine the neuromorphic stimulation model for sensory feedback. Highly localized pressure above a threshold β triggers the FS model, whereas the RS model is used in cases of more distributed fingertip pressure. The following pseudocode explains how the stimulation model is chosen, where β = 150 kPa, n is the number of active taxels, and pw is the stimulation pulse width:

if (P ≥ β and n < 2), then [nociceptor (Aδ) (FS: pw = 20 ms)]

else if (P ≥ β and n = 2), then [mechanoceptor (SA/RA) (FS: pw = 5 ms)]

else [mechanoceptor (SA/RA) (RS: pw = 5 ms)]

Prosthesis pain reflex

To mimic biology, we modeled the prosthesis pain withdrawal as a polysynaptic reflex (29, 30) in the prosthesis hardware. In our model, the prosthesis controller was treated as the spinal cord for the polysynaptic reflex. The nociceptor signal was the input, I(t), to an integrating interneuron Γ whose output IΓ(t) was the input to an α motor neuron, which triggered the withdrawal reflex through a prosthesis hand open command after ~100 ms of pain. Both neurons can be modeled as leaky integrate and fire with a synapse from the α motor neuron causing the reflex movement (Eqs. 4 and 5, and fig. S7), similar to the EMG signals generated during a nociceptive reflex (39).Embedded Image(4)Embedded Image(5)

Both neurons had time constant τm = 10 ms, resting potential E = −60 mV, membrane resistance R = 20 ohms, and a spiking threshold of vth = −40 mV. The time step was 5 ms, and the nociceptor signal was normalized, enveloped, and scaled by β = 0.2 mV. The prosthesis reflex parameters were chosen to trigger hand withdrawal after ~100 ms of pain to mimic the pain reflex in healthy humans (28). Fingertip pressure, the rate of contact, and the number of active sensing elements on each fingertip were used as features for an LDA algorithm to detect the different objects. Object 3 was labeled as a painful object. A taxel was considered active if it measured a pressure greater than 10 kPa. The pattern recognition algorithm was trained using sensor data from 5 trials of prosthesis grasping for each object and validated on 10 different trials.

Object design and fabrication

We created three objects of equal size with varying edge curvatures, defined by the edge blend radius, using a Dimension 1200es 3D printer (Stratasys, Eden Prairie, MN). Each object has a width of 5 cm but differed in curvature. Each object’s curvature followed a power law, where the leading edge of the protrusions varies in blend radii and ranges from flat to sharp. The radius of curvature, Rc, of the leading edge can be modified by the body power law exponent, n, whereEmbedded Image(6)

A is the power law constant, which is a function of n, and x is the position along the Cartesian axis in physical space. The objects for this study were designed to maintain a constant width, w (fig. S8), to prevent the ability to discriminate between the objects based on overall width. The three objects used had a power law exponent, n, of 1/4, 1/2, and 1 and were referred to as object 1, object 2, and object 3, respectively. More details and explanation of power law–shaped edges can be found in (40, 41).

Experimental design

Finger discrimination

The multilayered e-dermis was placed over the thumb and pinky finger of the prosthesis. Activation of each fingertip sensor corresponded directly to nerve stimulation of the amputee in the corresponding sites of his phantom hand. The participant was seated, and his vision was occluded. The experimenter pressed the prosthetic thumb, pinky, both, or neither in a random order. Each condition was presented eight times. The stimulation amplitude was 1.5 and 1.45 mA for the thumb and pinky sites on the amputee’s residual limb, respectively. Next, the experimenter pressed the prosthetic thumb or pinky with a light (<100 kPa), medium (<200 kPa), or hard (>200 kPa) pressure (figs. S4 and S5). Each force condition was presented 10 times in a random order for each finger.

Object discrimination

Fingertip sensors were placed on the thumb and index finger of a stationary bebionic prosthetic hand. The participant was seated, and his vision of the prosthesis was occluded. A stimulating electrode was placed over the region of his residual limb that corresponded to his thumb and index fingers on his phantom hand. The experimenter presented one of the three objects on the index finger of the prosthetic hand for several seconds. The participant responded with the perceived object and the perceived discomfort based on the tactile sensation. Each block consisted of up to 15 object presentations. The participant performed three blocks of this experiment. Each object was presented randomly within each block, and each object was presented the same number of times as the other objects. The participant visually inspected the individual objects before the experiment took place, but he was not given any sample stimulation of what each object would feel like. This was done to allow the participant to create his own expectation of what each object should feel like if he were to receive sensory feedback on his phantom hand.

Pain detection task

In the benchtop PDT, the prosthesis was mounted on a stand with the multilayered sensors on the thumb and index finger. The object was placed on a stand, and the prosthesis grabbed the object using a closed precision pinch grip. Each object was presented to the prosthesis at least 15 times in a random order. For the user-controlled PDT, the participant used his prosthesis to pick up and move one of the three objects. Each object was presented at least 10 times. The instances of prosthesis reflex were recorded. The participant took a survey at the end of the experiments (table S2).

Data collection

Each taxel of the multilayered e-dermis was connected to a voltage divider. Sensor data were collected by the customized prosthesis controller and sent through serial communication with a baud rate of 115,200 bps to MATLAB (MathWorks, Natick, MA) on a PC for further postprocessing and plotting. Each sensing element in the e-dermis was sampled at 200 Hz. Responses from the psychophysical experiments were recorded using MATLAB and stored for processing and plotting. The prosthesis controller communicated with MATLAB through Bluetooth communication with a baud rate of 468,000 bps. Sixty-four–channel EEG data were recorded at 500 Hz by a SynAmp2 system (Compumedics Neuroscan, Charlotte, NC) and processed in MATLAB using the EEGlab Toolbox (Swartz Center for Computational Neuroscience, University of California, San Diego, San Diego, CA). EEG data were downsampled to 256 Hz and band-pass–filtered between 0.5 and 40 Hz using a sixth-order Chebyshev filter. Muscle artifacts were rejected by the Automatic Artifact Rejection (AAR) blind source separation algorithm using canonical correlation approach. Independent component analysis was performed for removal of the eye and remnant muscle artifacts to obtain noise-free EEG data. Results from data collected over multiple trials of the same experiment were averaged together. Statistical P values were calculated using a one-tailed, two-sample t test. Error bars represent the SEM, unless otherwise specified.

SUPPLEMENTARY MATERIALS

robotics.sciencemag.org/cgi/content/full/3/19/eaat3818/DC1

Fig. S1. Sensory mapping over time.

Fig. S2. Stimulation thresholds over time.

Fig. S3. EEG activation.

Fig. S4. Amputee pressure discrimination.

Fig. S5. Average fingertip pressures.

Fig. S6. Custom prosthetic arm.

Fig. S7. Prosthesis pain reflex.

Fig. S8. Power law object edge radius of curvature.

Table S1. Scaled comfort responses.

Table S2. Amputee survey.

Movie S1. Dynamic EEG activity during nerve stimulation.

Movie S2. Neuromorphic transduction during grasping.

Movie S3. Prosthesis PDT with reflex.

Movie S4. Amputee PDT with reflex.

REFERENCES AND NOTES

Acknowledgments: We would like to sincerely thank the participant who volunteered and selflessly dedicated his time for this research to help enhance the current state of the art of prosthetic limb technology for the betterment of current and future users. We would like to thank B. Skerritt-Davis for help with the EEG experiment and M. Hodgson for support with the prosthesis controller and hardware. Funding: This work was supported in part by the Space@Hopkins funding initiative through Johns Hopkins University, the Johns Hopkins University Applied Physics Laboratory Graduate Fellowship Program, and the Neuroengineering Training Initiative through the National Institute of Biomedical Imaging and Bioengineering through the NIH under grant T32EB003383. Author contributions: L.E.O. and N.V.T. conceptualized the idea of quantifying and conveying pain and tactile sensory signals. L.E.O. designed the studies, developed the hardware and software, performed the experiments, analyzed the data, and wrote the paper. A.D. analyzed EEG data and assisted with writing the paper. H.H.N. assisted in collecting and analyzing data. J.L.B., C.L.H., and R.R.K. assisted in analyzing the data and writing the paper. N.V.T. supervised the experiments and assisted in designing the studies, analyzing the data, and writing the paper. Competing interests: L.E.O. and N.V.T. are inventors on intellectual property regarding the multilayered e-dermis, which has been disclosed to Johns Hopkins University. N.V.T. is a cofounder and R.R.K. is the chief executive officer of IBT. This relationship has been disclosed to and is managed by Johns Hopkins University. The other authors declare that they have no competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper or the Supplementary Materials. Data and software code can be made available by materials transfer agreement upon reasonable request.
View Abstract

Navigate This Article