Research ArticlePROSTHETICS

Biomimetic sensory feedback through peripheral nerve stimulation improves dexterous use of a bionic hand

See allHide authors and affiliations

Science Robotics  24 Jul 2019:
Vol. 4, Issue 32, eaax2352
DOI: 10.1126/scirobotics.aax2352

Abstract

We describe use of a bidirectional neuromyoelectric prosthetic hand that conveys biomimetic sensory feedback. Electromyographic recordings from residual arm muscles were decoded to provide independent and proportional control of a six-DOF prosthetic hand and wrist—the DEKA LUKE arm. Activation of contact sensors on the prosthesis resulted in intraneural microstimulation of residual sensory nerve fibers through chronically implanted Utah Slanted Electrode Arrays, thereby evoking tactile percepts on the phantom hand. With sensory feedback enabled, the participant exhibited greater precision in grip force and was better able to handle fragile objects. With active exploration, the participant was also able to distinguish between small and large objects and between soft and hard ones. When the sensory feedback was biomimetic—designed to mimic natural sensory signals—the participant was able to identify the objects significantly faster than with the use of traditional encoding algorithms that depended on only the present stimulus intensity. Thus, artificial touch can be sculpted by patterning the sensory feedback, and biologically inspired patterns elicit more interpretable and useful percepts.

INTRODUCTION

State-of-the-art upper-limb prostheses have become capable of mimicking many of the movements and grip patterns of endogenous human hands (13). Although these devices have the capabilities to replace much of the motor function lost after hand amputation, the methods for controlling and receiving feedback from these prosthetic limbs are still primitive (4, 5). The advent of neuromuscular implant systems capable of recording efferent motor activity and stimulating afferent sensory nerve fibers improves the transfer of sensorimotor information to and from a user’s peripheral nervous system, paving the way for more dexterous bionic hands (69).

Conveying sensory feedback through an electrical interface with the peripheral nervous system has been shown to confer functional benefits (916). However, demonstrations of these improvements are limited, and the sensory encoding algorithms themselves are often unsophisticated. The human hand is innervated by several types of tactile nerve fibers that each respond to different aspects of skin deformations. Manual interactions generally activate all of the fiber types, and tactile percepts are shaped by complex spatiotemporal patterns of activation across the different afferent populations (17, 18). One of the notable features of the aggregate afferent activity is the massive phasic bursts during the onset and offset of contact and the far weaker response during maintained contact (1922). Most extant sensory encoding mechanisms track sensor output (e.g., the absolute pressure, force, or torque from a prosthetic device) by modulating stimulation intensity and thus disregard this important and salient aspect of natural sensory feedback (9, 10, 12, 2329). To the extent that artificially induced sensory signals mimic natural ones, they are likely to elicit more naturalistic percepts and confer greater dexterity to the user (15, 30).

In the present study, we first demonstrate that closed-loop sensory feedback improved performance on dexterous tasks and enabled sensory discrimination during active manipulation of objects. We then show that artificial sensory experiences were enriched when the stimulation regimes were designed to mimic the natural patterns of neuronal activation that are evoked during manual interactions with a native hand. These results constitute an important step toward the development of dexterous bionic hands and have broad implications for neural interfaces and prosthetic devices.

RESULTS

We implanted one Utah Slanted Electrode Array (USEA) in the median nerve and another in the ulnar nerve, plus eight electromyographic recording leads (iEMGs) in the forearm muscles of an individual with a transradial amputation halfway between the wrist and elbow. The participant used this neuromyoelectric interface to control and sense through a state-of-the-art dexterous sensorized prosthetic hand and wrist (LUKE arm, DEKA; Fig. 1). Control signals were obtained using the filtered iEMG recordings as input to a modified Kalman filter (29, 31). The participant was able to control all six DOFs of the prosthesis independently, proportionally, and simultaneously in real time, achieving performance comparable with those of clinically available prosthetics in the modified Box and Blocks test (fig. S1) (32)—a standard test of manual dexterity—and efficiency comparable with that of able-bodied participants in a novel foraging task (fig. S2) (33). Recordings of muscle activation remained reliable over the entire duration of the study (14 months). Using muscle recordings rather than neural ones as control signals eliminates the problem of stimulation artifacts and allows for uncompromised sensory feedback.

Fig. 1 Participant and sensorized bionic hand.

A transradial amputee (A) had two total USEAs (B) implanted, one each, into the residual median and ulnar nerves of the arm. Activation of contact sensors on the DEKA LUKE arm (C) triggered stimulation of individual USEA electrodes or groups of USEA electrodes so that the amputee felt a sensation on his phantom hand at the corresponding location. For example, when contact was made with the index fingertip sensor, current was delivered through USEA electrodes with projection fields on the phantom index fingertip. Thus, when the prosthetic index fingertip made contact with an object, the participant experienced a sensation on the index fingertip.

Electrical stimulation of the residual nerves evokes sensations on the phantom hand

Electrical stimulation of the residual nerves through the chronically implanted USEAs evoked localized sensations that were experienced on the phantom hand. The participant reported up to 119 sensory percepts distributed over the hand and varying in their quality (Fig. 2 and fig. S3). As might be expected given the known patterns of innervation of the skin, a preponderance of percepts originated in the fingers and particularly the fingertips. The quality of the percepts also varied; some were described as “vibration” (36%), “pressure” (29%), or “tapping” (3%), which were likely associated with activation of cutaneous tactile nerve fibers; others were described as pain (16%), presumably reflecting activation of nociceptive fibers; and a few were described as “tightening” (12%) and joint movement (3%), presumably reflecting activation of proprioceptive nerve fibers such as muscle afferents. Activation of contact sensors on the prosthetic hand triggered stimulation of individual USEA electrodes or groups of USEA electrodes with congruent receptive fields. For example, when contact was made with the index fingertip sensor, a current was delivered through USEA electrodes with projection fields on the index fingertip of the phantom; that way, when the prosthetic index fingertip made contact with an object, the participant experienced a sensation on the index fingertip.

Fig. 2 Centroids of the projected fields for cutaneous percepts (circles) and location of proprioceptive percepts (black arrows) evoked by stimulation through individual USEA electrodes in the residual median or ulnar nerves.

A total of 119 sensory percepts were evoked (72% from median nerve) 2 weeks after the implantation surgery. The quality of the evoked percepts varied across electrodes: 37% vibration (red), 29% pressure (green), 16% pain (blue), 12% tightening (orange), 3% movement (arrows), 3% tapping (yellow), and 1% buzzing (black). A map of the complete projected fields can be found in fig. S3.

Sensory feedback improves grasping performance

The grip force required to grasp an object depends on its mass and on the coefficient of friction between skin and object: Heavy and slippery objects are gripped with more force than are light, high-friction ones (34). With our native hands, we are exquisitely proficient at exerting just enough pressure on an object to grasp it, an ability for which we rely on the sense of touch (34).

Some tests of manual dexterity do not benefit from tactile feedback. For example, performance on the modified Box and Blocks test is only slightly improved with touch because visual feedback is available and no penalty is incurred for exerting too much force on an object. However, other tests of manual dexterity are highly dependent on tactile feedback. In one such test, a participant moves an object from one location to another, as in the modified Box and Blocks test (fig. S1). However, the object is “fragile” and “breaks” if squeezed too hard (fig. S4) (35, 36). The participant moved the object without breaking it significantly more often with sensory feedback than without (32 of 40 times versus 22 of 40 times; Pearson’s χ2 test, P = 0.017; Fig. 3) and did so more rapidly (9.13 ± 0.44 s versus 11.14 ± 0.49 s per trial; t test, P < 0.001; Fig. 3).

Fig. 3 Sensory feedback improves object manipulation.

The participant’s task was to move a fragile object that breaks if the grip force is too strong. With sensory feedback, the participant moved the object more often without breaking it and did so more rapidly (basic). With divided attention (distr.), the feedback-induced boost in performance was maintained, but only the effect on duration remained statistically significant. *P < 0.05, n = 80 for both basic and distr. cases. Data show means ± SEM.

Performance of activities of daily living (ADLs) often involves dividing attention between multiple simultaneous subtasks—e.g., holding a jar while twisting off its lid—so sensory feedback that is attentionally demanding is inappropriate (37). To test whether the sensory feedback conveyed through nerve stimulation was resistant to divided attention, we had participants perform the fragile object test while counting backward. We found that the feedback-induced boost in performance was maintained with divided attention but only the effect on duration remained statistically significant under this condition (5.91 ± 0.20 s versus 7.68 ± 0.42 s; t test, P < 0.001; Fig. 3).

Another way to assess the impact of sensory feedback on object interactions is to characterize the degree to which we exert a consistent amount of force on an object upon repeated grasping (38). To test this capability, we had the participant repeatedly grasp a load cell with the prosthetic hand. Sensory feedback was provided on some experimental blocks but not others. The participant’s grip performance was more precise with sensory feedback than without, as evidenced by less variable grip force on six of eight objects (Fig. 4). Furthermore, sensory feedback significantly reduced the coefficient of variation (ratio of grip precision to grip force) across all objects [Fig. 4 and fig. S5 show the standardized Grasping Relative Index of Performance (GRIP) for this test] (38).

Fig. 4 Sensory feedback improves grip precision.

(A) Forces (means ± SD) generated by the participant when grasping a load cell while viewing one of eight different virtual objects. Sensory feedback improved grip precision, as evidenced by less variable grip force on six of eight objects. Without sensory feedback, the participant erred on the side of caution and underestimated desired grip force for fragile objects (bread, eggs, and open water bottle). (B) Coefficient of variation (means ± SEM) of grip force across all eight objects. Sensory feedback significantly reduced the coefficient of variation (i.e., the ratio of grip precision to grip force). Asterisk (*) indicates different means (P < 0.05), and sharp (#) indicates different SDs (P < 0.05); n = 40 for each object.

Sensory feedback enables haptic perception

When we manipulate objects, we acquire information about their shape, size, and texture through sensory signals from our hands (39, 40). Haptic perception relies on an interplay between exploratory movements and the sensory consequences of those movements (41). To assess the degree to which the prosthesis could convey object information, we developed a closed-loop sensorimotor task in which the participant actively manipulated one of two objects with the prosthetic index finger (fig. S6). Stimulation was at a fixed frequency and amplitude and was delivered as long as contact with the object was maintained. On each trial, one of two objects was presented—a golf ball or a (larger) lacrosse ball—and the participant’s task was to report the size of the object (small versus large). Alternative sensory cues were reduced or eliminated by having the prosthesis mounted externally on a table (rather than being worn by the participant) and by having the participant wear an eye mask and headphones. The participant was able to perform this task almost perfectly with the sensory feedback, correctly reporting the size on 31 of 32 object presentations (binomial test, P < 0.0001).

To further assess haptic perception, we developed a closed-loop sensorimotor task in which the participant actively manipulated one of two objects—a soft foam block or a hard plastic block—and discriminated the compliance (soft versus hard; fig. S7). In this experiment, the amplitude of electrical stimulation increased linearly with the output of the sensor. The participant was able to distinguish between the two objects significantly better than chance (60 of 80 trials; binomial test, P < 0.0001) and did so after squeezing the object several times (Fig. 5), highlighting the interplay between motor behavior and sensory feedback.

Fig. 5 Biomimetic sensory feedback improves performance on object discrimination tasks.

(A) Example force (top; blue) and change in force (top; red) when the participant actively manipulated a soft foam block. Note the repetitive waxes and wanes in force (e.g., at ~2 s), associated with the participant’s active exploration of the object. Traditional linear encoding tracks force only (bottom; light blue), whereas the first-order biomimetic encoding incorporates the first derivative of force (bottom; light red) and second-order biomimetic mimics the aggregate responses of tactile nerve fibers (bottom; light green). Linear algorithms were scaled (doubled) such that peak stimulation amplitude and frequency were matched to the biomimetic algorithms; arrows highlight the time when peak stimulation occurs for the different algorithms. (B) Biomimetic sensory feedback improved response time relative to its nonbiomimetic counterpart in size and compliance (comp.) discrimination tasks. Performance across experiments varied because of changes in stimulation parameters, but biomimetic stimulation consistently outperformed nonbiomimetic stimulation. *P < 0.05, n = 32 for binary versus biomimetic 1, n = 48 for linear versus biomimetic 1, and n = 32 for binary versus biomimetic 2. Data show means ± SEM.

Biomimetic peripheral nerve stimulation improves object discrimination

In the studies described above, sensory feedback provided either a contact signal or a signal proportional to the contact force. Although both regimes of stimulation led to significant improvements in closed-loop sensorimotor tasks, neither regime is liable to produce naturalistic patterns of activation in the nerve. Interactions with objects are characterized by a strong burst of activation at the onset and offset of contact and much weaker activation during maintained contact (42). This initial onset conveys important information about the shape of the object (40). The aggregate response of tactile nerve fibers is determined not only by the degree to which the skin is indented but also by the rate at which the skin is indented, and the latter component dwarfs the former one.

We therefore sought to implement a sensory feedback algorithm that incorporates this temporal property of natural tactile signals. As a first-order approximation, we developed a sensory feedback algorithm in which the intensity of stimulation was proportional not only to the contact force but also to its rate of change. This first-order biomimetic algorithm leads to stronger stimulation at the contact onset, when the rate of change is highest to mimic the phasic bursts observed in natural nerve activation during contact transients. To test this simple biomimetic algorithm, we had the participant discriminate the size and compliance of objects, and we compared his performance with that using the standard sensory encoding algorithms (contact tracking and force tracking). We found that the participant was able to perform these tasks significantly faster with the biomimetic feedback than with its nonbiomimetic counterparts. Biomimetic sensory feedback improved response time by 24% for size discrimination (11.78 ± 0.75 s versus 8.94 ± 0.79 s; t test, P < 0.05; Fig. 5) and by 44% for compliance discrimination (14.16 ± 1.05 s versus 7.91 ± 0.81 s; t test, P < 0.005; fig. S7).

In the above implementation of biomimetic feedback, the peak intensity of stimulation was higher than with nonbiomimetic feedback because the overall charge was approximately matched. One possibility, then, was that the improvement in performance with biomimetic feedback was a consequence of the higher peak stimulation intensity. Although a higher peak firing rate might itself be more biomimetic, improved discrimination would not necessarily depend on differences in temporal firing patterns between the biomimetic and nonbiomimetic encoding schemes. To distinguish between these possibilities, we implemented a version of the biomimetic algorithm such that the peak stimulation intensity (pulse amplitude and frequency) was matched to that of the nonbiomimetic algorithms. Even with matched peak intensity, the biomimetic feedback led to a 46% improvement in performance (7.56 ± 1.08 s versus 4.64 ± 0.77 s; t test, P < 0.005; Fig. 5). Another potential confound is that biomimetic algorithm might peak faster than the nonbiomimetic ones, leading to faster performance. However, the improvement in response time was on a longer time scale than the shift in peak stimulation, so this effect was not a trivial consequence of the timing of stimulation. Rather, it reflects an improvement in the intuitiveness and informativeness of the artificial sensory signals, which capture some of the essential temporal characteristics of natural tactile signals.

The above results suggest that dynamics of the response evoked through electrical stimulation—if it mimics a natural response—can lead to more interpretable and useful sensory feedback. However, the above biomimetic algorithm captured some aspects of the natural tactile feedback—namely, the increase in sensitivity to contact transients—but not others, borne out of the idiosyncratic properties of the different classes of tactile nerve fibers and their respective innervation densities. In light of this, we tested another sensory encoding algorithm that sought to more faithfully mimic natural nerve activations. Briefly, this algorithm is designed to reflect the measured sensitivity of populations of nerve fibers to skin indentation and its two derivatives (rate and acceleration) (19). With this second-order biomimetic feedback, the participant identified object compliance 56% faster than with the traditional linear feedback (6.71 ± 1.47 s versus 2.93 ± 1.37 s; t test, P < 0.05; Fig. 5). These results further demonstrate that biomimicry improves the intuitiveness of the artificial sensory feedback.

Participant successfully performed a variety of ADLs

An important concern in laboratory demonstrations of neuroprosthetic control is whether tasks that are used to assess the performance of the prosthesis are ecologically valid. With regard to the present study, will improvements in performance with sensory feedback on laboratory tasks translate to improved performance on ADLs? We evaluated this by having the participant complete several ADLs over 3 days of testing. With just the prosthesis alone or in conjunction with his intact hand, he performed basic ADLs (feeding and dressing) (43), instrumental ADLs (housework, meal preparation, and technology use) (44), and ADLs that he had found challenging without the prosthesis (loading a pillow into a pillowcase, hammering, donning, and doffing a ring; Fig. 6). Improvements are difficult to quantify with ADLs, but the participant noted that sensory feedback was particularly useful when manipulating fragile objects (e.g., eggs and grapes) and spontaneously reported that he enjoyed the sensation of “feeling” objects in his hand.

Fig. 6 Sensory feedback supports ADLs.

The participant performed several one- and two-handed ADLs while using the sensorized prosthesis, including moving an egg (A), picking grapes (B), texting on his phone (C), and shaking hands with his wife (D).

DISCUSSION

In the present study, we demonstrate that artificial sensory feedback improves fine motor control and confers to the user the ability to sense object properties through a bionic hand. Furthermore, these artificial sensory experiences are enriched when the sensory feedback is designed to mimic the nervous system’s natural language. By capturing some of the essential characteristics of natural tactile signals, biomimetic stimulation improves the intuitiveness and informativeness of the sensory feedback, as evidenced by swifter object discrimination capabilities.

The present results build on previous work, showing that sensory feedback leads to improved grip and handling of fragile objects (10). We extend these previous findings by showing that grasp force is achieved faster and more accurately and that fragile objects are transferred faster with sensory feedback than without. These improvements are augmented when the sensory feedback is biomimetic. Although previous studies have demonstrated that object properties can be sensed through a prosthetic hand (12), we extend these previous findings to a different sensorimotor task—compliance discrimination—and directly demonstrate the improvement of biomimetic feedback relative to its nonbiomimetic counterpart. In this respect, our work is consistent with a recent study, showing that biomimetic stimulation leads to more naturalistic percepts, leads to greater embodiment, and improves performance on object manipulation tasks (30). In the present study, we extend these previous findings to a new technology and a new task, an important replication of the benefits of sensory feedback and biomimicry, given that the relevant studies thus far have involved a single participant (12, 13, 15, 30, 45).

Amputees have expressed a desire for sensory feedback to reduce their dependence on visual feedback (37). The ability to feel grip force while grasping and holding objects is the most important aspect of sensory feedback for amputees (46). The sensory feedback provided here allowed the participant to perform object discrimination tasks without visual or auditory feedback and enabled the participant to exert grip forces more precisely.

Ideally, sensory peripheral nerve interfaces and encoding algorithms would activate each afferent nerve fiber selectively and independently, so as to replicate the spatiotemporal pattern of neural discharges that would be transmitted from an intact hand. The ability of different USEA electrodes to activate a large number of different percepts (Fig. 2) increases the ability to provide more biomimetic sensory input. The present experiments used relatively simple receptive fields and sensorimotor tasks to study the importance of temporal aspects of sensory encoding at a population level in isolation and hence did not fully explore these capabilities. However, such capabilities may prove increasingly useful with richer sensorimotor tasks and with the advent of prosthetic hands with greater numbers and varieties of sensors.

In addition to sensorimotor functional improvements, closed-loop sensorized prostheses often bring psychological benefits (9, 10, 29, 4749). The same participant in this study reported decreased phantom pain and increased embodiment of the prosthesis as a result of the sensory feedback (29). After the study, the participant highlighted the emotional impact of artificial touch when he used the bionic hand to shake hands with his wife and felt her touch through it for the first time. The functional and emotional benefits of dexterous motor control and biomimetic sensory feedback are likely to be further enhanced with long-term use, and efforts are underway to develop a portable take-home system (50).

MATERIALS AND METHODS

Study design

We considered the participant for this chronic implant study due to the transradial level of his amputation, his willingness to volunteer, and overall good health. Termination of the study and explantation of the electrodes were voluntary or would occur if the implants were causing a health risk as indicated by a qualified physician or at 14 months after the implant date. Previous studies from this group (6, 51, 52) were limited in duration (less than 5 weeks) for safety considerations; because no health risks emerged from these previous studies, the University of Utah Institutional Review Board and the participant agreed to a 14-month duration for this study.

The experiments performed in this study were completed in 2- to 3-hour sessions, one to three times a week, across the 14-month duration of the study. The number of replicates per experiment was matched to that of previous studies involving fragile object manipulation (10), object discrimination (12), and the Grasping Relative Index of Performance (38). Data were considered outliers if they fell outside three SDs from the mean (38).

Human participant and implanted devices

A male left transradial amputee, whose amputation occurred 13 years before the onset of the study, underwent surgeries and performed experiments with informed consent and under protocols approved by the University of Utah Institutional Review Board and the Department of the Navy Human Resources Protection Program. Under general anesthesia, two 100-electrode USEAs were implanted in the median, and ulnar nerves of the residual limb, proximal to the elbow, and eight iEMGs, with four electrical contacts each, were implanted in the upper forearm with attempted targeting of each lead to different lower-arm extensor or flexor muscles. Additional information and figures regarding the devices and implantation procedure can be found in the Supplementary Materials and (29), which reports on the same participant in this study.

Decoding motor intent

Motor intent was decoded from residual forearm muscles recorded at 1 kHz, while the participant actively mimicked prosthetic hand movements, as previously reported in (6, 29, 31). Every 33 ms, the mean absolute value (MAV) over a 300-ms window was calculated for the 32 iEMG electrodes and the 496 possible differential pairs. A total of 528 features were generated (MAV for 32 single-ended and 496 differential pairs). To save computational time and reduce potential overfitting, the 528 features were then down-selected to the best 48 features using a Gram-Schmidt channel-section algorithm (53). These 48 features served as an input to a modified Kalman filter (MKF)–based decode that uses customizable, non-unity thresholds and gains (29, 54). The output of the MKF was used to directly control the position or velocity of the six DOFs of the prosthesis. The ability to proportionally control position or velocity was toggled on a DOF-by-DOF basis. More information regarding the prosthetic control algorithm can be found in (54) and the Supplementary Materials.

Mapping of USEA-evoked percepts

Electrical stimulation was delivered via USEAs using the Ripple Neuro LLC Grapevine System with Micro2+Stim front ends. All stimulation was delivered as biphasic, cathodic-first pulses, with 200- to 320-μs phase durations and a 100-μs interphase duration. The stimulation frequency varied between 10 and 500 Hz, and stimulation amplitudes were in the range of 1 to 100 μA.

USEA stimulation threshold maps were collected roughly every 4 to 8 weeks, during which each electrode of the USEAs was stimulated in isolation at increasing amplitudes. Electrodes that evoked a sensory percept at less than 100 μA were noted, and the location, quality, and intensity of each percept were documented as well as the threshold amplitude at which the percept was evoked. For these mappings, stimulation was delivered in a pulsed fashion with a 500-ms train of 100-Hz stimulation being delivered every second. Additional descriptions for electrode mapping (6) and the stimulation parameters we used (29) exist elsewhere. Sensory percepts were stable over the course of these experiments and persisted 14 months after the implant (fig. S8). More information regarding the stability of the USEA-evoked percepts is available in the Supplementary Materials.

Encoding sensory feedback

Stimulation through a single USEA electrode typically evoked a single percept with a distinct receptive field (e.g., sensations were isolated to only the index finger, or only the middle finger, but not both fingers). Occasionally, stimulation of a single USEA electrode would evoke multiple percepts in distinct receptive fields (e.g., stimulation of a single USEA electrode evoked sensations on both the index and middle fingers); these electrodes with multiple distinct percepts were not used for real-time sensory feedback.

The distinctly evoked percepts were then assigned to a single contact (cutaneous) or motor (proprioceptive) sensor on the prosthesis with a corresponding receptive field. For example, if stimulation through USEA electrode X evoked a pressure-like percept on the middle finger and if separately stimulating through USEA electrode Y also evoked a percept on the middle finger, then both electrodes X and Y would be assigned to the middle finger contact sensor on the prosthesis. We stimulated between 1 and 12 USEA electrodes that had overlapping receptive fields with a given sensor on the prosthesis (Table 1). Because of the time-intensive nature of assigning all electrodes, a subset of sensors on the prosthesis were used for each task; the specific sensors used for a particular task are detailed in the corresponding section for that task. Activation of sensors resulted in biphasic, charge-balanced stimulation (200- or 320-μs phase durations, cathodic first, with a 100-μs interphase duration). We encoded percept intensity by modulating the frequency or current amplitude of stimulation with either linear or biomimetic encoding algorithms (see next section). For all encoding algorithms, the intensity of the sensation increased with increasing stimulation amplitude and frequency, but there were no reported changes in perceptive field location or sensory modality.

Table 1 Stimulation parameters used for each task.

View this table:

Stimulation parameters were adjusted at the start of each experimental session to maximize the naturalism and perceived intensity range of the stimulation. To the extent possible, the participant’s sensory experience (e.g., perceived intensity range, perceptive field, etc.) was kept consistent across days. Stimulation typically produced natural-feeling pressure sensations on the palmar aspects of the hand. The exact parameters (electrodes, encoding algorithm, amplitude, frequency, and pulse duration) used for each task are summarized in Table 1.

Sensory encoding algorithms

For binary sensory encoding, the stimulation was fixed at the specified amplitude (100 μA, 320 μS) and frequency (100 Hz) as long as any contact was made. For traditional, linear sensory encoding, the stimulation frequency and amplitude increased solely on the basis of the absolute sensor value. For biomimetic 1 sensory encoding, the stimulation frequency and amplitude increased on the basis of the absolute sensor value and on positive rate of change of the sensor; stimulation tracked the current sensor value plus 10 times any positive finite difference between the current and previous sensor value. For scaled, traditional, linear sensory feedback, the stimulation frequency and amplitude were multiplied by a constant factor (=2) such that the range was comparable with that of the biomimetic stimulation (Fig. 5). Stimulation amplitude and frequency increased together over their respective ranges (see Tables 1 and 2).

Table 2 Sensory encoding algorithms.

Ft, frequency at time t; At, amplitude at time t; ct, normalized contact value at time t; vt, velocity at time t; at, acceleration at time t; min, minimum value; max, maximum value. Note that for all algorithms, sensory feedback is off and no stimulation occurs when ct = 0.

View this table:

The biomimetic 2 sensory encoding algorithm was developed from recordings of nonhuman primate cutaneous afferents in response to physical contact with the fingertip (19). This computationally inexpensive model describes the instantaneous firing rate (i.e., stimulation frequency) of the afferent population using the contact stimulus position, velocity, and acceleration. Similar to other biomimetic algorithms (30), the biomimetic 2 sensory encoding algorithm leverages TouchSim (55) to simulate the responses of all tactile fibers to any spatiotemporal deformation of the skin of the hand. This model—dubbed TouchMime—provides a more computationally efficient approach to the aggregate response of the nerve to time-varying pressure applied to the fingertip, allowing for high-accuracy biomimetic sensory encoding in real time. In addition, the parameters of the model were tuned for the sampling rate of the DEKA LUKE arm sensors (30 Hz) and for USEA stimulation (i.e., intrafascicular stimulation at 30 Hz) at a fixed, suprathreshold stimulation amplitude, further improving the accuracy of the biomimetic encoding (19). Additional details regarding the model development and validation can be found in (19).

Both models presented here are distinct from those used in (30). The biomimetic 1 algorithm concurrently modulates frequency and amplitude most closely replicating the responses of populations of slowly adapting type 1 (SA1) and rapidly adapting (RA) fibers. The biomimetic 2 algorithm provides a more faithful replication of a complete aggregate nerve response, keeping the population size constant (fixed stimulation amplitude) and mimicking the aggregate firing rate of SA1, RA, and Pacinian fibers within that population of the nerve. Both models are computationally efficient, allowing for real-time biomimetic sensory encoding. Analytic formulations for each encoding algorithm are provided in Table 2.

We did not attempt to measure the intuitiveness or naturalism of the sensory encoding algorithms, nor did we track the participant’s ability to interpret this feedback. Experimental sessions were kept under 2 hours, and no learning effects were observed in this time frame.

Fragile object test

The fragile object test [originally introduced in (35)] has been used as a variant of the modified Box and Blocks test (36) to show the benefits of sensory feedback (14, 30, 36). Our implementation of this test differed from its predecessors in that the object was much heavier and the ratio between the weight and breaking force was much smaller, rendering the overall task more difficult. In (36), the fragile object weighed 8 g and broke if a force of 10.7 ± 1 N was applied to it (ratio of 1.34 N/g), and in (14, 30), the object weighed ~80 g and broke with a force of 1.23 ± 0.02 N (ratio of 0.15 N/g). In contrast, the object used in this study weighed 630.57 g and broke at 14.79 ± 0.34 N (ratio of 0.02 N/g).

The participant used only thumb abduction/adduction, and artificial sensory feedback was provided on the basis of the thumb contact sensor. Trial failure was defined as “breaking” the object, which occurred when the compression force exceeded 14.79 ± 0.34 N, or an inability to move the object in 30 s. Trial success was defined as a trial in which the participant lifted and placed the unbroken object within an adjacent circle on the table (~10 cm away) within 30 s. In half of the sets, the participant was intentionally distracted by having to count backward by twos from a random even number between 50 and 100.

A single trial was performed once every minute. A single experimental block consisted of eight trials with or without artificial sensory feedback. The participant completed five experimental blocks with and without sensory feedback for both the basic and distracted conditions. The experimental blocks were counterbalanced to reduce order effects. Under all conditions, the participant was able to use audiovisual feedback to help locate and grasp the object, as well as to identify when the object broke.

Statistical analyses were run separately for the basic and distracted conditions. A 50% binomial test was used to determine whether performance was significantly greater than chance alone. For comparison of completion time for the successful trials, response times showed no deviations from normality (Anderson-Darling, Jarque-Bera, and Lilliefors tests). Unpaired t tests (unequal sample size due to different success rates) were then used to compare completion times.

Object discrimination tasks

For size discrimination, the participant had to distinguish between a “large” lacrosse ball and a “small” golf ball (fig. S6). The two objects were chosen so that they represented real-world interactions, minimized differences in compliance, and maximized differences in size while still requiring some degree of active flexion to make contact. Relative to the index finger’s full range of motion, the large object required a 19% decrease in joint angle to make contact, and the small object required a 49% decrease. Response time was measured from the start of the trial to when the participant verbally reported the object’s size.

For compliance discrimination, the participant had to distinguish between a “soft” foam block and a “hard” plastic block (fig. S7). The soft block was cut to match the size of the hard block so that stimulation due to initial contact occurred at the same degree of index flexion. Response time was measured from the start of stimulation (i.e., measurable contact was made with the object and the participant felt the object) to the time when the participant verbally reported the object’s compliance.

We did not attempt to quantify how many levels of size and compliance the participant was able to discriminate. With traditional linear feedback, the just-noticeable difference of the neural stimulation would bind the discrimination capabilities. Instead, we focused on quantifying improvements in the intuitiveness of the sensory feedback (measured by response time) when biomimetic stimulation regimes are used.

For both, the output of the modified Kalman filter was used to directly control the position of the index finger. Position control (i.e., postural control) provided improved performance relative to velocity control (fig. S6), which is consistent with the natural encoding schemes of the hand (56). The participant received cutaneous sensory feedback from the index contact sensor; proprioceptive sensory feedback was not provided, although endogenous proprioception of residual forearm muscles and efference copy may have been present. The participant was blindfolded and wore headphones, and the physical prosthesis was detached from his residual limb, so that external cues about the object were eliminated.

A single trial was performed once every minute. For each trial, the participant was given 30 s to complete the task. A single experimental block consisted of eight trials using a single algorithm. The participant completed two experimental blocks for each size discrimination algorithm and two to six experimental blocks for each compliance discrimination algorithm. The order of the objects was pseudorandomized such that equal numbers of both appeared in the experimental block. The experimental blocks were counterbalanced to reduce order effects.

Statistical analyses were run separately for each algorithm comparison. Because of limited time with the participant, direct comparisons were limited to biomimetic 1 versus traditional linear, biomimetic 1 versus scaled traditional linear, and biomimetic 2 versus scaled traditional linear. A 50% binomial test was used to determine whether performance was significantly greater than chance alone. For algorithm comparisons, response times showed no deviations from normality (Anderson-Darling, Jarque-Bera, and Lilliefors tests). Paired t tests were then used for these comparisons on a trial-by-trial basis to control for order effects and sensory adaptation (57). Statistical analysis of response times with biomimetic and nonbiomimetic encoding algorithms was confined to algorithms using the same stimulation parameters on the same day to avoid any variations in evoked sensations that may have occurred across days.

Grasping Relative Index of Performance test

A detailed description of the GRIP test is reported elsewhere (38). Briefly, a screen was placed between the participant’s line of sight to the prosthesis and the load cell to eliminate audiovisual cues from the prosthetic hand. In contrast to the fragile object test, the GRIP test measures the ability to modulate grip force without audiovisual feedback. The participant was presented with pictures of one of eight objects (Fig. 4) and instructed to grab the load cell with a force appropriate for gripping the object shown in the picture. The participant grabbed each of the eight objects 20 times without sensory feedback and 20 times with sensory feedback. Outliers and trials with preemptive grasps were not included in the analysis (38). Peak grasping forces showed no deviations from normality (Anderson-Darling, Jarque-Bera, and Lilliefors tests). Unpaired t tests were used to compare means, and Levene’s test was used to compare SDs.

DEKA LUKE arm and ADLs

The DEKA LUKE arm, as used in this study, has 6 moveable DOFs (Table 3), 6 position sensors, and 13 contact sensors (Table 4). The prosthetic is interfaced via a controller area network communication protocol with 100-Hz update cycles. The accuracy of the movements is dictated by the precision of the motor commands (Table 3). For more information regarding the accuracy of the control algorithm, see the Supplementary Materials and (54).

Table 3 Motor control specifications.

View this table:
Table 4 Sensor specifications.

View this table:

The DEKA LUKE arm, in its transradial configuration, weighs about 1.27 kg (58), slightly more than that of an intact human hand. There are no temperature or pain sensors on the DEKA LUKE arm. Furthermore, electrical stimulation of sensory afferents preferentially activates larger diameter fibers first (59), making USEA-evoked pain or temperature percepts uncommon.

The six position sensors correspond to the six moveable DOFs. The 13 contact sensors are made of nine torque sensors for contact applied to the fingers and four force sensors for contact applied to the hand. There is a torque sensor for digits D2 to D5 that detects torque applied to the finger opposing flexion (e.g., during grasping of an object) and a torque sensor for the lateral portion of D2 (e.g., during a key grip). D1 also has four additional torque sensors to detect contact due to adduction, abduction, reposition, or opposition.

ADLs were performed with the DEKA LUKE arm mounted to a custom socket that fit to the residual limb of the participant. With only the prosthesis or with in conjunction with his intact hand, the participant performed basic ADLs (feeding and dressing) (43), instrumental ADLs (housework, meal preparation, and technology use) (44), and ADLs that he had found challenging without the prosthesis (loading a pillow into a pillowcase, hammering, donning and doffing a ring; Fig. 6). All ADLs were performed with audiovisual feedback to best approximate real-world use. Traditional linear sensory feedback was provided because ADLs were performed before implementing the biomimetic encoding algorithms. Because of limited patient time and an inability to precisely quantify performance, ADLs were not repeated with biomimetic sensory feedback.

Statistical analyses

All statistical analyses were run with significance as α = 0.05. Data were checked for normality to ensure that the appropriate parametric analysis or nonparametric equivalent was used. Subsequent pairwise analyses were corrected for multiple comparisons using the Dunn-Šidák approach. All data are shown as means ± SEM, unless otherwise stated.

SUPPLEMENTARY MATERIALS

robotics.sciencemag.org/cgi/content/full/4/32/eaax2352/DC1

Stability of the USEA

Decoding intended movements with a modified Kalman filter

Surgical procedure

Fig. S1. Modified Box and Blocks test.

Fig. S2. Prosthesis efficiency and profitability task.

Fig. S3. Projected fields of electrically evoked sensations.

Fig. S4. Fragile object test.

Fig. S5. Grasping Relative Index of Performance task.

Fig. S6. Size discrimination task.

Fig. S7. Compliance discrimination task.

Fig. S8. Stability of USEA-evoked sensations.

References (6066)

REFERENCES AND NOTES

Acknowledgments: We thank the participant in this study who freely donated 14 months of his life for the advancement of knowledge and for a better future for amputees. Funding: This work was sponsored by the Hand Proprioception and Touch Interfaces (HAPTIX) program administered by the Biological Technologies Office (BTO) of the Defense Advanced Research Projects Agency (DARPA), through the Space and Naval Warfare Systems Center (contract no. N66001-15-C-4017). Additional sponsorship was provided by the NSF through grant no. NSF ECCS-1533649 and NSF GRFP award no. 1747505. Author contributions: J.A.G. developed and implemented sensory encoding algorithms, designed experiments, collected data, and drafted the manuscript. D.T.K. developed the software for the prosthesis, designed experiments, collected data, and assisted with drafting of the manuscript. T.S.D. developed the software for closed-loop prosthesis control. E.V.O. and Q.H. developed the sensory encoding algorithm based on afferent recordings. C.C.D. provided clinical support and expertise throughout. D.T.H. implanted and explanted the USEAs and iEMGs, provided clinical oversight throughout the study, and oversaw the development of experiments and protocols. Z.C.T., D.T.B., and P.D.M. provided the equipment, analysis, and guidance for GRIP and PEP tests (under DARPA contract no. N66001-15-C-4015). S.J.B. oversaw the development of sensory encoding algorithms, assisted in the experimental design, and aided with drafting of the manuscript. G.A.C. oversaw and led the development of all methods, experiments, and protocols and assisted with experiments and drafting of the manuscript. All authors contributed to the revision of the manuscript. Competing interests: D.T.K., T.S.D., S.M.W., C.C.D., and G.A.C. are inventors of the prosthetic control algorithm used in this study (international application no. PCT/US2017/044947). D.T.K., S.M.W., D.T.H., and G.A.C. are inventors of the protective carrier devices used during the implant procedure (international application no. PCT/US2017/044427). G.A.C. holds patents on the implants’ signal referencing and antinoise architecture (U.S. patent nos. 8639312 and 8359083). T.S.D. is a former consultant for Ripple Neuro LLC. D.T.K. is now an employee of Blackrock Microsystems, which provided implanted materials for this study. Data and materials availability: All data needed to evaluate the conclusions are available in the paper or the supplementary materials. Data and materials requests should be sent to G.A.C. (greg.clark{at}utah.edu). Requestors may need to be approved by the human-subjects research committees (e.g., local institutional review board and Department of the Navy Human Resources Protection Program) to comply with Health Insurance Portability and Accountability Act requirements.
View Abstract

Stay Connected to Science Robotics

Navigate This Article