ReviewSENSORS

Electronic skins and machine learning for intelligent soft robots

See allHide authors and affiliations

Science Robotics  22 Apr 2020:
Vol. 5, Issue 41, eaaz9239
DOI: 10.1126/scirobotics.aaz9239

Abstract

Soft robots have garnered interest for real-world applications because of their intrinsic safety embedded at the material level. These robots use deformable materials capable of shape and behavioral changes and allow conformable physical contact for manipulation. Yet, with the introduction of soft and stretchable materials to robotic systems comes a myriad of challenges for sensor integration, including multimodal sensing capable of stretching, embedment of high-resolution but large-area sensor arrays, and sensor fusion with an increasing volume of data. This Review explores the emerging confluence of e-skins and machine learning, with a focus on how roboticists can combine recent developments from the two fields to build autonomous, deployable soft robots, integrated with capabilities for informative touch and proprioception to stand up to the challenges of real-world environments.

INTRODUCTION

Skin plays an essential role for biological systems as a barrier between an organism’s external environment and its internal components. Embedded within its layers is a dense network of mechanical, chemical, vibrational, temperature, and pain receptors, which work in coordination to enable somatosensation in skin. These capabilities would also be incredibly useful for robots. Electronic skin (e-skin) research was originally motivated, in part, by a desire to understand biological sensing, but the lessons learned can help improve the design of robotic systems. To sense, plan, and act, robots require a variety of sensors embedded throughout their bodies so that they can obtain information about their environment.

The field of soft robotics (1) studies the use of flexible and compliant materials as components for building robots, instead of traditionally rigid components such as metals. Soft robots often draw inspiration from nature, which has evolved organisms that can operate in unstructured environments. In contrast, current robotic systems are usually confined to structured laboratories or warehouse environments. In addition, natural environments typically contain several objects of varying material properties that further complicate tasks such as object interaction and locomotion.

The overlap between e-skins, soft robotics, and machine learning is continually growing, and recent advances are summarized in Fig. 1. Soft actuation has improved tremendously in capabilities (Fig. 1 bottom), and soft sensors and e-skins exhibit a wide range of complexities (Fig. 1 left).

Fig. 1 Trends in the intersections between e-skins, soft robotics, and machine learning.

(Left) A range of e-skins and soft sensors that increase in complexity, from bottom to top, by metrics including density, resolution, and fabrication (19, 21, 2630). (Middle) Soft robots and e-skins that merge actuation and sensing (10, 3745, 47), from left to right and top to bottom. (Bottom) Soft robots that focused primarily on actuation (3135), from left to right. Red boxes indicate work that has leveraged machine learning in the processing of their sensor information. [CREDIT: ADAPTED BY A. KITTERMAN/SCIENCE ROBOTICS FROM MATERIAL PUBLISHED BY IEEE (10, 26, 27, 32, 37, 4042, 45, 109), SPRINGER/NATURE (19, 20, 35), WILEY (28), THE NATIONAL ACADEMY OF SCIENCE (31), AND AAAS/SCIENCE (21, 29, 33, 34, 39, 43, 44, 47), LORI K. SANDERS/HARVARD UNIVERSITY (35), A. ECKERT/TUM (109)]

SEE FIG. 1 CAPTION FOR CREDIT INFORMATION

Several recent advances have combined principles from each field, often physically manifesting in the form of sensorized fingers and grippers (Fig. 1 top). Future breakthroughs in the field may come from further integration of sensors and actuators as roboticists move toward designing systems that rival the abilities of biological organisms.

Several reviews have covered related topics on e-skins and perception in soft robots, including design and fabrication of e-skins (2, 3), wearable sensors (4), e-skins for interactive robots (5, 6), and future directions in sensing and perception for soft robots (7, 8). This Review examines recent developments in skin-based sensing for soft robots, covering hardware and fabrication techniques and machine learning techniques that translate robot perception into action planning. To limit the scope of this Review, we consider a soft robot skin to be skin sensors directly mounted on the surface [e.g., (9)] or embedded in a thin layer beneath the surface of the body of a soft robot [e.g., (10)]. To highlight the opportunities at the intersection of e-skin and soft robotics research, we cover a variety of interdisciplinary topics including fabrication, learning, and control.

INTERDISCIPLINARY TOOLS

Design and fabrication of integrated e-skins

Compared with rigid robots, the high mechanical compliance of soft robots enables safer and more efficient human-robot interaction (HRI) because they can seamlessly interact with the human body (11). Further advancement of soft robots requires high-performance electronics and sensors that can stretch continuously with their bodies. Recent research in artificial skin has mainly focused on making individual sensor devices with better performance, such as sensitivity, stretchability, and reliability over many use cycles (Fig. 2). To realize fully biomimetic skin for soft robotics, artificial skin should contain sensor arrays that are stretchable, cover large areas with a high spatiotemporal resolution, and have multiple functions that mimic diverse receptors of the human skin (Fig. 2A). These features should enable robots to use data-driven methods to extract rich information about their environment.

Fig. 2 Sensor arrays enable e-skins to extract information about their environment.

(A) Human skin with various receptors used to sense stimuli. (B) A scalable tactile glove containing a passive matrix of 548 force sensors for the collection of large datasets (12). (C) 2D sensor array used to generate a profile of pressure intensity from experimental mapping of the pixel signals using an active matrix (126). The icons at the bottom represent biological analogies: Merkel disks, Meissner corpuscle, and free nerves. (D) A 3D array of electronic sensors assembled from 2D electronics (127). (E) Multimodal sensor array that can capture both pressure and temperature information (24).

CREDIT: ADAPTED BY A. KITTERMAN/SCIENCE ROBOTICS FROM MATERIAL PUBLISHED BY SPRINGER/NATURE (126, 127) AND SCIENCE ROBOTICS (24) AND UNPUBLISHED MATERIAL FROM THE AUTHORS OF (12)

Increasing sensor density and quantity normally requires a larger number of interconnecting wires. To reduce this burden, sensor arrays are normally designed in matrix form. For example, a recently developed tactile glove comprising 548 force sensors was constructed using readily available materials and simple fabrication tools (Fig. 2B) (12). This sensor array recorded a large-scale dataset of tactile maps (about 135,000 frames), which was used to identify objects using convolutional neural networks. This work highlights the ability of large-scale datasets collected by a high-density sensor array to enable not only a sense of touch but also the intelligent extraction of information from touch. Increasing the sensor density simply by scaling down a passive matrix architecture will reduce the amplitude of analog signals while increasing cross-talk between interconnects. If multiple sensors are sampled simultaneously, each line will produce electromagnetic noise, which will corrupt the signals being carried on neighboring conductive traces. Furthermore, the large number of addressing lines will be difficult to manage as the number of sensors increases substantially. These problems can be addressed with an active matrix that pairs each sensor with a transistor to provide local signal amplification and allows sensors to take turns transmitting information (1316).

Active matrices with multiplexed signal transduction typically consume less power than passive matrices because they require fewer sampling lines and do not need external circuitry (17). However, stretchable e-skins could allow better coverage of curved robot surfaces while allowing sensing of complex texture information through detection of deformation and vibration, mimicking biological skin. Recent advancements in organic electronics by Wang et al. (18) led to the creation of an intrinsically stretchable transistor array with 347 transistors per square centimeter. Their proof-of-concept demonstration illustrated that such a conformable active matrix could accurately map the force applied on each sensor. These capabilities indicate that stretchable active matrices containing soft sensors and transistors are a promising step toward soft robotic skin with high resolution and high data fidelity.

Making multilayered sensor arrays in a three-dimensional (3D) lattice can further increase the sensor areal density and allow greater integration of different sensor modalities. Just as receptors in biological skin are embedded at various depths, engineers can embed sensors that are sensitive to different stimuli in different spatial locations. For example, pressure, shear, and strain sensors can be distributed in different layers of the e-skin to achieve optimized sensitivity. Huang et al. (19) demonstrated that stretchable electronics integrated in 3D can be built with a layer-by-layer method using transfer printing of predesigned stretchable circuits on elastomers with vertical interconnects. This stretchable human-machine interface had a four-layer design that offered multimodal sensing and had integrated circuits for wireless data transfer. Using strain engineering methods, 2D structures can also be assembled into 3D electronic systems with sensing capabilities. Semiconductor materials can play critical roles in this context, through demonstrations of complex, mechanically assembled 3D systems for light-imaging capabilities that can encompass measurements of the direction, intensity, and angular divergence properties of incident light.

3D printing has also been used to directly print sensors in soft robots to improve both exteroceptive and interoceptive capabilities (20). This work highlights how a 3D integration framework enables a higher integration density on stretchable substrates than single-layer approaches and allows new functionalities that would be difficult to implement with conventional layer-by-layer designs.

Processing complex tactile information from a sensor array requires efficient signaling and sampling methods. In human skin, stimulation of the receptors is converted into a series of voltage pulses sent to the nerves. This inspired researchers to develop artificial receptors and afferent nerves to convert tactile information to digital pulses at the site of sensation (21, 22). The signal could potentially be perceived by a user’s nerves and brain, thus directly linking the human brain with soft robotic prosthetics. For example, Kim et al. (23) recently developed a flexible artificial afferent nerve that can effectively collect pressure information from arrays of pressure sensors and convert them to action potentials to activate muscles.

Biological skin contains receptor networks that can detect various stimuli, such as vibration, humidity, and temperature. Several studies on e-skin sensor arrays focused on the classification of a single type of information, such as force, shape, or direction of motion. The next generation of e-skins should integrate multimodal sensor arrays to capture richer sensory information than their predecessors. Recently, Lee et al. (24) reported a neuromimetic architecture that enabled simultaneous transmission of both tactile and thermotactile information (Fig. 2E). The pressure- and temperature-sensitive transducers can both be communicated through the pulse signatures by a single electrical conductor. As a biomimetic signaling method, this approach is promising for reducing wiring and computational requirements when a robot is covered with thousands of sensors. Multimodal sensing could also be achieved through integration of multiple stretchable optical fibers, which has been shown to be effective at localizing and estimating force in soft actuators (25).

Overall, many innovations are required for realizing high-density and multifunctional sensor arrays for soft robots. A close collaboration between roboticists and materials scientists is needed to develop high-performance stretchable conductors for electrodes and interconnections and stretchable semiconductors for building active matrices and signal amplifiers. Different sensing modalities and integration architectures should also be explored. Lastly, hardware and algorithms for data processing should be considered during the design of sensory systems, and their performance should be evaluated on a holistic range of practical robotic tasks.

Skin-based sensing for soft robots

As sensors are increasingly integrated into soft robots, we can imagine a conceptual plane that categorizes research based on the sophistication of actuation and sensing independently (Fig. 1). Stand-alone sensors lie on the y axis; some consist of simpler strain sensors [the bottom three images in Fig. 1 left (2628)], whereas others have more sophisticated sensing schemes, including distributed or multimodal sensing [the top four images in Fig. 1 left (19, 21, 29, 30)]. The x axis, representing actuation-focused soft robots, shows examples of increasingly complex soft systems that can walk (31, 32), grow (33), swim (34), and operate autonomously on chemical fuel (35) (Fig. 1 bottom). Last, many recent works have begun exploring the intersection of the actuation and sensing (Fig. 1 middle). Several of them embedded strain sensors for state estimation or tactile sensing in a finger-like structure (10, 3644), whereas the others mounted their skins externally (4547). As both areas progress, we envision further integration of increasingly sophisticated actuation and sensing, extending into the top-right quadrant of the conceptual plane.

Access to higher-resolution data about touch will increase the ability of soft robots to perceive the complex deformations that they experience during tasks, including locomotion and manipulation. Today’s discrete sensors, which are built with high sensitivity and selectivity, can be tailored to sense deformation modes in a localized region or known environment with high confidence (46, 48). However, this sensing paradigm is insufficient in dynamic or unknown environments where robots will experience substantial deformation because robots do not yet have the level of sophistication of human skin receptors or the human brain to collect a broad range of information. In addition, many robots are unable to process the volume of information to accurately determine the environment or the object being sensed. The transition from discrete to continuous sensing and the shift from structured to unconstrained environments both require e-skins that can rapidly collect and process large amounts of information. The added complexity from both transitions compounds the processing required to interpret the signals.

Several designs of skin-like sensors have been used in soft robotics. Many of these sensors contain conductive and stretchable materials to produce resistive or capacitive strain sensors (10, 26, 49, 50). Other groups have used optical devices such as cameras and optical fibers to sense deformations within an actuator (5153). Several of these existing sensors are well suited for measuring characteristics such as strain, pressure, and bending but do not enable the high sensor densities or resolutions that have been demonstrated in e-skins. Soft robots would benefit from integration with e-skins, such as the skin-like sensor arrays that have been deployed in medical applications or directly on skin (22, 29, 5456).

Currently, soft skin-like sensors have been deployed in several ways. Some groups used their sensors as wearables; Mengüc et al. (57) used liquid metal sensors to measure human gaits using a sensor fabrication process first presented by Park et al. (27). The resistance of these sensors increases as the embedded microchannels inside the elastomer matrix are stretched because of the increased length and decreased area of their bulk liquid-metal channels. Others incorporated their sensors with robots: Boutry et al. (58) paired a shear force sensor with a robot arm to allow robotic hand control. Booth et al. (47) demonstrated reconfigurable, actuatable e-skins that could control the motion of deformable inanimate objects from their surface. Zhao et al. (43) embedded optical sensors within soft pneumatic fingers, which they then integrated with a Baxter robot. As skin-based sensing capabilities continue to improve, the goal is to develop capabilities that match or outperform biological systems (top right corner of Fig. 1).

Machine learning for soft e-skins

As e-skins increase in resolution, their signals could be processed to detect higher-order deformation modes and higher-level notions about the environment, such as material type. However, obtaining this information requires algorithms that can extract useful information from large quantities of data. To handle the vast amount of data that e-skins can provide, machine learning is emerging as a versatile tool for making sense of large quantities of data (Fig. 3). For example, Piacenza et al. (59) obtained high-resolution data from a robotic fingertip and used ridge regression to process this data to estimate the locations of indentations. Similarly, Larson et al. (60) used convolutional neural networks to learn deformations on a sensor array that can interpret human touch in soft interfaces. At the level of abstraction of the entire robotic system, Van Meerbeek et al. (53) tested various learning algorithms to estimate the twist and bend angles in sensorized foam, finding that k-nearest neighbors (kNN) outperformed other common algorithms, including support vector machines (SVM) and multilayer perceptrons. In addition, researchers have also focused on recurrent neural networks, which have been shown to be advantageous for learning patterns in time series data (36, 39, 61, 62).

Fig. 3 Machine learning techniques for processing raw sensory information, different levels of abstraction to aid in robot perception, and action planning.

The level of abstraction depends on the task, and the most effective type of learning architecture depends on the quality and structure of the sensor signals. The level of abstraction depends on the task, and the most effective type of learning architecture depends on the quality and structure of the sensor signals. Higher-level processes can include parallel execution of lower-level processes. End-to-end architectures [e.g., (61)] without mid-level and low-level pipelines would likely be faster and more effective but are computationally expensive to develop.

CREDIT: A. KITTERMAN/SCIENCE ROBOTICS

Because of the complexity of the mapping between raw sensory information and relevant functional abstractions, information theory and machine learning will play a large role in bringing tactile sensing to human-like performance levels. In particular, the subfield of reinforcement learning (RL) will be important for developing closed-loop control for tactile feedback. Suitable algorithms and architectures for analogous tasks in soft robotics can be developed by learning from biological processes. For example, in computer vision and machine learning, the hierarchical nature of visual processing (corresponding to compositional functions) (63) has recently enabled deep neural networks to achieve human-like performance across a variety of visual processing tasks (64). Processing signals from arrays of tactile sensors may benefit from similar techniques, as sets of sensor readings have information encoded in spatial relationships that can be naturally represented using matrices.

Tactile exploration can benefit from recent developments in learning-based simultaneous localization and mapping algorithms. Notably, Mirowski et al. (65) used an asynchronous advantage actor-critic algorithm for navigating in a complex environment and additionally solved auxiliary prediction tasks that made the RL problem faster and more data efficient. Chen et al. (66) showed a direct policy learning algorithm with spatial memory and bootstrapped with human-mediated imitation learning without explicit task rewards. In the absence of continuous reward functions, actor-critic algorithms are preferred because they require fewer samples.

Similarly, tactile manipulation tasks can use insights from learning-based manipulation controllers. A general trend observed in such works is the success of model-based RL (67) or learning by demonstration (68), approaches that leverage techniques from control theory or human knowledge, respectively. There has been a successful solution for the direct learning of control policies for dexterous manipulation, but it relied on the availability of an accurate simulation environment (69). Until robot simulators can model soft-body dynamics that reliably transfer to real robot hardware, such approaches are difficult to apply to soft robots and deformable objects. For specific simple tasks, it might be easier to find a direct policy than to fit a general-purpose model of the system dynamics (70).

APPLICATIONS OF E-SKINS

Shape sensing

Whereas environmental sensing helps a robot understand its surroundings, having a self-model of the robot’s body is important for planning trajectories and actions within that environment. For robots primarily composed of rigid components, the geometry of each segment remains the same throughout the robot’s lifetime, and relative rotations or translations of links provide enough additional information to fully specify the overall changes in shape. However, for soft robots, individual segments can continuously change their shapes, via both intentional and unintentional deformation modes, which complicates modeling and sensing schemes. Complementing recent work on soft sensing (7), the direct sensing of surface deformations would enhance the functionality of soft robots.

One approach to sensing the shape of soft robots involves pairing a model with a relatively low number of sensors, typically on the same order of magnitude as the number of controllable degrees of freedom in the system. A great deal of progress has been made in modeling manipulators that can be parameterized by a curve in 3D space (71). These models have even been coupled with sensing mechanisms to enable closed-loop control of continuum manipulators (72). Some approaches embedded sensors into other soft robotic components, such as bending actuators, to achieve closed-loop control in a low-dimensional task space (73).

The primary drawback of this type of approach is that when other unplanned deformation modes are introduced, such as buckling or a change of material properties through damage or natural material aging, the models accumulate error. In addition, it is unclear how to generalize these advances to reconfigurable soft robots (74) or robots that have more complex morphologies. For example, recent simulations suggest that there is a wide range of soft robot morphologies that could produce useful locomotion, including quadrupedal shapes and various oddly shaped exteriors (75). All these classes of robots would benefit from sensing mechanisms with fewer assumptions about the robot’s mechanical properties.

The ideal shape-sensing system could stretch with the robot’s surface without affecting its kinematics or dynamics, sense shape without external components, and be thin. E-skins designed for wearable applications should accommodate the strains of about 55 to 75% experienced by biological skin (22), and a similar range should be suitable for most soft robotic applications, although different robots experience different surface strains. Although a perfect solution for shape sensing of soft robots does not currently exist, recent advances in the field of flexible shape-sensing e-skins (Fig. 4) have the potential to greatly improve the capabilities of soft robots.

Fig. 4 E-skins that can sense their shape in 3D.

Recent advances in shape-sensing e-skins use several sensing modalities. (A and B) Accelerometers and/or magnetometers on rigid PCBs can rotate relative to each other and reconstruct their shape at discrete points. (A) “3D capture sheet” (76). (B) Hexagonal PCBs with integrated accelerometers (48). (C and D) Continuously flexible devices can sense deformation throughout their surface and estimate their resulting shape. Data-driven methods were then used in these examples to estimate the continuous shape of the e-skin. (C) Fiber Bragg gratings in silicone (79). (D) Plastic optical fibers in silicone foam (53).

CREDIT: ADAPTED BY A. KITTERMAN/SCIENCE ROBOTICS FROM MATERIAL PUBLISHED BY IEEE (48, 76, 79), ICS/TUM (48), AND SCIENCE ROBOTICS (53)

In contrast to that of soft skins, most work on shape-sensing e-skins treats the skin as an inextensible sheet of rigid elements joined by known axes of rotation (Fig. 4, A and B). The primary challenge is thus estimating the relative orientation between sections with known geometries to determine the spatial locations of discrete points within the sheet. In one early study, Hoshi and Shinoda (76) arranged 24 printed circuit board (PCB) “nodes” into a mesh and estimated internode rotations using accelerometers and magnetometers (Fig. 4A). Building upon this work, Mittendorfer and Cheng (48) developed rigid sensorized hexagonal PCBs that could be integrated into semiflexible sheets and wrapped around robots (Fig. 4B). The nodes contained accelerometers similar to the work by Hoshi and Shinoda (76) and had similar assumptions (PCBs are free to rotate but cannot be stretched), but rotations between neighboring PCBs were calculated by obtaining at least two orientations of the skin-per-skin shape and solving a constrained Procrustes problem for aligning matrices of data points in real time. Hermanis et al. (77) then used a grid-like arrangement of accelerometers and gravitometers on a flexible fabric sheet. The sheets were demonstrated in a dynamic state estimation task where a user wore a shirt equipped with the shape-sensing sheets while bending and crouching.

In contrast to the discrete sampling methods mentioned above, other approaches leveraged techniques from machine learning and statistics to process various sensing signals and extract a continuous estimate of the shape of the skins (Fig. 4, C and D). This kind of data-driven technique will be increasingly useful as the sensory spatial density increases, as discussed throughout this Review. For instance, Rendl et al. (78) used regularized least squares to process data from 16 piezoelectric bend sensors on a plastic sheet [polyethylene terephthalate (PET)] to approximate the shape of the sheet as a combination of several shape primitives. This created a flexible system that could sense the bent state of the sheet with a roughly centimeter-level accuracy over an approximately A4-sized sheet. Another study used relatively inextensible optical fiber Bragg gratings arranged in a circle on the top and bottom of a silicone e-skin (Fig. 4C) (79). The relation between the strains on the fiber and the shape of the sheet was extracted from training data using a feed-forward artificial neural network containing one hidden layer for computation between the input and output layers. In a similar spirit, an array of optical fibers were twisted through an elastomeric foam, and their outputs were sent to several machine learning algorithms (including kNN, SVM, neural networks, and decision trees) to predict the mode of deformation and angle of deformation of their structure (Fig. 4D) (53). These approaches all dealt well with a limited set of deformations and, in principle, should work for a wider range of deformations when paired with a more expressive (deeper) network. However, none of these existing works can mechanically accommodate large in-plane strains, primarily because of the inextensibility of the optical fibers used.

Toward feedback control of soft robots

The intrinsic material compliance of soft robots can protect both the robot and the environment from damage when interacting with unstructured environments. This property makes soft robots appealing in contexts such as HRI and robotic manipulation, where safety around fragile objects can be important (11, 80). E-skins have great potential to enable soft robots to interact intelligently with their environment.

In addition, tactile information obtained through skin is vital for a variety of general robotic control tasks. The type of sensor modality to be used, the processing algorithm, and the response from the body all depend on the task at hand (81). These tasks can be divided into three broad categories depending on the flow of information or energy and the primary system of concern (Fig. 5).

Fig. 5 Closed-loop tasks where tactile sensing is essential.

These tasks primarily differ depending on the system that determines the objective (denoted by the shaded boxes). The middle row consists of biological demonstrations of the tasks. The bottom row contains examples (69, 128, 129) of these capabilities in current rigid robots, which we expect to further improve in parallel with the integration of e-skins, soft robotics, and machine learning. Note that the presented division is not strict, and real-world tasks often involve a combination of all three elementary tasks. Yellow arrows indicate energy flow; blue arrows indicate information flow. (A) Manipulation involves altering the state of an external object to a desired set point using internal actuators. (B) Exploration involves motion of the body to account for uncertainties in the environment based on somatosensory feedback. (C) Reaction involves estimating and responding to environmental cues such that the body remains in a desired state. [CREDIT: ADAPTED BY A. KITTERMAN/SCIENCE ROBOTICS FROM MATERIALS PUBLISHED BY JOHAN63/ISTOCK (ELEPHANT), MIKE VEITCH/ ALAMYSTOCK PHOTO (OCTOPUS), BENJAMIN TORODE/GETTY IMAGES (CAT), OSADA/SÉGUIN/DRAASM/STANFORD UNIVERSITY (DIVER), SAGE [ROBOT HAND (69)], IEEE [DIVER (128)], AND SCIENCE ROBOTICS (DRONE) (129)]

SEE FIG. 5 CAPTION FOR CREDIT INFORMATION

Manipulation

Robotic manipulation involves altering the state of an external object to a desired set point using internal actuators. The role of tactile sensors is mainly to obtain state information of the external object. As energy flows to the environment, stability of the object is of high concern. Grasp force optimization and stabilization is one of the most basic manipulation tasks involving tactile sensors (82). Early works were built on the estimation of normal and tangential forces on the hand to detect slip and react accordingly (83). Recent works used learning-based methods for slip onset prediction with adjustment and grasp failure detection with adjustment because of the ability of these methods to handle complex multimodal sensory information (84) and their generalizability (85).

Other manipulation studies used low-dimensional sensor space representations to improve performance in certain situations. Van Hoof et al. (86) used autoencoders to generate a low-dimensional representation of their complex and continuous tactile data. Control policies learned using this latent space representation required fewer rollouts and were more robust to noise. Another study was on calibration and self-modeling of a fully sensorized body for whole-body manipulation (87). Recent work has shown that this process can be fully automated using control signal information and other sensor modalities, including inertial measurement units (88).

Perhaps the most complex manipulation task is in-hand manipulation, which imposes strict requirements on the body, brain, and sensors (89). Current progress in in-hand manipulation using tactile sensors is primarily limited to rolling circular objects (90). On the other hand, notable developments toward in-hand manipulation have been achieved with external visual tracking systems (69). However, control policies trained using vision alone are scene dependent and require large quantities of training data, motivating further research into using tactile sensing during in-hand manipulation.

Exploration

Tactile exploration is the process of voluntary motion of the body based on the somatosensory feedback for identifying environmental properties (91). The environmental property of interest could be low-level features, such as surface texture (92, 93) or temperature (94), or midlevel tasks, such as object classification (95, 96). However, to be fully autonomous, the higher-level process of selecting the best actions for obtaining better sensory information, also known as active exploration, must be considered. This is not trivial because the concept of an objective function and a reward function becomes difficult to define.

It is currently conjectured that human exploration is driven by a combination of extrinsic and intrinsic reward variables (97). Extrinsic rewards are task specific, such as classification of objects, whereas intrinsic rewards are task independent and hence more general, such as curiosity-driven exploration. Experiments suggest that humans primarily use six types of exploratory movements when exploring objects to determine their properties (98). Hence, there have been studies on acquiring these specialized closed-loop policies based on intrinsic rewards such as curiosity (99) or extrinsic rewards such as texture discrimination ability (100). To achieve efficient exploration with soft robots, a combination of tactile and proprioceptive feedback will likely be useful for effectively implementing such reward functions.

A first step toward an autonomous tactile exploration control architecture, referred to as tactile servoing by the authors, was proposed by Li et al. (101). By framing the control objective as the problem of following a trajectory in the sensor feature space, various autonomous sensory exploration strategies emerged. The emergent exploration strategies included maintaining contact with an object, edge tracking, and shape exploration of an unknown object. Exploration has also been framed as a force and pose control problem on an unknown object using tactile sensors for feedback (102). Additional tactile information obtained during the process was then used to estimate the compliance of the object. Recent works integrated active exploration with object discrimination (103). However, the midlevel processes were independent from the high-level exploration strategy, and the proposed algorithm was therefore relatively inefficient and slow. The next challenge in this area is to develop exploration strategies that run simultaneously and are regulated by the tactile feature extraction process. Such an algorithm would allow robots equipped with e-skins to efficiently process their sensory information to make informed decisions on how to move within the world to gather information and achieve at least locally optimal exploration strategies.

Reaction

Whole-body tactile skins are required for reacting to active environmental forces applied by external agents (104). Here, the control objective is to estimate and react to external forces such that the body remains stable while executing a behavior. Often, the safety of the external agent, typically a human, becomes a higher priority than robot stability (105). Because reaction typically involves HRI, additional challenges arise from safety, context prediction, and adaptation (106, 107). Otherwise, closed-loop reaction using tactile sensing is similar to the closed-loop manipulation problem and is often implemented in parallel with manipulation tasks as in the case of slip detection (108).

The main challenges in whole-body sensing are the organization and calibration of many spatially distributed multimodal tactile sensing elements (109). Spatial calibration can be manually performed or automated using robot kinematics and action inference techniques (110). Data-driven methods are also promising for end-to-end models without an explicit kinematic/dynamic calibration (111). The most recent and comprehensive whole-body tactile sensing research was able to self-organize and self-calibrate 1260 multimodal sensing units and implement a hierarchical task manager composed of the fusion of a balance controller, a self-collision avoidance system, and a skin compliance controller (112).

OPPORTUNITIES AND OUTLOOK

The fields of e-skins and soft robotics have both experienced rapid progress in recent years. However, incorporating advances from both fields to produce intelligent, autonomous soft robots is a challenging task that will require progress in several key areas (Fig. 6). Here, we outline major open questions in this area and identify areas of research that could provide solutions.

Fig. 6 Potential capabilities and technologies that could be achieved with e-skins and soft robotics.

(A) Multimodal sensing would be useful during manipulation for detecting gripper states, object properties, and events such as contact and slip. (B) E-skins with an integrated human-robot interface could enable seamless assistive wearable robots and intuitive teleoperation of anthropomorphic robots. (C) When paired with the appropriate actuators, shape sensing would enable closed-loop changes of shape. (D) Closed-loop control algorithms would enable soft robots equipped with e-skins to succeed when performing complex tasks, including in-arm manipulation.

CREDIT: A. KITTERMAN/SCIENCE ROBOTICS

Design and fabrication

The primary future challenges of developing sensor arrays for soft robots will be to design stretchable sensory arrays with wide bandwidth and high dynamic range, resolution, and sensitivity. In addition, multimodal sensing would increase the robots’ knowledge of their environment, leading to richer HRI (Fig. 6, A and B). Sensing of pressure, shear, and vibration and even detecting the presence of chemical and biological markers in the environment would be useful for a wide range of applications, including manipulation, disaster response, and manufacturing. Recent efforts on integrating bacteria cells into soft robots have made it possible to directly detect and display chemical information on soft robots (44). Other major design challenges include choosing how many sensors to integrate into a skin and deciding how to place them intelligently. Resources are limited and require careful allocation.

Machine learning and information processing

Advancing the intelligence of soft robots will also require computational models that can extract useful information from sensor arrays. However, the details of how to develop and implement such algorithms are unclear. For example, deciding which algorithms can most efficiently accomplish tasks in classification, regression, and fault detection; whether neural networks should be used; which architectures are easiest to train; and whether there are trade-offs between efficiency and reliability are all open questions that need to be addressed. Answering these questions will necessitate collaboration among computer and data scientists, materials engineers, and neuroscientists. The result will be robots that are more aware of themselves, their environment, and their interactions with humans, yielding richer and more productive experiences for human end users.

Affective touch is a crucial form of nonverbal communication that humans use daily and is one application that would benefit from the combination of e-skins, soft robotics, and machine learning. In contrast, most robots currently are unable to understand gestures such as a pat on the back because either they do not have the sensors necessary to measure the interaction or they are not able to make sense of the affective contact.

Shape sensing

Despite the recent progress in shape-sensing e-skins, it is unclear how to extend these advances to the wide range of soft robots presented in literature. Soft robots experience large strains and complex deformations; key challenges include increasing the stretchability of shape-sensing skins and improving the resolution of sensors to detect small curvatures.

Once the field has reliable solutions for soft robot proprioception, it is conceivable that shape feedback would enable controlled shape change in robots. Current soft robots are not able to morph into specific configurations, yet even simple shape change has led to innovative solutions for a wide range of tasks, such as obstacle avoidance (9), rolling locomotion (113), underwater locomotion (114), and camouflage (115). Larger shape changes could result in robots that switch between morphologies and corresponding locomotion gaits on demand (Fig. 6C).

Feedback control

Using sensorized skins to close the loop has the potential to improve the ability of soft robots to react to their environment, to locomote, to explore, and to manipulate objects using their deformable bodies (Fig. 6D). The use of soft tactile sensors for closed-loop control is still in its nascency. The few relevant studies in this area used low-dimensional soft strain sensors for closed-loop kinematic or force control (73, 116, 117). This is surprising given the wealth of literature on soft sensing technologies and considering the intended application of these sensors (7). One reason for this discrepancy could be that soft sensors were originally developed for wearable devices and therefore used only for state estimation. Another reason could be the demanding performance expectations placed on soft sensors. Although it would be useful to develop drift-free, linear sensors with high gauge factors, biology suggests that workarounds are possible. For example, the human tactile sensing system is hysteretic, nonlinear, time varying, and slow. Nature adapted to these drawbacks by developing hyper-redundant sensing networks and intelligent data processing techniques (118).

Along the same lines, various sensor design strategies can be found by observing nature. Tactile exploration likely requires the highest spatial resolution (around 2 mm), as evident from the dense mechanoreceptor distribution at the human fingertip (119, 120). On the other end of the spectrum, tactile reaction likely requires the lowest spatial resolution, as suggested by the poor spatial resolution across other parts of the body. Tactile manipulation lies in between, with an expected spatial resolution of 5 mm (121).

The type and the distribution of mechanoreceptors across the body also suggest the type of sensor technologies that would be useful for a particular task. Humans use distinct sensors for static and dynamic cues. Low-bandwidth mechanoreceptors (10 to 50 Hz) can be found mainly in the fingertip and would be essential for tactile exploration (122). Higher-bandwidth mechanoreceptors (50 to 400 Hz), which respond to the vibrations induced during object slippage, are distributed primarily at the palm of the hand (123). The response and the sensing areas of the mechanoreceptors are strongly dependent on the skin morphology. Hence, it is vital to consider the design of the body and the motion capabilities for mimicking the dynamic receptors in our body.

Other insights can be gained by extending such an analysis to invertebrate biological organisms, such as octopuses. An octopus has several receptors, primarily chemoreceptors, located on each sucker. In addition, the octopus has strain receptors associated with its muscles and a relatively large brain for processing its receptor information. Despite these capabilities, it has a poor proprioceptive sense and cannot estimate the overall shape and location of external objects that it is handling. There is local proprioceptive feedback in each arm for low-level control, but the only feedback to the central nervous system comes through vision (124). Wells (125) conjectured that in flexible animals, motor control is hierarchical and proprioceptive information must be used locally. Contrary to popular belief, the performance of an octopus in manipulation tasks is poor. Therefore, it might be necessary to incorporate rigid components in fully soft robots, if they are to be used for tactile-based closed-loop control tasks.

Outlook

Researchers have developed many interesting forms of actuation that more closely mimic the functionality and capabilities found in nature. The next step for the field is to develop biologically inspired tactile sensing for soft-bodied robots that can safely interact with, and explore, their environments. Current work tends to concentrate on the design and fabrication of soft robots and explores how machine learning can enhance soft robot perception. In the short term, the field can focus on deployable, high-resolution sensor skins, algorithms for processing the dense sensor information, and reliable feedback control for soft robots. The longer-term goal is robots that can touch and feel with the sensitivity and perception of natural systems.

We believe that future societies will include robots tightly integrated with humanity. This includes in-home, assistive robots that can sense and understand gestures such as a pat on the back, collaborative robots that work alongside humans, and exploratory robots that can navigate the unpredictable real world.

REFERENCES AND NOTES

Acknowledgement: We also thank members of the University of California San Diego Bioinspired Robotics and Design Lab and the Yale Faboratory for helpful comments. Funding: B.S. and M.T.T. acknowledge support from the Office of Naval Research (grant nos. N00014-17-1-2062 and N00014-18-1-2277). D.S. was supported by a NASA Space Technology Research Fellowship (grant 80NSSC17K0164). T.G.T. was supported by the SHERO project, a Future and Emerging Technologies (FET) program of the European Commission (grant agreement ID 828818). Z.B. acknowledges support of related research by the Air Force Office of Scientific Research Materials Chemistry Program (grant no. FA9550-18-1-0143). R.K.-B. acknowledges support of related research by the NSF Emerging Frontiers and Multidisciplinary Activities Program (award no. 1830870). Author contributions: B.S., D.S., J.L., and T.G.T conceptualized the review and wrote the manuscript. Y.-L.P., F.I., Z.B., R.K.-B, and M.T.T guided the topics of discussion and structure of the review. All authors provided feedback and revisions. Competing interests: The authors declare that they have no competing financial interests. Data and materials availability: Correspondence and requests for materials should be addressed to M.T.T. (email: tolley{at}ucsd.edu).

Stay Connected to Science Robotics

Navigate This Article