Research ArticleSENSORS

Soft magnetic skin for super-resolution tactile sensing with force self-decoupling

See allHide authors and affiliations

Science Robotics  24 Feb 2021:
Vol. 6, Issue 51, eabc8801
DOI: 10.1126/scirobotics.abc8801

Abstract

Human skin can sense subtle changes of both normal and shear forces (i.e., self-decoupled) and perceive stimuli with finer resolution than the average spacing between mechanoreceptors (i.e., super-resolved). By contrast, existing tactile sensors for robotic applications are inferior, lacking accurate force decoupling and proper spatial resolution at the same time. Here, we present a soft tactile sensor with self-decoupling and super-resolution abilities by designing a sinusoidally magnetized flexible film (with the thickness ~0.5 millimeters), whose deformation can be detected by a Hall sensor according to the change of magnetic flux densities under external forces. The sensor can accurately measure the normal force and the shear force (demonstrated in one dimension) with a single unit and achieve a 60-fold super-resolved accuracy enhanced by deep learning. By mounting our sensor at the fingertip of a robotic gripper, we show that robots can accomplish challenging tasks such as stably grasping fragile objects under external disturbance and threading a needle via teleoperation. This research provides new insight into tactile sensor design and could be beneficial to various applications in robotics field, such as adaptive grasping, dexterous manipulation, and human-robot interaction.

INTRODUCTION

Human hands are amazingly skilled at recognizing texture and handling objects with different shapes and sizes. Designing a flexible mechanical hand and a system with the same ability as a human hand is a grand challenge for engineers and scientists. Over the years, although remarkable progress has been made in robotic manipulation (1, 2), achieving fine tactile feedback (e.g., contact force) and dexterous daily operation (e.g., adaptive grasping) remains a major challenge. One of the main reasons is that artificial hands lack skin-comparable tactile sensors. Human skin, which mainly consists of four types of mechanoreceptors (SA-I, SA-II, RA-I, and RA-II) (3, 4), is able to perceive tactile stimulus at a spatial resolution higher than the average spacing between mechanoreceptors in the finger (known as the tactile hyperacuity or super-resolution) (5) and to simultaneously distinguish the normal force and the shear force (6). Through the rich tactile feedback from the skin, humans can precisely and reliably recognize, grab, and manipulate everyday objects by dynamically adjusting the finger pose and contact force (7). Hence, designing tactile sensors with skin-like characteristics is of considerable importance toward improving robot capabilities and would benefit a wide variety of industrial and domestic applications.

Recently, many artificial tactile sensors have been proposed for perception based on a wide variety of transduction principles—including capacitance (812), piezoresistance (1319), optics (2022), magnetics (2325), giant magnetoresistance (26, 27), and barometric pressure (28) (see Table 1)—within which machine learning technologies have played an important role to map the raw sensory information to relevant functional abstractions for these sensors (2936). Among all, the tactile sensor arrays (8, 28) and the latest optical-based sensors (20) can achieve super-resolution comparable to human skin, but the shortcomings in bulky structure (20) or solely normal force sensing (8, 28) limit their applications in robotic tasks that require compact sensor structure and involve both the normal and shear forces. Recent progress in materials science has enabled shear force detection through a complicated electrode structure (9, 10, 1517), and such materials include carbon nanotubes (9, 1517), copper/tin-coated textile (10), graphene oxide (15), buckypaper (16), and so on. However, the normal and the shear forces for these sensors are not naturally decoupled, and only the normal (or shear) force under a constant shear (or normal) load, rather than all load combinations, is calibrated. Although an approximated decoupling model can be built by fitting the coupled data of all combinations of normal and shear loads, its practical application remains a big challenge because the calibration is time-consuming and is prone to overfitting. Mechanical decoupling (11, 12, 18, 19, 22) could be a reliable solution for accurately separating the normal and tangential forces, but sensors based on mechanical decoupling are not suitable for daily grasping and manipulation tasks, considering the complex and bulky structure as well as the undeveloped super-resolution ability.

Table 1 Comparison of state-of-the-art tactile super-resolution methods and force decoupling methods.

NA, not applicable; CCD, charge-coupled device; LED, light-emitting diode; PD, photodetector.

View this table:

To tackle this challenge, we introduce a soft tactile sensor with both the super-resolution and self-decoupling capabilities mimicking the force-sensing abilities of human skin. These features are achieved by a sandwich structure that consists of a flexible magnetic film, a silicone elastomer layer, and a Hall sensor. The flexible magnetic sheet is sinusoidally magnetized with multiple pairs of north-south magnetic poles, which successfully overcomes the inherent problem of the strong cross-coupling effects in the conventional magnet-based tactile sensors (2325). As a result, our sensor can naturally decouple the normal and shear forces and simplifies the sensor structure and the calibration process compared with other existing sensors. In addition, it can achieve a 60-fold super-resolved accuracy enhanced by deep learning, which is the best among the state-of-the-art tactile super-resolution methods (listed in Table 1). With our tactile sensor mounted at the fingertip, robots can leverage the large amount of tactile feedback to accomplish challenging tasks such as stably grasping fragile objects under external disturbance. Moreover, the sensor can act as a pointing device, by which the input from the human hand can be applied to a virtual or teleoperated environment. For example, a robot can be teleoperated to accomplish a needle-threading task by applying the input from the human hand via the tactile sensor.

RESULTS

Tactile sensor design

Inspired by the functionalized multilayer structure of human skin (Fig. 1A), we designed the tactile sensor as a sandwich structure to sense external forces. The top layer is made of a flexible magnetic film (E ~ 2 MPa and thickness, 0.5 mm) with a one-dimensional (1D) sinusoidal magnetization profile. The middle layer is made of the soft silicone elastomer (thickness, 3.5 mm; Ecoflex 00-50) with a lower elastic modulus (E ~ 83 kPa). When an external force is applied to the top layer, the Hall sensor embedded in the middle layer senses the deformation of the magnetic film according to the change of magnetic flux densities. The bottom layer is a hard printed circuit board (PCB; thickness, 1.6 mm) with a Hall sensor mounted, which can be replaced with a flexible one (flexible printed circuit) when necessary. To optimize the sensor performance in terms of the measurement range and sensitivity, we carefully chose several parameters including the thickness of the top layer d, the magnetic period of the magnetic film T, and the thickness of the middle layer h.

Fig. 1 Illustration of the tactile sensor.

(A) Illustration of the human skin structure and the soft tactile sensor with a flat surface in a 3-by-3 array, where SA-I and SA-II refer to slow-adapting receptors and RA-I and RA-II refer to rapid-adapting receptors. (B) Illustration of the working principle of the tactile sensor. Left: The initial state (no loading) of the sensor. Middle: The partial deformation of the sensor under a spherical indentation (normal load). Right: The overall deformation of the sensor under the shear load (contains both normal forces caused by the weight of the object and the shear force applied externally). (C) The magnetic strength curve of the flexible magnet along the z axis where the inset is the magnetic strength change (ΔB%) under the displacement load along the z axis (ΔZ) with different elastomer thickness h. In the expression of magnetic flux density B, parameters k, d, and z are the wave number, the thickness of the magnetic film, and the distance away from the bottom surface of the film, respectively. (D) The magnetic ratio (RB) curve of the flexible magnet along the x axis, where Bx and Bz are magnetic flux densities along the x and z axes, respectively.

As illustrated in Fig. 1B, when an external force F is applied to the flexible magnet (top layer), the magnetic flux under the flexible magnetic film changes. The deformation, either from the normal or shear direction, can be sensed by the embedded Hall sensor at the middle layer and then converted to electric signals via the PCB. For partial contact (Fig. 1B, middle), the relationship between the normal force Fz and the magnetic field rotation angle α (or displacement dz along the z axis) can be obtained after calibration, which will be introduced in detail in the section on machine learning–enhanced tactile super-resolution. For full contact (Fig. 1B, right), the relationship between the shear force Fx and the translational movement of the magnetic field dx (displacement component of d along the x axis) can be simply described by Hooke’s law: Fx = SG∙(dx/h), where S is the contact area and G is the shear modulus.

According to the magnetic properties of the Halbach arrays (37), the overall (resultant) magnetic flux density B(x,z) and the ratio RB(x,z) (equal to Bx/Bz, where Bx and Bz are the magnetic flux densities along the x and z directions, respectively) at any point (x,z) under the flexible magnetic film can be described as[B(x,z)RB(x,z)]=[M0(ekd1)ekztan(kx)](1)where M0 is the magnetization magnitude and k is the wave number (k = 2π/T). In this work, we set d as 0.5 mm to balance the tradeoff between the sensor thickness and the remnant magnetic strength of the magnetic film. The experimental results indicate that such a value can ensure both the flexibility and the robustness of the sensor; i.e., Eq. 1 is always valid during the overall deformation of the sensor and the magnetic film can maintain a high magnetic strength (as shown in fig. S1). T is relevant to the sensor’s sensitivity and the measurement range. In particular, when we increase the magnetic period T, the sensitivity of the sensor increases and the measurement range decreases. To balance the measurement range and the sensitivity along both normal and shear directions, we chose the magnetic period as T = 6 mm (i.e., the wave number k is 2π/T = 1.04), which is an integer value (for ease of magnetization with the ready-made machine) between the balance period T1 = 5.81 mm for the normal force and T2 = 6.24 mm for the shear force (details can be found in fig. S2). After determining the parameters of the flexible magnet, we set the elastomer thickness h and the relative position between the flexible magnet and the Hall sensor.

Figure 1C shows the magnetic strength curve of the flexible magnet along the z axis, indicating that the magnetic strength B decreases rapidly when leaving the bottom surface of the magnet. Specifically, the attenuation of the magnetic strength B, compared with the magnetic strength Bmax at z = 0 mm, is 95, 98, and 99% at z = 2.9 mm, z = 3.8 mm, and z = 4.5 mm, respectively. Hence, h is chosen within 3.8 mm where B ≥ 0.02Bmax to ensure that the magnetic flux density signals are valid and detectable to the sensor. Figure 1C also illustrates the magnetic strength change (ΔB%) under the displacement load along the z axis (ΔZ) with different elastomer thickness h, where ΔB% is defined asΔB(%)=|BB0B0|×100%(2)

Here, B and B0 are the magnetic strength sensed by the Hall sensor with and without the external force. The sensitivity can be thereby represented as S = Δ(ΔB%)/ΔP, where the pressure change ΔP = E∙(ΔZ/h) and E is the elastic module of the silicone. It can be found that ΔB% does not change with different h, suggesting that the sensitivity is directly proportional to h. With this feature, we can choose a larger h for a larger measurement range and better sensitivity.

The above analysis verifies that the sensitivity and measurement range of the sensor can be designed to meet the requirements of diverse real-world applications by adjusting the elastic modulus E (for sensitivity) and/or the elastomer thickness h (for measurement range) accordingly. Here, we chose h = 3.5 mm and E = 83 kPa to obtain the largest measurement range and a sensitivity of 0.01 kPa−1 (P ≤ 120 kPa) for the normal force and 0.1 kPa−1 (P ≤ 10 kPa) to 0.27 kPa−1 (10 kPa < P ≤ 16 kPa) for the shear force with the response time ~15 ms (figs. S3 and S4). Such a high sensitivity in the shear direction is especially important for detecting subtle shear forces and preventing slippage in applications such as grasping and manipulation.

Note that (i) the sensitivity and measurement range of the sensor can also be tuned at the very beginning of the sensor designing stage, i.e., by adjusting the parameters of the first layer (d and T), to match the requirement of different working scenarios of the robot (e.g., requiring the measurements of large forces with a low sensitivity or subtle forces with high sensitivity). (ii) In our setup, we align the centers of the Hall sensor and the magnetic pole to obtain a symmetric measurement range of the shear force (as shown by the symmetric magnetic ratio RB curve in Fig. 1D). For applications that only involve the shear force along one direction, the centers of the magnetic pole and the Hall sensor can be designed to be misaligned to increase the measurement range. (iii) Besides working as a single taxel as described above, the tactile sensor can be easily extended to the form of sensor arrays (e.g., a 3-by-3 array in Fig. 1A), and such a property is essential for tactile super-resolution and applications that require sensor scalability and large sensing areas.

Sensor self-decoupling mechanism

The sensor’s self-decoupling ability is enabled by a proper design of the magnetization direction in the flexible magnetic film (i.e., the top layer). Here, we magnetized the flexible magnet in a sinusoidal pattern, resulting in a magnetic field distribution under the film that has two self-decoupled components in terms of the magnetic strength B and the magnetic ratio RB (Fig. 2A), as described by Eq. 1. In this case, the overall magnetic flux density B(x,z) is independent of the X coordinate and the magnetic ratio RB(x,z) is independent of the Z coordinate, indicating that B(x,z) and RB(x,z) are inherently decoupled in the x-z plane, and we can thus simplify their notations as B(z) and RB(x), respectively. The simulation results (by ANSYS Maxwell 3D) and the experimental measurements of the magnetic distribution under the flexible magnet are shown in Fig. 2 (B and C, respectively).

Fig. 2 Self-decoupling mechanism.

(A) Illustration of the magnetic strength planes and magnetic ratio planes (the ratio of Bx and Bz) under the bottom surface of a sinusoidally magnetized film used in our tactile sensor. (B) Simulation results (in ANSYS Maxwell 3D) of the magnetic field under the magnetic film, where the red arrows indicate the magnetization direction (also see fig. S4). (C) Experimental measurements of the magnetic distribution under the magnetic film along the x axis, where z = 1.5 (or 3.5) indicates that the measurement plane is 1.5 mm (or 3.5 mm) away from the bottom surface of the magnetic film. (D) Illustration of the deformation of a sensing unit under the full-contact loading, where ΔX and ΔZ denote the displacements along the x and z directions, respectively. (E) The mean values of predicted loading angles (derived from the outputs of the normal force Fz and the shear force Fx) versus the actual angles of the loading that is applied by a glass stick across 10 trials, where the lower and upper error bars represent the minimum and maximum values of the predicted angles, respectively. (F) Relationship between the normal force output and the magnetic strength measurement B under the normal load Fz, where c1 and b1 are parameters of the curve for Fz fitted with Eq. 3. (G) The magnetic strength B under both shear and normal forces. It shows that B increases as Fz increases but remains constant at different Fx, which suggests that the change of magnetic strength is independent of the shear load. (H) The normal force output under displacement loads of both ΔX and ΔZ, where the normal force increases only when ΔZ increases, which suggests that the predicted normal force is independent of the shear load. (I) Relationship between the shear force and the magnetic ratio measurement RB under the load along the x direction, where c2 and b2 are parameters of the curve for Fx fitted with Eq. 3. (J) The magnetic ratio RB under loads of both Fx and Fz. (K) The predicted shear force under both ΔX and ΔZ.

As illustrated in Fig. 2 (D and E), when the flexible magnetic film shifts from the position (x0,z0) to (x1,z1) under an external force F, the relationship between the force components and the change of magnetic signals can be derived as (see formula derivation in text S1)[FzFx]=[c1SEkhlnB(z1)B(z0)+b1c2SGkharctan RB(x1)+b2](3)where S is the contact area and E, G, and h are the elastic modulus, the shear modulus, and the thickness of the elastomer layer, respectively. c1 and c2 are the compensation coefficients for the elastic modulus and the shear modulus. b1 and b2 are bias coefficients to ease the calibration of the sensor in different applications. For isotropic elastic materials, the shear modulus satisfies G = E/(2(1 + V)), where V is the Poisson’s ratio of the material (~0.5 for the elastomer layer). Equation 3 indicates that the normal force output Fz only depends on the magnetic strength B and the shear force output Fx is only dependent on the magnetic ratio RB. Because B and RB are independent of each other, this theoretically verifies that the sensor outputs of the normal force and the shear force are decoupled.

This self-decoupling property was also verified experimentally. Figure 2F illustrates the relationship between the normal force output Fz and the magnetic strength B. Data from the experiment match Eq. 3, with the root mean square error (RMSE) between the theoretical and measured forces being only 0.15 N. Moreover, as shown in Fig. 2 (G and H), the magnetic strength B (or the normal force output) increases as the normal load Fz (or ∆Z) increases, but it remains constant when the shear load Fx (or ∆X) grows. This implies that the measurement of the normal force is independent of the shear load. Similarly, corresponding results were obtained for the shear force output Fx in Fig. 2 (I to K). These results reveal that the measurements of the normal and shear forces are independent of each other. Here, we only demonstrate the force decoupling ability of the sensor in two axes (x-z). We note that it can be theoretically extended to three axes (x-y-z) by using a flexible magnet with a sinusoidal magnetization pattern along the radial direction (fig. S6), and such a desired pattern could be generated by an electrical magnetizer with proper circular coil design. Compared with the traditional tactile sensors that are either mechanically decoupled (using multiple sensing units) or decoupled by data-fitting methods (using complicated models), our sensor is able to measure both the normal force and the shear force in a decoupled way with a simpler sensor structure and more time-saving fabrication and calibration processes.

Machine learning–enhanced tactile super-resolution

We covered the 3-by-3 Hall sensor array with a continuous magnet sheet analogous to the continuous structure of human skin to compare the responses to external stimulus of our sensor and the human skin. Our sensor exhibits high similarity to the human skin in the receptive fields (Fig. 3, A and B), which thereby leads to similar responses to the external load (Fig. 3, C and D). In particular, the responses of both the human skin (in terms of the impulse frequency) and the tactile sensor (in terms of the magnetic flux density Bz) reach the peak when a spherical object contacts the sensor’s hot spot (i.e., the location with the maximum sensitivity) and decrease as the contact shifts away from the hot spot. Taking the indentation depth of 200 μm, for example, as the contact shifts away from the hot spot by 3 mm, the impulse frequency of human skin decreases from 70 to ~0 Hz, and the sensor response (Bz) drops from 318 to ~0 mT. At the same time, the responses get weakened for both the human skin (from 138 to 20 Hz) and the tactile sensor (from 1067 to 318 mT) as the indentation depth decreases from 500 to 200 μm at the hot spot.

Fig. 3 Tactile super-resolution method.

(A) Top: Illustration of the skin structure of the human finger. Bottom: The receptive field of a single SA-I afferent in the fingerpad [redrawn from (39)]. (B) Top: A modified version of the tactile sensor from Fig. 1A, where the discrete magnetic films in a 3-by-3 array are replaced with a continuous magnet sheet mimicking the human skin. Bottom: The receptive field of a single taxel of the sensor, which is similar to that of the human skin. (C and D) Responses of the human skin and the tactile sensor to the spherical indentation of different depths and locations. (E) Top and middle: (i) A ball placed on the tactile sensor is first targeted on taxel S5 because the magnetic flux density along the z direction of the taxel S5 (BzS5) is the largest among all taxels; (ii) then, the contact position of the ball can be further reduced from the whole area of the taxel to the right side of the taxel according to the sign of BxS5 (here, BxS5 > 0). (iii) Similarly, the ball is localized to the upper side of the taxel because ByS5 > 0, which leads to a coarse estimation of the contact position in the upper right corner of taxel S5. Bottom: (i) Find two neighboring taxels of S5 (S6 in the x direction and S2 in the −y direction); (ii) feed the corresponding magnetic flux densities as inputs into two neural networks for estimating the X coordinate and Y coordinate, respectively, and the regression error is shown in Fig. 4A. (iii) Look up the indentation depth according to the contact position (X, Y) and the current readings of magnetic flux densities from the lookup tables, as shown in Fig. 4C; then, find the magnitude of the normal force corresponding to the indentation depth from the lookup table, as shown in Fig. 4D.

Similar to the biological hyperacuity of humans (5), the artificial tactile super-resolution is a technique that leverages overlapping receptive fields of neighboring taxels to perceive stimuli details better than the sensor’s physical resolution (8). For traditional tactile sensor arrays, the physical resolution is usually coarse because of the low taxel density, but a higher taxel density will lead to crowded wire connections, a longer time for data acquisition, and potentially cross-talk effects. As an alternative, the artificial tactile super-resolution technique can improve the sensor’s spatial resolution without introducing a high density of taxels.

As illustrated in Fig. 3E, when a ball contacts the sensor surface, the taxel (S5) that measures the largest magnetic flux density in the z direction (BzS5) would be the closest one to the contact point (i.e., S5 is activated the most among nine taxels). Then, the contact area can be further narrowed down to the upper right corner of the taxel S5 by merely checking the sign of the corresponding magnetic flux densities measured by the sensor, i.e., the Bx reading (BxS5 > 0) and the By reading (ByS5 > 0). Using above qualitative analyses, the localization accuracy can be improved by twofold (from 6 to 3 mm) without precalibration (figs. S7 to S9). Moreover, such a coarse localization method is insensitive to the object shape and the involvement of the shear force, which eases many practical applications, such as the real-time position tracking of a rolling ball (see movie S1).

For estimating the accurate X and Y coordinates of the contact position, two neural networks with the same architecture were used (Fig. 3E). To be specific, inspired by the fact that human touch attains super-resolution based on the population responses of multiple mechanoreceptors, the responses of both the taxel S5 and its neighboring taxels (S6 in the x direction and S2 in the y direction) were considered as the inputs of the neural networks. Each neural network is composed of three layers (except for the input layer of 6 neurons), with 64 hidden units in each of the two hidden layers and one hidden unit in the output layer, and the activation function of each hidden neuron is the rectified linear unit. More details about the localization procedure can be found in fig. S10.

To investigate whether By (or Bx) is beneficial to the localization accuracy in the x (or y) direction, we compared the X (or Y) localization accuracy in two cases: By (or Bx) is unknown versus known. In the first case, the inputs are two neighboring taxels’ magnetic flux densities {Bx, Bz} for the X localization and {By, Bz} for the Y localization; in the second case, the inputs are {Bx, By, Bz} for both the X and Y localization. The results show that the localization errors (in terms of the mean absolute error) for both X and Y coordinates are much smaller when using {Bx, By, Bz} as inputs than using {Bx, Bz} or {By, Bz} as inputs (Fig. 4A), suggesting that By (or Bx) contributes to the localization accuracy in the x (or y) direction. The fivefold cross-validation error is 0.09 mm for the X localization and 0.11 mm for the Y localization on average, suggesting that the localization accuracy can be theoretically improved by ~60-fold from 6 mm (i.e., the physical resolution of the sensor) to around 0.1 mm. By contrast, we replaced the inputs of the neural networks by the magnetic flux densities of every individual taxel, and then, the localization error increased (fig. S11). This suggests that taking into account the responses of two neighboring taxels produces a higher localization accuracy than just considering a single taxel, which is in line with our previous assumption—i.e., considering the population response of multiple sensing units can improve the localization accuracy.

Fig. 4 Evaluation of the proposed tactile super-resolution method.

(A) The fivefold cross-validation error (in terms of the mean absolute error) of the proposed tactile super-resolution method, where the error bars represent SDs across 10 trials. (B) Validation results of the contact localization with super-resolution. An indenter of a spherical head is pressed on the different locations of the sensor surface within a 1 mm–by–1 mm contact region in a pattern of CITYU, and we show the localization error at each contact point. (C) Relationship between the magnetic flux density, the contact location, and the indentation depth, from which we can look up the indentation depth according to the current magnetic flux density and the contact location estimated with the neural networks. (D) Relationship between the normal force and the indentation depth. We fit a quadratic function for the measured data and use it to predict contact forces from the indentation depth.

To validate the performance of the deep learning–enhanced super-resolution in real-world applications, we used a spherical indenter (8 mm in diameter) to press different locations of the sensor dot by dot within a 1 mm–by–1 mm area in a pattern of “CITYU” (Fig. 4B). The experimental results indicate that the localization errors at most contact points are smaller than 0.06 mm and in the worst case is around 0.1 mm, which means that the proposed localization method achieves at least a 60-fold improvement of the localization accuracy as expected.

Besides providing the precise contact location, our sensor can also measure the force magnitude with a high resolution (with RMSE of 0.01 N). Given the contact location obtained from the proposed super-resolution method, we can then inversely look up the pressing (indentation) depth according to the readings of magnetic flux densities and (X, Y) coordinates from the precalibrated lookup tables as shown in Fig. 4C. Using the same method, we can look up the force magnitude corresponding to a particular indentation depth from Fig. 4D. Missing data can be filled in by applying linear interpolation. For example, if the contact location is (0.2 mm, 0.4 mm) relative to the center of the taxel S5 and the measured magnetic flux density BzS5 is 1475 μT, then the corresponding indentation depth would be 0.6 mm in Fig. 4C and the contact force related to the indentation depth (0.6 mm) would be 0.87 N in Fig. 4D.

Empowered by the deep learning–enhanced tactile super-resolution method, our sensor can achieve a 60-fold improvement of the localization accuracy and measure the force magnitude with a high resolution, which is the best among the state-of-the-art super-resolution methods (see Table 1). The key insight behind such a high super-resolution and the force measurement ability is that the signal source of our sensor, i.e., the flexible magnetic film, has a 3D continuous and smooth magnetic distribution in the space compared with the 1D discrete signals (capacitance, light intensity, and barometric pressure) in most conventional methods. The main advantage of using the continuous magnetic signals is that we can achieve the same super-resolution as with other methods while using the least number of taxels, which is in agreement with the mission of the super-resolution technology: improving the physical resolution with the least number of sensing units. Furthermore, the ability to measure the force magnitude makes the super-resolution technology more complete because most of the traditional super-resolution methods can only measure the force (contact) location.

Adaptive grasping and teleoperated needle threading

Grasping a soft or fragile object has been widely demonstrated by dexterous robot hands. However, it remains a great challenge for robotic grippers to adapt to the external disturbance applied on the object (e.g., dragging force imposed on the object) because the shear force and the normal force cannot be accurately detected simultaneously. With the self-decoupling ability, our tactile sensor can meet such challenging requirements in robotics.

As shown in Fig. 5A, we mounted the sensor on a robotic hand (Robotiq Hand-E) and demonstrated the grasping of an egg under dynamic dragging force. At the initial stage (0-t1), the gripper is open and does not grasp anything; thereby, both the gripping force Fz and the shear force Fx are zero (here, Fx is measured from one side of the gripper, and the actual measurement of the external force Fx′ along the x direction should be twice of Fx). Then, the gripper grabs the egg at t1 and lifts it at t2, where Fz increases sharply from 0 to 3.1 N, and Fx increases slightly from 0 to 0.17 N to counteract the egg’s weight (Fx′ ≈ 0.34 N). After the egg is grabbed stably, we dragged the egg downward by giving a random force Fx′, the maximum value of which is 1.6 N (about five times larger than the initial value of Fx′). The change of the dragging force Fx′ is reflected in the growth curve of the shear force Fx, and the ratio Fx/Fz rapidly approaches the boundary of the friction cone, which indicates that the egg is close to slipping from the gripper (t3-t4). In this dynamic procedure (t3-t4), the gripper adaptively increases the gripping force (from 3.1 to 9.7 N) until Fx/Fz drops into the safe area inside the friction cone to preclude slippage (38). When the dragging force is removed at t4, the gripper reduces the gripping force gradually back to 3.1 N until Fx/Fz increases above the lower bound to prevent crushing the egg. For comparison, we also conduct a similar experiment without using the sensor feedback (t5-t8). It shows that the egg starts to slip at t7 and drops at t8 because the gripping force Fz (3.1 N) or the corresponding friction force (f = μFz ≈ 0.62 N, where the friction coefficient μ ≈ 0.2) is not adequate to counteract the external dragging force Fx′ (0.9 N); i.e., Fx/Fz falls outside the friction cone.

Fig. 5 Adaptive grasping and teleoperated needle threading.

(A) We mounted the tactile sensor on a robotic gripper for an egg-grasping task. The output of the normal force Fz indicates the real-time grip force, and the output of the shear force Fx indicates the current shear force on one side of the gripper. It shows that, with the decoupled force feedback from the tactile sensor, the egg is stably grasped even under external disturbance (in terms of the dragging force). Without the sensor feedback, the egg slips from the gripper under the dragging force. (B) An experiment of holding a bottle while filling water into it. Top: The bottle is stably held in the gripper with the force feedback from the tactile sensor. Bottom: By contrast, the bottle slips during water filling without the force feedback. (C) Demonstration of teleoperated needle threading with a single cell of the sensor. When a human finger touches the different regions of the sensor, the corresponding tactile signals are interpreted as different movement commands, which are then used for controlling the movement of the robot hand and threading a needle successfully.

To further demonstrate the ability of our sensor in handling soft and slippery objects, we conducted an experiment to hold a plastic bottle being filled with water using the same control strategy. As the experimental results show (Fig. 5B), with the sensor feedback, the gripper can stably hold the bottle while filling ~60 ml (60 g) water into it, which is about 12 times larger than the initial weight (~5 g). In contrast, the bottle slips from the gripper when the volume of the water reaches ~30 ml when there is no sensor feedback. These experiments demonstrate that the real-time feedback of both the normal and shear forces is crucial for slip prevention and effective grasping in robotic tasks, and at the same time, our tactile sensor exhibits the ability to improve the grasping stability and flexibility of existing robotic hands, which is important for robots to perform delicate operations especially in unstructured setups.

The sensor’s super-resolution ability is also demonstrated by a needle threading task. Here, a flexible thread (with a diameter of 0.2 mm) was pushed through a needle tip (with an eye diameter of 0.5 mm) using a robot hand that was remotely controlled by pressing the tactile sensor at different regions, which requires high flexibility and accurate teleoperation. A laptop connecting the tactile sensor and the robot operated as the control center to receive and process the touch signals and, subsequently, to send movement commands to the controller of the robot (see movie S4). As illustrated in Fig. 5C, the overall procedure was conducted as follows: (i) The tactile sensor encoded the contact information as tactile signals in terms of 3D magnetic flux densities when a human finger touched the sensor in different regions; (ii) the tactile signals were decoded by a super-resolution algorithm according to the sign (positive/negative) and relative magnitude of the 3D magnetic flux densities and then converted to a series of movement commands for the robot hand; and, last, (iii) the robot controller executed the movement commands to move the robot hand toward a desired orientation with some appropriate step size.

The movement commands are programmable, and in this case, they are defined in six directions, i.e., left, right, up, down, front, and back. This was implemented by processing the tactile signals with the super-resolution algorithm as shown in Fig. 5C, where ΔBx is the difference between the current magnetic flux density Bx and Bx0 in the unloading condition, the activation threshold c1 was set as 100 to make sure that no command can be incorrectly activated when nothing touches the sensor, and c2 was set as 1000 to differentiate light and heavy touches. The needle threading task was accomplished by using only a single 8 mm–by–8 mm taxel. Compared with conventional sensor arrays that normally require six taxels, our sensor reduces the number of sensing units and the system complexity, making it suitable to serve as a compact pointing device on laptops and mobile devices, like a soft TrackPoint.

DISCUSSION

Precise tactile sensing plays a key role in manipulation, handling, and interaction using robot hands. Although vast kinds of soft tactile sensors have been developed in recent years, they are still not comparable to the sensing ability of the human skin, which boasts the unique capability of simultaneous normal/shear force decoupling and tactile super-resolution. The proposed sensor here can inherently decouple the normal and shear forces and provides a tactile super-resolution with a 60-fold super-resolved accuracy, which outperforms existing techniques (see Table 1). Its human skin–like ability helps accomplish several challenging tasks in robotics, including the contact position tracking, adaptive grasping with external disturbance, and teleoperated needle threading.

The top layer of our tactile sensor is made of a soft magnetic film to perceive the applied force. Because the magnetic field is continuously distributed in 3D space, the induced deformation of the film can be recognized by the Hall sensor through the magnetic field change, and because the deformation is directly related to the material and structure, the measurement range and sensitivity can be largely adjusted by tuning the sensor’s design parameters, i.e., either the thickness d and magnetic period T of the flexible magnet or the thickness h and the elastic modulus E of the elastomer layer. Thus, more diversified tasks can be accomplished by designing a proper measurement range and sensitivity according to the actual application requirements. The tactile sensor can be easily extended to the form of discrete sensor arrays or even continuous electronic skin that can cover the entire body of the robot, if we replace the rigid PCB and discrete flexible magnet of the sensor with flexible printed circuits and a continuous flexible magnetic sheet. Moreover, the sensitivity and measurement range of such a sizeable electronic skin can be designed to vary in different regions by tuning the magnetic period of the flexible magnet locally without changing the thickness of the entire e-skin. This property is similar to the diversified sensitivities and ranges of the human skin in different parts of the human body, e.g., at the fingertip or palm, which will be of great importance for applications that require different sensitivities and ranges of the robotic skin (e.g., at the robot hand or arm).

Our sensor can provide two types of tactile super-resolution strategies in terms of coarse localization and fine localization. The coarse localization method is insensitive to the object’s shape and surface conditions, and it can improve the localization accuracy by twofold with no need for precalibration. For an object of a certain shape, the localization accuracy can be further improved after training neural networks on the object (~60-fold for spherical object), named fine localization. We note that the neural networks are required to be retrained if the shape of the contact object is changed.

Three kinds of tasks—contact position tracking, adaptive grasping, and teleoperated needle threading—have verified the sensing capability of our sensor. Further improvements can be made for these tasks by fully leveraging the properties of the sensor. In the adaptive grasping tasks, the response frequency of both the robotic hand and arm are limited at 1 Hz because of the communication restrictions between the robot and the sensor. The response frequency of the robot can be further improved to match the maximum sampling rate of the tactile sensor (more than 100 Hz) in the future by optimizing the communication procedure. We also point out that, in this demonstration, we only considered the shear force in the x direction, limited by the one-directional shear force sensing ability of our sensor. This issue can be addressed by magnetizing the flexible magnet in a sinusoidal magnetization pattern along the radial direction.

In summary, this work reports a soft tactile sensor with both self-decoupling and super-resolution abilities, which are comparable to those of the human skin. Our sensor can enable the implementation of several challenging tasks in robotics, such as contact position tracking, reliable grasping with external disturbance, and teleoperated needle threading. This research provides a promising strategy for tactile sensor design for many fields in robotics, especially delicate grasping, flexible manipulation, and human-robot interaction.

MATERIALS AND METHODS

Fabrication of the flexible magnetic film and the elastomer layer

The flexible magnetic film is a mixture of polydimethylsiloxane (PDMS; SYLGARD 184, Dow Corning) and neodymium (NdFeB) magnetic powders with a weight ratio of 1:3. To avoid agglomeration, the NdFeB magnetic powders were first dispersed in an organic solvent, heptane, via high-frequency ultrasonic agitation, and then, the PDMS base monomer was added into the mixture of NdFeB powers + heptane and mixed thoroughly at 2000 rpm in a planetary mixer. After the organic solvent was evaporated, the curing agent (10 weight % with respect to the PDMS base monomer) was added into the mixture of NdFeB powders + PDMS base monomer and then mixed at 2000 rpm for 5 min, followed by defoaming in a vacuum machine. Last, the mixture was cast into an aluminum mold and cured for 2 hours at 80°C, after which the flexible magnetic film was magnetized under a strong rotating magnetic field generated by the pulse magnetizer. The Hall sensor (MLX90393) used for measuring the magnetic field is commercially available from Melexis (Belgium). The elastomer sandwiched between the flexible magnet and the Hall sensor on PCB is Ecoflex 00-50 (Smooth-On Inc.), which was directly cast on the PCB.

Assembly of the flexible magnetic film and the elastomer layer

The flexible magnet was bonded onto the elastomer by using an intermediate prepolymerized mixture of PDMS and Ecoflex 00-50. Before the mixture cured, the position of the flexible magnet could be freely adjusted until the centers of the magnetic pole and the Hall sensor were perfectly aligned, and then, the entire structure was heated to cure the mixture and keep the alignment permanently.

Sensor characterization

The sensor was calibrated on a manual three-axis moving platform. The 3D-printed indenters were used to apply the displacement loads, and the ATI Nano 17 6-axis F/T sensor (ATI Industrial Automation, Apex, NC, USA) was used as the ground truth of the displacement loads.

Characterization for self-decoupling

To validate the self-decoupling property of the sensor, 7-by-7 combinations of displacement loads were applied within seven rounds. Specifically, a constant displacement load along the normal direction (∆Z1) was first applied on the sensor by a 3D-printed indenter of a square shape, and then, a successive of displacement loads along the tangential direction (from ∆X1 to ∆X7) were applied with an increment step size of 0.25 mm. In the next round, the constant displacement load along the normal direction was increased to ∆Z2 with a step size of 0.25 mm, and the tangential displacement loads were then applied in the same way as in the first round (from ∆X1 to ∆X7). We repeated this procedure until the normal displacement load was increased to ∆Z7 so that the responses of the sensor under 7-by-7 combinations of displacement loads were collected (Fig. 2, G to K).

Characterization for super-resolution

The flexible magnet (18 mm by 18 mm) was sampled at 0.2-mm spacing in both the x and y directions, generating 8100 sampling points (90 by 90) in total. Then, the 3D-printed indenter of a spherical head (diameter, 8 mm) was pressed at each sampling point with five indentation depth (from 0.2 to 1 mm), which produced 40,500 sets of data for training the neural networks, where each set of the data includes three magnetic flux densities (Bx, By, and Bz) of nine taxels, i.e., 27 magnetic flux densities for each set of the data.

Demonstration setup

For the adaptive grasping task, the sensor was equipped on a robotic gripper (Robotiq Hand-E) for grasping an egg and a water-filling bottle, and the opening/closing speed and step size of the gripper were set as 150 mm/s and 0.2 mm, respectively. For the needle threading task, a robotic gripper was mounted on a commercialized robot arm (Universal Robot 10e). The flexible thread (with a diameter of 0.2 mm) held in the robotic gripper was remotely controlled to push through the needle tip (with an eye diameter of 0.5 mm) by reading the input signal from the tactile sensor. In these two tasks, the tactile signals were acquired with the Arduino Mega 2560 via I2C protocol and were then sent to a laptop to control the movement of the robotic arm (or gripper) via TCP/IP. In addition, the response frequencies of both the robotic gripper and arm were limited at 1 Hz, and the sensor sampling rate was set as 10 Hz.

SUPPLEMENTARY MATERIALS

robotics.sciencemag.org/cgi/content/full/6/51/eabc8801/DC1

Text S1. Formula derivation.

Text S2. Sensor responses when touching ferromagnetic materials.

Fig. S1. Magnetization loop of the flexible magnetic film.

Fig. S2. The sensing range and sensitivity of the sensor under different magnetic period of the flexible magnet.

Fig. S3. Sensitivities of the sensor for the normal and shear forces.

Fig. S4. Response time of the sensor (15 ms).

Fig. S5. The simulation result of the magnetic flux density distribution of the flexible magnet.

Fig. S6. Relationship between the magnetization manner of the flexible magnet and the corresponding force decoupling ability of the sensor.

Fig. S7. Receptive field (x directional response Bx) of all nine taxels.

Fig. S8. Receptive field (y directional response By) of all nine taxels.

Fig. S9. Receptive field (z directional response Bz) of all nine taxels.

Fig. S10. Localization procedure using responses of two neighboring taxels.

Fig. S11. Localization error using responses of every individual taxel as the inputs of the neural networks, where the error bars represent SDs of the localization error across 10 trials.

Fig. S12. Hysteresis curve of the sensor output during a cycle of loading and unloading (0 to 230 kPa).

Fig. S13. Repeatability test of the sensor over 30,000 cycles.

Fig. S14. Signal-to-noise ratio model of the sensor.

Fig. S15. Schematic illustration of the data collection process.

Fig. S16. Measurements of the magnetic strength on the strong and weak sides of the magnetic film.

Movie S1. Real-time contact position tracking of a rolling ball.

Movie S2. Adaptive grasping of an egg with dynamic disturbance.

Movie S3. Adaptive grasping of a water-filling bottle.

Movie S4. Teleoperated needle threading.

Movie S5. Sensor responses when touching ferromagnetic materials.

REFERENCES AND NOTES

Funding: J.P. acknowledges support from NSFC/RGC Joint Research Scheme N_HKU103/16, General Research Fund 11207818 and 11202119, and AIR@InnoHK–Center for Transformative Garment Production (TransGP). Y.S. acknowledges support from NSFC 61922093, NSFC U1813211, General Research Fund CityU 11211720, and Shenzhen (China) Key Basic Research Project JCYJ20200109114827177. Author contributions: Y.Y. designed, fabricated, and characterized the sensor; conducted the experiments; and analyzed the data. Z.H. developed the super-resolution algorithm. Z.Y. and C.S. provided critical feedback on the self-decoupling. W.Y. provided critical feedback on the tactile super-resolution. J.P. and Y.S. proposed the concept and lead this project. All the authors wrote and read the manuscript. Competing interests: Y.Y., Y.S., and J.P. are inventors on patent application PWG/PA/918/11/2020, approved and to be submitted by the City University of Hong Kong, that covers a soft tactile sensor by flexible magnetic film. The other authors declare that they have no competing financial interests. Data and materials availability: All data needed to evaluate the conclusions are available in the article and the Supplementary Materials.

Stay Connected to Science Robotics

Navigate This Article