Research ArticleSOFT ROBOTS

An anthropomorphic soft skeleton hand exploiting conditional models for piano playing

See allHide authors and affiliations

Science Robotics  19 Dec 2018:
Vol. 3, Issue 25, eaau3098
DOI: 10.1126/scirobotics.aau3098

Abstract

The development of robotic manipulators and hands that show dexterity, adaptability, and subtle behavior comparable to human hands is an unsolved research challenge. In this article, we considered the passive dynamics of mechanically complex systems, such as a skeleton hand, as an approach to improving adaptability, dexterity, and richness of behavioral diversity of such robotic manipulators. With the use of state-of-the-art multimaterial three-dimensional printing technologies, it is possible to design and construct complex passive structures, namely, a complex anthropomorphic skeleton hand that shows anisotropic mechanical stiffness. We introduce a concept, termed the “conditional model,” that exploits the anisotropic stiffness of complex soft-rigid hybrid systems. In this approach, the physical configuration, environment conditions, and conditional actuation (applied actuation) resulted in an observable conditional model, allowing joint actuation through passivity-based dynamic interactions. The conditional model approach allowed the physical configuration and actuation to be altered, enabling a single skeleton hand to perform three different phrases of piano music with varying styles and forms and facilitating improved dynamic behaviors and interactions with the piano over those achievable with a rigid end effector.

INTRODUCTION

There is increasing interest in the study of nature to provide biological inspiration for the development of robots with physical and cognitive abilities comparable to biological systems (1, 2). Animals interact in highly complex and varied ways with rapidly evolving, information-rich environments (3). Previous work on biologically inspired robotics has demonstrated that the complexity in animals’ behavior results from reciprocal interactions between the controller (brain), the body, and its interactions with the environment (4, 5). Complex behavior does not result from the controller or brain alone but from complexity distributed across the entire system, including the mechanical body (6).

The mechanical properties and design of systems play a considerable role in the intelligent functioning of animals and machines. This can be observed in passivity-based robot control (7, 8). Passivity can, for example, be used to achieve a pendulum-like swing of legs for locomotion, requiring no explicit active control to achieve stable bipedal walking (9). High-functioning passively controlled robots have achieved a range of different behaviors, such as robotic swimming, flying, and manipulation (10). Smart mechanical design enables systems to show exquisite and complex behaviors that are self-stabilizing and energetically efficient at reduced computational cost (11).

Achieving functional behaviors through passivity is crucial for the survival of biological systems; however, as a design method for robotic systems, passivity is known to intrinsically restrict the range of behaviors (12). Underactuated control provides a compromise and can expand the range of behaviors by introducing a coupling between passive mechanics and limited joint actuation (13, 14). This creates behaviors that are highly environmentally dependent and sensitive to changes but limits behavioral diversity, typically with a one-to-one mapping between environment and behavior imposed (15, 16). This limitation can be particularly seen in robotic manipulation and hand design, where passive control and underactuated mechanical design allow only a single (1719) or, at best, a limited number of behaviors to be achieved (2024). To leverage the intelligence of passive mechanical bodies, a method for generating a range of behaviors in variable environments is required.

Achieving behavioral diversity in robotics, while using passive dynamics, remains a fundamental challenge. There have been several recent approaches that used passivity to achieve complex (i.e., varied and adaptive) dynamic behaviors, where the complex behavior emerged from many hard-to-control independent mechanical components. One approach was to actively control the mechanical dynamics of the robots by implementing variable stiffness mechanisms that allowed adaptation of the passive behaviors to varying environments (2527). Although this approach allowed different behaviors to be achieved, the inclusion of actuators limited the scalability and required more complex control (2831). A second approach was the use of materials to alter or adapt the behavior (32). Soft deformable materials were integrated into robots to expand the diversity of achievable behavioral patterns (33, 34). Behaviors of robots using soft deformable materials were generated through the mechanical dynamics of interactions between the environment and materials (35). The increased compliance of the soft materials provided more flexibility, enabling a wider variety of mechanical dynamics. However, the inherent flexibility of soft materials can result in behaviors that are ill defined and highly variable. A key challenge, therefore, is controlling the mechanical compliance when using softer materials (36, 37). This has been demonstrated with soft robots by using variable stiffness materials to achieve a range of movements and to modulate interactions with the environment (3840). The synergy between soft bodies and actuation methods could then be used. This allows the movement of soft bodies to be limited or constrained, in turn limiting the requirement for complex additional actuation sources. In particular, work on adaptive synergies (4143) and tendon routing (44) shows notable breakthroughs and developments with respect to robotic manipulators. Although many of these approaches provide methods for exploiting mechanical passive dynamics, they do not provide a framework for significantly scaling complexity and diversity in behavior.

This paper proposes an alternative approach, using hybrid soft-rigid mechanical structures, where the stiffness of the structures can be set heterogeneously across the body. This builds on well-understood techniques, such as flexure joints, that use anisotropy (25, 45, 46). By taking advantage of state-of-the-art multimaterial three-dimensional (3D) printing techniques, we constructed complex hybrid mechanical structures (4749). Given appropriate environmental and actuation conditions, this heterogeneity of stiffness could be exploited, with the conditions restricting the joint space such that the observed model of the structure, termed “conditional model,” varies (Fig. 1). In addition to the conditional actuation applied to the system, there is a second observable actuation, the joint actuation, that determines the passivity-based dynamic interactions of the system. The passive interactions arising from the joint actuation may lead to a change in the physical configuration of the structure altering the conditional model (Fig. 1B). This approach provides a conceptual understanding of how complex behaviors can emerge from a passive-based hybrid structure while also informing the design and control necessary to achieve different conditional models and accompanying behaviors.

Fig. 1 Representation of the conditional model.

M indicates a motor that provides actuation, and T indicates the resultant torque. (A) A conditional model occurs when a conditional actuation is applied to a physical configuration (e.g., geometry and materials). This model represents the interaction between the system and the environment. There is a secondary internal actuation of the system, the joint interaction, which is dependent on the restriction of the joint space of the conditional model. This results in the passive-based output behavior, leading to a change in the initial physical configuration of the system. (B) For each resultant conditional model, the joint actuation is dependent on different physical configurations from which a second conditional model can be achieved. In this way, it is possible to achieve conditional models that then allow other conditional models to emerge.

The ability to show many conditional models enables diverse and complex mechanical dynamics from a single system. This ability mirrors how the human hand can show highly varied dynamics; for example, a strong fist can be formed to hit a rigid wall, or a soft finger can be used to touch a smooth surface. The range of conditional models that can arise from one passive structure is dependent on the mechanical design, actuation, and environmental conditions. Certain conditional models can only be achieved by first triggering previous conditional models and associated output behaviors; hence, a typical one-to-one mapping of control inputs and behavior can no longer be used (Fig. 1B).

This article investigates the behaviors achievable through the emergence of conditional models. The mechanical complexity of structures, in this case the many interacting mechanical parts with varying stiffnesses, plays a crucial role in the emergence of conditional models. The complexity enables adaptive environmental interactions yet also allows the emergence of specific conditional models and behaviors. The greater the variety of mechanical dynamics within the body of a robot, the wider the variety of conditional models that can be determined through different physical configurations and actuation conditions. Mechanical behavior is bounded by the physical design and geometry of the system, for example, the joint design and the material properties, whereas the environment and surroundings impose conditions on the complex mechanical system contributing to the behavior (41, 50, 51). This approach to designing and controlling a mechanical body leads to richer behavioral diversity in comparison with the previously discussed passivity-based and soft robotic approaches. The diversity of behavior originates from the complexity of the mechanical design while simultaneously reducing the complexity of the required control. To demonstrate this concept, we considered a 3D-printed anthropomorphic robotic hand interacting with a complex environment. Existing anthropomorphic hands often require complex actuation or oversimplify the model such that complex joint behaviors are lost (5255). In this work, we introduce an approach to producing a near-exact replication of human bone and ligaments by 3D printing, with the behavior dominated by passive dynamics.

To validate the proposed approach, we present a case study of the dexterous robotic hand playing a piano. Piano playing emerges through the coupling between the biomechanics and neuromuscular dynamics of the pianist (mechanical impedance of the finger) and the dynamics of the piano itself (56). Piano playing thus relies on the interaction between the environment and the mechanics of the players’ hand. Piano-playing robotics research dates back to the 1980s (57), with many piano-playing robots developed with a focus on both mechanical and algorithm development (5862). Most of the robots used rigid finger joints with no compliance such that a high accuracy of finger positioning could be achieved. However, the control required to achieve a variety of more nuanced playing styles, ranging from highly precise rapid movements to softer more adaptive playing, has not been explored thoroughly. Successful expressive and varied piano playing within the fixed environmental conditions provided by the piano posed a rigorous test for the conditional model framework and robotics in general and demonstrated the contributions of this concept.

RESULTS

Designing internal conditions: Anthropomorphic soft hand skeleton

The process of designing and building systems with anisotropic stiffness that allow a range of different conditional models requires a different approach than that of building conventional rigid actuated systems and is more challenging. Biological systems show a diverse range of complex joints, including highly mobile ones such as shoulder or hand joints. These provide an excellent starting point for the design of joints that show anisotropic stiffness and can have many different observable modes (63). The combination of bone-bone interactions and ligaments creates complex passive behaviors (64, 65). Unlike pin joints typically used in conventional rigid robotic systems, these structures can exhibit anisotropic behaviors depending on the actuation applied and the environmental interaction. For this piano-playing case study, we developed an anthropomorphic hand that used these complex interactions. The design of physical system (the physical configuration) focused on the joint design and the material properties of the hand skeleton.

Figure 2A shows the model of the anthropomorphic hand skeleton used. This was directly inspired by human anatomy, with bones and ligaments placed as they are in nature (66). Every finger joint was encapsulated by ligaments constraining the movement and stiffness of the joint yet allowing the two bones to move independently and to interact. Unlike many other anthropomorphic robotic hands, this design allowed both osteokinematics, observable movement of bone shafts, and anthrokinematics, movements at joint surfaces that cannot be directly observed and are considered to be passive (67). By 3D printing this model with varying ligament stiffnesses, we could vary the anisotropic properties of each finger (Fig. 2B). By printing joint ligaments with sufficient compliance, the anisotropic stiffness allowed for complex behaviors influenced by external actuation and interaction with the environment. Movie S1 shows the multimaterial 3D printing process and the flexibility and range of movement of the printed hands.

Fig. 2 The piano-playing hand.

The hand has been printed at the same scale as a human hand, where the middle finger has a length of 6 cm. (A) Anthropomorphic model of the hand showing the three groups of ligaments that influenced the three behavior primitives investigated. (B) 3D-printed hand conditional stiffness system attached to the UR5 robot arm, which provided the external actuation, and the piano environment used.

Figure 3A shows the experimentally determined stiffness of a 3D-printed index finger when force was applied perpendicular to the fingertip at varying orientations. A static force was applied to the fingertip while the metacarpus was fixed, allowing the overall perpendicular stiffness behavior to be measured. Three different materials (with finger joint Young’s modulus, EJ, of 1, 2.5, and 50 MPa) were used to print the ligaments (see more details in Materials and Methods). As shown in Fig. 3A, varying the physical configuration (ligament stiffness) or the environmental conditions (angle of force applied) gave rise to different observable models, which provided different output behaviors. Because of the geometry of the bones and the interactions between the bones and the ligaments, the stiffness was greater in the ventral-dorsal direction in comparison with the lateral direction. In particular, we observed lower compliance in the horizontal plane, where lateral movement led to a “jamming” between the two bones that limited the compliance of the joint in that direction (68). The stiffness was not completely symmetrical around the vertical axis, reflecting the nonsymmetric bone-bone interaction when the heads of the bones interact. Thus, for different conditions, the conditional model of the system changed.

Fig. 3 Demonstration of the conditional model.

(A) Compliance behaviors of a single finger (distal phalange to metacarpal) printed to scale with varying ligament stiffnesses. Varying compliance allows the emergence of different conditional models for single-finger interactions. (B) Directional compliance of a thumb, showing how the different environmental interactions and physical configurations lead to the emergence of different conditional models.

Similar to the stiffness deformation landscape of a single finger shown in Fig. 3A, the anisotropic nature of the thumb stiffness showed similarly diverse and complex behaviors, with a larger range of different possible conditional models. Behavioral diversity of this finger was generated by exploiting the variable conditional models imposed by the environment, actuation, and physical constraints. A similar design strategy was applied to the other parts of the complex hand skeleton. The thumb joint was more complex and allowed similar exploitation of anisotropic stiffness; the larger number of ligaments contributed to the directional stiffness, allowing a much greater range of arthrokinematic behavior (Fig. 3B). The different conditional models that could be achieved with the thumb joint were more complex and showed greater variation than those possible with a single finger.

The ligaments in the skeleton model were grouped into three types (Fig. 2A): those that contribute to finger joint stiffness, span stiffness, and thumb abduction/adduction stiffness. The material properties of these ligaments (denoted EJ, ES, and ET for each group, respectively) controlled the stiffness of these joints, influencing the overall behaviors of the passive hand. As demonstrated in Fig. 3A, the collateral ligaments and other associated finger ligaments contributed to the anisotropic stiffness and hence conditional models of the finger, which could be controlled by varying EJ. The variable span stiffness was controlled by the deep transverse metacarpal ligaments, which was determined by ES. Last, the stiffness of the complex thumb joint and the range of motion were controlled by ET of the palmar carpal-metacarpal II ligaments and surrounding ligaments.

To “engage” the anisotropic stiffness and select a specific conditional model, we needed an active component to provide external actuation to the passive system. In this case study, we used a multi–degree-of-freedom UR5 robotic arm to provide wrist actuation with the passive hand attached to the arm (Fig. 2B). The wrist actuation allowed for dynamic changes in hand position with respect to the environment. This promoted varying interactions and dynamic coupling between these two components and allowed different conditional models to be observed. Figure S2 shows the overall architecture used to demonstrate the case study.

To experimentally determine how the physical configuration and environmental interaction impose restrictions on the joint and thus determine the joint actuation from specific conditional actuation, we investigated the influence of material properties on piano-playing behaviors. Within our experiments, we also investigated the coupling between the mechanical system and the environment by varying the wrist dynamics and material properties of the hand.

Analysis of conditional model concept for behavioral diversity

To allow systematic analysis, we investigated three behavior primitives: single-finger tapping, thumb adduction/abduction, and hand span behavior. The combination of these primitives enables a wide range of playing behaviors and different conditional models. These three behavior primitives map to three ligament groups for which the material properties are varied: finger joints (EJ), thumb ligaments (ET), and span ligaments (ES).

Single-finger behavior

The single finger exhibits a wide variety of conditional models and hence behaviors depending on the conditions provided. The first series of experiments involved a single finger playing a single note, tapping one piano key where wrist actuation was only applied in the vertical plane. Considering a right-handed reference frame centered on the wrist with the z axis vertically upward, this corresponds to wrist movement in the z axis. The control parameters of this wrist actuation include the frequency (or playing speed) and the displacement of vertical motions. For different control parameters, the output frequency, rate of force applied, and the maximum force at the fingertip in contact with the surface of piano key were observed. The focus of this analysis was the ability of the internal conditions to control the overall performance of piano playing with one finger. In particular, we analyzed the change in output behavior for fingers with varying ligament stiffnesses. Four single fingers were tested, each 3D-printed with ligaments of different Young’s moduli (EJ).

Figure 4A (left) shows the input-output frequency response of the finger, which is an important metric when considering the achievable tempos during piano playing (69, 70). The range of output frequency in each finger was measured by actuating the wrist vertically, oscillating with a fixed amplitude in the z axis. When increasing the input frequency, the output frequency of the rigid finger could be considered to be the same as the input frequency within the range of reasonable playing frequencies. The range of conditional models that the rigid finger could achieve was low, whereas for the more compliant fingers, which exhibited a more complex and nonlinear relationship between the conditional actuation and the joint actuation, a greater range of models was possible. Because the stiffness of the ligaments was reduced, the anisotropic stiffness was such that the system showed some nonlinear behavior, with the damping effect limiting the maximum achievable frequency. Lower-stiffness fingers experienced lower-frequency environmental interaction, and a different conditional model was achieved in comparison with that of the high-stiffness finger. By lowering the stiffness, the conditional model had a limited range of playing frequencies in comparison with the fully rigid finger, where the model was capable of playing music with a greater range of frequencies. However, a stiffer finger had a smaller range of different conditional models, resulting in a trade-off of other playing capabilities and stylistic behaviors.

Fig. 4 Experimental testing of behavioral primitives.

(A) Single-finger playing experiments, with 3D-printed fingers with varying material stiffnesses (EJ) showing the effects of varying different control parameters: frequency of playing (playing speed), rate of note playing (playing style, e.g., legato/staccato), and maximum force detected on the fingertip (volume). The force was measured with FSRs on the piano keys. These results highlight the difference in joint actuation for the different models. bpm, beats per minute. (B) Abduction/adduction distance measured between the tip of the thumb and the tip of the first finger when the wrist was actuated horizontally after the thumb was moved vertically down such that it pressed the key. Experiments were undertaken with hands with varying thumb ligament stiffnesses (ET). (C) Hand span stiffness demonstrated with a single finger (left), where the displacement between the second and third finger was measured, with the second finger playing a note and the wrist actuated horizontally. Whole-hand playing (right) when the wrist was actuated at varying amounts and the stiffness changed. The right graph shows the varying output forces when these conditions are changed.

Figure 4A (middle) shows the rate of tapping force at piano key contact with respect to different frequencies (or actuation speeds). The rate of tapping force indicated the articulation of sound, which directly influenced the transition between notes ranging from slurred/legato to staccato. The lower rate of force change resulted in smoother transition between two notes. This experiment highlights the salient differences between rigid and softer, compliant fingers. Although rigid fingers could achieve a conditional model that exhibited a larger range of force changes for a greater range of input frequencies, they could not achieve lower playing rates, especially at a higher frequency. These results indicate that a soft finger is necessary to play fast slurred or legato pieces, because it provides more suitable conditional models, whereas a more rigid one should be used for articulated music.

Similar behavior characteristics could be seen for the peak force when playing using a single finger, which indicates the volume of note. Figure 4A (right plot) shows that a rigid finger could generate a larger variety of peak forces, whereas the subtle control of volume could be easier when a softer finger was used. The results demonstrate that the mechanical properties of the piano affect the achievable conditional model, limiting the ultimate ranges of the behavior. Actuation can be used to trigger a given conditional model and hence behavior or responses within these physical limits.

These results highlight the various trade-offs between different stiffness finger ligaments and actuation combinations. We show how anisotropic stiffness can be exploited through the physical configuration and environmental interaction to enable specific conditional models. There is no unique combination of actuation and mechanical properties for one conditional model, behavior, or playing style. However, the choice of the physical configuration (geometry and materials) does limit the range of behaviors that can be achieved. Movie S2 highlights the range of movement of a single finger, showing the finger trajectory and tone when playing a single note with fingers of different joint stiffnesses.

Thumb adduction/abduction behaviors

For these experiments, we specifically focused on the thumb and index fingers, because they exhibit a rich variety of motions in comparison with other human hand movements, with high mechanical complexity providing a large range of conditional models. The thumb abduction and adduction movement are particularly interesting, because the range of movements and behavior reflects the complexity of the thumb anatomy and provides a great deal of functionality for hands (71) and, more specifically, for piano playing. Movie S3 demonstrates the abduction/adduction behavior that could be achieved with the passive hand structure when using the coupling between environment and actuation. To achieve these abduction/adduction behaviors, we needed to pass through specific conditional models that enabled other conditional models to be observed.

In this experiment, finger movement was articulated into two phases: First, the wrist was actuated downward in the z axis with a certain stroke and a fixed speed such that the thumb fingertip could press the key down all the way. Second, the wrist was then moved horizontally to the keys in the x axis such that the index finger is moved over the thumb. The thumb abduction behavior was possible because, in the previous conditional model, the thumb finger was prevented from moving sideways by the neighboring key during the second movement. In this way, we used the actuation to achieve different conditional models and, through these, then accessed other models, showing some emergence of different conditional models. After these wrist actions, we measured the horizontal distance between thumb and index fingertips as an indication of the adduction/abduction behavior of the hand. This type of behavior is used by human players to allow smooth transitions when moving sequentially over notes, for example, in scales or when performing jumps and rapid movements (72).

As in the single-finger experiments, we investigated the effect of ligament material properties on the behavior of the hand. Only the material properties of the adduction/abduction ligaments were varied while keeping the others the same. Figure 4B shows the distance between the two fingers with respect to different horizontal displacement of the wrist joint for four different ET values ranging from 1 MPa to 2 GPa.

When rigid ligaments were used and horizontal displacement was applied to the wrist, the two fingers had limited ability to move relative to each other. Thus, when the wrist was moved, the maximum adduction distance was found experimentally to be about 18 mm, with the thumb only able to move on the pressed key until the neighboring key prevented further movement. In contrast, when decreasing the stiffness of the ligament, the maximum adduction distance could be significantly extended, with more than 80-mm adduction seen with the 1-MPa ligament enabled by a significantly different conditional model. The width of an ordinary piano key is 13 mm, so the ligament stiffness could influence the capability of playing between five keys with abduction. The softness of the ligament also influenced the nonlinear relationship between wrist movement and abduction distance resulting from complex bone-ligament interactions. The softer the ligaments, the greater the nonlinearity of the abduction behavior of the thumb joint and the lower the horizontal displacement required to achieve a given range of abduction. For these softer ligaments, there was a more complex, nonlinear relationship between the conditional actuation and the joint actuation, with greater differences between different conditional models. These results show how different behaviors can emerge from different conditional models, with some behaviors only able to be achieved from specific conditional models.

A similar behavior is observed for thumb abduction. The rigid ligaments provided limited abduction, with the distance again determined by the piano keys. The lower-stiffness ligaments allowed more than 50 mm of extension between the thumb and finger with the 1-MPa ligaments. The greatest nonlinearity was also seen with the lower-stiffness ligaments: The maximum abduction of this conditional model was limited by the physical mechanics of the thumb joint.

Hand span behavior

The next experiment considered the ability of the hand to compress/stretch laterally, allowing passive translation and rotation of the fingers, enabling jumps and smooth transitions with variable jump lengths between notes. In addition, translation of fingers allowed the hand to be rotated such that the fingers were playing on the side, with the whole hand contributing to playing the note. Designing a passive anthropomorphic skeleton for these tasks was more challenging because a larger portion of the hand was involved in this behavior. The finger joint stiffness, EJ, contributed to this hand behavior; however, the deep transverse metacarpal ligaments (primarily contributing to hand span stiffness) provided additional stretch and were the determining factor in the behavior of this primitive.

The ability of fingers to move laterally enabled smooth sideways transitions between notes and sideways note playing. Similar to the previous experiments, the hand stretch behavior can be achieved by a two-phase articulation of wrist actuation. First, the wrist was actuated downward in the z axis by a given amount at a given rate so that the key was fully pressed with the second finger, with sideways movement limited due to the piano keys that remained unpressed. The wrist was then actuated horizontally such that the finger moved laterally, with the key kept pressed down such that the angle of the finger to the hand varied depending on the span stiffness. Again, two different consecutive conditional models were used to achieve a specific behavior. A series of experiments were conducted by using four 3D-printed hands with different ES values: 1, 2.5, and 50 MPa and 2 GPa. The horizontal displacement between the second finger and the middle finger was measured. For all experiments, EJ was kept as low as possible at 1 MPa to make the largest stretch possible. The measurement of lateral displacement for the four hands is shown in Fig. 3C (left), where the stretch was measured at every 5-mm increment of sideways movement. Movie S4 shows how compression in the span of the hand could be used to achieve jumps of varying lengths and how angled hand playing could be used.

The hands with the 2-GPa and 50-MPa hand span stiffness exhibited very small displacement under 10 to 20 mm most of the time, with limited change in the conditional model. Because the stiffness was reduced, the stretch range increased significantly, with a maximum recorded stretch of more than 40 mm. The response was initially linear; however, for lower stiffness, the response of the ligaments became increasingly nonlinear as material and geometric constraints were reached and the range of the joints was restricted, resulting in a different conditional model. Varying the stretch ligament stiffness significantly affected the possible range of lateral movement with a single finger, because there was a physical limit on the movement of the finger joint.

This span stiffness also influenced whole-hand playing, more specifically little finger playing, where the wrist is at an angle to the piano (Fig. 4C, right picture). When playing chords with jumps, whole-hand playing is often used, and the hand can be angled to achieve playing of different styles. Here, the span stiffness dominated the playing behavior. The key force was investigated when the hand was actuated downward by 15 mm, pressing the key, for different wrist angles. When the hand was perpendicular to the piano key, the force was highest because the span stiffness was fully engaged and acting in full compression. At lower angles, the stiffness was lower, often significantly so. The increase in force with angle was far greater for the stiffer span stiffness, with an increase of more than 30 mN. Varying the angle of interaction and span stiffness affected the force that could be applied to the piano keys and allowed different conditional models to emerge.

Utilization of conditional models for complex piano playing

The previous experiments have shown how conditional models for one or two fingers could be achieved. The next challenge was how to integrate these individual conditional model mechanisms to create a robot with a significantly increased range of behaviors. This integration challenge was difficult, because some of the physical and actuation conditions necessary for certain behaviors interfered or were not compatible with others. The integration process was not simply the aggregation of individual mechanisms; conditional models could not always be treated in an additive manner. The resultant behavior was dependent on the interactions between the different subsystems. Therefore, a consideration was to avoid conflicts of conditions while integrating the required behaviors without compromising behavioral performances. In the following experiments, we present a case study in which such decoupling of conditional models can be used to achieve a range of complex piano-playing behavior.

The case study considered the design of an anthropomorphic skeletal hand that could play three different pieces of music without changing the mechanical and material properties (Fig. 5). The first piece was the four bars of Toccata by Scarlatti. This is a fast-paced melody, where single-note staccato playing is repeated with periodic shifts in pitch. The second phrase of music was selected from Alligator Crawl by Fats Waller. This requires consecutive smooth playing of notes an octave (eight notes) apart with a shift between each octave played. The final phrase of music was from Rhapsody in Blue by Gershwin—the archetypal glissando (rapid slide of thumb finger between consecutive notes), which requires thumb abduction smoothly and rapidly sliding over piano keys. Each of these musical phrases required different patterns and combinations of conditional models.

Fig. 5 Case study demonstrating playing three musical phrases.

(A) Results from playing Toccata with a staccato style. The key force for a human playing and robot playing using the second finger for hands with varying stiffnesses, the average response (solid thick lines) and individual force profiles (thinner lines) for varying EJ values of the finger joints, and the repeated musical pattern that forms the basis of this phrase are shown. Representations of the approximate conditional models (CMs) with the conditional actuation (CA) and joint actuation (JA) are also shown. The robot playing was repeated 20 times. (B) Results from playing the two notes, which form the basis of the Alligator Crawl refrain. The response from the force sensors measuring the thumb force, which is used to play the first note; the little finger used to play the second note for different stiffness for all joints (EJ, ES, and ET); the conditional models; and both conditional and joint actuation are shown. The robot playing was repeated 20 times. (C) Force sensor results for playing the glissando (slurred section) in Rhapsody in Blue. The average force sensors results over three keys forming part of this slurred section, which was played with the thumb using hands of different ET values, and the different conditional model states used to achieve the playing behavior are shown. The robot playing was repeated 20 times. (D) Stiffness parameters required for the various components of the hand to play all three phrases of music closest to human playing.

For a skeletal hand to play these pieces in a style similar to humans, we had to identify the stiffness requirements of different parts of the hand for each of these three pieces. The first piece, Toccata, requires a high-frequency and high-force staccato playing style, which generally requires a conditional model with high stiffness in the finger, because lower stiffness cannot convey high-frequency strong actuation to the keys. Figure 5A shows the force profiles, measured through a sensor attached to a key, to compare the trajectories when the material stiffness of finger joints was varied. We measured force over 20 cycles to allow variations and deteriorations to be observed. Compared with an experienced human piano player (see the dotted red line), the stiffer finger with a 50-MPa ligament exhibited the performance closest to human playing, whereas the softer fingers (2.5 and 1 MPa) could not reach sufficient strength and articulation with adequate temporal length. Therefore, for playing Toccata, a single finger with the higher-stiffness (50 MPa) material is necessary for the finger joint ligaments to achieve the higher-stiffness conditional model. Because of the material surrounding the single finger, performing the playing was critical. To limit playing or interaction of surrounding keys, we maximized the stiffness of the playing fingers while lowering that of the surrounding fingers. This highlights how the environment and mechanics of the hand are coupled together and how conditional models could be used to achieve varying outputs.

The second phrase of music from Alligator Crawl by Fats Waller requires a low hand span stiffness to allow sideways translation of the finger to achieve the octave spread. Conversely, it is also important to keep these fingers sufficiently stiff to achieve reasonably well-articulated notes, to minimize the pause between notes. This required a complex conditional model that showed variation in the stiffness of different joints. To verify these requirements, we printed three hands, with ET values of 1, 2.5, and 50 MPa while keeping the hand span stiffness at 1 MPa, and measured force profiles exerted on two keys when two consequent notes were being played. Figure 5B shows the experimental results compared with the force profile of a human player. This experiment demonstrates how, for the same conditional model, different joint actuations could be achieved. The lower-finger stiffness with 1 MPa showed poorly defined notes, leading to slurred and weak articulation. The 2.5-MPa finger stiffness, in contrast, showed the closest similarity to the human’s. The height and length of the two notes are similar, and there is a comparable pause in between.

The final piece, Rhapsody in Blue, requires rapid succession of playing, with a slurred transition between notes achieved by the thumb finger sliding over the string of notes, showing adduction-like behavior. To achieve in the artificial hand a smooth transition similar to humans, we set the thumb abduction/adduction stiffness appropriately. While the wrist is moving horizontally, the thumb finger needs to interact with consecutive keys in a smooth and repetitive manner, resulting in a set of soft slurred sounds. In this case with consistent actuation, the environmental interaction led to different conditional models and joint actuation. Again, we used three printed hands with different thumb abduction/adduction stiffnesses (1, 2.5, and 50 MPa) in the experiment, where force profiles over three piano keys were recorded. Figure 5C shows that the thumb abduction stiffness of 1 MPa had temporal force profiles comparable to those of human player, whereas stiffer joints resulted in a less-smooth force profile with larger articulated forces.

By combining the optimal joint material required for each of the three phrases of music, a single hand can be printed that allows all three phrases to be played (see Fig. 5D). The thumb must have low stiffness (1 MPa) to allow abduction when playing the glissando, and a relatively low span stiffness should be used (2.5 MPa). The finger used to play the single notes in the Toccata (the first finger) must have high stiffness (50 MPa). In addition, the joint stiffness of the thumb and little finger must be 2.5 MPa to allow sufficient articulation to play Alligator Crawl. All remaining fingers (index and fourth finger) were kept at low stiffness (1 MPa) such that the other fingers interact preferentially with the piano. For this case study, the three different stiffness requirements were mostly complementary, and the complex nature of the hand skeleton allows varied stiffness to be achieved across one physical model. The actuation conditions required to achieve the playing of these phrases of music are detailed in figs. S6 to S10. A single 3D-printed hand playing the three phrases of music is shown in movie S5, with further details of the actuation given in table S2.

DISCUSSION

Despite extensive robotic manipulation research for the past half-century, dexterous hand manipulation remains an unsolved research question. Many of today’s advanced robots are not capable of manipulation tasks that small children perform with ease. We hypothesized that complex passive mechanical structures could be used to address this gap in capabilities. Biological systems use mechanical design of bodies to achieve a wide variety of behaviors, and thus, this should be reflected in manipulation research. Recent advances in multimaterial 3D printing technology allow systematic investigation of the complexity of passive mechanical structures. This approach allowed the printing of a passive anthropomorphic hand that provided the ability to reproduce complex human hand capabilities, that is, various piano-playing techniques, and to show hand behaviors that cannot be performed by other conventional robots. This included the ability to achieve thumb abduction/adduction to perform jumps when piano playing. The utilization of the environment and physical conditions makes this technique applicable across the robotics field.

Piano playing has proven to be a complex and nuanced challenge requiring a significant range of behaviors and playing styles. It is a challenge that demonstrates how the physical configuration (mechanical geometry and material properties) and system-level actuation (wrist actuation) can be coupled with the environment (the piano) to achieve playing in a variety of styles through the emergence of different conditional models. This approach has enabled robotic piano playing with a fluidity and range of behaviors that has not previously been demonstrated in robotic piano-playing work (5861). Although this work investigated only piano playing, the conditional model framework is transferable to other applications, especially to those that require diverse dynamic interactions with complex physical environment under severe restrictions of using sensory motor feedback. We have presented first steps toward clarifying the role of actuation in complex soft-rigid hybrid systems.

The introduction of soft material elements to robotic systems is particularly important when designing complex passive mechanical structures. The same principle can be applied to robotic systems and manipulators. Continuum deformable bodies have the potential to generate large varieties of behaviors; however, to generate useful and controllable systems, a new design methodology is required to unleash the potential. The inclusion of lower-stiffness ligaments in the hand design has demonstrated the advantages of softer materials. Use of hybrid soft-rigid mechanical systems with 3D printing technologies has enabled the design and fabrication of structures with anisotropic stiffness properties. In addition to the use of soft-rigid materials, the concept of conditional model provides a method of using the complexity offered by such structures. The combined conceptual approach and the manufacturing technique open up new exciting directions and methods of research.

The concept of conditional models provides insight into how such anisotropic soft-rigid hybrid structures should be systematically investigated and designed. It provides a framework that identifies the three underlying components necessary for the conditional model approach: physical configuration, actuation, and environmental conditions. All of these must be considered to achieve a specific conditional model that enables passive dynamic interactions in soft-rigid hybrid structures. If these conditions are exploited adequately, then a passive mechanical structure can achieve complex piano playing. Such exploitation is largely dependent on the complexity of mechanical structures, which maximizes the range of possible conditional models that can emerge and is possible from state-of-the-art multimaterial 3D printing technology.

The research presented in this article can also be used to better understand biological systems, by building bioinspired robotics that explore biological design and function. An obvious criticism might arise from the oversimplification of the anthropomorphic skeletal structure when compared with the biomechanics of human hands playing a piano. There are a number of discrepancies in the playing mechanisms (for example, our system did not consider the roles of muscle activities and skin frictions). Despite these limitations, the proposed approach allowed investigation of the underlying principles of skeletal dynamics to achieve highly challenging manipulation tasks. Previous work has stated that the sound produced by the piano emerges through coupling between the biomechanics and neuromuscular dynamics of the pianist (mechanical impedance of the finger) and the dynamics of the piano itself (56). Piano playing thus relies on the interaction between the mechanics of the piano keys and the player’s hand, which can be studied further by using the proposed system.

The proposed architecture can be extended to other application and research directions. The role of the physical configuration should be more thoroughly investigated to analyze the limits of what passive mechanical systems can achieve. In particular, automation of the mechanical design process would enable a more scalable and consistent approach. Another promising area of future investigation is identifying how passive properties can be used to selectively choose a specific environment and, through that, vary the conditional model. A key challenge is the integration of active stiffness control, sensory feedback, and motor learning to closely mimic biological systems. Although these advanced motion-control capabilities are important, the underlying mechanical complexity enables passive behaviors “for free,” and thus, the exploitation of these is the most important consideration.

MATERIALS AND METHODS

The case study outlined in this paper demonstrates how conditional models can be used to achieve a broad range of behavioral outputs with limited control input and presents materials and methods for producing such systems. The production of the robotic system used novel 3D printing techniques to allow the printing of variable stiffness complex systems, such as the anthropomorphic hands used in this case study. The mechanical design of this hand model, the enabling 3D printing methods, the integration of the full robotics system, and details of the experimental methods are provided in this section.

Anthropomorphic hand skeleton design

The anthropomorphic hand skeleton was adapted from a commercial 3D model purchased from TurboSquid (www.turbosquid.com). The model was used as the initial basis for the hand mechanical structure, with various modifications, such as changes in material stiffness; however, we kept changes to a minimum. Starting with an anthropomorphic computer-aided design (CAD) model of the full hand and wrist—including bones, ligaments, tendons, and muscles—we removed the tendons and muscles to leave just the passive dynamic system, which was formed from the coupling of the rigid bones and more flexible ligaments. The ligaments surrounding the finger joints (the collateral ligaments) were adapted to simplify the joints and to provide increased stability and robustness. The remaining hand model was kept fully anthropomorphic to allow the mechanics of the joint interactions to be fully explored and exploited. To allow the materials of individual parts of the CAD model to be set individually, we made the CAD model such that all parts were uniquely separable. The specific material properties of each bone and ligament could be set individually, allowing control of the internal conditions of the hand. More details of material properties used are in the Supplementary Materials.

3D printing

Multimaterial-fused deposition modeling 3D printing is an increasingly used technology (73, 74) that allows the rapid construction of 3D models with materials with varying mechanical properties. It enables durable 3D parts to be produced with high accuracy and repeatability while allowing different components of a model to be printed with varying Young’s moduli by blending the base materials. This method allows printing of complex CAD models, such as the anthropomorphic hand, in a single print, where all parts of the model are fused together. Support material was required to achieve functional compliant joints and ligament structures but must be removed chemically in a postprocessing step. The material used for each component of the CAD model could be determined individually, allowing the material properties of the ligaments to be varied individually and modulating the range and dynamic behavior of the hand.

We used a Stratasys Connex 5000 3D printer and Vero White (Stratasys), a photopolymer with high strength (tensile strength, 60 to 70 MPa) and stiffness (flexile strength, 75 to 110 MPa) to print the rigid bone structures (75). Vero White can be blended with other lower-stiffness materials to print plastic with variable stiffness. In particular, it can be blended with Tango Black, which simulates thermoplastic elastomers with flexible, rubber-like qualities and has a Shore hardness in the range of 26 to 28 scale A, allowing up to 220% elongation at break. The ligaments were printed with the Tango Black Material blended with varying ratios of Vero White. The properties of the blends of materials used for the experiments are given in table S1. The printing process took about 10 hours, with a further 4 hours required for effective mechanical and chemical removal of the support material. This allowed rapid iteration of hand designs with minimal manual postprocessing required. The hand was then attached to a UR5 robot arm (Fig. 2) to allow wrist actuation and control. 3D printing allowed the rapid and repetitive production of hand mechanical structures in which passive dynamic behaviors could be tailored with minimal additional construction or development work required.

Experimental setup

For the experiments conducted in this article, we mounted the 3D-printed hand skeletons directly on a UR5 robot arm. Using the arm allowed precise static and dynamic control of the hand skeleton and a focus on the wrist kinematics without consideration for the rest of the arm. The on-board inverse kinematic of the UR5 was used, with a Python API used to allow control of the position, movement between poses, and speed. The acceleration and deceleration of movement could be controlled in addition to the steady-state speed, allowing the dynamic and frequency response of the system to be measured. Positions corresponding to keyboard positions and fixed trajectories for chord jumps and thumb abduction movements have been collated into a database. Within this, the correct control parameters (speed, range of movement, and frequency) were set to achieve the selected music with the correct playing style for the specific material property required. More details of control parameters of the arm can be found in the Supplementary Materials with a block diagram of the system (fig. S3).

To measure the details of the hand skeleton behaviors and a ground truth of human playing, we equipped the piano with force-sensing resistors (FSRs). A load cell was used to perform calibration of the sensors. Analog digital converters on an Arduino microcontroller were used to register the response from these sensors in real time, whereas the data were synchronized with the arm motion commands.

Mechanical characterization experiments

For the experiments of the single note playing behavior primitive (Fig. 4A), we investigated the playing of a single key equipped with an FSR. The index finger was used in the experiments, with differing ligament material (with different values of EJ) for the distal interphalangeal joint, the proximal interphalangeal joint, and metacarpophalangeal joints. For these experiments, a single finger was used to allow the properties of the finger to be isolated from that of the rest of the hand, because finger properties are determined only by EJ. The wrist control parameters (frequency, speed, and displacement stroke distance) were varied for the different experiments and the response for the FSR on the key measured. The frequency of the force sensor signal was determined by measuring the average period of the output note as determined from the force sensor response, with the rate of changed used for the dynamic response. The maximum achieved force was used to indicate the achievable volume of playing. Ranges of frequencies, volumes, and playing rates were chosen to map to typical human values and to be within the capabilities of the arm providing the wrist actuation. All experiments were repeated five times, with the average given.

For the thumb abduction experiments, we varied the ligament stiffness surrounding the joint (EJ), keeping ligaments in the rest of the hand at the lowest stiffness. Although other joints, in particular the finger joints, contribute to the measured abduction, this was minimal and was kept constant across all experiments. The wrist was moved such that the thumb was playing a note (middle C) with a fixed downward displacement and an angle of inflection with the piano to allow sideways movement of the fingers. The wrist was then moved horizontally such that the thumb movement was limited by the key it was pressing, exposing the movement of the fingers relative to the thumb. By using a camera fixed above the piano, we measured the horizontal displacement between the thumb tip and the tip of the first finger. This was repeated five times, with the average distance recorded.

The final characterization related to the hand span behavior primitive, with the material properties ES varied. Similarly, the remaining joint materials were maintained at the lowest stiffness. The second finger was moved downward such that it was pressing the middle C key. The wrist was then moved horizontally, with the second finger trapped. The distance between the second and third finger was measured to provide a measurement of the compliance of the hand span. Last, to investigate how this span stiffness affects the playing behavior, the wrist was rotated and the little finger was used to play notes with a fixed vertical displacement of 15 mm, with the key force measured with a force-sensitive resistor.

Piano-playing experiments with integrated hand skeleton

The three excerpts of music were chosen on the basis of the variety of playing modes and the three behavior primitives investigated in Fig. 4. For each phrase of music, the wrist location and movements were determined from the note requirements. For each piece of music, the optimum material properties and control parameters to achieve the playing style were chosen from the results shown in Fig. 4. More details on control parameters for the arm can be found in the Supplementary Materials.

For registering the experimental data, the same force-sensitive resistors were used on the piano keys. One, two, and three sensors were installed to obtain Fig. 5, A, B, and C, respectively.

SUPPLEMENTARY MATERIALS

robotics.sciencemag.org/cgi/content/full/3/25/eaau3098/DC1

Fig. S1. Hand CAD model showing the finger ligament designs.

Fig. S2. Full experimental setup.

Fig. S3. Block diagram of the system for piano playing.

Fig. S4. Experimental method to determine the compliance of a single finger.

Fig. S5. Toccata music separated into regions.

Fig. S6. Flow chart of the motion planning required for playing Toccata.

Fig. S7. Alligator Crawl.

Fig. S8. Flow chart for Alligator Crawl.

Fig. S9. Adapted music for Rhapsody in Blue and motion planning flowchart.

Table S1. 3D-printed materials used and their material properties.

Table S2. Summary of arm control parameters.

Table S3. Summary of arm parameters used in the arm control for playing Toccata.

Table S4. Summary of arm parameters used in the arm control for playing Toccata.

Table S5. Summary of arm parameters used in the arm control for playing Rhapsody in Blue.

Movie S1. Fabrication.

Movie S2. Single-finger behavior.

Movie S3. Abduction/adduction behavior.

Movie S4. Hand span playing behavior.

Movie S5. Three pieces of music playing.

REFERENCES AND NOTES

Acknowledgments: Funding: This work was funded by the United Kingdom Engineering and Physical Sciences Research Council (EPSRC) MOTION grant (EP/N03211X/2) and the EPSRC CDT in Sensor Technologies (grant EP/L015889/1). Author contributions: J.A.E.H. designed and performed all experiments. J.A.E.H., F.I., and P.M. wrote the paper. Competing interests: The authors declare that they have no competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper or the Supplementary Materials. Altered CAD models and specific robot control code can be obtained from J.A.E.H.
View Abstract

Stay Connected to Science Robotics

Navigate This Article