Research ArticleSOFT ROBOTS

An anthropomorphic soft skeleton hand exploiting conditional models for piano playing

See allHide authors and affiliations

Science Robotics  19 Dec 2018:
Vol. 3, Issue 25, eaau3098
DOI: 10.1126/scirobotics.aau3098
  • Fig. 1 Representation of the conditional model.

    M indicates a motor that provides actuation, and T indicates the resultant torque. (A) A conditional model occurs when a conditional actuation is applied to a physical configuration (e.g., geometry and materials). This model represents the interaction between the system and the environment. There is a secondary internal actuation of the system, the joint interaction, which is dependent on the restriction of the joint space of the conditional model. This results in the passive-based output behavior, leading to a change in the initial physical configuration of the system. (B) For each resultant conditional model, the joint actuation is dependent on different physical configurations from which a second conditional model can be achieved. In this way, it is possible to achieve conditional models that then allow other conditional models to emerge.

  • Fig. 2 The piano-playing hand.

    The hand has been printed at the same scale as a human hand, where the middle finger has a length of 6 cm. (A) Anthropomorphic model of the hand showing the three groups of ligaments that influenced the three behavior primitives investigated. (B) 3D-printed hand conditional stiffness system attached to the UR5 robot arm, which provided the external actuation, and the piano environment used.

  • Fig. 3 Demonstration of the conditional model.

    (A) Compliance behaviors of a single finger (distal phalange to metacarpal) printed to scale with varying ligament stiffnesses. Varying compliance allows the emergence of different conditional models for single-finger interactions. (B) Directional compliance of a thumb, showing how the different environmental interactions and physical configurations lead to the emergence of different conditional models.

  • Fig. 4 Experimental testing of behavioral primitives.

    (A) Single-finger playing experiments, with 3D-printed fingers with varying material stiffnesses (EJ) showing the effects of varying different control parameters: frequency of playing (playing speed), rate of note playing (playing style, e.g., legato/staccato), and maximum force detected on the fingertip (volume). The force was measured with FSRs on the piano keys. These results highlight the difference in joint actuation for the different models. bpm, beats per minute. (B) Abduction/adduction distance measured between the tip of the thumb and the tip of the first finger when the wrist was actuated horizontally after the thumb was moved vertically down such that it pressed the key. Experiments were undertaken with hands with varying thumb ligament stiffnesses (ET). (C) Hand span stiffness demonstrated with a single finger (left), where the displacement between the second and third finger was measured, with the second finger playing a note and the wrist actuated horizontally. Whole-hand playing (right) when the wrist was actuated at varying amounts and the stiffness changed. The right graph shows the varying output forces when these conditions are changed.

  • Fig. 5 Case study demonstrating playing three musical phrases.

    (A) Results from playing Toccata with a staccato style. The key force for a human playing and robot playing using the second finger for hands with varying stiffnesses, the average response (solid thick lines) and individual force profiles (thinner lines) for varying EJ values of the finger joints, and the repeated musical pattern that forms the basis of this phrase are shown. Representations of the approximate conditional models (CMs) with the conditional actuation (CA) and joint actuation (JA) are also shown. The robot playing was repeated 20 times. (B) Results from playing the two notes, which form the basis of the Alligator Crawl refrain. The response from the force sensors measuring the thumb force, which is used to play the first note; the little finger used to play the second note for different stiffness for all joints (EJ, ES, and ET); the conditional models; and both conditional and joint actuation are shown. The robot playing was repeated 20 times. (C) Force sensor results for playing the glissando (slurred section) in Rhapsody in Blue. The average force sensors results over three keys forming part of this slurred section, which was played with the thumb using hands of different ET values, and the different conditional model states used to achieve the playing behavior are shown. The robot playing was repeated 20 times. (D) Stiffness parameters required for the various components of the hand to play all three phrases of music closest to human playing.

Supplementary Materials

  • robotics.sciencemag.org/cgi/content/full/3/25/eaau3098/DC1

    Fig. S1. Hand CAD model showing the finger ligament designs.

    Fig. S2. Full experimental setup.

    Fig. S3. Block diagram of the system for piano playing.

    Fig. S4. Experimental method to determine the compliance of a single finger.

    Fig. S5. Toccata music separated into regions.

    Fig. S6. Flow chart of the motion planning required for playing Toccata.

    Fig. S7. Alligator Crawl.

    Fig. S8. Flow chart for Alligator Crawl.

    Fig. S9. Adapted music for Rhapsody in Blue and motion planning flowchart.

    Table S1. 3D-printed materials used and their material properties.

    Table S2. Summary of arm control parameters.

    Table S3. Summary of arm parameters used in the arm control for playing Toccata.

    Table S4. Summary of arm parameters used in the arm control for playing Toccata.

    Table S5. Summary of arm parameters used in the arm control for playing Rhapsody in Blue.

    Movie S1. Fabrication.

    Movie S2. Single-finger behavior.

    Movie S3. Abduction/adduction behavior.

    Movie S4. Hand span playing behavior.

    Movie S5. Three pieces of music playing.

  • Supplementary Materials

    The PDF file includes:

    • Fig. S1. Hand CAD model showing the finger ligament designs.
    • Fig. S2. Full experimental setup.
    • Fig. S3. Block diagram of the system for piano playing.
    • Fig. S4. Experimental method to determine the compliance of a single finger.
    • Fig. S5. Toccata music separated into regions.
    • Fig. S6. Flow chart of the motion planning required for playing Toccata.
    • Fig. S7. Alligator Crawl.
    • Fig. S8. Flow chart for Alligator Crawl.
    • Fig. S9. Adapted music for Rhapsody in Blue and motion planning flowchart.
    • Table S1. 3D-printed materials used and their material properties.
    • Table S2. Summary of arm control parameters.
    • Table S3. Summary of arm parameters used in the arm control for playing Toccata.
    • Table S4. Summary of arm parameters used in the arm control for playing Toccata.
    • Table S5. Summary of arm parameters used in the arm control for playing Rhapsody in Blue.
    • Legends for movies S1 to S5

    Download PDF

    Other Supplementary Material for this manuscript includes the following:

    • Movie S1 (.mp4 format). Fabrication.
    • Movie S2 (.mp4 format). Single-finger behavior.
    • Movie S3 (.mp4 format). Abduction/adduction behavior.
    • Movie S4 (.mp4 format). Hand span playing behavior.
    • Movie S5 (.mp4 format). Three pieces of music playing.

    Files in this Data Supplement:

Stay Connected to Science Robotics

Navigate This Article