Research ArticleSOCIAL ROBOTICS

Modeling engagement in long-term, in-home socially assistive robot interventions for children with autism spectrum disorders

See allHide authors and affiliations

Science Robotics  26 Feb 2020:
Vol. 5, Issue 39, eaaz3791
DOI: 10.1126/scirobotics.aaz3791
  • Fig. 1 Long-term, real-world SAR intervention setup.

    In this month-long, in-home study, child participants with ASD played math games on a touchscreen tablet, while a socially assistive robot used multimodal data to provide personalized feedback and instruction (37).

  • Fig. 2 Engagement by participant.

    This study observed a significant variance in engagement across participants (error bars, SD) (A) and for each participant (B). A decreasing trend (P < 0.01) in engagement was also observed over the month-long intervention (B), indicating the need for online engagement recognition and response.

  • Fig. 3 Model results.

    Generalized models trained on different users and individualized models trained on early subsets of the intervention achieved comparable AUROC values with those of models trained on random samples of all users’ data (A) but had much lower recall for disengagement (B).

  • Fig. 4 Comparing data across participants, sessions, and engagement states.

    A visualization of limited overlap in data of two participants (A), two sessions from the same user (B), and across engagement states (C). Higher variances in data were also observed when users were disengaged (C). Visualized data are those with high face detection confidence, compressed to two dimensions using PCA; statistically significant (P < 0.01) differences in means and variances were determined using the complete dataset.

  • Fig. 5 Sequences of engagement and disengagement.

    Median duration of ES (11.0 s) was significantly (P < 0.01) higher than the median duration of DS (4.0 s). Despite the prevalence of shorter sequences, DS longer than the upper quartile (9.5 s) accounted for 75% of the total time users were disengaged.

  • Fig. 6 Reengagement strategy results.

    Post hoc analysis of strategy to reengage users if predicted engagement probability was less than a threshold on average over a window. As shown by the orange line in (A) and (B), a threshold of 0.35 and window of 3 s would have reengaged users in not only 73% of long DS but also 15% of ES. Varying window lengths (A) and thresholds (B) illustrates the trade-off between maximizing reengagement in disengaged sequences and minimizing reengagement in engaged sequences. Results based on generalized models trained on six participants.

  • Fig. 7 Results across different modalities and model types.

    All modalities together outperformed each modality separately, but visual features were the most significant (A). Tree-based models were the most successful among conventional supervised ML model types (B).

Supplementary Materials

  • robotics.sciencemag.org/cgi/content/full/5/39/eaaz3791/DC1

    Note S1. List of multimodal features.

    Fig. S1. Comparing data across users.

    Fig. S2. Comparing data across sessions.

    Table S1. Participant demographic information.

    Table S2. Engagement annotation criteria.

    Table S3. Model hyperparameters.

    Table S4. Generalized model results.

    Table S5. Individualized model results.

    Table S6. Random sampling model results.

    Table S7. Reengagement strategy evaluation using generalized models.

    Table S8. Reengagement strategy evaluation using individualized models.

    Data S1. Dataset for engagement models.

    Data S2. Descriptions of columns in data S1.

    Reference (38)

  • Supplementary Materials

    The PDF file includes:

    • Note S1. List of multimodal features.
    • Fig. S1. Comparing data across users.
    • Fig. S2. Comparing data across sessions.
    • Table S1. Participant demographic information.
    • Table S2. Engagement annotation criteria.
    • Table S3. Model hyperparameters.
    • Table S4. Generalized model results.
    • Table S5. Individualized model results.
    • Table S6. Random sampling model results.
    • Table S7. Reengagement strategy evaluation using generalized models.
    • Table S8. Reengagement strategy evaluation using individualized models.
    • Legends for data S1 and S2
    • Reference (38)

    Download PDF

    Other Supplementary Material for this manuscript includes the following:

    • Data S1 (.csv format). Dataset for engagement models.
    • Data S2 (.csv format). Descriptions of columns in data S1.

    Files in this Data Supplement:

Stay Connected to Science Robotics

Navigate This Article