Key Points

Question Can a wearable artificial intelligence intervention designed for use in the home to reinforce facial engagement and emotion recognition improve socialization in children with autism spectrum disorder?

Findings In this randomized clinical trial of 71 children with autism spectrum disorder, children treated at home with the wearable intervention showed a significant improvement in socialization over children only receiving standard of care behavioral therapy.

Meaning The mobile intervention, which teaches the recognition and relevance of emotion in the child’s natural setting, can augment standard of care therapy to achieve higher socialization in children with autism spectrum disorder.

Abstract

Importance Autism behavioral therapy is effective but expensive and difficult to access. While mobile technology–based therapy can alleviate wait-lists and scale for increasing demand, few clinical trials exist to support its use for autism spectrum disorder (ASD) care.

Objective To evaluate the efficacy of Superpower Glass, an artificial intelligence–driven wearable behavioral intervention for improving social outcomes of children with ASD.

Design, Setting, and Participants A randomized clinical trial in which participants received the Superpower Glass intervention plus standard of care applied behavioral analysis therapy and control participants received only applied behavioral analysis therapy. Assessments were completed at the Stanford University Medical School, and enrolled participants used the Superpower Glass intervention in their homes. Children aged 6 to 12 years with a formal ASD diagnosis who were currently receiving applied behavioral analysis therapy were included. Families were recruited between June 2016 and December 2017. The first participant was enrolled on November 1, 2016, and the last appointment was completed on April 11, 2018. Data analysis was conducted between April and October 2018.

Interventions The Superpower Glass intervention, deployed via Google Glass (worn by the child) and a smartphone app, promotes facial engagement and emotion recognition by detecting facial expressions and providing reinforcing social cues. Families were asked to conduct 20-minute sessions at home 4 times per week for 6 weeks.

Main Outcomes and Measures Four socialization measures were assessed using an intention-to-treat analysis with a Bonferroni test correction.

Results Overall, 71 children (63 boys [89%]; mean [SD] age, 8.38 [2.46] years) diagnosed with ASD were enrolled (40 [56.3%] were randomized to treatment, and 31 (43.7%) were randomized to control). Children receiving the intervention showed significant improvements on the Vineland Adaptive Behaviors Scale socialization subscale compared with treatment as usual controls (mean [SD] treatment impact, 4.58 [1.62]; P = .005). Positive mean treatment effects were also found for the other 3 primary measures but not to a significance threshold of P = .0125.

Conclusions and Relevance The observed 4.58-point average gain on the Vineland Adaptive Behaviors Scale socialization subscale is comparable with gains observed with standard of care therapy. To our knowledge, this is the first randomized clinical trial to demonstrate efficacy of a wearable digital intervention to improve social behavior of children with ASD. The intervention reinforces facial engagement and emotion recognition, suggesting either or both could be a mechanism of action driving the observed improvement. This study underscores the potential of digital home therapy to augment the standard of care.

Trial Registration ClinicalTrials.gov identifier: NCT03569176

Introduction

Autism spectrum disorder (ASD) prevalence has increased dramatically in the United States in the past few decades to 1 in 59 children.1 While the symptoms of ASD vary widely, difficulty with socialization is a core deficit.2 Children struggle to engage in joint attention, sustain eye contact, and recognize facial expressions.2-5 Experts recommend 20 hours per week of applied behavioral analysis (ABA)2,6 often incorporating naturalistic developmental behavioral interventions7 with a behavioral therapist for at least 2 years.2 Although these behavioral interventions are effective, they are costly (between $40 000 and $60 000 per child per year8), and many children face challenges generalizing therapy to real-world contexts.7 Furthermore, the increase in prevalence of ASD has outpaced availability of behavioral therapists, creating wait-lists of up to 18 months in the United States.9 Learning aids based on novel ubiquitous technologies using machine learning can begin to address these problems by creating opportunities for therapy that are accessible outside of the clinician’s office. Such tools may generalize to the natural environment where the social skills are used and act as a care bridge while children wait for standard therapy.

We designed a wearable social learning aid for children with ASD to encourage facial engagement and provide feedback to the child during social interactions at home. A computer vision system runs on Google Glass that is wirelessly connected to a smartphone app to provide the intervention to the child wearing the glasses. We confirmed the feasibility of fit and its augmented reality form factor in an in-laboratory feasibility study10 and in a field test of home usage by children with ASD.11,12 These design studies supported the hypothesis that children aged 4 to 17 years can comfortably wear the glasses and process the video and audio cues provided through the smart glasses unit. Furthermore, the field test study suggested that three 20-minute play sessions per week for 6 weeks could improve social behavior, with a mean improvement of 7.4 points on the Social Responsiveness Scale, Second edition (SRS-II) (P < .01). Finally, these studies generated a consensus by the participating children on the name, Superpower Glass (SG).

In the present study, we tested the hypothesis that children with ASD randomized to use SG at home for 6 weeks along with their ongoing ABA therapy will achieve greater improvements in socialization compared with children of the same age range engaged in ABA therapy alone. Although studies have established the feasibility of using digital technology platforms to deliver therapy to children with ASD,13,14 with the exception of speech-generating devices in the home,15 there have been few clinical tests of the efficacy of digital tools as an intervention to improve core symptomatic deficits in children with ASD.13,14 To our knowledge, this is the first randomized clinical trial with an intention-to-treat (ITT) analysis designed to test the efficacy of a wearable machine learning tool for intervention on a core ASD deficit in the natural home environment.

Methods

Study Design

Participants were randomized via a computer script in a 1:1 ratio after all intake measures were completed into either the SG intervention or a treatment as usual control group (both groups received ABA therapy at home at least twice per week) for 6 weeks. A clinical coordinator assigned the participants to their condition. Another clinical coordinator who was blinded to the assignment of participants recorded all primary and secondary outcome measures (see below) at the start of condition (intake), end of 6-week condition (posttest 1), and after 6-week follow-up (posttest 2). The baseline measures Social Communication Questionnaire and abbreviated IQ were collected only at intake. We included a crossover option for the control arm participants, where their posttest 1 appointment served as the start of their treatment period. The study protocol is provided in Supplement 1, and more details are provided in eMethods 1 in Supplement 2.

Eligibility and Screening

Families were recruited between June 2016 and December 2017 and enrolled if they were within driving distance of Stanford University, had a child with ASD between age 6 and 12 years currently receiving ABA therapy at least twice per week at home, scored greater than 15 on the Social Communication Questionnaire,16 and consented to participate. Parents provided written informed consent at their intake appointment. We did not use IQ for eligibility. The study was conducted in accordance with Stanford University’s institutional review board.

Intervention

The SG intervention is worn by the child with ASD at home. The system tracks faces, classifies the emotions of the child’s social partners,10-12,17-19 and provides 2 forms of cues to the child in real time. First, a green indicator box illuminates the smart glasses unit’s peripheral monitor when a face is detected within the outward-facing camera’s field of view. Second, an emoticon appears in the display and a robotic voice audio cue is played through the bone-conducting speaker of the glasses when a face is classified as expressing 1 of 8 emotions by the machine-learning model: happy, sad, angry, scared, surprised, disgust, “meh,” and neutral (Figure 1). Families can opt to disable audio feedback. The system operates at a frame rate of approximately 15 to 20 frames per second, so facial expressions are typically recognized within 100 milliseconds.16

We designed the mobile app to be the control center that allows the child’s caregiver to manage the system. The app receives images, runs the emotion classifier, and saves video and usage data. We provided 3 engagement activity modes: (1) capture the smile, during which the child is prompted by audio to find an emotion in a family member’s face by, for example, telling a joke to elicit the happy emotion; (2) guess the emotion, during which the caregiver asks the child to guess the emotions they are acting and controls the response manually; and (3) free play, an unstructured activity during which the child receives emotional cues for all individuals interacting with them. We automatically logged the activities chosen and compiled the videos of each session for families to review, upload, or delete.

Those in the SG group were asked to use each of the 3 engagement activities at least once and to use the device at home for 20 minutes 3 times per week with family members and once per week with their ABA behavior interventionist (BI), for a total of 4 times per week. Because we gave families the option to delete videos for privacy reasons, we measured usage through the number of days on which at least 1 video was created, whether saved or deleted, during the intervention period. This was an open-label trial; families and their BI were unblinded to the treatment. The clinical coordinator remained blinded for the entire study.

Primary Outcome Measures (Unordered)

The SRS-II20 total score is a 65-item survey completed by a child’s caregiver to identify the presence and severity of social impairment in children across 5 social domains. Scores below and above 60 were consistent with typical development and ASD, respectively.

The Vineland Adaptive Behavioral Scales, Second edition (VABS-II),21 socialization subscale is an assessment that measures communication, daily living, socialization, motor functioning, and adaptive behavior skills. The VABS-II has been clinically validated for use to track change over time22,23 and is used as an outcome measure in several randomized clinical trials for children with ASD.24,25 A higher score indicates greater adaptive functioning. We set the socialization subscale a priori as a primary end point and considered the full scale as a secondary outcome measure.

The Developmental Neuropsychological Assessment, Second edition (NEPSY-II),26Affect Recognition Domain measures facial affect recognition of 6 emotions (happy, sad, angry, fear, disgust, and neutral) from a standardized set of photographs of children’s faces over 4 tasks. Higher scores indicate a higher ability to correctly compare facial expressions.

The Emotion Guessing Game (EGG) was designed by our team to evaluate a child’s ability to correctly label emotions expressed by a live human actor. The child’s score is calculated as the number of correct guesses made on 40 facial expressions randomized to include 5 examples of the 8 emotions. The assessment was done in person by the blinded clinical coordinator. In addition, we included the complete Child Behavior Checklist27 and VABS-II21 composite score as secondary endpoints.

Intake Screening Measures

We used the Social Communication Questionnaire16,28 to screen for ASD and determine eligibility for the present study. A score greater than 15 is consistent with an ASD diagnosis. The abbreviated IQ29 measures a child’s standard IQ score based on a nonverbal fluid reasoning task and a verbal knowledge task.

Analysis Sets

We performed analysis on 4 sets of participants. The primary ITT analysis was performed on what is henceforth referred to as the ITT cohort, including all participants who were randomized into the study. We performed secondary analyses on the completers cohort, which included treatment and control participants who completed all intake and posttest 1 measures; the treatment-first cohort, which included all participants assigned to treatment (ITT); and the full-treatment cohort, which included all participants who received the intervention and completed at least their first posttest appointment after the intervention, including crossover participants (eMethods 2 in Supplement 2).

Primary Analysis

To test the efficacy of SG plus ABA vs ABA alone, we used a generalized linear mixed-effects regression model and assessed each outcome separately. The model included all available measurements for all participants randomized in the study, including withdrawn participants and those who did not adhere to the intervention. Specifically, for each outcome measure M in the measures section above, and individual i at time of measure t, we applied the following model: M it = γ 0 + γ 0 i + γ 1 weeks it + γ 2 treatment i + γ 3 weeks it × treatment i + ε it , where the individual-specific random effect, γ 0 i , was included to account for correlation across repeated measurements within an individual over time, weeks it represents the number of weeks elapsed since randomization for the ith individual at the tth time point, and treatment i is an indicator representing whether the ith individual belongs to the treatment-first group. To test for significance, we use a 2-sided Wald test on γ 3 (representing the impact of treatment over time) at the Bonferroni-adjusted level of .0125 to account for the testing of our 4 primary outcomes. We excluded 3 participants from the ITT cohort owing to a randomization error at the intake appointment; testing was performed to ensure that their removal does not affect significance in any of the primary analyses.

We further assessed whether abbreviated IQ, age, or sex were modifiers of treatment in the regression analysis. For this purpose, we augmented the above analysis to include the modifier (eg, age) and interaction terms with treatment, week, as well as a 3-way interaction term with treatment and week. Likelihood ratio tests then assessed whether the modifier significantly impacted the treatment effect. In addition to the primary ITT cohort, we applied the same mixed-effects model analysis to the completers cohort to evaluate the impact of the intervention on outcomes among those who completed the study.

Six-Week Follow-up Analysis

To test if a return to baseline occurred in the treatment group, we applied a model that tracks gains from intake to posttest 1 and posttest 2 only among those assigned to the treatment group: M it = γ 0 + γ 0i + γ 1 weeks it × [posttest 1] it + γ 2 weeks it × [posttest 2] it + ε it , where a Wald test on γ 2 allowed us to test whether gains at 12 weeks were (still) significantly different from the intake baseline. The sign and magnitude of the coefficient indicates the point value of the change. We also performed a follow-up analysis on the entire treatment cohort, including control participants who crossed over (eMethods 2 in Supplement 2).

Secondary Measures

We applied the ITT model from the primary analysis to the secondary measures, Child Behavior Checklist and the full VABS-II scale. We also performed the same moderator analysis on the secondary outcome measures.

Treatment Cohort Analysis

We performed further exploratory analyses on the treatment-first cohort and then on the full treatment cohort, which included the crossover group. Using mixed-effects linear regression methods, we compared changes in outcome measures to baseline values of abbreviated IQ, age, usage, and the level of BI involvement to evaluate additional trends, such as increased effect of SG use on particular age groups and developmental levels.

Results

A total of 474 families were screened for eligibility. Seventy-four were initially enrolled, but 3 were later excluded owing to a randomization error that occurred early in the study (their data do not affect significance in any of the primary analyses). Of the 71 enrolled participants remaining, 40 (56.3%) were randomly assigned to treatment, and 31 (43.7%) were randomly assigned to control. One participant reported an adverse reaction to the glasses. Participants received between 15 to 20 hours of standard ABA per week. The consort flow diagram of the study participants is in Figure 2. Participant demographics are reported in Table 1.

The mean (SD) treatment time between intake and posttest 1 was 6.81 (1.85) weeks. Families in the treatment group used the device on a mean (SD) of 12.12 (5.80) times through the 6-week treatment period, 51% of the requested dosage of 24 days. Participants played guess the emotion and capture the smile in 39.8% and 23.8% of the sessions, respectively, suggesting a potential preference for structured games over the unstructured free play option, which was chosen 36.4% of the time. Families ran a mean (SD) of 3.9 (3.38) sessions with their ABA BI, 65% of the recommended dose. Families did not report problems with the emotion classification, and our empirical measurement of accuracy (eMethods 1 in Supplement 2) was 72%.

Primary Analysis

Results from the primary ITT analysis of 71 individuals and the completers cohort of 52 individuals are presented in Table 2. Composition of the cohorts is presented in eTable 1 in Supplement 2. The VABS-II socialization subscale score significantly increased between start and end of the intervention in treatment-to-control comparisons (mean treatment impact: 4.58 points, P = .005, mean learning effect: −1.56 points in the ITT cohort; mean treatment impact: 5.38 points, P < .001, mean learning effect: −1.58 points in the completers cohort). Emotion Guessing Game, NEPSY-II–Affect, and SRS-II showed larger positive mean changes in treatment participants compared with controls, but those improvements were not significant (Table 2). Moderator analyses for abbreviated IQ, age, and sex showed a moderation effect for sex for the EGG, with girls showing greater improvement (likelihood ratio test P = .004; eTable 2 in Supplement 2).

Six-Week Follow-up Analysis

Results from the follow-up analysis conducted on the treatment-first group for all ITT participants (n = 40) and completers (n = 27) are presented in Table 3. We observed a reduced mean improvement from intake to follow-up on the VABS-II socialization subscale and loss of significance. We observed significant gains on EGG (mean [SD] treatment change over 12 weeks, 5.647 [1.166]; P < .001) and SRS-II (mean [SD] treatment change over 12 weeks, −2.832 [0.951]; P = .003), from intake to posttest 2 but not intake to posttest 1, like in the primary analysis. We present results on the full-treatment cohort including crossovers in eTable 3 in Supplement 2.

Secondary Measures

No significant changes from intake to posttest 1 were observed on secondary measures (eTable 4 in Supplement 2). The VABS-II adaptive composite score showed a moderation effect for age, with younger participants showing slightly greater improvement (likelihood ratio test P = .04, eTable 2 in Supplement 2).

Treatment Cohort Exploratory Analysis

We observed no noteworthy exploratory interactions beyond the ITT moderator. Specifically, we found no significant correlation between increased BI involvement in therapy and social skills gains. Usage in the entire treatment completer cohort, including 12 additional crossover participants, was similar to the treatment-first cohort (eResults in Supplement 2).

Discussion

An ITT analysis revealed that children with ASD receiving ABA who were randomized into the SG intervention showed significant improvements between intake and conclusion on the VABS-II socialization subscale, a clinical measure sensitive to changes in socialization.22-25 These gains were similar to those exhibited in other studies.24,25 Significant differences between cohorts were not observed on the other primary end points. While there was a positive change on VABS-II socialization between intake and 6 weeks after the conclusion of the SG treatment, this was not significant. Additional gains at posttest 2 were observed for SRS-II and EGG, but given a lack of posttest 2 control data, we cannot rule out a practice effect for these measures. Overall, these results support the hypothesis that the SG intervention can improve social skills of children with ASD between the ages of 6 and 12 years as an augmentation to standard of care therapy.

On average, participants used the device on half as many days as initially recommended. Potential reasons for this include decreased motivation by parents to use the device over time and/or decreased engagement from children with the device over time. These factors should be examined in future studies by improving system engagement and further examining the impact of the intervention.

The SG intervention has at least 2 potential mechanisms of action: reinforcement that faces have variation in emotion (salience of emotion) and training on how to differentiate emotions. Analyses of potential correlations between outcome measures and usage or participant demographics yielded no clear conclusions about best responders. Nevertheless, we hypothesize that the dual mechanisms of action may have contributed to the observed socialization gains despite use in fewer sessions than the prescribed usage. Beyond these 2, it is possible that the learning aid generally encourages social interaction in the family around face contact and emotion, thereby increasing social acuity in the child.

Limitations

While our change was in line with previous reports using the VABS-II, the poststudy empirical variance suggested that the study may be underpowered by a factor of 2. Further limitations include low participant adherence to the recommended treatment dosing and the restricted recruitment of individuals to within driving distance of Stanford where the population may be enriched for familiarity with technology. Additionally, owing to the inherent demographic and behavioral heterogeneity of children with ASD and differing amounts of ABA therapy received prior to the intervention, children began the intervention with varying levels of social skills. Finally, because control participants did not have a second posttest appointment before crossing over into treatment, we were unable to perform direct comparison tests for sustained gains. While data from the treatment group points toward a need to increase the 6-week treatment period, further research is required.

Conclusions

This is the first randomized clinical trial to demonstrate efficacy of a wearable behavioral intervention for children with ASD, to our knowledge. The intervention teaches children emotion recognition, facial engagement, and the salience of emotion, suggesting the potential for multiple mechanism(s) of action driving the observed improvement in social behavior. Results of this study underscore the potential of digital home therapy to augment the current standard of care.

Back to top Article Information

Corresponding Author: Dennis P. Wall, PhD, Department of Pediatrics, Division of Systems Medicine, Stanford University, 1265 Welch Rd, Medical School Office Bldg x14, Stanford, CA 94305 (dpwall@stanford.edu).

Accepted for Publication: January 17, 2019.

Published Online: March 25, 2019. doi:10.1001/jamapediatrics.2019.0285

Open Access: This article is published under the JN-OA license and is free to read on the day of publication.

Author Contributions: Mr Voss and Dr Wall had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Voss, Daniels, Kline, Haber, Washington, Robinson, Feinstein, Winograd, Wall.

Acquisition, analysis, or interpretation of data: Voss, Schwartz, Daniels, Kline, Haber, Tariq, Robinson, Desai, Phillips, Feinstein, Wall.

Drafting of the manuscript: Voss, Schwartz, Daniels, Kline, Haber, Wall.

Critical revision of the manuscript for important intellectual content: Voss, Schwartz, Daniels, Kline, Washington, Tariq, Robinson, Desai, Phillips, Feinstein, Winograd, Wall.

Statistical analysis: Voss, Schwartz, Kline, Haber, Tariq, Robinson, Desai, Wall.

Obtained funding: Voss, Haber, Wall.

Administrative, technical, or material support: Voss, Schwartz, Daniels, Kline, Haber, Washington, Tariq, Wall.

Supervision: Voss, Daniels, Feinstein, Winograd, Wall.

Conflict of Interest Disclosures: Mr Voss reports personal fees from Cognoa, Inc. outside the submitted work; reports a pending patent (US 62/333,108); and participated in advanced contractual discussions, planning to license the patent and software enabling the Superpower Glass intervention to Cognoa, Inc. Mr Voss has been a technological consultant at Cognoa, Inc. since August 2018 and Cognoa has prepared a US Food and Drug Administration breakthrough submission based on the intervention in October 2018. As a co-owner of the intellectual property and patent with Stanford University, Mr Voss will receive licensing fees and royalties if and when the intervention is commercialized and sold as a medical product. Cognoa has not participated in any study design or analysis. The statistical analysis of this study occurred before and independent from any involvement with Cognoa and licensing discussions were started independently from the randomized clinical trial outcomes. Ms Schwartz reports no conflicts. Ms Daniels reports no conflicts. Mr Kline reports personal fees from Cognoa, Inc. outside the submitted work; reports a pending patent (US 62/333,108); and participated in advanced contractual discussions with Cognoa, Inc. with the intent to license the patent and software enabling the Superpower Glass intervention. Mr Kline has been a technological consultant at Cognoa, Inc. since September 2018 and Cognoa has prepared a US Food and Drug Administration breakthrough submission based on the intervention in October 2018. Mr Kline will receive additional milestone compensation if (a) that US Food and Drug Administration process proceeds successfully and the intervention is commercialized and sold as a medical product and (b) other company milestones not related to the Superpower Glass intervention are also met. Dr Haber reports; personal fees from Cognoa, Inc. outside the submitted work; and reports a pending patent (US 62/333,108); and participated in advanced contractual discussions, planning to license the patent and software enabling the Superpower Glass intervention to Cognoa, Inc. Dr Haber has been a technological consultant at Cognoa, Inc. since September 2018 and Cognoa has prepared a US Food and Drug Administration breakthrough submission based on the intervention in October 2018. As a co-owner of the patent with Stanford University, Dr Haber will receive licensing fees and royalties if and when the intervention is commercialized and sold as a medical product. Mr Washington reports no conflicts. Ms Tariq reports no conflicts. Dr Robinson reports no conflicts. Dr Phillips reports no conflicts. Dr Feinstein reports personal fees from Trayt.io outside the submitted work and other support from Riverside County Mental Health, Riverside University Health, and Behavioral Health Services outside the submitted work. Dr Winograd reports a pending patent (US 62/333,108). Dr Wall reports a pending patent (US 62/333,108); and participated in advanced contractual discussions, planning to license the patent and software enabling the Superpower Glass intervention to Cognoa, Inc. Cognoa is considering licensing this technology but the negotiations are not complete. Dr Wall is the founder of Cognoa, Inc. No other disclosures were reported.

Funding/Support: The work was supported by material support (35 units of Google Glass) from Google Inc and grants from the National Institutes of Health (grants 1R01EB02502501 and 1R21HD091500-01), the Lucile Packard Foundation for Children’s Health, the Hartwell Foundation, the Wallace H. Coulter Foundation, the Stanford Precision Health and Integrated Diagnostics Center (PHIND), the Stanford Predictives and Diagnostics Accelerator (SPADA), the Stanford Bio-X Center Program, the Stanford Beckman Center, the Walter V. and Idun Berry Postdoctoral Fellowship Program, the Bill and Melinda Gates Foundation, Peter Sullivan, and Bobby Dekeyser. We thank David Orr and Imma Calvo for their generous support in obtaining funding.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Data Sharing Statement: See Supplement 3.

Additional Contributions: We thank each family that participated in this research study for their time and contributions. Additionally, we sincerely thank Geraldine Dawson, PhD (Duke University), for her input during the preparation of the manuscript. Dr Dawson was not compensated for this work.