Realtime Voice Analysis with Google Glass

Twillio’s recent Autism Hackathon inspired me and several others to try to harness real time sensor analysis, including voice, to help people with autism be more self sufficient. We used Google’s new wearable computer, Google Glass. Staff from the Autism Speaks organization, and another gentleman who works full time providing life training for people with Autism Spectrum Disorder, said our hack would be great for mock interview training! Here’s how it works:

We built on top of an excellent open source sound analysis app called Audalyzer. Here’s what the basic Audalyzer screen looks like running on Google Glass with a whistle at a certain tone in the background:

Screens we added then help someone keep their voice calm and level during an interview, comparing the current loudest pitch to the average:

We also used other sensors. For example, the orientation sensor is used to track gaze level. Here is looking at your feet:

Here is a level gaze:

A scoring system rewards users who get things right and ace their interview!

The code is all open source on Github. Most of the new code is in the mymonitor package with some ugly hooks into the main app due to time constraints at the hackathon. Code is broken into separate analyzers that can warn the user about behavior. For example, here’s a simple analyzer about speaking too loud:

public class TooLoudAnalyzer implements Analyzer { private static final float TOO_LOUD = -20f; private static TooLoudAnalyzer instance = new TooLoudAnalyzer(); private Float currentVolume; public static synchronized TooLoudAnalyzer getInstance() { return instance; } @Override public synchronized String getLabel() { final Boolean okLoudness = isConditionGood(); return null == okLoudness ? "Measuring loudness..." : okLoudness ? ("Good Job Lance!

(" + Math.round(currentVolume) + "dB vs " + Math.round(TOO_LOUD) + "dB)") : ("Please keep your voice down.

(" + Math.round(currentVolume) + "dB vs " + Math.round(TOO_LOUD) + "dB)"); } @Override public synchronized Boolean isConditionGood() { if (null == currentVolume) { return null; } return currentVolume < TOO_LOUD; } public synchronized void setPower(float power) { if (Float.isInfinite(power) || Float.isNaN(power)) { return; } currentVolume = power; } @Override public synchronized Integer getDrawable() { return R.drawable.loud_card; }

Thanks for checking out our team’s hack! This seems to be a really interesting area to work in! Rock Health recently funded a company called Spire, which helps lower stress and make people more productive by measuring breathing patterns and letting people train themselves to be calmer. Measuring voice has similar interesting propositions, not just for keeping your own voice calm, but also for detecting stress or pain levels in a conversation partner or patient’s voice.

-Lance

Interested in working on cutting edge Android and Glass apps at startups and multi-billion dollar companies? New Frontier Nomads is hiring! Check out our job listing.