One of the coolest phenomena in sensory neuroscience is sensory substitution, or the brain’s ability to make sense out of non-natural sensory information. For instance, when an algorithm is used to convert video into an audio signal, people can learn to use the generated audio to navigate and pick up objects—and even activate the visual cortex while doing so! This ability is likely owed to the brain’s remarkable ability to detect statistical patterns, which allows it to quickly work out how patterns of tones correspond to the spatial features of the environment.



Amazingly, this ability can extend even to senses that we don’t have; for instance one team found that people were able to improve their navigation performance when fitted with a device that communicated the direction of magnetic north through vibration. After six weeks of training, people reported not only learning to interpret the vibration signals but also developing a new intuitive sense of spatial orientation; tests of eye movements also showed that some subjects integrated information from the vibration signals into their automatic eye-stabilization reflexes.





The possibility of augmenting our sensory systems has become particularly exciting now that most people carry around a phone constantly connected to the internet and loaded with sensors capable of detecting everything from infrasound to ultraviolet.



I recently started experimenting with this technology using a smartphone and a Wear OS smartwatch to actually deliver output (vibration) to my wrist. As a first “sense” I wanted to add, I developed a program that used the phone’s radio to detect the beacons broadcast by Bluetooth devices and translate them into specific vibration patterns that were played on the watch.



While using the system initially just felt like weird buzzing on my wrist, after using the system for a few weeks I began to notice something like what the subjects in the magnetic-north trial did. I started to develop an intuitive ability to attribute the signals to specific device; to notice people moving behind me based on their Bluetooth transmissions, and to recognize specific people from specific Bluetooth patterns.



Building a navigation app



While this was cool, it wasn’t necessarily useful. To put together something more practical, I decided to build a smartwatch-based version of Feelspace’s original navigation app. The app uses the heading derived from GPS measurements to generate a vibration signal which corresponds to the cardinal direction you are moving, where the vibration frequency decreases as the travel direction goes from south to west to north to east.













If you want to try this app, you can download it here! You’ll also need to download Sixth Sense which provides the utilities to actually generate the vibration.



I also decided to convert the code that handles communicating with wearable devices and generating vibrations to a separate app (called Sixth Sense), which provides services that can be used by any app that wants to uses sensory substitution functions. Interfacing with wearable devices can be complicated and annoying, so having a dedicated “output” app allows many “sensing” apps to easily generate and execute vibration patterns without needing to know about the lower-level details. In addition to permitting other developers to easily write apps, this means the Sixth Sense system is platform agnostic—for instance, it can generate vibrations on a Wear OS smartwatch if one is present, but also use the phone’s internal vibration motor.









Building new sensory substitution apps

This section talks about developing apps that work with Sixth Sense. If you’re not interested in that it’s probably not worth it.





Sixth Sense allows any app to generate vibration patterns on a wearable simply by sending a broadcast as shown below





Intent intent = new Intent(); intent.setAction("neurelectrics.sixthsense.SEND_PACKET"); intent.putExtra("sequence", "50,0,50,0"); intent.putExtra("priority", 1); intent.putExtra("duration", 100); intent.putExtra("expires",System.currentTimeMillis() + 1500); sendBroadcast(intent);

This command will cause whatever device the user has selected to output vibrations to vibrate according to the parameters passed in extras described below. All parameters are required.



Sequence: A String of arbitrary length, containing the vibration intensity at each time point in a comma-separated list. Vibration intensity can range from 0 (motor off) to 255 (maximum).



Not all output devices support variable intensity, and these devices interpret any value above 0 as “on”. Therefore it’s important to make sure that important info is not encoded in sequences like “50,100,50,100”, which will just produce a continuous vibration on devices without intensity control.



Priority: An int larger than 0 describing how time-critical this output is. Outputs with a smaller value will preempt outputs with a higher value. Because multiple apps can be sending data to SixthSense at the same time, an output can be preempted by an output from another app.





Duration: An int specifying how long each time point in the sequence lasts. This value must be less than 255. Longer point durations can be achieved simply by duplicating each point (i.e. a 500 ms vibration could be generated with a sequence of “100,100” and a duration of 250)





Expires: long value describing at what time this output should be discarded if it has not been transmitted, with the time specified in the format of System.currentTimeMillis(). Because outputs can be delayed due to system lag or preemption by higher priority outputs, the expires value allows you to control when the data is considered “stale” and should not be transmitted.





There are also a few other things to keep in mind when building sensory substitution apps.





Minimize memory load: While it’s possible to encode information in complex ways (i.e. intensity ratio between two pulses), these are difficult for users to learning because they require the user to remember the length of the first pulse while waiting for the second). Simple manipulations (like changing the frequency) are easier to learn.





Provide an option to control the speed of vibration sequences. Sequences can be easier to learn when they are initially presented slower.





Be cautious of high frequencies: The mechanical properties of many devices act like a low pass filter, so encoding critical information above 20 Hz is not recommended.



Be mindful of power use. Continuously monitoring many sensors can drain the phone battery. The impact can be greatly reduced by optimizing for the circumstances when the app is used (i.e. NavSense turns on GPS only when the accelerator detects motion).