Gesture Tracking

Smartwatches may be all the rage, but they’re limited in terms of how users can interact with them. (It’s difficult to type on a smartwatch display!) So computer scientists and electrical engineers have teamed up to develop a technology that can accurately track finger movements to within about 8 millimeters using sonar.

Dubbed FingerIO, the system uses the microphones and speakers already built into a smartphone or smartwatch. As the speaker sends an inaudible signal that bounces off the finger, the microphone records the “echo” to calculate the location of the finger in space. The result is an ability to interact with mobile devices by merely writing or gesturing on a tabletop, a piece of paper, or even in mid-air.

The use of sonar has several advantages over cameras. First of all, sound waves do not require a line of sight in order to transmit information, so users can interact with their device through the fabric of a shirt, for example. Secondly, sound waves travel slower than radio waves or light, so processing bandwidth is greatly reduced. And finally, the capability can be achieved without any special hardware, as most devices already have a speaker and a microphone built in.

In tests on a typical Android device, where users were asked to draw a variety of shapes, the average difference as tracked by FingerIO was 8 millimeters for a smartphone and 12 millimeters for a smartwatch – more than adequate for tracking finger movements, given the fact that a finger is about 10 millimeters thick. The researchers plan to expand the tracking abilities to multiple fingers moving simultaneously, and extend the technology to three dimensions by adding additional microphones.