Part 1 Recap

In part 1, I covered the importance of using data insights in helping to create a world-class fingerprint sensor solution. In that journey, five different data areas were relied on -specifically:

BigQuery data User trial observations User trial surveys Key Performance Indicators (KPIs) Customer sentiment

In this next article, I'll go over the BigQuery data collection, reports, and insights that were gathered to understand, triage, and improve fingerprint sensor performance. There is a lot of ground to cover, so will break this out into multiple articles.

Big Data Environment

Infrastructure basics

Motorola had created an end-to-end pipeline that took data (events) generated on a smartphone and uploaded periodically to the cloud, where it was hosted in a Google BigQuery data warehouse environment. Commercially, this system interacted with tens of millions of live devices that generated petabytes of data. But this same system was also available to engineers building the next generation of products. I leveraged this system to collect data from internal users on prototype hardware and analyzed data in a sandbox environment.

Data flexibility

Part of the BigQuery data insight process involved defining what data to collect. In reality, there was some data instrumented "up front" - i.e. things that you knew were going to be relevant to collect. Other data elements were added over time, to help triage and solve issues that were being raised by internal users, that could not be addressed with the initial data dictionary.

At a high level, data was collected surrounding:

Fingerprint enrollment events Fingerprint unlock events

The data collections could also be analyzed in flexible groupings, such as:

Observing individual devices or across a collection of devices

Performance on a single software release or across multiple software releases

Performance viewed on a single calendar date, per multiple calendar dates, or consolidated over a period of time

Various combinations of the above





BigQuery Data Analytics for Fingerprint Sensor (FPS)

Unlock Activity

During the product development process, it was common for internal users to personally report observations and issues to me. The Unlock Activity report helped give a general summary of what type of user this was - e.g. occasional or heavy user, and to get an idea of how regularly they were using the device. This helped set the stage to better understand statements such as "the FPS is working better/worse than before" or "the FPS is unreliable" because they could be fact-checked against data for this particular user.

The core metrics captured in the Unlock Activity report included:

Reject total - total of failed attempts to unlock Success total - total of successful attempts to unlock Unlock attempts total - a sum of Reject + Success totals Reject percentage - percentage of unlock attempts that were rejected

Again, this report was more to help characterize the user and perform initial cross-check of feedback claims. It could not, for example, adequately tell if the user enrolled with one finger and then tried to unlock with a different finger, which could explain high rejection rates.

Below screenshot shows an example Unlock Activity report

Unlock Efficiency

A great fingerprint sensor solution is one that always successfully unlocks on the first touch attempt. While perfection is desired, in real-world usage, this is not realistic. If the user can unlock on that second attempt, as long as the bulk of the distribution still favors a single touch, this is still a good solution because the majority of overall usage will still be faster than the alternative, which at the time was PIN unlock.

The Unlock Efficiency report shows the percentage of unlocks that occurred on the 1st touch (0 rejections), the 2nd touch (1 rejection), all the way up to a 5th touch (4 rejections) if applicable, summing up to 100%. Note that a 6th touch attempt (or greater) was not allowed in our design because five touch failures in a row would be classified as a transaction failure. In this scenario, the fingerprint sensor would be locked out for a short period of time, partly to help slow down any brute force attacks to unlock the device using a fingerprint spoof.

Below screenshot shows an example Unlock Efficiency report. Note that for device identified with barcode 0022081928 on software build MPJ24.139-63 experienced unlocking 78% of the time on a first touch and 20.82% unlocking on the second touch. In essence, 98.82% of the time the user was able to unlock their device as quick or quicker than if using a PIN unlock method.

Poor Quality - Incidental Touches

How does a fingerprint sensor know to when to run? And by run, I mean starting the process to capture an image of whatever is in contact with the sensor, which it hopes is a genuine fingerprint. Keeping a sensor continuously on and imaging is expensive in terms of battery drain - which has to be avoided in portable consumer electronics. Most capacitive touch fingerprint sensors get around this by including a dedicated "wake-on-touch" (WOT) circuit, which can run at low power and periodically check for a possible finger touch. When the sensor WOT circuit triggers, it wakes the rest of the sensor system to start grabbing full fingerprint images, run compensation algorithms, extract features, and perform match checks against enrolled fingers.

The sensor solution from our fingerprint sensor vendor could generate a "quality score" of the print image captured following successful WOT. A threshold mechanism could prevent any further processing of the print image when the quality score was considered too low and unlikely to be a real fingerprint.

Several very interesting observations were made during development, which tied back into monitoring this quality score:

Initially, users who pocketed their phones often found upon retrieval that there were failed unlock attempts or transaction failures. We eventually discovered that the phone would bounce around in their pocket and image through the pocket cloth material against their leg. In some cases, people who exercised with their phone generated hundreds of WOT events, but with low quality fingerprint image scores. Knowing that the sensor could wake and image through clothing materials and mistake non-fingerprint skin areas as potential fingerprints, resulted in a side analysis of material and skin areas (e.g. palm) to learn the susceptibility to false and incidental touches.

Based on this, there were several tuning iterations to discover what made an effective quality threshold --

We wanted to set the quality threshold high enough to avoid full system wake-ups, "invalid" unlock failures, and outright transaction failures.

On the other hand, setting the quality threshold too high was shown to result in rejecting true fingerprint touches. This was especially tricky as not everyone's fingerprints were of the same quality; dialing up too high of a threshold made the sensor unusable for some people.

Ultimately, we learned that having a one-size-fits-all quality threshold was at times still problematic. To better address, we took inspiration from this Motorola patent, where additional sensors were leveraged to detect a stowed (aka in-pocket) state. When the sensor recorded two consecutive false matches, there was a check for stowed state, and if true would disable the fingerprint sensor. Once the phone detected it was no longer stowed, the fingerprint sensor was reenabled. This helped cut down on incidental touches as well on pocketed phones.

One half of the Poor Quality Touch report counted incidental touches, which was defined as the number of times the sensor was activated by a WOT but print quality did not meet threshold for match processing. By tuning the threshold, observing BigQuery total incidental touch data and cross-checking against user trial survey data, we were able to optimize the quality threshold, which lead to a good user experience.

Poor Quality - Finger Moved Too Fast

If you are a fan of early arcade games, you'll be familiar with the classics like Asteroids, Space Invaders, and Galaga. One thing in common with these games is that there is a joystick for movement and a fire button, the latter which would get tapped as fast as humanly possible most of the time, at least when I played.

A fun challenge of fingerprint sensor solution development is the variability and unpredictability of human interaction. Having worked with so many sensor vendors, I could take any solution on the market and achieve near perfect unlock performance. This was possible because I knew the workings of the technology and how to move robotically & methodically during enrollment and unlock to practically eliminate false rejects.

However, most users are not like me at all. During product development, there were some vocal users who complained of poor performance. I followed up with them to observe usage first hand. One thing in common I noticed was that they were tapping the sensor to unlock just like smacking the fire button on an 80's arcade game - as fast as possible. The WOT would activate the sensor for full imaging, but those fingers were already flying off the sensor before their fingerprint could ever be reliably imaged.

Some outcomes of this experience --

Whenever a user moved too fast, feedback on the phone's unlock screen was added to help guide users that their "finger moved too fast" with the hope they would settle their finger longer and more deliberately the next time. It triggered investigation into what optimizations and tuning could be made with the fingerprint sensor vendor to grab images more successfully in quick touches, without compromising matcher performance. Poor Quality Touch report was expanded to count the number of times these events occurred. Like the Unlock Activity report, this data helped characterize the type of user and their interaction style with the sensor.

Below screenshot is an example of the Poor Quality Touch report, which includes breakdowns of total incidental touches and total finger moved too fast counts.

Coming up next

In the next article, I'll go over the remaining BigQuery fingerprint sensor core reports, which cover the areas of:

Transactional Failures Finger Enrollments Unlocking Speed

Finally, I'll share details on the SQL queries, which were used to generate the report data.