Back in 2012, during the early days of KeepSafe, we sought to implement an encryption scheme for our Android App. Through many iterations and prototypes, we found a sweet spot of sorts by leveraging the power of the JNI (Java Native Interface.) We decided to write our interface into the encryption library we utilized in Java, calling into the library via the JNI solely for the purpose of encryption and decryption. We opted for an on-the-fly solution, minimizing the impact on user experience as much as possible. Once we were happy with our solution, we decided to deploy it into our production app. We rigorously tested our code and were confident that everything would go smoothly; that is, until things beyond our control broke.

Enter the Dreaded “UnsatisfiedLinkError”

As we anxiously refreshed our crash reports following our release, we started to notice a recurring error. Users were running into an “UnsatisfiedLinkError”, which means that either A) the native library we were calling into did not exist or B) the native method we were calling did not exist. Since B) would almost always be caught via compiling and basic testing, we were immediately perplexed at the fact that users’ installations did not have the native libraries we shipped within the APK.

Here are some samples of the exceptions we ran into, they ranged from standard…

java.lang.UnsatisfiedLinkError: Couldn’t load stlport_shared from loader dalvik.system.PathClassLoader[dexPath=/data/app/com.kii.safe-1.apk,libraryPath=/data/app-lib/com.kii.safe-1]: findLibrary returned null

at java.lang.Runtime.loadLibrary(Runtime.java:365)

at java.lang.System.loadLibrary(System.java:535)

at com.kii.safe.Native.<clinit>(Native.java:16)

… 63 more Caused by: java.lang.UnsatisfiedLinkError: Library stlport_shared not found

at java.lang.Runtime.loadLibrary(Runtime.java:461)

at java.lang.System.loadLibrary(System.java:557)

at com.kii.safe.Native.<clinit>(Native.java:16)

… 5 more Caused by: java.lang.UnsatisfiedLinkError: Cannot load library: get_lib_extents[760]: 1305 — /mnt/asec/com.kii.safe-1/lib/libstlport_shared.so is not a valid ELF object

at java.lang.Runtime.loadLibrary(Runtime.java:434)

at java.lang.System.loadLibrary(System.java:554)

at com.kii.safe.Native.<clinit>(Native.java:15) Caused by: java.lang.UnsatisfiedLinkError: Library cryptopp not found

at java.lang.Runtime.loadLibrary(Runtime.java:461)

at java.lang.System.loadLibrary(System.java:557)

at com.kii.safe.Native.<clinit>(Native.java:17)

…to the rather obscene…

Caused by: java.lang.UnsatisfiedLinkError: Cannot load library: reloc_library[1286]: 1748 cannot locate ‘쯰ҷЦf1Ϙ˗˞ք᣼0Ⱉض夘Ϛ.͏闑㥁ج뭫ර⓻в^ӎ3c`+W#Ҽ?-Bַˌ֕꼠’…

at java.lang.Runtime.loadLibrary(Runtime.java:370)

at java.lang.System.loadLibrary(System.java:535)

at com.kii.safe.Native.<clinit>(Native.java:17) Caused by: java.lang.UnsatisfiedLinkError: Cannot load library: reloc_library[1312]: 1327 cannot locate ‘Pܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭXߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭXߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#׶ʭX Ϛߐܝ#

at java.lang.Runtime.loadLibrary(Runtime.java:434)

at java.lang.System.loadLibrary(System.java:554)

at com.kii.safe.Native.<clinit>(Native.java:17)

There was no clear pattern as to what library was at fault as it seemed all libraries were throwing the exception. It was not specific to an Android version and it did not only occur on specific devices. Moreover, in certain instances some of the native libraries were loaded properly, but not all. At this point, we were furiously searching the internet for answers or help, only to come up empty-handed. We started to release hotfixes, mostly filled with speculative fixes, additional tracking data so we had a better understanding of what exactly the environment was when the crash occured, and a piece of code that specifically checked for the native libraries in their expected install location.

Moreover, in certain instances some of the native libraries were loaded properly, but not all.

We gathered that, yes, the native libraries were not there, it was not some weird fluke with the System class or file system error, and that user’s devices were seemingly normal.

Idea #1 — User Devices are Running out of Space

As we thought through the possible causes for this exception, we started to think that maybe people just had no room for the native library to be installed correctly, and thus it never was installed. A quick diagnostic check before attempting to load the libraries proved this idea wrong very quickly. Users had more than enough space on their device for the libraries we were shipping.

Idea #2 — Native Libraries were not being included in Updates

The second plausible cause for our problem was that Google Play was mangling our APK when serving it to users’ devices. We had some reinforcement for this idea after reading such reports as this one, detailing that Google Play supposedly sent out a notice to all app developers affected by an issue preventing users from launching their apps after an update due to native libraries being incorrectly installed. The only problem was that this report was in August, and we were dealing with the problem months later. We also never received a notice from Google Play mentioning any sort of error on their part. Additionally, this is something that would be very hard to verify.

Idea #3 — Debug the issue directly with a real user

Since we were unable to reproduce the issue on any of the 10+ devices of ranging variety we had on hand, we decided that we would reach out to a user who was experiencing the problem. One user graciously decided to help us and noted that the app was working fine prior to the latest update. The problem here was that the app version the user noted as working was a version that included our encryption code and native libraries, just to add to our confusion and the overarching puzzle. We decided to supply the user with an APK file directly, where we verified that all native libraries were present in the APK file. The user installed the APK, launched the app, and ran directly into the same UnsatisfiedLinkError exception again. This confirmed that Google Play was not the issue, the issue was with Android’s PackageManager installation process. At some point during the installation, something would go wrong and the native libraries inside of the APK would not be extracted.

[…] the issue was with Android’s PackageManager installation process.

Getting to a Solution

Since we figured out that the problem was with the installation process, we decided that we would replicate the part of the installation process that extracted the native libraries inside of our App’s code. Luckily you can get a reference to your App’s APK file on the device very easily via

Context.getApplicationInfo().sourceDir;

which we would then use to extract the native libraries to an internal storage location. Since APK files are just ZIP files, this was a simple matter of writing some ZIP extraction code. We quickly implemented the extraction code and shipped it, resulting in a major drop in the number of crashes!