Uncovering the Secret Tools in Google’s Oreo NDK

Using the hidden tools!

8/21/2017 wasn’t only a day for celestial bodies to be aligned. It was also the day that Google announced their latest Operating System to grace the mobile devices of the world. Google’s use of the eclipse to create extra fanfare around Oreo was not lost on the media.

Every few releases, something notable is changed that has direct positive impact on developers. Oreo was one such release. With Oreo a subtle change left nearly undocumented, yet drastically affects how Native Mobile developers should approach creating professional change management pipelines.

Memory sanitization, unit and integration testing, and profiling were the realm of the most adventurous developer. Oreo’s release delivers a shift where any native developer can now utilize a classical professional tooling environment.

Most notably, VR development is in constant pursuit of getting everything done in constant, smooth, 60 Frames Per Second (FPS). Anything slower than that frame rate is a trip down the vomit comet app trail of bad reviews. This is why game developers are so at home in VR development as classical game development is a high FPS optimization love affair.

Under the desire of faster FPS, certain techniques have been developed over the years that, while highly efficient, yield complexity. Complexity is the path of error-prone code.

Threading, memory management, networking, audio, graphics, and logic in general at this frame rate provides fertile ground for cryptic bugs that always seem to look right. The more complex an app gets, with the more team members, the more bugs creep in that are usually among the more difficult to solve.

With VR development’s highly complex code requirements, getting help from an automated tool is vital towards wrangling the bugs. In Oreo, if an APK is packaged with wrap.sh, the OS will execute that wrap.sh script when launching the app. That gives developers a chance to use their normal classical tools to launch the app and put it under a watchful eye.

The tools that all developers know and love, from memory sanitizing tools (ASan, Valgrind), thread sanitizing tools (TSan), unit testing tools (gtest, mockito), and even profiling tools (prof) can now stand at the ready in your developer’s toolbox to help diagnose problems at runtime.

Each of these robust, mature tools are normally used in other platforms. They were designed years ago, with the idea of launching an application, capturing that applications’ output, memory allocation, or system performance. It is also the model that debuggers frequently use; however, for wise security reasons, Android has restricted this behavior from end user devices.

The workflow of yesterday to utilize these tools, involved first creating a setup of each project with Valgrind rebuilt for each ABI under test. Then running the app via the adb command line disjoint from the Android Studio debugger. Finally, create some custom scripting which involves a dance of shipping C symbols, .oat files and android os root level access magic, all in order to achieve a simple app wrapping start script.

Getting Valgrind to work on Android was always one of the more time consuming tasks with getting any project configured. In this article, I’m going to detail how to use ASan, a tool that works similarly to Valgrind and helpfully is shipped with the NDK.

Prior to Oreo’s wrap.sh, the method for getting an app startup script involved being able to write to the /system partition. Unfortunately, that is a major security vulnerability so commercial devices strongly lock that access down, as well they should.

However, as a legitimate use case for having that access, I will break with what I normally tell users. The first step, was root the device, then unlock the bootloader, then install a eng/dev version of the OS (which usually doesn’t exist), and finally unlock the ability to write to /system. All of that prior to getting an ASan startup script and binary installed.

Google recognized this was a problem a few years back and started shipping the emulator with a eng/dev version of the OS. This is probably the best recommended approach for running bugs. Load up the app in a startup script running on the emulator which allows for eng/dev system level write access.

Now VR development isn’t normal development. We have our own problems that the emulator can’t solve. Oculus Mobile is a Samsung only thing and the SDK checks for a Samsung OS, thus it won’t even run on the emulator. In addition, the emulator doesn’t have an ability to “look around” which is a core requirement for VR. Thus we’re a device only requirement for most of our tooling.

How the world of our tooling changed when Google released Oreo’s ability to have wrap.sh! Praise be and pass the sanitizers! Unfortunately, despite this great news, not all is completely easy for VR developers even with Google’s latest Android incantation.

Google shipped with a bug in the first version of Oreo surrounding wrap.sh. This bug required us to have root capabilities in order to turn off selinux (temporarily). This bug has been addressed! The most recent version of Oreo 8.1 includes support without rooting! Ensure you have version 17 of the NDK (very latest) and Android Studio. Everything can be done from Android Studio and your individual app.

The rest of the way, I’ll demo how to use ASan in any project. First your app must be compiled with “-fsanitize=address -fno-omit-frame-pointer” flags. Next it must be linked with “-fsanitize=address” link flags. NB: these flags will cause the app to NOT work without ASan, so making a build flavor is highly recommended (as is only doing this for debug build types). As the app will only run if it can find an ASan runtime, we must package the APK with the ASan runtime library and use LD_PRELOAD to instruct the OS where to find the ASan runtime library.