The Journey Begins

These past few weeks have been quite a whirlwind — flying across the country on a last-minute trip, navigating a sea of documentation, and scaling a mountain of technical challenges.

The difficulties came from all sides: a new device, new tools, new workflows, unfamiliar operating system, and EXA’s technical requirements. My “getting started” challenges were compounded by various factors — travel, splitting work between computers, a few stubborn issues, a few tangents, and the need to also create a native C++ audio plugin (which is outside the scope of most Unity projects). Stumbling over one obstacle after another, my initial journey on this project hasn’t been pretty.

Not long ago, I faced a similar situation with my EXA multiplayer work. I was getting frustrated, and when my kids asked why, I told them I was getting tired of failing over and over again. They looked concerned, so I decided to change my tune. Sometimes we try to do difficult things, and make lots of mistakes, I told them. But every time we fail, we can learn a little more, and get a little closer to the goal. What’s important is to keep failing forward.

That seemed to resonate with them, and the phrase has stuck with me, too. It also happens to describe my initial journey with Magic Leap development quite well: learning fast and failing forward, all the way to my first milestone.

Bootcamp

The first phase of the Creator Program was to attend a “bootcamp” event hosted by Magic Leap. The event was optional, but it made sense to go — I would meet the program’s leaders, get to know the other teams, receive direct help from Magic Leap engineers, watch some presentations, and generally jump-start my project.

A miscommunication about bootcamp locations required me to make some last-minute travel arrangements, and I was soon on a flight from Michigan to California. Even though I’m not a big fan of traveling, and even though I booked a hotel room with a barely-usable shower, the trip was definitely worth it.

Bootcamp packed quite a bit into two-and-a-half days. We had time to work on our projects, with Magic Leap staff on-hand to answer questions about anything from compiler errors to interaction design. Each team had an extended technical discussion with Magic Leap engineers, tailored to our project’s specific needs. We filmed optional interviews to describe our projects and goals. We got to know the other teams, learned about their projects, received an overview of the Creator Program, and attended break-out sessions that seemed relevant to our projects.

Bootcamp introduced me to a few great contacts for technical issues, a handful of projects that might have some challenges similar to mine, and of course, dozens of amazing Creators. I also learned that the program has only two solo Creators, myself included, and that the timeline for my project seems to be shorter than most. No pressure.

Because bootcamp was hosted at SVVR (Silicon Valley Virtual Reality), there were some virtual reality stations available. I installed EXA on two of them, and found a few people willing to try it out. It’s so much more interesting to show people how the app looks and feels, rather than just describing it… and even better to explore it together in multiplayer. There happened to be some musicians at bootcamp, so we also had a couple impromptu EXA jam sessions together via the local network.

Bootcamp was largely a positive experience, but was also a time of mounting frustration and concern for me. As I dove into my project for the first time, I met an increasing array of issues. With each new obstacle I attacked, a few more would appear. Questions began to creep in. How long is this going to take? Is this even something I can solve? Working with so many new pieces, it wasn’t always clear whether issues were coming from my side, or theirs; it was tough to know whether I was even on the right track.

Setup

Before anything with my project could begin, I had to setup and configure my computers and the Magic Leap One device for development work. I decided to get this started before leaving for bootcamp, so I could focus on more important things while I was there.

Magic Leap’s documentation for getting everything setup was quite good. Still, there were quite a lot of steps, including installing a custom version of Unity, and several new pieces and tools and workflows to learn.

To further complicate things — which seems to happen quite a lot with my project — I needed to perform setup for both my PC development machine and my Mac laptop. This doubled my setup time, of course, and I had to be careful to follow some different instructions for each operating system.

I wouldn’t have bothered to setup on Mac, but I needed my laptop for the upcoming bootcamp. I prioritized the Mac setup for this reason, but also because I couldn’t get my PC to recognize the device.

For a week after returning from bootcamp, that PC issue took its toll on my efficiency. I’ve spent years refining my PC development environment, with tools and shortcuts how I want them, everything streamlined and familiar— not so much with my Mac. I finally got tired of feeling slow and clumsy, and set aside development to fix the PC problem.

I tried performing just about every variation of starting the device, plugging into USB, opening the “remote” tool, and searching for connected devices. I finally found my answer in two separate comments, mixed among several other suggestions in this discussion thread. I deleted some Windows registry keys, uninstalled a generic-looking USB device driver, and finally, there it was: a connected Magic Leap device.

Along the way I also learned about, experimented with, and incorrectly used things like debugging tools, command line interfaces, Magic Leap plugins for Visual Studio, and new Unity configurations. For each, I gradually figured out what I needed, what I didn’t, and how to use it properly for my project. This was all a bit overwhelming at first, until I started to see how all the pieces would fit together in my development workflow.

Finding A Milestone

Once my initial setup was complete, I took a long-shot for my first real build for the device. I made a full copy of my EXA project source, ripped out the support for virtual reality devices, disabled anything that wasn’t essential, dropped in the Magic Leap assets, and hit the build button.

It compiled, but it definitely didn’t work.

I was still very new to everything Magic Leap at this point, and didn’t even know how to obtain the app’s debugging logs. All I knew was: when I launched my build on the device, nothing happened at all.

That’s a tough place to start, so I went in the opposite direction. Within an hour, I had created a simple build with physics-enabled cubes floating around me, all waiting to be bopped around with the controller. The documentation and assets from Magic Leap were all clear, and making this scene was very straight-forward. I spent a few minutes wishing EXA could somehow be that simple, enjoyed a few minutes playing with my augmented reality cubes, then decided to get on with it.

My first all-or-nothing approach was impractical, so I settled on a middle ground: I’d get a “watch-only” EXA performance to play on the device. To do this, I could use a tool that I had already built, called “EXA Remix”, which allows third-party Unity apps to load, customize, and play performances that were originally created using EXA.

This “watch-only” concept worked well as my first major milestone—it incorporated the app’s basic setup, many of its graphics and effects, and the loading and parsing of files (both JSON and audio data). Importantly, it would also require me to address my biggest known technical challenge: modifying EXA’s native audio system to work on Magic Leap’s Android-based operating system.

Logging

My first step toward this milestone was to figure out how to access my app’s debugging logs — development work would be hopeless without those.

A search for “debug logs” in the documentation took me on a long detour with a debugging tool called DERP. This required installing the tool, adding and configuring some related assets into my Unity project, and connecting my build to the tool over local network. I got it working, only to realize there’s an easier way. DERP has some benefits, like sending commands into the running build, but I’d rather not have spent the time on it.

The easier way to access logs is to use the MLDB command line tool. Here are some commands [in brackets] that might be helpful:

[mldb log] prints logging output directly into the terminal. Use keyboard CTRL+C to exit the printing mode and return to your terminal prompt.

prints logging output directly into the terminal. Use keyboard to exit the printing mode and return to your terminal prompt. [mldb log -c] clears the logs.

clears the logs. [mldb log -c && cls && mldb log] clears the logs, clears the terminal, then starts printing again. The “&&” conditionally chains commands together on a PC terminal.

clears the logs, clears the terminal, then starts printing again. The “&&” conditionally chains commands together on a PC terminal. [mldb log > C:/path/to/log.txt] pipes the logging output to a file. This one’s my favorite, because it allows me to use an app like BareTail to search, filter, and color-code logs in real-time. Highly recommended.

pipes the logging output to a file. This one’s my favorite, because it allows me to use an app like BareTail to search, filter, and color-code logs in real-time. Highly recommended. [mldb terminate com.companyname.appname] quits the specified app, via its package ID, so you don’t have to do it within the device. This is nice for debugging, especially when there’s runaway audio playback that needs to stop. Immediately.

quits the specified app, via its package ID, so you don’t have to do it within the device. This is nice for debugging, especially when there’s runaway audio playback that needs to stop. Immediately. [mldb] provides a full listing of its sub-commands and options.

These commands are slightly different on Mac — the [&&] becomes [;], and the [cls] becomes [clear]. The keyboard CTRL+C stays the same, however; it does not become Command+C, as I kept trying at first.

Loading Files

With logging ready to go, I started working toward my first milestone, and immediately encountered an unexpected obstacle — I couldn’t get files to load on the device. A music app that can’t load its instruments or sounds is not very useful, so this issue became a blocker during my time at the bootcamp, and a few days after.

EXA uses a standard approach for loading files: it places the files (including pre-installed performances, instruments, and sounds) into the Unity “StreamingAssets” folder, and uses typical .NET methods for accessing them (such as FileStream and StreamReader). However, when trying to load these files on the device, I received an UnauthorizedAccessException error. Essentially, this means that the app does not have permission to load the requested file or directory.

Given that this error hasn’t occurred for EXA before, and that the files were located within EXA’s own data directories, I thought it may be an issue on Magic Leap’s side. At bootcamp, I asked the technical staff about it — we looked at some code together, tried setting various permissions, and even found a not-quite-what-I-needed workaround. The staff were all very helpful, but we couldn’t solve the issue.

After returning from bootcamp, I continued searching for clues that might help. Eventually, I realized that opening a FileStream with a FileMode.Open parameter will request both “read” and “write” access for the file by default. Providing an optional FileAccess.Read parameter, however, will request “read” access only. After everything else I’d tried, this was all I needed to do to resolve the file-loading errors.

Once I could access files, I also spent some time failing forward with errors related to JSON parsing. When running on the device, my app couldn’t seem to locate the simple classes that represent the JSON data structure. I learned that, because building for the Magic Leap One uses IL2CPP, the builds exclude methods and constructors that are only accessed via reflection. So, to fix the parsing errors, I needed to actually use the data class constructors in the code, as well as constructors for a few flavors of List<T>.

Native Audio

EXA’s native audio system consumed more time than all the rest, but I’m exhausted just thinking about it, so I’ll keep it short: hours and hours of tweaking audio buffer sizes and sampling rates and float-to-byte-to-int16 conversions and buffer callbacks and compilation options — all combined in various ways until each new piece would work.

After listening to more choppy, garbled, or (worst of all) silent audio than I care to remember, I got it all working. The big picture: I used the Magic Leap C-based Audio API, integrated it within PortAudio as a new audio host, and compiled it all via MABU script (similar to a “make” file) for Magic Leap’s Android-based operating system.

Rather than describing my development slog, here are some of the key things I learned along the way: