Google’s ARCore platform can create virtual objects, blending them with the real world through your device’s camera. Follow along as we explore some key ARCore concepts and delve into creating an augmented reality app from scratch.

With the release of ARCore, Google has provided us with a brand new platform for building augmented reality apps for Android, but how does it work and where do we get started?

ARCore makes use of three main technologies to achieve this:

Motion tracking: ARCore can detect and track visually distinct features in the real world to understand it’s position in space

Environmental understanding: This allows the platform to detect flat surfaces like the floor or tables

Light estimation: ARCore understands the lighting of the environment allowing you to adjust colours to give virtual objects an increased sense of realism

Augmented Reality vocabulary you need to know

There are five main concepts to understand before diving into the details.

Feature points

When you move around with your device, ARCore uses the camera to detect “visually distinct features” in each captured image. Those are called feature points. ARCore uses these points in combination with the device sensors to figure out your location in the space and estimate your pose.

Pose

ARCore uses pose to refer to the position and orientation of the camera. It needs to align the pose of the virtual camera with the pose of your device’s camera so that virtual objects are rendered from the correct perspective. This allows you to place a virtual object on top of a plane, circle around it and watch it from behind.

Plane

When processing the camera input stream, apart from feature points ARCore also looks for horizontal surfaces, like tables, desks or the floor. Those detected surfaces are called planes. We’ll see how you can use these planes to anchor virtual objects to the scene.

Anchor

For a virtual object to be placed it needs to be attached to an anchor. An anchor describes a fixed location and orientation in the real world. By attaching the virtual object to an anchor, we ensure ARCore tracks the object’s position and orientation correctly over time. An anchor is created as a result of a hit test when tapping on the screen.

Hit test

When the user taps on the device’s screen ARCore runs a hit test from that (x,y) coordinate. Imagine a light ray coming from the point your finger touched and going straight into the camera’s view. ARCore will return any planes or feature points intersected by this ray, plus the pose of that intersection. The result of a hit test is a paired collection of planes & poses that we can use to create anchors and attach virtual objects to the world.

Here's a screenshot of the sample app featuring all those concepts

What do you need to start developing an ARCore Android app?

Android Studio 3.0 or higher

A device with Android 7.0 or higher

Camera permission

ARCore feature requirement

You can develop ARCore Android apps the same way you develop normal Android apps. There’s no need for any extra tools. What changes is the way you render your app’s views, because we are going to be rendering virtual objects on top of our camera image stream. For that we need to use a SurfaceView and OpenGL

The sample

In this example we are going to showcase a simple app that does all of the following:

Check ARCore dependencies (ARCore SDK and Camera permissions) are satisfied before running an ARCore Session

Display detected Feature Points

Display detected Planes

Capture user’s taps to perform Hit Tests

Create Anchors

Attach virtual models to anchors

This flowchart provides a high level overview of what the app does:

What are the main classes you need to understand?

When you develop an app for ARCore it behaves pretty much like a game: there is a surface in which you draw updates every frame. That is what happens inside the green box in the above diagram.

The ARCore main entry point is a class called Session . A new Session is created when the Activity starts and it’s responsible for handling AR system state and session lifecycles.

This means every time draw frame is called you need to ask this Session object for the AR state: are there any feature points? Any planes?

The sample app draws a background, feature points, planes and 3D objects. This is the sequence of operations executed for each draw frame call:

Update ARCore data: in this block we’re grouping together several housekeeping operations we need to execute to ensure AR is running correctly: session.update() this provides you with a Frame , which you can use for... frame.getCamera() this provides you with Camera , which you can use for... Calculate projection and view matrices. These matrices would be used later on.

Draw the background: take the Frame from step 1 and use it to draw the background of your app. This is basically the image we are receiving from the camera. If we remove everything else and only keep these two steps what we will end up having is a surface view displaying the camera view.

from step 1 and use it to draw the background of your app. This is basically the image we are receiving from the camera. If we remove everything else and only keep these two steps what we will end up having is a surface view displaying the camera view. Draw tracked feature points: take the Frame and ask for the “current set of estimated 3d points attached to real-world geometry”. Then it draws this point cloud using the camera projection and view matrices as reference.

and ask for the “current set of estimated 3d points attached to real-world geometry”. Then it draws this point cloud using the camera projection and view matrices as reference. Draw planes: ask the Session for all the trackables with type equals to Plane and draw them (For now only Plane and Point are considered Trackable , but this makes us think more types will be added in the future).

for all the trackables with type equals to and draw them (For now only and are considered , but this makes us think more types will be added in the future). Draw models: this step can be divided in two: Check if the user has tapped the screen, a plane has been hit and create an anchor at the hit point. Frame has a method hitTest that receives an Android MotionEvent and returns the list of successful hits for that event. Then it’s just a matter of iterating through that list and creating an anchor if the hit is a Plane . Iterate through all of the existing anchors and draw a virtual model at the anchor’s coordinates.



Can you use an emulator?

Yes, the emulator will generate a virtual house interior scene where you can move around and simulate plane and points detection.

Can you test it?

As ARCore was released by Google very recently documentation in relation to automated test support is currently non-existent and material surrounding the topic across the web just as scarce. However, this doesn't mean we can't test our ARCore applications.

Unit testing forms the foundation of the testing pyramid and is traditionally where the majority of our automated test coverage/effort originates. When it comes to testing our ARCore based applications this should be no different. The code we're writing is still Java after all.

Moving up the testing pyramid we come to integration and user interface testing. Unfortunately, there’s isn’t currently a clear path towards driving user interaction with our ARCore application programmatically, however, purpose built strategies and tools to accomplish this should become more apparent and available as time passes and the technology is adopted on a wider scale. In the meantime, we can focus a larger proportion of our testing effort towards manual based exploratory testing.

Pro’s and con’s

ARCore technology used to detect planes, points and anchoring 3D models within a space is impressive. Using all of these features is easy and intuitive once you learn all of the vocabulary and related terminology.

The main pain point we found for Android developers was rendering 3D models. The basic sample project uses openGL which is not the nicest thing to work with. It operates at a very low level and you’re required to have a good understanding of matrices transformations and complex 3D space maths to work with it.

There are third party 3D rendering libraries available for Android that you can use instead of raw openGL. None of them is officially recommended by Google, so is up to you to choose. On top of that there’s no Android Studio integration for any of them, which doesn’t help in making things easier.

Something else to worry about is the fact that we didn’t find a strategy to automatically test ARCore apps.

We’d love to see how ARCore evolves in the future. We hold high expectations for this technology and hope to see easier ways to render models in the future as well as better Android Studio support or integrations.

Learn more!

If you want to learn more check the Google quick start for Android and this presentation from GDG Europe’17

Finally, if you’re curious on how to do AR on iOS check out https://blog.novoda.com/getting-started-with-arkit/ by our colleague Berta