At my day job I’m an iOS developer. I was enthralled by AR when Apple released ARKit last summer, and immediately began learning everything I could on it. My AR journey brought me to learning 3D art with Blender, and game development with Unity. I used these skills to begin working on Escape The Room — AR, my AR experience coming soon to iOS. It took me some time to figure out what I needed to learn, and I could have used a little direction in learning AR/VR development. The goal of this article is to give you a good idea of some major concepts that will help accelerate your AR/VR development!

Modeling

Not the modeling that involves runways and chique clothes, this modeling means 3D modeling. 3D modeling is the base of the objects that you see in 3D. That chest in a pirate game? That was modeled. That gun in a shooting game? That was modeled. Most physical objects in games are 3D models. The structure of 3D models is determined by a “mesh”. You’ve probably seen a photo like this before

Thats a 3D “mesh”. A mesh consists of vertices, edges, and faces. By manipulating these vertices, edges, and faces, 3D artists are able to form anything they want! All those characters in your favorite Pixar movie? Made with meshes!

Starting off in AR development, you might not want to jump into making your own models right away, and truthfully you won’t have to! There are tons of 3D models you can get online for free; at TurboSquid, or if you’re using Unity, the Asset Store.

Key Ideas of Models

Most of the time your users in AR or VR will be interacting with models. Whether they’re pressing a button, inspecting an object, or viewing the scenery, those are all models. Models don’t only consist of meshes, the data that defines the structure of the model. Models will often be paired with “materials”. I’ll go into materials in the next section. But first:

How To Make Your Own Models

Models are made using 3D modeling software. Blender, Maya, and Cinema4D are some of the most popular.

If you start to enjoy AR or VR development you might be inclined to learn how to make your own models if you want to make something original, so you’re not hamstrung by the assets that are provided online. For this I would recommend downloading and learning Blender. Blender is a free open source 3D software package. It has an awesome community and a huge amount of tutorials to kick start your learning. I personally believe Blender will continue to grow and become a staple of 3D development as AR and VR start to proliferate and reach mass market penetration. Other 3D creation platforms are extremely expensive. Maya, one of the game industry standards, is a few thousand dollars for a yearly license… I’d rather stick with the awesome and free Blender community.

Now that we have a mesh, the next step is to make our object look good! This requires materials.

Materials

A mesh without a model looks naked.

A screenshot of a model I made in Blender. So bare!

How to make this look better? We need materials! Materials allow objects to be rendered in ways that lets the lights, color, and reflection of the object look how we want. If you have a mesh of chair, using materials you can make that chair look like its made out of wood, or made out of metal. Materials put a polished touch on your AR experience.

How To Make Materials

Making materials depends on how you’re developing your app. Unity’s material system is different than Unreal Engine’s. And Blender materials are developed differently than Unity. If developing in Xcode with Swift, materials are different from all of the above.

I could write a whole article on materials, so I’ll try to keep this short. For this article I’ll discuss PBR Materials. PBR stands for Physically Based Rendering. PBR Materials are the sum of adding different images together that will render an object in the way we want. Each image has one specific aspect of the material that its trying to change. These images are often called “maps”. These maps or images are often referred to as textures. It’s a lot of jargon but you’ll get used to it!

The “Albedo” map of a material changes the color of an object. Want an object to look like wood? Add an albedo that looks like wood and your material will look like this.

Ad an albedo that looks like metal and your material will look like this

The “Normal” map of a material determines if there are any bumps in a material.

Then there are many other maps for Specular(how shiny), Reflection (how much light an object reflects), and some more complex variables you won’t need right away. You can find free materials in the same places you can find free models. If you want to make your own materials, all you need are images that look the way you want an object to be rendered. You can sign up for subscription services that give you access to a huge amount of super high quality PBR images, so you can make awesome materials. I use Megascans and love it.

The result of adding materials to an object.

Above is the finished product of the model I posted earlier. I used a normal map and roughness map to get the scratches on the object. I changed the color of the object using the albedo map, and I made the alien symbols glow using the emission property on a material. Materials are an art, and super fun, so just play around with them until you make something cool!

3D Space

This section will be short, but it’s important. Developing in AR/VR is different than web or mobile development because there is an extra dimension that users get to interact with. In AR, users move through 3D space, and the origin of that 3D space is often determined as where a user starts a session of the app.

3D Development will make you best friends with this little guy

On a 2D mobile app you know where a user can go. You built that interface for them. With AR, they could be anywhere and go anywhere. Coordinate space differs between engines, some use (X,Y,Z), some use (X,Z,Y). For this example we’ll use (X,Y,Z) X is forward, Y is up, Z is to the right. If a user starts at coordinates of (0, 0, 0) the might immediately turn around, so the object that you were planning on putting right in front of them, lets say (5, 0, 0), might not be where they are looking.

You’ll need to work with the Camera property of whatever engine you’re working in. In Unity, it’s easy to grab the Camera property of where a user is looking, and place objects based on it’s rotation and position. For AR you need to be flexible and write code in ways that takes into account a multitude of environments that a user could be in. This is more a discussion of design vs. development, but it can save you lots of wasted time.

Originally I was going to set up the scene of Escape The Room by having a user scan their floor, and the objects would be placed automatically on the horizontal plane that was detected. What I found was this was WAY harder than expected. Users could be facing different directions, plane sizes would shift, objects would be to close to walls and become inaccessible. What I ended up doing is having users place objects directly onto the floor around them, so you know things are in a solid position before things kick off.

VR is a bit different, because the objects are already placed in the scene, but its good to think about how you, or any user, would move through 3D space for the first time, and what they would experience.

Remember, we are in the early days of XR, so its key to cleanly introduce users to how to move through 3D space. Users will do things in your world you never expected, so build things to be flexible!

Lighting

Lighting is like good manners at the dinner table. You never notice it when it’s good, but you certainly notice it when it’s bad. AR and VR diverge here a bit, because in VR you determine the lighting, and in AR you are at the mercy of the user having lighting. It’s a key design challenge for AR, so you should tackle it head on to give a great experience!

The first thing you can do when developing in AR is tell a user if they have bad lighting! You can receive updates from ARKit if the camera is having a hard time detecting the environment, and you should tell the user about this! They don’t know they’re doing anything wrong if you don’t tell them, and it will lead to a better overall experience if you can get them to flip an extra light on or move rooms.

Lighting is key to detecting planes in AR, which will help with the start of an experience, but lighting also makes an experience feel real. I talk about this briefly in my article on making metal look good in AR, and I’ll pull some key points from there.

This is an object with bad lighting in AR:

It looks fake and empty and out of place. Work with the lighting a bit and you can get your object to look much better:

Ahh much more real!

You can add your own lights to a scene that will help light your objects up, but be careful with this! Adding lights can be taxing on performance, especially on mobile. Reflection Probes in Unity can help make your metal objects look real, as discussed in the above article. There have been some developments in making your objects feel like they’re in the room with you using lighting and reflections and I plan on diving into that soon.

That’s all for this article, I hope you learned something! Reach out with any comments and follow on Twitter and Instagram to stay in touch and get updates on Escape The Room — AR. Also join the Discord server to chat about everything AR/VR! Happy developing!