Now we will get familiar with features and functionalities of AR Core

Get ready!! Now we will focus on the state of AR. Below are the few questions which might be hitting you mind at this point of time.

Who’s behind it?? how effective is the technology?? what hurdles need to be overcome??

Don’t worry at all. Hopefully contents below will overcome all your curiosity regarding AR Core.

How AR Core tracks?

Let’s now talk about tracking. AR relies on computer vision to see the world and recognize the objects in it. The first step in the computer vision process is getting the visual information, the environment around the hardware to the brain inside the device. The process of scanning, recognizing, segmenting, and analyzing environmental information is called tracking, in immersive technologies.

For AR, there’s two ways tracking happens, inside-out tracking & outside-in tracking.

Outside-in tracking

In case of Outside-in Tracking, cameras or sensors aren’t housed within the AR device itself, instead they are mounted elsewhere in the space. Typically, they are mounted on walls or on stands to have an unobstructed view of the AR device. They then feed information to the AR device directly or through a computer. The external cameras or sensors can be as large as you want, at least theoretically. Outside-in Tracking overcomes some of the space and power issues that can occur with AR devices. Suppose while tracking your headset loses connection to the outside sensors for even a moment, then they can lose tracking. Due to that the visuals will suffer breaking immersion.

2. Inside-out tracking

In case of Inside-out tracking, cameras and sensors are built right into the body of the device. Smartphones are the most obvious example of this type of tracking. They have cameras for seeing and processors for thinking in one wireless battery-powered portable device.

On the AR headset side Microsoft’s HoloLens is another device that uses inside-out tracking in AR. The HoloLens frames include five cameras for analyzing the surrounding environment, one camera for measuring depth, one HD video camera, one light sensor and four microphones but all that hardware takes up space, power, and generates heat. The true power of standalone AR devices will emerge when they become as ubiquitous and as useful as smartphones. In the meantime, smartphone-based AR will be the primary method for most of the world to engage with AR content.

How motion tracking happens in AR Core?

Whether it’s happening on a smartphone or inside a standalone headset, every AR app is intended to show convincing virtual objects. One of the most important things that systems like AR Core do is motion tracking. AR platforms need to know when you move. The general technology behind this is called Simultaneous Localization and Mapping or SLAM. This is the process by which technologies like robots and smartphones analyze, understand, and orient themselves to the physical world.

SLAM processes require data collecting hardware like cameras, depth sensors, light sensors, gyroscopes, and accelerometers. AR Core uses all of these to create an understanding of your environment and uses that information to correctly render augmented experiences by detecting planes and feature points to set appropriate anchors.

AR Core uses all of these to create an understanding of your environment and uses that information to correctly render augmented experiences by detecting planes and feature points to set appropriate anchors. In particular, AR Core uses a process called Concurrent Odometry and Mapping or COM. That might sound complex, but basically, COM tells a smartphone where it’s located in space in relationship to the world around it. It does this by capturing visually distinct features in your environment. These are called feature points . Feature points can be the edge of a chair, a light switch on a wall, the corner of a rug, or anything else that is likely to stay visible and consistently placed in your environment. Any high-contrast visual conserve as a feature point. This means that vases, plates, cups, wood textures, wallpaper design, statues, and other common elements could all work as potential feature points.

That might sound complex, but basically, It does this by capturing visually distinct features in your environment. These are called . Any high-contrast visual conserve as a feature point. This means that vases, plates, cups, wood textures, wallpaper design, statues, and other common elements could all work as potential feature points. AR Core combined, it’s new awareness of feature points with the inertial data, all the information about your movement, from your smartphone. Many smartphones in existence today have gyroscopes for measuring the phones angle and accelerometers for measuring the phones speed. Together, feature points in inertial data work together to help AR Core determine your phones pose.

Many smartphones in existence today have gyroscopes for measuring the phones angle and accelerometers for measuring the phones speed. Pose means any object’s position and orientation to the world around it. Now that AR Core knows the pose of your phone, it knows where it needs to place the digital assets to seem logical in your environment. Remember, virtual objects need to have a place and be at the right scale as you walk around them.

For example, the lion needs to have its feet on the ground to create the illusion that it is standing there, rather than floating in space.

How AR Core understands Environment?

Environmental understanding, is AR Core’s process for seeing, processing and using information about the physical world around an AR device.

The process begins with feature points. The same feature points used for motion tracking. ARCore uses your phone’s camera to capture clusters of feature points along a surface. To create what’s known as a plane.

Plane finding is the term for ARCore’s ability to detect and generate flat surfaces. AR Core’s awareness of those planes, is what allows it to properly place and adjust 3D assets in physical space, such as on the floor or on a table otherwise objects would just float. This process enables you to do things like, see how a plant would look on your desk or place a virtual human in front of you for a conversation. Once you know where the planes in the world are, you can hit test or re-cast to see what plane the user is tapping on. This allows you to place objects on top of the floor or on your desk making them follow the same rules of physics as real solid objects.

Gyroscopes and accelerometers combined with your smartphone’s camera and the ARCore’s unique software, all add up to the discovery and detection of planes. This ability is unique to smartphone powered AR, as it requires the use of all those internal components that are already built into the system. This is how ARCore addresses the issue of context awareness.

So, what is Context awareness?

The most important component of realism, Context Awareness, is the most difficult to achieve. As we discussed earlier, AR hardware has to be aware of essentially every single object in its environment. It needs to understand that there is a desk, a chair, and a table next to a bookcase, a vase, and a television. It needs to know which of these items is taller, shorter, fatter or wider than the others, and how this changes when the subject moves around in space. That’s a lot to track. Generating this awareness quickly and without a drop in the digital objects fidelity, smoothness or functionality is one of the biggest challenges facing AR creators today. Companies like Google are investing in software tools like AR Core to help address some of these issues.

Multi-plane detection and spatial mapping

ARCore is able to take multiple surface areas such as the table, sofa, and the floor all at the same time, if desired assets can be placed on any of the surfaces and each has the same anchoring and posing capabilities to keep the objects behaving realistically. Notice how the hit test against different planes gives accurate 3D poses for the assets that had the graphic system to render them at different sizes and depth from the camera’s perspective.

For example , you can see that the object farthest away look smaller than the closer ones. Since AR Core is constantly learning from the environment, the longer you’re using your phone to spatially map the environment, the better the pose is understood.

, you can see that the object farthest away look smaller than the closer ones. Since AR Core is constantly learning from the environment, the longer you’re using your phone to spatially map the environment, the better the pose is understood. For example , placing a hand in the scene close to the camera may cause AR Core to map the planes to your hand. This will cause issues as soon as you move your hand because AR Core assumes planes are not moving. Digital objects are typically projected to be a few feet away from you, so putting your hand in front of them will just highlight the lack of occlusion and might confuse the system. In general, it’s best not to place an object until the room has been sufficiently mapped and statics of faces have been defined.

, placing a hand in the scene close to the camera may cause AR Core to map the planes to your hand. This will cause issues as soon as you move your hand because AR Core assumes planes are not moving. Digital objects are typically projected to be a few feet away from you, so putting your hand in front of them will just highlight the lack of occlusion and might confuse the system. In general, it’s best not to place an object until the room has been sufficiently mapped and statics of faces have been defined. How simple surfaces challenge AR?

AR is constantly trying to find interesting and distinct feature points to see, track, and remember for orienting the device and digital assets in augmented experience. This means that the software will have more trouble with something plane like a white wooden coffee table than it would with a knotty wooden table with a coffee mug on it. Distinct texture is important for providing the contrast needed to create feature points. To put it simply, the more differentiation there is on surfaces and a given space, the better AR apps will function.

How to place and position assets?

There are a few basic rules that augmented reality developers need to remember about the way objects behave in AR. These behaviors are the key to merging the real and digital worlds seamlessly. The first of these behaviors is placing of assets.

Stationary AR objects need to stick to one point in a given environment. This can be something concrete such as a wall, floor, or ceiling, or it could be suspended somewhere in mid air. Whatever the case, placing means these objects stay where they’re positioned. Even when users are in motion. The mug on your coffee table doesn’t jump around when you move your head. If you look away, it’s right there when you look back again. For AR to maintain the illusion of reality, digital objects need to behave in the same way real ones do.

Solid augmented assets

AR objects need to appear solid. This may sound obvious, but it takes conscious effort to achieve. As you engage with and ultimately create AR content, keep this in mind that AR objects should never overlap with real-world objects nor should they appear to be floating in thin air if they’re not something like an airplane or balloon. If either of these missteps is in your app, it will break immersion for your users.

User interaction: hit-testing and pose

Now let’s talk about hit-testing. Hit-testing lets you establish a pose when watching objects and is the next step in AR Core user process after feature tracking and clean finding. Hit-testing works by drawing a line out from the phone and moving outward in that direction until it hits the plane. When it establishes this connection it then allows AR Core to establish position and orientation or both for digital objects.

Why scaling and sizing is important for virtual assets?

In addition to sticking wherever they are placed in the real world, AR objects need to be able to scale.

Let’s think about it like this. When a car is coming toward you from a distance, it starts out small and gets bigger. A painting viewed from the side looks very different when you walk around and face it head on. Our physical distance from a given object and our orientation around it changes how they appear to us. A well-constructed AR experience will incorporate objects that are not only appropriately placed, but will look different if you stand right next to it, below it, above it, or view it from afar. So, this is called scaling.

Together, placing and scaling are what take AR objects from digital novelties, to assets that could potentially replace real world counterparts.

How important is environmental conditions(lightning) for AR Core?

Light estimation

Have you ever noticed how your phone screen automatically dims or brightens depending on where you are standing. It happens because many smartphones have a light sensor. Light sensors allow for features on a phone like brightness management and automatic screen lock when the phone rises to your ear. Current AR technology only allows you to make a global estimate of the lighting, such as brightness, color and temperature.

The way AR Core uses light estimation is by scanning the camera images pixels to determine an average of incoming light which helps to decide how to provide the perfect lightning for an AR object inside of a specific environment. Light and shadows are a big part which helps your eyes to visualize that an object is real. If you’ve ever been able to tell that actors in a film are standing in front of a green screen rather than in a depicted environment, that’s because the lighting is incorrectly matched. Wide estimation is yet another way AR core allows users to create more believable AR apps, games and experiences.

Lighting for increased realism

Just like a real world object, objects in AR need to respond to different patterns of lighting to make sense in our minds. The colors, shading, and shadows cast by these objects all need to behave properly both in the initial lighting of a scene and in the case of a lighting change.

For example, if you dim the lights during an AR experience, then the AR objects should change in color and shading appearance. Similarly, if you move an object around, the shadows needs to move accordingly just like they would in real life.

How low-light conditions limit AR?

AR is still a new and emerging technology, particularly for smartphones. There’s a lot that the AR Core platform can do, but let’s take a moment to learn about some of its current limitations.

So just like your own eyes, AR needs light to see. In order to find those feature points and planes, your phone’s camera and other sensors need to be able to grab a good picture of what’s actually located in the world around you. This means that in dim or dark environments, most Augmented Reality devices will struggle to properly understand your environment, and therefore be unable to properly render the experience.

Low light conditions are a problem for every AR tracking system that exists today. The key to these technologies is their ability to orient themselves by seeing and understanding the real world. To do that the real world needs to be well lit and visible. This will require advancements in camera technologies, and computer vision.

AR’s technical constraints: size, power, heat

The technical details of what makes AR challenging from a technical standpoint are complex but they can be boiled down to three simple words; size, power, and heat. We’ve come a long way when it comes to miniaturized processors and graphics cards, but we’re still not quite at the level we need to be to make high-end everyday AR, a well reality.

Rendering an AR experience takes a lot of power. Just think about how much your cell phone battery drains when you’re streaming video. Now imagine if it wasn’t merely streaming images but generating those images and doing so while also tracking every other object in a room and re-calibrating the image every time your head changes it’s position or rotates.

Again, another solution is to have an external battery pack that clips to your belt. That might be working for now, but ultimately, the power problem is one we’re going to have to overcome within the frame of the device if we want AR to reach its full potential. You’ve probably noticed that every PC or laptop you’ve ever owned as a fan inside. Computing generates heat, a lot of it. In fact, the more power used, the more heat that gets generated, and the smaller the device, the slower it gets rid of that heat.

AR is a highly complex process, and therefore generates a lot of heat and that heat in turn can slow down processors or even short them out altogether. Managing this heat is even harder with the limited size and structural requirements of an AR headset. There isn’t much room for heat sinks and fans in a device that’s supposed to eventually light-weighted and look like an ordinary pair of glasses. Some AR headset manufacturers are packing all of the processing power into the frames of the visor itself. Meanwhile, other manufacturers are using external hardware like battery packs to address the issue.

Computer vision limitations

The final stumbling block is Computer Vision which AR needs to overcome on its path to maturity and it is also the most challenging part to be solved.

Computer Vision is the term for the hardware, software, and processes that allow computers to see and understand the physical world. For example, you might be able to search Google for dogs and find dogs, but that’s because Google’s unique search algorithms and tools have categorizes images as dogs.

For example, you might be able to search Google for dogs and find dogs, but that’s because Google’s unique search algorithms and tools have categorizes images as dogs. However, Computer Vision processes would actually allow search engines to see the pictures they are searching and recognize a dog on their own but that is a simple example with huge implications. Computers that can see, can recognize pedestrians and stop signs to make sure autonomous cars work the way they’re supposed to and they can look at the world around you to place digital assets where they would naturally belong. This is an amazing concept but it is also very difficult to achieve. Currently, Computer Vision is a fast growing but limited technology. Giving computers the ability to recognize the full catalog of earthly objects at any time of day and segment them into useful groups, just isn’t something we’ve completely pulled off yet.

Occlusion

Occlusion refers to what happens when an image or object is blocked by another. Let me explain you this, move your hand in front of your face, so you have just occluded the computer screen with your hand. However, imagine if you moved your hand in front of your face and the screen was still visible, you’d probably get a little concerned.

AR objects have to play by the rules of occlusion if we want them to seem real. This means that AR hardware has to not only understand where the object is in the room but also its relative distance from the user compared with any other objects physical or digital.

Occlusion means hiding virtual objects behind other virtual objects, and ones in the real world.

Say you move behind a wall while using AR and if you still see the AR objects, it will break your sense of immersion. Seamless occlusion requires constant re-calibration as users can move in any direction at any given moment, that is why it’s one of the trickiest aspects of building successful AR content.

Occlusion between virtual assets

Let’s think what happens when you place an object into the scene. The asset placed behind the first one is currently blocked from you. Remember, Occlusion occurs whenever one solid object moves in front of another. Creating occlusion is essential for establishing AR realism. Even as you move your phone around the environment, the assets will continue to behave the way we would expect them to, if they were real. However, for now this occlusion is only possible between digital objects. The real-world object would not block the watching object. A good work around for this current limitation would be to design experiences that real world objects are not expected to occlude your watching ones.

Constraints of occlusion and shading

Well, another thing to remember is that currently, AR Core cannot occlude digital objects when real-world ones block them from view. This means that even if your character should technically appear behind a desk, they’ll instead float in front of it. The dream of AR is for this type of occlusion to occur naturally but in the meantime, it’s important to know the limitation in order to find creative workarounds. Speaking of creative workarounds, another feature AR Core does not currently support is shadows. The good news is they are supported in 3D game engines like Unity, which allows mobile AR developers to create more realistic content for AR Core.