For better or for worse, augmented reality ("AR") is charging forward in the consumer space—but there’s a place for AR in the industrial world as well. We’re not quite at the point of putting Microsoft HoloLens kits on the heads of roughnecks working out on oil rigs, but when it comes to complex machinery out in remote locations, augmenting what field engineers can see and do can have a tremendous impact on a company’s bottom line.

By way of example, GE is focusing efforts on constructing an extensible "field maintenance manual" intended to be used for industrial equipment. The use case being tested in the labs is with oil and gas; researchers in GE’s Research Center in Brazil are building software that they hope will replace the need to deal with bulky printed maintenance manuals—manuals which have to be kept up to date and which lack any kind of interactivity.

To learn a bit more about augmented reality in industry, Ars spoke with Dr. Camila Nunes, a scientist in charge of software and productivity analytics with GE Brazil. Nunes has an extensive background in oil and gas, having done graduate and postdoctoral work with Petrobras, the largest Brazilian energy corporation (and, indeed, the largest company in the Southern Hemisphere). At GE, Nunes works on bringing to life the interactive field maintenance manual concept.

Christmastime

To elaborate on how GE is using augmented reality in oil and gas, Nunes explained that frequently offshore oil and gas workers are called upon to install "Christmas trees," which are complex assemblies of pipes and valves used for a variety of purposes, including monitoring wells or injecting fluids into them. Owing to the wide range of functions, these are necessarily complex devices, and their installation can take more than a hundred hours under ideal conditions.

Currently, explained Nunes, the installation and servicing of a Christmas tree is done using paper manuals and checklists, and GE is aiming to change that by making things electronic and interactive. The interactive field maintenance manual concept is a multi-pronged beast, with a front end in the field and a back end that contains a tremendous variety of hardware and which can be continually updated as new equipment enters the field or new procedures are devised.

Nunes demonstrated this with a tablet in the augmented reality lab and a small 3D-printed duplicate of a piece of well hardware. The maintenance manual app used the tablet’s camera to figure out what kind of hardware it was looking at, and then was able to track the component as the tablet moved around it. The operator could look up installation procedures and see steps demonstrated in 3D on the parts each step involves, rather than having to refer to static printed diagrams.

More, the app enables an operator to take any part of the complex assembly being worked on and "explode" it, expanding it so that its interlinked component parts are visible. This kind of 3D exploded view is superior to a printed page because it can be zoomed and manipulated, and individual parts can be directly addressed in the app, rather than having to refer to additional printed pages.

Head-mounted versus hand-held

As things stand, the maintenance manual concept currently exists on tablets—iOS and Android. The back-end where the bulk of the data lives is based on Java Web Services, so when the tablet is connected to the Internet, it can pull up data on any piece of equipment in the entire library. However, in the field—particularly on sea-based drilling platforms—Internet access is never a sure thing. Because of this, the app can be preloaded with data on the pieces of equipment that the operator expects to have to work on. Then, it can function without a network connection.

Using commodity tablet hardware lets GE do things in software that would have been very difficult or impossible a decade ago; in particular, the GPUs of most consumer-grade tablets are good enough that weaving in OpenGL-based 3D renderings on top of the tablet’s live camera feed is trivial. There’s also enough CPU power available to perform image recognition tasks—in the lab, the 3D printed miniature parts had QR-like barcodes on them that the app could use to know what it was pointed at, but in the field, the manual features full image recognition and can identify equipment purely through the tablet’s camera feed.

RAM is an issue—at least for now. Nunes explained that it was relatively easy for the GE developers’ reach to exceed their grasp at first, as the tablets’ graphical and computational capabilities led the developers to try to do multiple things at once without regard for memory management. They’ve learned from their earlier efforts, though.

This is going to be a lot more important going forward, because GE wants to eventually transition the current tablet-based app into a head-mounted display, moving from tablet augmented reality to actual in-your-field-of-view augmented reality. Nunes couldn’t disclose which vendors GE was working with to make this happen, but she did say that nothing on the market today was quite good enough to make the concept work as well as it does in tablet form.

Ahead of the class

Moving away from paper and into an augmented reality maintenance manual has another tremendous benefit: if used properly, it can shorten the training cycle of equipment technicians.

The goal here would be to actually put the exact same kind of AR manual in technician training classrooms as in the field. Prospective technicians can learn to service equipment not just by reading books and working with mock-ups, but by actually walking through virtual procedures—the same procedures they’d use out at sea on a platform. Further, instructors can "share screens" with them and assist with troubleshooting—just as remote experts could help when out in the field (network connectivity permitting).

Augmented Reality in industry faces different sets of challenges than it does in the consumer world. Where we’re concerned primarily with consumer AR being intrusive and distracting to the everyday person doing everyday things, industrial AR is a lot like the military’s use of the technology: it’s acceptable to implement it with the expectation that the user will have some relevant training and job skills. Plus, as mentioned, AR can be incorporated into the actual job training, getting technicians used to using it throughout their entire job.

The AR maintenance manual concept looks like a win—and the applications are clearly there beyond oil and gas. Any kind of industrial setting with complex machinery that requires installation and maintenance checklists could benefit; GE hopes to be able to roll the technology out in the near term.