Microsoft’s HoloLens and the use of reality meshes have got round tracking issues, paving the way for exciting new AR applications from Bentley, writes Greg Corke
Bentley Fellow Stéphane Côté lives and breathes Augmented Reality (AR). For years, he’s been a driving force behind AR in the built environment, and has led several research and development projects.
The applications have been far reaching — from construction planning and subsurface utility visualisation to asset queries and plant maintenance. All use the physical world as a visual gateway to digital data, whether that’s an augmented model, hologram, datasheet or task notes. Simply point a tablet or look through an AR headset and the relevant information is there, instantly, in context. No more looking up file names, searching for references, or working off paper drawings.
Côté admits that tracking issues — not being able to augment the physical world with sufficient accuracy — have held back developments.
Unless the augmentation appears, and behaves, exactly as if it were in the physical world, mistakes will be made, he says. And if mistakes are made, the technology will not be used.
Côté has experimented with various positioning techniques, including beacons, total stations, GPS and feature recognition, none of which have been perfect. However, as he revealed at the recent Bentley YII conference, there have been two major breakthroughs. For outdoor applications, Bentley has developed an alternative approach to AR that uses a reality mesh instead of a live camera feed. For indoor applications, it’s Microsoft’s HoloLens, the AR headset that everyone is talking about.
Outdoor AR – mesh augmentation
Most AR tracking techniques capture a video stream from a tablet’s camera to ‘recognise’ visual features in the physical world, such as a lamp post, hydrant or steel section. However, due to changing conditions, such as shadows and wind, or moving objects like cars, Côté’s team has found that augmentations can be inaccurate or ‘shaky’.
Rather than exploring more ways to master the physical world with a live video feed, this has led to a change in tack at Bentley – the use of the reality mesh for context.
Reality meshes are created in advance, by a collection of photographs taken by a drone and processed with Bentley ContextCapture.
Then, later, wherever the user points the tablet on site, it displays the corresponding part of the mesh. This is tracked by the tablet’s GPS.
While GPS isn’t 100% accurate at tracking position and orientation, it is good enough to enable the user to visually match features displayed on the mesh to those in the real world. The important thing here is that the mesh is always perfectly aligned to the augmentation. This means that accuracy can be quantified and certified, so users can rely on the information, which is essential for engineering applications.
In addition to ‘solving’ the tracking issue, Côté says there are other advantages. Augmentation sessions can be carried out during the night, or when there is snow or rain, as the technique does not rely on visual tracking. Also, augmentation can be done remotely. This, says, Côté, could enable teams to plan site visits in advance.
There are downsides. First, it requires the 3D reality mesh to be captured in advance, and this mesh must be kept up to date. Reality meshes for entire cities are already available, says Côté, citing a 1,000km2 city model of Paris that exisits, but he admits that the realism of the augmentations will depend on the resolution of the images used to produce the meshes — so drone photos will likely produce more detailed meshes than aerial photos. Check out this example video to see the technology in action.
Indoor AR – Microsoft HoloLens
The HoloLens is a self-contained, holographic AR headset, which displays a hologram directly in the sight line of the wearer, so it looks like the object is in the physical world. Users can simply look at objects to access data, pinch their fingers together to select, or use voice controls. The big advantage over a tablet is that both hands are kept free.
With multiple sensors, the device offers truly exceptional tracking for indoor applications, says Côté, solving Bentley’s long-standing tracking issue in an instant.
The HoloLens is not geo-referenced — i.e. does not know its location in the physical world. But it is very good at recognising interior spaces and its exact position within them.
Bentley has been exploring several applications for the HoloLens, including “Construction planning in context”, “X-Ray vision for building maintenance”, and “Plant maintenance”. These are best illustrated by video, so we strongly recommend you check out the Bentley Research YouTube channel here.
Côté says the HoloLens does not work that well outdoors, as it can be affected by sunlight. Its form also needs to change so it can be used on construction sites — i.e. ruggedized and built into a hard hat. However, as a first release technology, it’s very exciting. There are also many more AR headsets in development.
Conclusion
Bentley sees AR becoming a hugely important technology for the built environment — and we tend to agree. But it is only starting to scratch the surface of what is possible.
Now that it no longer has a tracking issue to solve, it can focus its R&D efforts on developing more practical use cases.
We expect 2017 to be a pivotal year for AR as Bentley starts to work with partners in industry on real-world applications, hopefully taking it one step closer to delivering a commercial AR solution.
This article is part of a Bentley Systems Year In Infrastructure special report. Read the other articles below
The age of the reality mesh: Bentley Systems is forging ahead with Reality Capture by embedding reality meshes into its foundation platform.
OpenRoads Designer: This new breed of civil design tool combines mesh data with detail design in one dynamic environment.
Cloud analytics: Knowledge is power Bentley is helping engineers ‘design out’ delays before they happen, by giving them access to real-time supply info.
LumenRT embraces VR: LumenRT, an easy-to-use real time viz tool with a powerful ‘game engine’ environment, now offers VR support.
Bentley Systems futures: Bentley Systems may still be preparing for an IPO, but there has been some important news on its relationship with Siemens that could have big ramifications in the future
If you enjoyed this article, subscribe to AEC Magazine for FREE
