Trimble is leading the pack in Mixed Reality for construction through industry partnerships and R&D projects to develop exciting new applications for the HoloLens. Greg Corke reports on what construction might look like in the future
Imagine stepping out on a construction site wearing a Mixed Reality (MR) headset and instantly being informed what’s behind schedule, what’s been installed in the wrong place or where there might be an issue or clash. This is automatic construction verification in real time using object recognition, machine learning and computer vision and could soon become a reality thanks to R&D work being undertaken by Trimble.
When the HoloLens was first unveiled in 2015, Trimble was one of the first companies to get on board, working closely with Microsoft and collaborating with several leading AEC firms, including AECOM and Gensler, to explore real world applications of the technology.
The release of SketchUp Viewer for HoloLens in 2016 applied MR technology to design but the most exciting development — in our eyes at least — came in January this year with the launch of Trimble Connect for HoloLens, which finally brought MR to the construction site.
This is Mixed Reality in the true sense, where digital models are displayed in the context of the real world – so the wearer of the HoloLens can see where things will be constructed or, more importantly for identifying costly mistakes early on, where they should have been constructed.
From design office to site
Trimble Connect is a cloud-based construction management solution that acts as a collaboration hub for AEC projects. It can read in files from many different sources – Autodesk Revit, Tekla Structures, SketchUp and more – then be used to co-ordinate models and manage projects through change orders, to do lists and RFIs.
The workflow with the HoloLens currently starts in the office where a QA issue might be identified, and someone sent out on site to investigate. On site, that person puts on the HoloLens, sees a ‘to do’ list inside the MR environment and then inspects the issue.
At the moment, this is a very manual process. The worker first has to align the real and virtual worlds by telling the HoloLens that certain surfaces it has scanned in the real world are structural columns, staircases or elevator shafts in the digital model.
Once everything is aligned, on subsequent site visits the mapping sensors built in to the HoloLens do an excellent job of automatically recognising it is in the same location. However, as construction sites are continually changing with new elements added on a daily basis, alignment often needs to be done over again.
Trimble is working on an R&D project to use machine learning to automate this process. The idea is the HoloLens is taught to ‘recognise’ objects in the real world — that a beam is a beam, a column a column, etc. — then automatically work out its location based on its knowledge of the design and align itself accordingly.
Automatic alignment would be an impressive leap forward, but the next step would be to use machine learning and AI to automatically identify issues in real time, a process that Trimble calls continuous alignment.
Rather than relying on human observation, Trimble is working on an R&D technology that would be able to tell the worker if something is being installed in the wrong spot, something wasn’t installed and should have been, or if there’s going to be a clash. This is automatic, real time construction verification, enabled through object recognition, where the technology not only knows that a column is a column, or a pipe is a pipe but the exact specification of that element. This will not only add a lot of value to construction, but also to asset and facility management.
The feedback loop
At the moment, Trimble Connect for HoloLens lacks a closed feedback loop for issue resolution, but Trimble is developing new functionality to address this.
Responding to a work order on site, the worker will be able to review the issue in context, then give recommendations for resolution using voice commands to add audio notes, videos or still images to Trimble Connect for HoloLens.
Once this information is synced back to Trimble Connect, a change order can then be sent to the designer or engineer back in the office so he or she can make the necessary amendments to the 3D model.
Location, location, location
The HoloLens, in its current incarnation, has relatively limited storage and memory so it’s not really practical to upload entire construction projects to the device. Trimble offers best practice advice to customers and it’s more likely that only specific issues within a project are currently brought into the system.
This looks set to change soon. A Trimble spokesperson told AEC Magazine that in the last 18 months it’s done so much R&D on things like streaming and dynamic Level of Detail (LOD) that in six months’ time this will likely be a non-issue.
At the moment getting workers to the precise location of a potential issue still requires knowledge of the site. In the future, it’s not beyond the realms of possibility that a worker could be guided to the next task by a virtual arrow on the surface of the ground. It could even plot the fastest route between issues.
Of course, site safety is of paramount importance, so a beep from the HoloLens could be used to alert workers when, for example, they get close to the edge of a deck or an elevator shaft, as it knows at all times where it is on site. It could also identify potential dangers through object recognition.
What you see is what you build
Applying MR, machine learning and AI on an ever-changing site is a major challenge but Trimble is already making big leaps with its technology in more controlled environments to help workers perform repeatable tasks.
It is currently working with Consolis, one of the biggest pre-fab manufacturers in Europe, to help optimise the production process for its rebar cages with a view to replacing traditional, and potentially error prone, workflows with more productive and accurate solutions.
On the factory floor, instead of working off drawings, the workers use the HoloLens to see exactly where to place complex rebar within the cage. There’s about 50 steps that need to be completed in sequence and, as the worker finishes each one, he or she can move to the next using a simple voice command. Everything that has been completed is greyed out until the whole cage is filled.
A team of three led by Dr Marianna Kopsida, who has been heading up many of the Mixed Reality R&D efforts at Trimble, was able to complete the task without having any prior knowledge of building rebar.
MR and machine learning could also bring QA to this process. Rather than having one team build the cage and someone else check it, the system could automatically perform quality control in real time, not only checking that the bar has been placed in the correct position but warning the worker if an incorrect bar is about to be used.
Conclusion
Trimble looks to be making big strides forward in Mixed Reality for construction, with technology designed to help ensure design data is implemented correctly on site and deliver new levels of efficiency, accuracy and quality control.
Trimble Connect for HoloLens is already being proven out on real life projects, but the most exciting developments look set to come in the future through the application of object recognition, machine learning and computer vision which will allow construction firms to use computers to track progress and identify issues on site, rather than relying solely on humans.
This isn’t just about using technology to drive efficiency, shorten project schedules and bring down costs. With the global workforce becoming less skilled and the complexity of projects increasing, mixed reality could eventually transform construction itself in a ‘what you see is what you build’ future where construction workers are shown precisely where to place elements without having to refer to documentation. We are already seeing the first glimpses of this in Trimble’s R&D collaboration with Consolis on its rebar cage project.
While the HoloLens has been made ‘construction ready’ thanks to the Trimble Hard hat, one has to remember it’s still very early days for the technology. In its current incarnation, the HoloLens is nearly three years old and has well documented limitations in terms of field of view and processing power, but HoloLens 2 is rumoured to be waiting in the wings to be unveiled later this year or next.
As we dream of a future with smaller, lighter and more powerful MR headsets, the important thing to remember is that Trimble, with its industry partnerships and R&D work, is laying a solid foundation for the future. 2019 looks set to be a big year for Mixed Reality as the industry moves out of the early adopter phase.
Safety first — the HoloLens hard hat
Earlier this year, Trimble, in collaboration with Microsoft, launched an official hard hat for HoloLens to ensure the Mixed Reality headset adheres to stringent on-site safety regulations.
It’s a neat solution that distributes the weight of the HoloLens well and is a lot more intuitive to put on than the mixed reality headset on its own.
Furthermore, as it looks and behaves just like an industry-standard hard hat, Trimble says it has been well received by construction workers.
For the time being, this type of device won’t be standard issue and will most likely to be worn by those with specific roles such as BIM manager, VDC manager or site manger.
The retail price is $300.
What is Mixed Reality?
Mixed Reality blends the real and virtual worlds in real-time. The Microsoft HoloLens leads the pack in terms of MR headsets and features semi-transparent glasses so workers can see 3D design or construction models aligned with real objects in the physical world. This is different to Virtual Reality (VR) where the wearer of a headset like the HTC Vive is completely cut off from the real world.
With HoloLens, users can simply look at objects to access data, use hand gestures or voice commands for control. The big advantage over a tablet, which can also be used for mixed or augmented reality applications, is that both hands are kept free which is useful when performing manual tasks.
One limitation of the current HoloLens is that it has a relatively limited field of view. It means the wearer of the device can only see holograms in their line of sight and not in their peripheral vision. This takes a bit of getting used to, particularly when you’re used to a VR headset which is properly immersive.
If you enjoyed this article, subscribe to our email newsletter or print / PDF magazine for FREE
