From drone-gathered site data to augmented reality (AR), advanced technologies are bringing new perspectives on real-world environments and works in progress, writes Randall S Newton
The construction industry in Japan has a problem: the shrinking birth rate is creating a shortage of skilled construction workers. So heavy equipment manufacturer Komatsu has created a Smart Construction division, building automated excavators and dozers. The robotic equipment pushes dirt without a human driver, following excavation plans with great accuracy. The automated heavy equipment is so fast and precise that existing methods of ground-based data gathering (surveying, ground-based scanning) cannot keep up.
To tackle this issue, Komatsu Smart Construction worked with San Franciscobased start-up Skycatch to improve data collection and processing. Skycatch specialises in industrial applications for Unmanned Aerial Vehicles (UAVs), commonly known as drones.
Now, whenever Komatsu Smart Construction vehicles are on site, they are guided by one or more drones gathering terrain data. Komatsu refers to the resulting workflow as the world’s first machine-to-machine automated construction system.
Skycatch’s drones photograph job sites, capturing imagery and automatically generating 3D site data. The data can be quickly laid over site drawings or models to automatically calculate area and volume of earth to be moved.
The results are transmitted as instructions to Smart Construction machinery for fully autonomous work on the site. Think of it as drone-to-phone construction management.
“Once we have efficiently planned the work amount, [clients] can use the Komatsu machinery. Smart Construction can successfully plan the efficient system from the beginning to the end so clients save significant amount of workload, time and costs,” says Fujitoshi Takamura, CTO of Komatsu.
Eyes inside the project
“We are a data company that happens to use drones to collect data,” says Patrick Stuart, director of products at Skycatch. The company’s combination of drones, mobile apps and cloud processing turns raw data from the drones into usable information for autonomous equipment or for use by engineers in existing CAD or BIM workflows.
For example, a utilities engineer can monitor how utilities are being installed in near real time. By tracking progress so closely, the engineer can see if and when a design change must be made, then take measurements and decide what to do. What took days or weeks with manual data gathering and oversight can be now done in minutes.
“We give people in construction something they never had before – eyes into an entire construction project,” says Stuart. Traditionally, construction managers gathered information from a variety of sources and methods, from discussions on-site to marked-up plans, to email and phone exchanges.
“It is like a phone tree game,” says Stuart. “Usually the data is late and not very accurate.”
The Skycatch technology stack comprises three layers: the drone hardware and control technology (Skycatch sells drones but also works with other brands); apps for data review and editing the gathered imagery; and a cloud intelligence platform, for turning imagery and scans into actionable data.
One example of the data processing is in creating topographic charts. There can be no trees, shrubs and so on in the final data. Removing all the flora by hand, using a computer workstation, can take hours or weeks, depending on the scale of the project. The Skycatch cloud platform does it in minutes, scaling to use as many processors as available to complete the task. The software can export to common AEC data formats for CAD and BIM.
Running robotic heavy equipment and monitoring construction progress are only two of the most obvious uses of UAVs. One Skycatch client started using the technology after ground had been broken and foundation work had begun on a campus-sized project.
Management started by using Skycatch technology to gather current site data, comparing it to the project design. On the first day, they found a $400,000 slab of concrete was eight inches misaligned. Work stopped; an investigation revealed flaws in the original survey that echoed through the design phase and into construction. If not found so quickly, a $400,000 concrete mistake could have ballooned, costing several times more than that sum, depending on when traditional construction oversight methods spotted it.
“I use the Skycatch data on the site daily if not hourly,” says Victoria Julian, a superintendent for DPR Construction. “We can take the Skycatch plan, colour-code it, mark it up, label it and use it for logistics planning, daily, weekly, monthly – whatever we need.”
“Everybody who sees this is amazed by it. Loves it. And is using it. It’s really an incredible tool that I don’t want to have to do without on the next big project.”
Recurring imagery as sequential data
Skycatch is one of several vendors working to unite the real and the virtual in construction. The new element is recurring imagery. A ‘single source of truth’ has always been the holy grail, but the construction process has made it elusive. The ability to add recurring highresolution actionable imagery data to BIM models or CAD drawings is a gamechanger. BIM no longer has to be only the design/engineering model; it can be the constantly updated record of construction.
The use of high-res recurring imagery can mitigate risk by providing evidence of deviation from the design, as in the case above. Instant aerial 2D linear and 3D volumetric measurements replace manual estimates or hands-on retrieval, with significant time savings and improved safety. Using imagery for job site flow analysis can identify high traffic areas immediately, allowing managers to create workarounds. Terrestrial 3D laser scanners have been in use on large projects for years, but the addition of drones adds both the new dimension of fly-over viewing and the ability to turn site analysis into a new form of sequential data.
A construction site is constantly changing; research shows the constant state of flux is one key reason construction has not made efficiency gains in recent years compared to other industries. Specifically, the issues of on-site coordination, planning and communication have not reaped significant automation gains, with the obvious exception of making data accessible on mobile devices. The creation of data and editing of data has remained a desktop activity. Using drones to gather recurring imagery means site maps can become daily reports. Planning logistics and managing assets is simplified when exact coordinates replace estimates. Communication with quality control, safety and other departments becomes faster and more accurate.
Flying Superintendent
Turner Construction is working with researchers from the University of Illinois (UI) to create a real-time construction monitoring system using drone imagery. Flying Superintendent uses both still images and video to guide quality control, safety compliance and logistics. “The analytics we conduct on these survey-grade 3D visual production models offer construction managers a transparent view into what’s happening on site each day, empowering them to improve reliability in short-term plans and eliminate problems before they happen,” says Mani Golparvar-Fard, lead principal investigator and an associate professor of civil engineering at UI. The research team is currently testing and refining Flying Superintendent on the construction of a new professional sports complex in Sacramento, California. The goal is to create a predictive computer vision system for construction site data analysis that runs with minimal human interaction and can navigate using construction site data. By not relying on GPS data as its only navigation method, drones can be used for indoor work where the GPS satellite signal is weak or non-existent. The research is being commercialised by a university spin-off, Reconstruct Inc. Flying Superintendent is also currently being used on a high-rise project in Arizona and by a leading Japanese construction company on multiple projects.
State of the art
The technology for aligning the real world of construction data with the virtual world of design and BIM is very much a work in progress. Start-ups like Skycatch and Reconstruct and others are working from the drone-and-imagery side, while experienced AEC technology vendors like Autodesk, Bentley and Trimble are working from the design data side. An overview of the leading vendors and their work reveals a variety of approaches to the challenge of aligning the real and the virtual in construction. Major players and technologies include:
3D Robotics
3D Robotics (3DR) was an early player in the commercialisation of drones and recently dropped out of the consumer market to focus on enterprise applications. It has received $99 million in venture capital funding and is working in close cooperation with Autodesk on SiteScan, an aerial analytics platform designed to integrate the drone-and-camera hardware with various Autodesk products including ReCap and A360. Earlier this year a SiteScanenabled 3DR drone created 3D imagery of the famed Red Rocks Amphitheatre near Denver as part of a renovation project. Autodesk helped scan Red Rocks a few years ago using terrestrial 3D laser scanners, but this update significantly improved upon the existing data because of the aerial dimension. Following a borrow- not-build ethic, 3DR is also collaborating with Sony (for mobile computing) and GoPro (for photography) as it builds out the capabilities of SiteScan.
Autodesk
Years of research and a variety of acquisitions has given Autodesk a rich treasure trove of relevant technologies for aligning real and virtual construction data. Until recently most of it was scattered in various divisions. Last year, Autodesk launched the Forge Initiative, designed to consolidate its various technologies and services related to mixing the use of realworld data (“reality capture” in Autodesk’s words) with design automation. Some of the technology is designed for product development and 3D printing, but there are also tools for AEC. ReCap 360 provides automatic stitching and registration of scan data to CAD/BIM files and can be used to measure items depicted in scans. A360 is a cloud collaboration tool that works in conjunction with ReCap, simplifying access to data for all stakeholders. The resulting data can be used in all Autodesk AEC and civil engineering applications including AutoCAD, Revit, Navisworks, Infraworks and others.
Bentley Systems
Bentley brings its enterprise approach to AEC and geospatial engineering with a line of software products for what it calls “reality modelling”. ContextCapture turns photos into detailed photo-textured 3D mesh models that integrate with MicroStation and other Bentley design and engineering products as well as CAD or GIS software from other vendors; the models are precise enough to measure from or use for comparing as-built with design. With ContextCapture, a set of photos up to 100 gigapixels in size can be converted into a single detailed model. The models are inherently georeferenced through both device positioning data and precise control points. Other modules in the ContextCapture line can create videos or interactive web models.
Trimble
It seems each of the major AEC vendors has a different phrase to describe using 3D site data in the construction workflow; for Trimble, it is “mixed reality”, to describe the blending of realworld objects with digital content from design in real time. Trimble’s roots are in construction, not design; in recent years it has been building a BIM portfolio from a construction tech point of view, and also working closely with other vendors where interests align. Trimble has interoperability agreements with both Autodesk and Bentley, and is using software from both companies in their mixed reality R&D. Trimble sees mixed reality as a form of spatial interaction, in which design teams and construction teams both review, interact and share using 3D models in the context of the physical environment instead of the virtual design space. “In the context of the building industry, this is the phase in which digital and real content co-exist,” says Aviad Almagor, Trimble’s director of Mixed Reality. “Architectural design collides with reality and construction teams transform digital content into physical objects.”
Trimble is working closely with Microsoft to bring Hololens technology to AEC. Hololens is a wearable, self-contained holographic display computer. Users can interact with 3D models that blend into the view of real objects. The Hololens display can be adjusted on a continuum from ‘real objects only’ to ‘digital model only’ – or anywhere in between.
Trimble is working from a deviceagnostic perspective. In addition to Microsoft Hololens, it’s also exploring the use of the Google Tango platform to develop augmented reality applications and Facebook Oculus Rift for virtual reality.
Two on-going pilot projects with very large companies highlight what Trimble is developing.
With engineering giant AECOM, Trimble is developing a system for collaborative review of engineering design from a construction perspective. Engineering data and site data collected by drones are overlaid for viewing inside a Microsoft Hololens display system. Design alternatives can be explored in a collaborative context and team members can immediately see how the changes will interact with the existing site.
In June 2016, Trimble announced a mixed reality pilot program with CSCEC Group 1, one of the largest construction firms in China. Similar to the AECOM pilot, drone data will be blended with design, but for construction operational overview, not engineering review. Given the need for on-site collaboration in the Group 1 pilot, the Google Tango tabletbased augmented reality technology may play the lead role from the display side.
Unlike the products from Autodesk and Bentley, the Trimble mixed reality technology is still in the proof-of-concept and pilot programme stage.
The Big Data future
Virtual reality (VR) and augmented reality (AR) are attracting a lot of attention in AEC at the moment. Project visualisation (whether for sales or architectural design review) is an obvious use for VR. Construction and operations are looking closely at AR as a new way to deliver assembly, operations and repair data to the job site. These are interesting applications, each worthy of their own articles. But there is something unique to realtime site information in the BIM model that transcends either specific display technology.
Virtual reality and augmented reality freeze data at a point in time, yet construction sites change constantly. Superimposing drone-gathered site data onto the BIM model in real time makes new technologies possible; Komatsu’s autonomous earth movers are just a first step. As all construction processes become accessible as data, technologies now used in so-called big data applications such as analytics and artificial intelligence can be applied. It should not take too long for autonomous robotic welders and riveters to assemble highrises and autonomous cement trucks to deliver just-in-time loads where they are needed on the job site without a specific advanced order or schedule. Adding other data sources such as weather and traffic could improve logistics and oversight. Add financial data to the mix and, over time, data analysis could more accurately predict costs for specific design styles and construction methods.
Randall S. Newton is Principal Analyst at Consilia Vektor. He has been writing about AEC since 1987.
If you enjoyed this article, subscribe to AEC Magazine for FREE
