HBIM and augmented information: towards a wider user community of image and range-based reconstructions

Abstract. This paper describes a procedure for the generation of a detailed HBIM which is then turned into a model for mobile apps based on augmented and virtual reality. Starting from laser point clouds, photogrammetric data and additional information, a geometric reconstruction with a high level of detail can be carried out by considering the basic requirements of BIM projects (parametric modelling, object relations, attributes). The work aims at demonstrating that a complex HBIM can be managed in portable devices to extract useful information not only for expert operators, but also towards a wider user community interested in cultural tourism.


INTRODUCTION
The use of handheld mobile devices (mobile phones, tablets, …) is no longer limited to personal and recreational purposes.Mobile devices are used for productive work in different disciplines (medicine, education, simulated training, …) for their high flexibility and a real-time access to information.Nowadays, it is normal to digitize notes, send and receive invoices, managing parts, scanning barcodes or make payments.This aim of this work is to investigate the possibility to handle accurate HBIM in portable devices for cultural heritage conservation and preservation policies with the aid of new technologies.The different operators (architects, archaeologists, restorers, historians, conservators, engineers, etc.) can exploit the advantage of the improved collaboration of BIM projects through the use of portable tools.On the other hand, practical applications are not only confined to professional operators.In this work particular attention is paid to a wider user community with different purposes, including built environment education, interactive learning, cultural tourism, and gamification, among others.Mobile devices can have a fundamental role in stimulating the interaction between people and digital cultural heritage in order to (i) connect people to heritage, (ii) create knowledge, and (iii) preserve the cultural heritage resource.There is growing interest in surveying, modelling, and visualization techniques for a better exploitation of the cultural content with the development of new digital applications and tools towards a more reflective society (Cuca et al., 2014).Although the integrated use of augmented information (including virtual reality and augmented reality) in mobile devices is not a new concept in the construction industry (Park and Wakefield, 2003;Osello, 2012), the integration of detailed HBIM poses new challenges right from the first phases of the reconstruction process.One of the big issues concerns the need of a very detailed model and its operational use in handheld mobile devices.This could require HBIM with a better level of detail than models for modern construction projects with predefined object libraries.Additional issues could arise in terms of memory occupation, making the exploitation of the model more complicated in mobile devices.The generation of an accurate and complete HBIM of a historic construction is a complex task for the lack of commercial software able to manage the geometric complexity (Fai et al., 2011;Murphy et al., 2013;Brumana et al., 2013Brumana et al., , 2014;;Oreni et al., 2014a) revealed by laser scanning and photogrammetric point clouds.Building Information Modelling is based on intelligent objects with relations to other objects, attributes, and parametric modelling tools.The use of advanced NURBS surfaces turned into complete HBIM objects was proposed by Oreni et al. (2014b) to avoid excessive simplifications resulting in models not useful for conservation.On the other hand, the information encapsulated into the model can be used to extract different kinds of solutions for different kinds of users.The advantages of BIM managed in mobile devices is very attractive not only for expert operators of the Architecture, Engineering and Construction (AEC) industry.The extension towards a complete HBIM can open up new opportunities and alternative approaches to data representation, organisation and interaction, also for the operators in the field of cultural heritage.Some mobile applications integrating BIM technology were proposed in Waugh et al. (2012), where a web-based augmented panoramic environment was developed to document construction progress.Park et al. (2013) presented a conceptual framework that integrates augmented reality with BIM to detect construction defects.Dunston and Wang (2005) proposed augmented reality as a tool to support all phases of the facility life cycle.As mentioned, this paper try to obtain a wider diffusion of the digital reconstruction.Augmented information is already used for built environment education, in particular with the provision of real knowledge that can be used in the real world (Tatum, 1987).Several examples have shown that augmented reality can aid tourist organizations and professionals, reaching a wider audience by serving as the delivery technology of advanced multimedia contents.Augmented reality information systems can help tourists with an easy access to valuable information, improving their knowledge regarding a touristic attraction or a destination, enhancing the tourist experience, and offering increased levels of entertainment throughout the process (Fritz et al., 2005).The work investigates the full pipeline for HBIM generation, starting form data acquisition till data delivery.Some mobile applications were tested to take into considerations both professional and casual users.The case study is Castel Masegra, a castle located in Sondrio (Italy).A detailed historical HBIM (500 MB in Revit) was derived from laser scanning point clouds (7.5 billion points).Building information modelling was carried out by dividing the different structural objects and their constructive logic.Chronological, material, and stratigraphic aspects were also taken into account.This step is not only useful for architectural purposes, but also for further static and dynamic simulations where the temporal evolution of the castle provides additional data about its logic of constructions (Barazzetti et a., 2015).The model was then exported and converted into several formats to exploit the possibility offered by the available portable applications.Two different mobile products were generated from the original HBIM.The first one is a complete HBIM integrated with technical data to support restoration and conservation projects.The second product is instead more oriented towards a wider user community.It aims at stimulating a larger interest in historical resources with remote navigations and on-site immersive visualizations.The obtained results showed that modern mobile devices are sufficient to handle advanced BIM reconstructions not only for technical purposes, but also for promoting cultural tourism.Fig. 2. The creation of a HBIM can be exploited for different purposes and user communities.

DATA ACQUISITION AND PROCESSING
The geometric survey of Castel Masegra was carried out with laser scanning and photogrammetric techniques.The complexity and the size of the castle required 176 scans registered with the support of a geodetic network.A total station Leica TS30 was used to create a robust geodetic network made up of 68 stations.The measurement phase took 4 days and included not only station points and some fixed points (mainly retro-reflective targets), but also chessboard targets for scan registration.Geodetic tripods were not moved during the survey to avoid repositioning errors.In all, 4,622 observations and 1,402 unknowns gave 3,220 degrees of freedom.Least Squares adjustment provided an average point precision of about ±1.2 mm.The network provided a robust reference system for scan registration.The instrument used is a Faro Focus 3D and the final point cloud is made up of 7.5 billion points.Scan acquisition took 5 days.Scans were registered with an average precision of ±3 mm by using chessboard targets measured with the total station and additional scan-to-scan correspondence (spherical targets).The survey was then integrated with some additional scans to capture the occluded areas after the first surveying phase.The limited time for scan acquisition (less than a week for the large and complex case study presented in the paper) proves the level of maturity reached by modern laser scanners.It is clear that the time needed for data processing (especially HBIM generation) is instead much longer, especially in the case of very detailed reconstruction.
Photogrammetric techniques were used to complete the reconstruction of some elements.Different digital cameras with different lens were used after a preliminary photogrammetric calibration.Images were extremely useful to extract dense point clouds which integrate laser scans.Photogrammetry was also used for some elements that required a representation with orthophotos (e.g. the umbrella vault in Fig. 4).The use of total station data allowed one to obtain a common reference system for the different acquisition techniques.A survey with a drone (Asctec Falcon 8) equipped with a RGB camera provided a set of images for the inspection of the roof the small surrounding hill.As the goal of the project is a HBIM useful for architectural and structural purposes, the surveying phase cannot be limited to the reconstruction of the shape.The presented measurement techniques can reveal the external layer of construction elements, whereas a HBIM is made up of objects with an internal structure.As the goal is the creation of an interoperable HBIM and its distribution among the different operators that work on the castle (engineers, architects, historians, archaeologists, restorers, etc.) the survey included a historical analysis, materials, construction phases, technological aspects, stratigraphic analysis, and information from other inspections such as infrared thermography or structural tests (flat-jacks, coring, etc.) (Binda and Tiraboschi, 1999;Colla et al., 2008;Gregorczyk and Lourenco, 2000;Rosina and Grinzato, 2001).

POINT CLOUDS TURNED INTO HBIM
The starting point for the generation of the HBIM is the set of dense laser scanning point clouds which reveal the geometric complexity of the castle.HBIM reconstruction must be carried out by considering the basic requirements of BIM projects.Detailed building information modelling cannot be carried out with the interpolation of the point cloud with mesh-based algorithms often used in photogrammetry and computer vision.Additional information (e.g.materials, construction stages, stratigraphy, etc.) has to be taken into account to create intelligent parametric objects with relations to other objects and attributes.Photogrammetric and laser scanning measurements are powerful tools for the generation of object surfaces.However, BIM projects are composed of solid elements with additional information about the internal layers.As point clouds reveal the external surface of the object, the use of other techniques (e.g.IRT), the inspection and analysis of the constructive logic, and architectural/structural interpretation are mandatory to correctly represent the different elements and their connections.The methodology for parametric BIM generation was based on a preliminary separation between simple and complex shapes.In the case of simple objects, the tools of most commercial software (Autodesk Revit in this case) were sufficient for an accurate reconstruction.However, the case of irregular objects (e.g.vaults, arches, etc.) was much more complicated for the lack of commercial solutions for BIM generation able to take into account the level of detail encapsulated into laser point clouds (Eastman et al., 2008;Lee et al., 2006).For this reason, the procedure proposed by Oreni et al. (2014b) was used to create parametric BIM objects based on surfaces made up of NURBS curves and NURBS surfaces (Piegl and Tiller, 1997;1999).Shown in Fig. 5 are some BIM objects used in the project: walls, vaults, columns, ceilings, beams and trusses, stairs, and decorations.Structural elements were classified following the predefined structure of the software database: category, family, type, and instance.3D modelling was carried out from slices and 2D drawings created from the laser cloud.The preliminary use of 2D drawings is a valid tool to distinguish areas where an accurate 3D modelling is required from parts that can be simplified with predefined objects.Plans, sections, and elevations correctly positioned in space provided the reference frame for the reconstruction of the model.This is a fundamental point towards the creation of an accurate BIM consistent with the preliminary products of the geometrical survey.Starting from the sections, the main deviations from verticality of exterior walls were identified, whereas the interior wall appeared reasonably vertical.Revit tools for openings ("windows" or "doors") were not directly used.The basic functions of the software allowed one to define a large number of predefined parameters.However, a limited correspondence was found for the complex openings of the castle.The accurate modelling of wooden frames required the definition of ad hoc families for the different types of openings.As things stand at the moment, the openings were modelled as "voids" in the BIM.One of the advantages of BIM is the opportunity to include different kinds of information in a dynamic common platform for the different operators involved in the project.A historical research was carried out to understand the different construction stages and their interconnections, which were represented in the final HBIM with multi-temporal elements.This means that the HBIM can provide a visualization of the changes and modifications occurred in the past, till the current shape.Obviously, the historical research cannot be assumed as a final solution.From this point of view the use of a HBIM is valid tool for the possibility to modify an initial hypothesis without redrawing (different construction stages can be set in the database).
The 3D visualization shown in Fig. 6 allows an immediate evaluation of the modifications occurred.This information can be used for different purposes.However, this paper focuses on cultural tourism (see next section) with the availability of historical data in mobile platforms.Another application was presented in Barazzetti et al. (2015), where the structural analysis via finite element models was illustrated.The simulation included the different construction stages, which are fundamental to understand the different connections between the elements of the castle.These examples confirm the suitability of HBIM technology as an common environment for both casual and expert operators.

FROM HBIM TO AR & VR
Different apps for mobile phones and tablets were taken into consideration in a exhaustive exploitation of the reconstruction carried out from images and laser scans.These applications are not limited to specific BIM solutions for professional operators.Casual and non-expert users more interested in interactive visualizations coupled with usable information can exploit some of the advantages of these representations.Augmented and virtual reality can be very important technologies for the collection, preservation, exploration and diffusion of cultural heritage.

BIM in portable devices: 3D models with information
Autodesk 360 (A360) is a mobile application directly connected to BIM projects generated in Autodesk Revit (Fig. 7).Different professional operators can exploit the advantages of such application, which offer a dynamic visualization of 2D and 3D drawings.On the other hand, the app is not only a viewer of geometric reconstructions.An efficient and simple visualization is carried out for BIM projects, including object properties and reports of the different activities.A HBIM project in Revit can be saved in a new DWF file format which preserves objects information.Different visualizations (3D views, sections, plans) can be set in the project to facilitate the access to the different parts of the model.Although some problems were found with object textures, A360 allowed an efficient visualization of the huge Revit file of the castle (more than 500 MB) thanks to a preliminary conversion in the DWF file without losing object information.BIMx is another application able to preserve some information during the conversion of the BIM project into a portable version (Fig. 8).The visualization of the model of the castle was very luid, with the opportunity to create dynamic sections.On the other hand, BIMx cannot open Revit models.For this reason, the model of the castle was converted into the interoperable IFC format.The IFC file was imported in ArchiCAD, from which the file for BIMx can be generated.The exchange of information between two BIM software resulted in some errors during the conversion of particular objects, including an information loss especially for CAD blocks, object textures and some material properties.

Virtual visit of the castle
iVisit3D is an user-friendly application able to create virtual visits with static images and panoramic views.A 3D model can be interactively navigated by setting some links between the predefined images.The use of static visualizations makes the navigation very fluid also for platforms with moderate performances (Fig. 9).The Revit file cannot be opened with the iVisit3D, which needed a preliminary conversion in a polygonal format.Rhinoceros was used to handle the geometric model.However, this generated a complete information loss including parametric modelling functions, object relations, and attributes.This is a fundamental difference between models created in BIM packages (Revit, ArchiCAD, Teckla, …) and software for 3D modelling (Rhinoceros, 3DS Max, SketchUp, …).After the generation of the 3D model, iVisit3D uses the powerful rendering engine Artlantis to create virtual tours, where texture can be applied to objects.Different illumination conditions can be simulated as well.Fig. 9.The virtual tour generated in iVisit3D required the preliminary conversion of the BIM with a complete information lossinformation loss.

Augmented reality from BIM
AR-media creates 3D visualizations based on augmented reality through mobile phones and tablets (Fig. 10).Two plugins are provided: the first one allows the conversion of a 3D model directly from a project generated in other 3D modelling environments.The second plugin allows the visualization by means of augmented reality techniques.The first plugin is available for different software, which are mainly pure modellers (Maya, SketchUp, etc.).The BIM model in Revit cannot be directly used by this package and a preliminary conversion in FBX is needed.This means that there is information loss.Finally, a marker can be set so that the package will show the corresponding 3D model when the camera capture the marker.Fig. 10.A 3D model without information can be imported in augmented reality applications.

Discussion
Table 1 shows a general comparison of the proposed applications.Some applications are mainly developed for BIM projects and are more oriented towards expert operators in the field of construction.A very important aspect concerns the lack of "perfect" interoperability during the exchange of information between different BIM packages.Conversion errors (information loss, attribute removal, modification of spatial location) were discovered for some objects, notwithstanding the availability of interoperable file formats (IFC in this case) for input and output.The HBIM can be intended as the central point of data processing.If the project requires an accurate BIM, technical products (plans, sections, …) and additional products (3D models for virtual reality, …) can be derived from the original BIM file.This means that the BIM and its sub products can be used by both expert and casual users.Future work is needed to better understand the potential of BIM technology not only in the case of expert operators, but also for a wider user community.The integration of other existing technologies and disciplines (e.g.QR codes, positioning systems integrated in mobile devices, advanced visual recognition systems, multimedia video glasses, etc.) is a very promising field of research to promote the interaction between users and cultural heritage.The main idea is an approach towards a reflective society which supports a sustainable management preservation and valorisation of built heritage (Fig. 11).
Fig. 11.The closed loop aims at stimulating a better interaction between people (expert operators, tourists, etc.) and built heritage through multimedia.

CONCLUSION
The growing interest in Building Information Modelling is a great opportunity for the expert operators in the field of photogrammetry, laser scanning, and 3D modelling.Accurate as-built BIM of existing constructions, including HBIM of historic constructions, can have a new role in digital documentation.This paper demonstrated that detailed HBIM can be derived from laser scanning and photogrammetric point clouds, obtaining rigours model that reveal the geometric complexity of the building.In other words, the generation of accurate models that reveal the geometry is not limited to few software for "pure" geometric modelling, whose output is not a BIM.On the other hand, new tools and functions were developed and implemented to overcome the lack of commercial software packages able to reconstruct complex shapes.Parametric modelling tools, relations, and attributes can be added to obtain a complete HBIM and not only a geometric reconstruction achievable with software for "pure" 3D modelling.The final model must be therefore an interoperable product for common BIM packages (Revit, ArchiCAD, Teckla, …), where parametric modelling tools, relations and attributes can be preserved with interoperable file formats.On the other hand, this work demonstrated that IFC files derived from commercial packages have compatibility issues, including modifications or information loss.This means that portable BIM viewers need a (partial) correction of the file saved in different formats.
The conversion of the model into pure geometric software was mandatory for virtual reality and augmented reality packages.In this case a complete information loss is inevitable.On the other hand, the first results with huge BIM files demonstrated that portable devices are sufficient to handle complex and detailed models (Fig. 12).
Mobile BIM applications can be a valid tool for expert operators interested in the conservation process, whereas 3D models and virtual tours can be generated from the HBIM, without redrawing a new model for these specific purposes.
The BIM can be therefore intended as a central tool for the accurate investigation of the building.Then, different sub products can be obtained without splitting data processing into different modelling projects for different purposes.A HBIM can be therefore the starting point for expert operators mainly interested in preservation and restoration or a wider user community interested in a product derived from the original HBIM.
Fig. 12.The use of the HBIM with an iPad: interactive visualization and queries can be easily carried out.

Fig. 3 .
Fig. 3. Top: the geodetic network measured with a total station (the average precision is ±1.2 mm) and a 3D view of laser scans (7.5 billion points).Bottom: panoramic visualization of a single scan.

Fig. 5 .
Fig. 5.The final BIM is made up of a combinations of objects with variable geometric complexity.

Fig. 6 .
Fig. 6.A detail of the BIM in Revit, where different colours correspond to different construction stages.

Fig. 7 .
Fig. 7.The BIM of Castel Masegra inspected with A360 on a tablet.The software preserves object information.

Fig. 8 .
Fig. 8.The object based visualization with BIMx is affected by an information loss for the preliminary file exchange from Revit to ArchiCAD.

Table 1 .
Comparison of the four applications.