A GEO-DATABASE FOR 3D-AIDED MULTI-EPOCH DOCUMENTATION OF BRIDGE INSPECTIONS

: The recent collapse of bridges in Italy has prompted numerous studies on monitoring and maintenance. Many structures in Italy have been in service for over 50 years, necessitating new approaches to ensure their safety. To address this issue, Italy's Consiglio Superiore dei Lavori Pubblici (Superior Council of Public Works) has developed the Guidelines for Risk Classification and Management, proposing a multi-level approach to bridge management within a complex geomorphological environment. The guidelines outline a multi-level process that includes surveying the structures, conducting detailed inspections, and assigning risk classes based on hazard, exposure, and vulnerability. Current inspection processes are time-consuming and costly. Therefore, alternative monitoring technologies are crucial. Unmanned aerial vehicles equipped with cameras, laser technologies, and GPS systems offer flexible and cost-effective solutions for visual inspection. These technologies enable the collection of both quantitative and qualitative data, such as size, material properties, and overall condition. In this context, efficient data management and exploration systems are necessary to handle the vast amount of geo-referenced information. Multi-epoch databases play a crucial role in documenting the conditions of bridges and supporting a maintenance and structural health monitoring workflow. These databases can be utilized within a Bridge Management System to aid road managers in decision-making processes. Additionally, 3D exploration platforms provide visual analysis and highlight areas of interest within the structure. This work presents a multi-epoch geo-database that adheres to the Italian guidelines, offering optimized data management and queryability for 2D and 3D information. The entire process is designed using open-source and reproducible


INTRODUCTION
Following the collapses observed on bridges in Italy in recent years (Malerba, 2022), the issue of monitoring and maintenance has become the subject of numerous studies and research.In fact, most of the bridges and viaducts on the Italian territory have been in service for more than half a century (Mannella, 2019).Such evidence highlighted the importance of the health and safety of infrastructure assets and the need for new approaches for their monitoring and safeguarding.Moreover, the reality of a complex geomorphological environment has led to the need of a formal framework for monitoring the health and safety of road infrastructures, optimising the efficiency of the investments to be made within a reference time horizon for periodic and urgent maintenance (Scalbi et al., 2023).This urgency led Italy's Consiglio Superiore dei Lavori Pubblici to design the Guidelines for Risk Classification and Management (MIT, 2022), proposing a multilevel (i.e., with 5 levels) approach to managing such infrastructures, within both the individual structure and the surrounding environmental context.It illustrates how risk classification is part of a general multistage approach, from a simple survey of the infrastructures and their positions to the determination of a class of attention based on detailed inspections (Giordano et al., 2022).In particular, these guidelines identified a series of levels with a progressive degree of detail to document the design characteristics of the structure (Level 0), record periodic observations on its health conditions through visual identification of damages (Level 1) and assign the bridge to a risk class based on hazard, exposure and vulnerability deduced from the previous steps (Level 2).The classes highlight different degrees of attention that imply different monitoring strategies, implying preliminary defects in-situ safety evaluation for medium to high-risk bridges (Level 3) and a detailed functional assessment (Level 4) (Natali et al., 2023).However, current inspection processes are often timeconsuming, costly and require disruption to the smooth operation of infrastructure systems (Agnisarman et al., 2019).Therefore, the choice of alternative monitoring technologies plays a critical role in the inspection and maintenance of civil infrastructure.Unmanned aerial vehicles (UAVs) can provide flexible opportunities for visual inspection of structures.When equipped with cameras, laser technologies and Global Navigation Satellite System (GNSS), they can collect both quantitative data, such as size and material properties, and accurate location, as well as qualitative information, such as physical appearance and overall condition (Feroz & Abu Dabous, 2021, Pinto et al., 2020).These non-destructive technologies have become very popular in research in recent years as they facilitate monitoring and inspection management.In fact, drone-based imagery and photogrammetry allow for 3D reconstruction of structural defects, such as cracks, by reducing the costs and potential hazard factors associated with manual inspections (Ioli et al., 2022).In addition, UAV photogrammetry and Laser Imaging Detection and Ranging (LIDAR) products allow the reconstruction and acquisition of point clouds that can be segmented and classified, by identifying the objects within scenes, assigning them to a specific class and using them for further studies (Barrile et al., 2019).The production of such a wide variety of geo-referenced information, including both 2D and 3D products, requires an efficient data management and exploration system.From a technical point of view, the presentation of information at different levels of detail, involving different types of data (e.g.text, 2D or 3D geometries) and media, provides a comprehensive view of the context in which a structure is located, with applications ranging from cultural heritage (Quintilla-Castán et al., 2022, Gaspari et al., 2023) to industrial structures (Bajauri at al., 2022).In this context, multi-epoch databases allow for documenting quantitatively the conditions of bridges by laying the foundations of a Bridge Management System (BMS) able to support a comprehensive maintenance and structural health monitoring workflow along the entire structure life cycle (Dayan et al., 2022).Such data inventory can further support road asset managers in the decision-making process with dedicated 3D exploration platforms that highlights visually portions of interest of the structure and allows further detailed visual analysis to be carried out a posteriori (Mohammadi et al., 2022).This work illustrates the definition of a multi-epoch geodatabase that implements the requirements of the Italian guidelines, defining a structure for an optimised management and queryability of data and an environment for 2D and 3D information interactions.The process has been designed with open source, reproducible and documented solutions in mind.

ITALIAN BRIDGES CONTEXT
The evaluation of the risk and safety of the existing national road heritage represents a crucial topic for agencies and asset managers engaged in the structural health assessment.This issue has acquired additional relevance following the recent collapse of bridges and viaducts in Italy (Bazzucchi et al., 2018, Ferro et al., 2022), leading to formulation of the Guidelines for Risk Classification and Management (MIT, 2020), followed by their operative application instructions (ANSFISA, 2022).These national directions defined a multi-level approach whose first three levels (Level 0, 1, 2) results in the classification of each bridge or viaduct into different risk classes, thus prioritizing monitoring and maintenance interventions on artefacts according to information and data processed at Levels 0 and 1 (Figure 1).The higher the assigned class, the higher the evaluated risk of collapse or deterioration linked to the bridge.In details, the Level 0 includes general information about the geographic location of the bridge, its material and structural composition, that also implies the attachment of original project design and official documentation of past inspections and maintenance operations.Each bridge also requires at least one detailed inspection report (Level 1), consisting in a table documenting the information retrieved from a visual check of the artifact.Hence, measurements and insights from the last inspection conducted serve as input for the identification of a series of defects on given structural elements of interest according to the guidelines, such as piles, abutments, slabs, beams, supports and joints.Each structural element is associated to different types of defects to be inspected, whose complete list is included in the guidelines.In addition, each individual defect is associated with a level of severity defined in the annexes to the guidelines.Based on the severity of defects found on portions of the bridge, a level of attention and risk is assigned to the entire structure.Eventually, this Level 2 classification results in the adoption of a dedicated surveillance level for the maintenance of bridges, with increasing degrees of attention according to higher risk.This strategy aims not only at identifying structure of higher priority but also at optimising the allocation of funds and operators for interventions and the adoption of appropriate monitoring of the structures (Natali et al., 2023).While the first three levels defined by the guidelines affect the management of all the structures present on the national territory, Levels 3 to 5 affects only bridges and viaducts in medium to high-risk classes.In particular, Level 3 encompasses initial safety evaluations to determine the need for more comprehensive analyses for medium-high and high risk bridge.Structures in the highest risk class, instead, are directly linked to Level 4, which is dedicated to conduct meticulous safety assessments in accordance with the latest technical standards, supplemented by specific guidelines for existing bridges.Eventually, the last Level 5 is applicable to highly significant bridges within the national road network, involving advanced resistance analyses and load tests.

METHODOLOGY
The proposed methodology defines a shared platform congruent with the first 3 levels in the structure of the Italian guidelines affecting the management of all bridges and viaducts on the national territory.It mainly consists of three components: data collection and preparation, database design and web platform implementation as depicted in Figure 2.Each step will be described in detail in the following sections.

Data collection and preparation
In order to document the current state of the artefact as well as its entire life cycle, the first step of the process is dedicated to the data gathering.In particular, the initialisation of the entire procedure is made possible by the collection of data concerning both the history of the project design and the geometry of each structure.The first reference source for this information is generally represented by a road administrator inventory where project documents, past maintenance intervention statements and inspection reports are stored by the local road managing agency.Such an archive usually allows decision-makers to reconstruct the approval timeline of the structural project, providing information on the bureaucratic operations that led to the construction of the bridge.However, in the case of very old artefacts or structure whose management has often been transferred to different agencies over time, older documents relating to the project design and previous maintenance operations may be missing or deteriorated.
On the other hand, in order to document the current state of a structure, a targeted survey of the artefact is required to obtain measurements and observations of its conditions and geometry, that may differ from the project design ones due to construction choices or deformations.Such a goal is achieved using various geomatics techniques for in-situ data acquisition and subsequent processing (e.g.3D model reconstruction, geometric accuracy assessment).As a result, accurate 3D scaled products like point clouds and meshes are obtained and georeferenced.In the Italian situation, the coordinate reference system is the European Terrestrial Reference Frame 2000 (EPSG:7930).This visual inspection is repeated in time on the structure according to the level of surveillance assigned in the latest steps of the Italian multi-level guidelines.
The information and products gathered and obtained in the first step of the methodology are the starting point for organising the structured data that populate a BMS.Hence, data coming from the road administrator inventory, that could be available as simple scanned documents or as structured digital files, identifies which general information required from the guidelines are missing.These resources are the core component for the population of the generic form for Level 0 and 1 of the Italian guidelines, providing the basic information on the project designer, administrative responsible and legal regulations to which the bridge is subjected.Instead, the geo-referenced products obtained from in-situ survey make up the basis for the implementation of the digital twin of the structure on which the 3D web viewer is built on in the next steps.The choice of the data is done on the basis not only of the guidelines' requirements but also of the users' preferences.For example, this could result in the adoption of bridge point clouds classified into different structural elements, facilitating the visual identification of portion of interest for the analysis of defects.Also, images georeferenced in 3D environments are required to objectively document the presence or absence of given defects at a specific moment in the bridge life cycle.

Database design
The database design phase is based on a requirement analysis that investigates the information types and structures required for Level 0, Level 1 and Level 2 steps of the guidelines (Figure 1).Such information organization requires for a data management system that supports its users in decision-making operations of data entry, update, and query.For these reasons, a relational spatially enabled database has been adopted for schematic implementation of the Italian guidelines.The one-to-many relationships of the relational model allow for a comprehensive documentation of the entire process, reconstructing the entire history of inspections of a single bridge and enabling the tracking of defect evolution in time.A particular need for the database is also represented by the defect form, a table to be filled for each structural element as a result of an inspection documented on the Level 1 form.The Italian guidelines provide a fixed template schema of distinct type of form defects for any of the 20 documented structural elements in different material.Indeed, the subset of the defects to be checked out of the total of the 121 varies from element to element, defining a different form schema whose conditions are defined through the relationship between the defects and the defect form answer table.With this assumption, the Entity Relationship Diagram (ERD) for the database has been designed (Figure 3).Such ERD contains a relationship loop, which is usually not recommended because it involves redundancy of information.However, in this case it was adopted because otherwise the defect form would have had too many attributes, seriously compromising its efficient manageability.Eventually, such schema was implemented using PostgreSQL as the backend free and opensource solution for relational database management system.The inclusion and manipulation of the geographic and geometric components of the tabular 2D data is assured by the PostGIS extension, with respect to the Open Geospatial Consortium standards.The implemented database is then hosted on a shared web server.

Web platform implementation
The web interactive platform in support of the bridge inspection documentation process consists of 2 distinct components: an application for form filling and data collection, and a 3D viewer for the exploration of the structure digital twin.The first component is developed using Django, a free and open-source web-framework based on Python that allows the development of custom web app through its admin portal for database management that provides a user-friendly application in support of the frontend interface (Figure 4).The structure of the application with specifications on each table model, relation and field is accessible at the following GitHub repository: https://github.com/labmgf-polimi/brescia_bridges_demo.At the same time, a custom 3D web platform is developed, based on the open-source WebGL Potree (Schuetz, 2016).Such solution starting from native functionalities of the chosen JavaScript library embedded custom options to facilitate the 3D navigation also for users without any technical background in geomatics, making the identification of portion of interest on the structure and the evaluation of possible defects or damages more accessible.In this way, users can interactively explore online the 3D model, executing routine measurements, filter applications or cross-sections directly on the 3D model without installing any specific additional software locally (Figure 5) (Gaspari et al., 2023).For instance, thanks to dedicated features integrated in the main graphic user interface, users are guided to pre-defined views on critical elements of the bridge, exploring with simplicity 3D products referring to it, such as the classified point cloud, images that depict that specific element and georeferenced annotations that univocally label the portion of the bridge.
To facilitate and easily reproduce each functionality, a web platform template named PONTI (Potree platform for Infrastructure Inspection) has been defined to fulfill the requirements of managing authorities.PONTI already provides three primary customizable features: displaying the point cloud of the examined structure, uploading template-based images, and annotating significant elements of the structure.The template and instructions for its adoption can be accessed from a dedicated GitHub repository located at https://github.com/labmgf-polimi/ponti.Relying on open-source code and documenting the process of implementation also allows to easily update the existing viewer for each bridge with new 3D products.Indeed, following the documented instruction it is possible to autonomously integrate multiple point clouds from different epochs in a single Potree scene, allowing the evaluation of the geometry of the structure over time.

PILOT STUDY
The described methodology and workflow were applied to a set of 4 bridges in the province of Brescia, in Northern Italy, during inspections commissioned by the local road owning public authority.The structures chosen for the pilot study consisted of reinforced concrete bridges with 3 to 9 spans (Figure 6).After a preliminary visit to the sites for identification of obstructions such as vegetation or other man-made objects, the in-situ surveys were conducted.First, for each structure, a topographic network was materialized in a local reference system, measuring with a total station a series of targets positioned on the structure deck, piles and abutments, as well as on the surrounding ground.Some of such targets were then adopted as Ground Control Points (GCPs) for the photogrammetric processing of images acquired in field with nadiral and oblique UAV flights, as well as with handheld camera.This double approach of aerial and ground-based photogrammetry was needed because portions of the bridges, especially in correspondence of their abutments, were close to dense vegetation that could have compromised the operativity of UAVs.Nevertheless, it is essential to address the operative constraints that arise during commercial drone-assisted bridge inspections, particularly concerning obstacles encountered, maintaining safe distances from bridge piles and decks.This constraint often leads to limited proximity to the structures, potentially impacting the resolution and quality of acquired data.In parallel, a set of point cloud acquisition sessions were conducted using a terrestrial laser scanning.Then, measurement of a subset of target with a GNSS antenna allowed the following georeferencing of survey products in a common global reference system through a roto-translation.A schematic description of techniques implied, equipment adopted, and corresponding data output is illustrated in detail in Table 1.The images acquired with the photogrammetric surveys were then processed with a Structure from Motion approach for 3D model reconstruction (Westoby et al., 2012).The quality of the obtained point clouds was assessed evaluating errors and residuals on targets not used as GCPs and adopted as Check Points (CPs), checking the required centimetric precision.Hence, the use of both photogrammetry and TLS was needed for accuracy assessment reasons, comparing the output results of these techniques in the processing phase to identify possible inconsistencies.Such operations were conducted by computing cloud-to-cloud distances between the dense clouds using M3C2 (Lague et al., 2013).Once the accuracy requirements were verified, the two distinct clouds were integrated and subsampled.The integration of the different data sources was needed because some small portions of the bridges were visible only in one of the two products, due to the specific peculiarities of each technique.Instead, the 1 cm-spacing subsampling of the final point cloud helped reducing the file dimensions, reducing the occupied storage space on the server for the web platform implementation.

3D viewer preparation
Once the survey products have been processed, a preliminary data preparation phase is required before implementing the Web 3D Viewer.In fact, the resulting point cloud is classified to identify the structural macro-components of the bridges, such as spans, piles and abutments.Similarly, the images used for the photogrammetric reconstruction of the model are processed and converted to a compressed .jpegformat so that they are free of distortion and their orientation and associated camera parameters saved into separate text files.After converting the dense point cloud into a Potree-compatible .jsonformat with the PotreeConverter tool, the input data for the main functionalities and the basic definition of the 3D data exploration platform is initialised with the standard HTML structure, that includes both the viewer and the sidebar which contains the native tools for measurement and cross-section extraction.A simple colour classification legend with checkboxes for each class is then added to the sidebar, allowing filtering operations on the point cloud view to hide portions of the structure not relevant to the specific studies.The classified dense point cloud is set as the predefined view (Figure 5) but the option in the sidebar always allows to move to the RGB visualization.This basic web page definition contains all the elements needed for the filling of Level 1 geometric information of the structure, enabling the possibility to extract cross-sections and measurements of pile heights, deck widths and span lengths in text (.txt, .csvfile) or .lasformat (Figure 7).In order to support the a posteriori visual inspection and defect identification on the structure, other two functionalities are implemented.The first one is the inclusion of the images used in the photogrammetric process and then oriented on the 3D reconstructed model.To facilitate the exploration of the data and avoid a crowded 3D environment, the selected images are loaded in the scene as distinct chunks referring to different portions of the bridge.Such operation allows users to inspect the high-resolution photos taken during the photogrammetric survey, zooming on particular details and making measurements on their 2D projection plane, reprojecting them directly on the 3D model view (Figure 8).This functionality is particularly useful to identify in the images defects like cracks and measure their extension or coverage.In this way the user can document and also geolocate the deficiency both qualitatively and quantitatively and fill with such information the form defects requested by the guidelines.Moreover, by labelling each image with its file name, it is possible to associate each identified defect to the images in which it has been recognized, guaranteeing a procedure that allows for objective and transparent documentation of the photo analysis that could be evaluated by several operators in later moments.An additional support to the structural defect documentation in the a posteriori visual inspection is provided by the usage of geolocated annotations (Figure 9).Such entities are defined through a title and a set of x, y and z coordinates of the global reference systems in which the 3D scene is defined.Labelling and enumerating each structural element on the bridge help linking each 3D component to its corresponding filled defect form, maintaining a unique identifier in the connected database of the entire structure.Moreover, such features enable the possibility to include custom HTML content in the label annotation description, allowing the integration of multimedia attachments.For example, external links to cloud spaces with more images of a particular structural element can be inserted, as well as thumbnail images or other documents.
Eventually, a dropdown menu with shortcuts to predefined views on elements to be inspected is defined and integrated in the graphic user interface, guiding users in targeted analysis of the bridge portion of interest, and showing annotations and image chunks that refer to the element.

Web-app for form filling
The designed Django-based web application adopted for the case study was tested locally for prototyping purposes.The backend PostgreSQL database was hosted online on Heroku -AWS server.The graphic user interface for the pilot test case is based on the default Django admin panel.From the home page (Figure 4), once the users have selected the form to be updated, they are guided through the editing of each field facilitated by the adoption of dropdown menu for value selection in case of pre-fixed options.The operation is completed only when all required (not null) attributes are updated.In the case of Level 0, as illustrated in Figure 10, users can easily insert geolocation information through a web map widget that also help visualise possible gross mistakes in inserted coordinates.Additionally, through simple buttons, users can inspect the inspection or maintenance storyline of a single bridge, as well as add new documents.Instead, defect identification is performed after the selection of the structural element conditions to be documented by the web application users (Figure 11).Hence, users are asked to check a given list of defects, evaluating their occurrence, extension and intensity.Each observation needs to be supported by the visual evidence of defect.Indeed, photo filenames have to be inserted on a given field.It is important to highlight that the defect form template, with its list of defects to be checked by inspectors, varies for different combination of structural elements and material.For example, in the case of the choice of a reinforced concrete beam, the defects that should be checked are only the ones belonging to the group of the same material and of generic deterioration (defects that could affect different materials).
The interaction between the web Django-based application and the 3D viewer is currently possible through a field of the Level 0 form with a hyperlink to the web address where the viewer is located.In this way, inspection operators can conduct aposteriori visual analysis with the support of oriented images in the Potree environment, easily identifying the localisation of damages and imperfections directly on the digital twin.

CONCLUSIONS
The proposed software architecture provides a free and opensource flexible approach to digital multi-epoch documentation of bridge inspections according to the Italian guidelines.It supports users in form filling with simplified widgets such as value maps, sliders and checkboxes, while making 2D and 3D data complementary.In addition, the local prototype can be adapted for a multi-user approach on a shared server, allowing users to collaborate within the managing authority and promoting cooperation between professionals and operators with different expertise.Such an approach, which documents in digital format each intervention and maintenance procedure throughout the bridge's life cycle, also prevents the deterioration and loss of official documents and reports, making it possible to reconstruct, if necessary, the main events that have affected the regular serviceability of the civil infrastructure.However, some components of the implementation workflow of the methodology, such as the bridge annotation positioning and damage identification on oriented images, still remain timeconsuming, representing a challenge for their application.Future developments of the project concern a more robust level of integration and dialogue between the form filling platform and the 3D viewer of the digital twin of the bridge.For instance, a dynamic integration between Potree georeferenced annotations of structural elements and the corresponding form on the database is under analysis.Indeed, a visual classification of the element annotations with colours linked to the severity of the defects identified on it (e.g.green, yellow, red) could further support a comprehensive understanding of the structure through its digital twin.
In addition, the integration of computer vision techniques for semantic segmentation of the point cloud and the oriented images can reduce the time required to complete the workflow, while making the entire process more automated and reproducible for non-expert users.Such technique could help easily classify the bridge point cloud into distinct structural elements as well as identify damages of different type on the oriented images and consequently on the scaled 3D model.Eventually, the Potree-based platform can be furtherly employed for the comparison of the current state of the bridge with the original project design, including 3D of different formats (e.g., BIM representations).

Figure 1 .
Figure 1.Multi-level approach defined by the Ministry of Infrastructure and Transport Italian Guidelines for the first 3 level of risk classification of bridges and viaducts.As a result of the classification, different levels of surveillance of the artefact are adopted.

Figure 2 .
Figure 2. Workflow of the proposed methodology.

Figure 3 .
Figure 3. ERD schema for the designed database compliant with the multi-level methodology proposed by the Italian Guidelines (Level 0-2).

Figure 4 .
Figure 4. Homepage of the web Django-based application for the filling of forms and tables with information required by the Italian guidelines for bridge inspection and monitoring.The application provides a simplified user interface for the editing and update of a backend PostgreSQL database.

Figure 5 .
Figure 5. Web 3D interactive viewer with a classified point cloud of a bridge and a toolbox for executing basic measurements operations on the model in the Potree WebGL environment.

Figure 6 .
Figure 6.Aerial views of the reinforced concrete bridges inspected in the Province of Brescia (Italy) for the pilot case study.

Figure 7 .
Figure 7. RGB visualization of a bridge digital twin and cross-section extraction along the road axis.Users can extract and save the obtained profile in different formats.

Figure 8 .
Figure 8. Execution of measurements on dimensions of interests of bridge elements using Potree functionalities directly on drone images oriented on the 3D model

Figure 9 .
Figure 9. Annotations geolocated on the structure identifying structural elements of interest for the bridge risk evaluation in accordance with the Italian guidelines.If clicked or hovered with a laptop mouse, the annotations show a custom HTML box.

Figure 10 .
Figure 10.Example of editing of form for Level 0 of bridge inspection documentation with bridge geometry, inspection, and maintenance sections.

Figure 11 .
Figure 11.Example of form defect filling with the support of the Potree-based 3D viewer.viewer.proposedby the Italian Guidelines (Level 0-2).

Table 1 .
Geomatics survey techniques and associated equipment implied for the pilot case study in the Province of Brescia (Italy).