A RAPID PIPELINE FOR PERIODIC INSPECTION AND MAINTENANCE OF ARCHITECTURAL SURFACES

: The contribution fits into the framework of Digital Heritage, within the diagnostic process, leading up to restoration. Control and monitoring of architectural heritage are still open topics, with respect to requirements as simplicity, non-invasiveness and cost-effectiveness of both procedures and equipment. A peculiar gap concerns the accessibility to the artefact, which could be compromised by bad state of preservation, poor hygienic/sanitary conditions, or unsafe post-emergency circumstances, as well as by considerable extent, complex morphology, or incidental access conditions, due to the simultaneous presence of multiple working groups. In recent years, literature is moving towards an optimization of the investigation phases, by implementing digital technologies for the acquisition, elaboration and interpretation of 2D/3D data, to assess the state of conservation of a building or its main components. Besides, digital image processing and artificial intelligence are progressively rationalizing the analysis of the collected data. Furthermore, easy common devices, like spherical cameras or smartphones, have been introduced for the virtual reconstruction and representation of architectural environments, with the purpose of assessing their morphology and conditions. However, there is still an absence of harmonized standard procedures. To address these issues, the paper proposes a rapid pipeline, involving easy-to-use devices and expeditious procedures, to remotely assess, quantify, and monitor the extension of surface decay of historical buildings. A workflow based on the use of 360° images and videogrammetry has been defined and tested on a representative case study, both for its cultural value and for its limited accessibility, demonstrating a great suitability to the topic.


Introduction and Literature Background
In the digital Cultural Heritage domain, recent/continuous developments and efforts concern the optimization of the diagnostic process, leading up to restoration. Control and monitoring of an artefact are quite crucial and open topics, with respect to specific requirements as simplicity, non-invasiveness and cost-effectiveness of both procedures and equipment. Particularly, when dealing with historical buildings, multiple criticalities might occur, which could reduce the operational feasibility, along the different phases of the process, entailing onsite operations and/or direct contact with the target. In this context, a number of gaps arises over others. First of all, the accessibility to the artefact, for the data collection, could be compromised by various factors, such as a bad state of preservation, poor hygienic/sanitary conditions, or unsafe postemergency circumstances, as well as by considerable extent, complex morphology, or incidental access conditions, due to the simultaneous presence of multiple working groups. In recent years, literature is moving towards a rationalization of the investigation phases, by implementing digital technologies for the acquisition, elaboration and interpretation of both two and three-dimensional data, with the aim of assessing the state of conservation of a building or its main architectural/structural components (Delpozzo et al., 2022), (L Barazzetti et al., 2022). Image or range-based techniques, like aerial or close-range photogrammetry and terrestrial laser scanning, on the one hand, are being increasingly widespread, improving the acquisition of reality-based geometrical and decay survey data. Besides, digital image processing and artificial intelligence are progressively growing, to rationalize and simplify the analysis of the collected data. Until now, machine/deep learning has been exploited, to recognize and evaluate damage patters, but mainly on 2D images, with the necessity of providing large amounts of training samples, and the lack of quantitative insights about threedimensional complex structures (Chaiyasarn et al., 2018) (Adamopoulos, 2021) (Hatir et al., 2020) (Bruno et al., 2023). In view of further simplification, easy or common tools, like spherical cameras or smartphones have been introduced, for the virtual reconstruction and representation of architectural environments. In addition, the implementation of Augmented or Virtual Reality (AR/VR) provides novel perspectives, both for documentation, knowledge and conservation purposes. Bekele et al. (2018) realized a comprehensive overview of digital technologies in Cultural Heritage, with particular attention to augmented and virtual reality, in terms of acquisition techniques, devices, systems or aims (Bekele et al., 2018). For example, in some cases 360° panoramas are integrated within 3D digital models, for the virtual documentation of museum collections (Nespeca, 2018). AR/VR are identified also as a faster and low cost alternative with respect to reality-capture techniques (photogrammetry and laser scanning), when the lasts are not viable, for multiple documentation purposes enclosing damage analysis (Lasorella et al., 2021) (Napolitano, 2018) (Trizio et al., 2021). Conversely, research is directed also to the retrieval of immersive virtual environments from laser scanning or photogrammetric 3D reconstruction, in order to operate a discrimination between two levels of detail: low definition for dissemination or fruition, and high definition for conservation (Valzano et al., 2018). Immersive environments (like the Immersia platform or the software Conceptouring) have been used also for the manipulation of 3D data, for the extraction and visualization of information (Barreau et al., 2017), (Trigilia et al., 2018). De Fino et al. add non-destructive diagnostic tests results into VR environments or photogrammetric models, within a complete diagnostic workflow, at increasing levels of detail from preliminary to in-depth knowledge (De Fino et al., 2018) ( Cantatore et al., 2020). Their results led to the definition of an immersive digital platform VERBUM, for the collection and management of a plurality of information . Moreover, Napolitano et al. (2019) used image-based documentation and augmented reality/virtual reality tools for structural health monitoring scopes, with the manual annotation of degradation maps on 3D data in virtual environments (Napolitano et al., 2019). For the benefit of simple and expeditious procedures toward data collection, a further strategy is emerging, which entails rapid smartphone video-acquisition, for the retrieval of multi-image 3D reconstructions (Sirmacek & Lindenbergh, 2014), (Repola et al., 2018) (Sun & Zhang, 2019), with the purpose of assessing their morphology and conditions (Torresani & Remondino, 2019) (Luigi Barazzetti et al., 2020) (Costantino et al., 2022). Indeed, in recent years, accurate 3D reconstruction of heritage assets has been realized, through the use of high-resolution smartphone-based videogrammetry, which demonstrates the suitability of this kind of output, with respect to heritage documentation (Murtiyoso & Grussenmeyer, 2021), (Ortiz-Coder & Cabecera, 2021a), (Ortiz-Coder & Cabecera, 2021b). Another line of research focuses on the implementation of supervised/unsupervised machine learning systems for the automatic segmentation or classification of 2D or 3D outputs, such as, on the one hand, UV maps, orthoimages or textures, and, on the other hand, point clouds. These works exploit the colour/light information enclosed in image outputs or in 3D data, for multiple purposes, like assessing the general state of conservation of a surface (like a wall) (De Fino & Fatiguso, 2020), the constructive elements and materials, and the masonry patterns (Eleonora Grilli & Remondino, 2019) (E. Grilli et al., 2018); or the classification of some forms of chromatic alterations on masonry surfaces (Malinverni et al., 2017) (Galantucci et al., 2020), (Musicco et al., 2021), (Valero et al., 2019) (Gong et al., 2021). However, there is still an absence of harmonized standard procedures, both in terms of acquisition, reconstruction and analysis of 2D/3D data, finalized to the condition assessment and monitoring of an artefact. Furthermore, it is worth to restate the need for simplified and rapid methodological pipelines, in order to overcome all the eventual criticalities related to the accessibility to the heritage.

MATERIALS AND METHODS
Based on the outlined literature background and related insights, the paper aims to address the above-mentioned issues, by proposing a simple and rapid pipeline, involving easy-to-use devices and expeditious procedures for onsite acquisition. This might enable to remotely assess, quantify, and monitor the extension of surface decay of historical buildings based on reliable digital replicas. The objective is to propose a straightforward and effective strategy to collect and manage data related to the damage assessment and monitoring of an architectural heritage, in a medium/long term perspective. To reach this goal, a processing workflow has been defined, based on the use of 360° images and videogrammetry, realized with spherical cameras and standard smartphones. Spherical/equirectangular images or photogrammetric polygonal meshes and high-resolution textures (retrieved from 4K videos) constitute a starting baseline, to perform a remote semi-automatic decay mapping process, with the purpose of rapidly derive qualitative and quantitative data about the decay patterns. In the end, an expeditious control and monitoring phase could be periodically implemented on spherical images, by acquiring and comparing the same outcomes throughout the service life, in order to support decision-makers on suitable conservation strategies. In detail, the methodological approach is focused on the role of virtual environments and photogrammetric 3D data, in support of the assessment, mapping and interpretation of decay evidence, within the building diagnostic process, in order to reach a consolidated level of knowledge about the artefact. The procedure is oriented to the creation of innovative protocols for the restitution of onsite conditions, with reference to their spatial and morphological configuration, in order to understand the origin and evolution of a phenomenon, like for example a static instability or a humidity problem. Indeed, the possibility to survey and classify pathologies through quantitative metric, qualitative comparison and optical information contained in equirectangular images and 3D data, entails a critical understanding of damages, and involves a simplification and rationalization of the diagnostic process and of the plan of investigations. Spherical/equirectangular images and coloured point clouds represent the input for the application of digital image processing and machine learning routines, with knowledge purposes, oriented to analyse the state of conservation of architectural heritages. The proposed workflow ( Figure 2) is articulated into four macrophases: • On-site survey/data acquisition (360° panoramas, 4K videoacquisition); • 3D reconstruction through elaboration of 4K video frames by Structure for Motion (SfM) and Dense Multi-View 3D Reconstruction (DMVR) algorithms; • Automatic decay mapping on both 3D meshes and 360° images, based on texture-based supervised machine learning systems and resulting in quantitative (textures) and qualitative (panoramas) assessment of surface alterations. • Periodic monitoring of decay evolution, based on comparison of 360° panoramas overtime, in terms of deviation and percentage variation of alteration patterns. The four macro-phases are detailed below for a representative case study.

Building typology and Case Study
The workflow illustrated in Figure 2 is conceived for an expeditious monitoring especially of interior environments of historical buildings, like convents, noble residential palaces, farmhouses, or underground areas of archaeological sites. They are often characterized by several rooms/areas, where a bad conservation state, with widespread humidity problems and sometimes also static instabilities, requires the onsite activities to minimize the permanence of operators in the site and maximize the reliability of the acquired information. The proposed pipeline properly fits to this kind of situations, because of the simplicity of usage of the equipment and the rapidity in the acquisition, which represent an advantage for the surveyors safety, avoiding a prolonged stay in difficult to access, unhealthy or dangerous places. For these reasons, the workflow has been applied and tested on the Cryptoporticus of the archaeological site of Egnatia (Fasano, Brindisi, Italy) an ancient Messapian, Roman and later Paleo-Christian city, located near the seaside, dating back to the I century B.C (Figure 1). It is an underground structure, with four arms of different lengths, partially excavated in the rock and partially realized with opus incertum (cement, stone and mortar), covered with plastered barrel vaults. On the inner sides of the arms, openings are located at regular distances; two pairs of entrances are placed on the longer side. The structure can be interpreted as covered ambulatory or as deposit for cereals (horreum). The whole structure of the Cryptoporticus is considerably affected by the presence of water and the lack of air and light, which led to extended humidity patterns. For this reason, parts of the arms of the structure have been analysed, by applying the expeditious assessment workflow.

Figure 2. Methodological workflow
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-M-2-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy

On-site survey/data acquisition
The methodology starts with a first phase, concerning the acquisition of 2D/3D data, through 360° spherical cameras and photogrammetry, in order to obtain virtual three-dimensional reconstructions of the architectural environments, in the form of: • Equirectangular/360° images • Coloured point clouds • Texturized polygonal meshes.
As far as the 360° panoramic views are concerned (Figure 3), the acquisitions have been planned starting from previous graphic drawings of the analysed asset. In detail, the elaborated sphericalimage scheme defines: i) the station point (Xs), according to the coordinates (x, y, z) of a local reference system; and ii) the number of stations, in elevation and plan. In addition, the grip point acquisition plan is elaborated according to an appropriate overlap between consecutive spherical images, both of the same environment and of contiguous environments. However, for each environment, according to a local coordinate system, a known point of coordinate (0,0,0) has been identified (e. g. the corner of a masonry). Hence, with respects to the identified point of origin, it has been calculated and associated the correspondent grip point (Xs). This approach is particularly useful for the periodic monitoring of the state of conservation of the building under investigation as discussed in 2.5. Consequently, spherical images have been acquired with a Samsung Gear 360 camera, mounted on a tripod, and controlled from a mobile device, through the use of a specific app (Street View). Furthermore, for the photogrammetric pipeline, an expeditious procedure has been privileged, which entails the use of a standard smartphone (Android/IOS), which can acquire video at a 4K resolution. For each selected environment, a video acquisition has been realized, following a specific itinerary, in order to reproduce, at a quite constant velocity, the whole environment, vaults and walls included. The path followed parallel paths with respect to the walls, at different heights, and two paths looking towards the vault, in order to cover the whole arm, within a reduced time interval (only few minutes). The acquisition campaign has been realized in 2019, concerning both spherical/equirectangular images and video data of the four arms of the cryptoporticus. However, given the low-lighting condition, guaranteed only by the entrance to the underground structure and a few small openings, in order to realize the whole survey, supplementary diffuse LED lights have been used.

3D reconstruction from videogrammetry
Starting from the image dataset, in form of 4K videos, a frame selection has been performed on each video, based on the frame extraction rate (or frame step), namely the number of frames to be extracted in one second. This choice has to be made on the basis of a preliminary inspection of the recorded data, considering the duration of the video, and the velocity of the acquisition, in relation to the extension of the environment, which directly affects the degree of overlapping, and thus the number of needed images to properly cover the entire structure. Video-images/frames have been used as source images, to reconstruct the 3D photogrammetric model of each architectural unit, according to the photogrammetric process (Luhmann et al., 2006). The adopted software was Agisoft Metashape (Agisoft LLC, 2019).
As already mentioned in section 2.2, the outputs of the photogrammetric process are: • Point clouds, unordered sets of points characterized by 3D coordinates, normal vectors, and chromatic information. • Texturized polygonal meshes, 3D models in which surfaces are sub-divided into regular polygons, enriched by highresolution textures, reproducing the photorealistic aspect of the artefact. The reconstructed dense point cloud of one wing is made of about 220 million of points, providing a ground resolution of about 0.5 mm/pixels; while the mesh models are characterized by 2 million faces (Figure 4).
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-M-2-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy

Automatic decay mapping
The outputs of the acquisition phase, and, particularly 360° panoramas and high-resolution textures, represent the main inputs for an expeditious automatic decay mapping approach, through a state-of-art texture-based supervised machine learning systems, following the procedures described in (Eleonora Grilli & Remondino, 2019) (E. Grilli et al., 2018) (De Fino & Fatiguso, 2020. In particular, a supervised segmentation, based on the application of random forest, has been performed to equirectangular images, on the one hand, and on photogrammetric-based textures, on the other hand, to detect various alterations on chromatic basis. The first step concerns the scaling of the 2D output, which then can be imported in the supervised classification system, in order to realize a manual annotation of representative samples (i.e. regions of pixels), for all the decay classes. Indeed, a number of classes has been identified, which corresponds to the different typologies of alteration (i.e. moist area, biological patina/colonization, soiling,…). In addition, in order to get a better classification, further categories have been identified, apart from the decay morphologies, like for example, unaltered surfaces, soil, background (elements that do not enter in the classification) ( Figure 5). The manually annotated samples and labels were necessary for the following training of the supervised classifier. At this stage, a feature selection has been performed, entailing the extraction of meaningful features (Gaussian blur, Sobel filter, Hessian, difference of gaussians and membrane projections) for the recognition of the classes. After the training, the segmentation has been achieved through the application of random forest to the non-labelled data, whose result is a pixel level clustering of the 360 panoramas/highresolution texture, with the automatic labelling of the remaining pixels of the image by the algorithm, on the basis of the previously identified classes (Breiman, 2001) (Witten et al., 2011). Fast Random Forest has been chosen as classifier, on the basis of the results of previous research works, which identifies it as the most performing among a selection of different machine learning classifiers (Eleonora Grilli & Remondino, 2019), (Adamopoulos, 2021). Hence, the outcomes are represented by classified 2D equirectangulars/high resolution textures ( Figure 6 and Figure 7), which can be visualized within a virtual immersive environment (for the 360°), for a qualitative observation of the damage distribution, or re-projected in a three-dimensional environment (for the high-resolution textures), for a quantitative measurement of their extension.

Figure 6
Classified texture of one wing of the cryptoporticus (top), with the isolation of two classes (moist area in yellow and biological patina in magenta) The classified scaled texture led to the separation and quantification of the classes, corresponding to the categories of alterations, both in terms of percentage and area. For example, for one of the wings the classes have been identified as follows: moist area: 142 m 2 (60%), biological patina: 55 m 2 (26%), soiling: 38 m 2 (16%). For the development of the 360°/texture-based segmentation, Weka engine has been adopted, through the software add-on embedded in Fiji (Image J) (Witten et al., 2011) (Arganda-Carreras et al., 2017.

Monitoring
The last phase of the methodological workflow entails a rapid procedure to control and monitor the evolution of the detected morphologies of decay over time. This approach is conceived by performing a new 360° acquisition, maintaining unaltered all the parameters and locations of the first campaign (as illustrated in section 2.2), in order to retrieve almost identical panoramas, to be subjected to the same supervised machine learning system. In this case, the classifier is already trained. Thus, the classes should be the same, expect from some new alterations that have arisen in the considered time interval. The classified regions are quantified in terms of percentage, so that it is possible to compare the same regions and verify whether some percentages have varied, meaning that some decay morphologies have modified their aspect/extension. As far as the equirectangulars are concerned, this kind of quantification is relative, and it is useful to highlight changes between two situations, within a time interval. However, given the strong deformations to which the object represented in the image are subjected, especially in side areas, it is not possible to retrieve an absolute quantitative information about the damage. Instead, in particularly critical conditions, new videogrammetry acquisition could be performed. In this case, high-resolution textures of the new photogrammetric model can be processed, following the same pipeline described in section 2.3 and 2.4. The results, in form of classified textures, with quantified alterations, are collated and compared with the previous ones, in order to get a more precise and absolute information about the distribution and extension the detected phenomena.

DISCUSSION AND CONCLUSIONS
The assessment of the state of conservation of heritage buildings and sites is paramount to identify the actions to be undertaken according to the well-established principles of compatibility, low intrusiveness and reversibility. As a matter of fact, depending on the origin, magnitude and severity of the occurring pathologies, technicians and decision-makers are asked to set a preservation strategy that might concern both prompt treatments and periodic monitoring. Particularly, the periodic monitoring might be preferred if further investigation is required before selecting the most suitable interventions, if the pathologies are supposed to be caused by factors that are not acting anymore, or if the treatments must be scheduled according to a multi-year maintenance programme. Nevertheless, monitoring activities might be required even after prompt treatments are carried out, in order to assess their medium-long term durability. Consequently, the standardization of the condition assessment and control processes is highly desirable, especially if applied to datasets by different operators in different periods for different purposes.
The described workflow has meant to address the abovementioned issues, by the identification of operational microphases -on-site survey; 3D reconstruction; automatic decay mapping; periodic monitoring -that could be ultimately run by different specialists throughout data collection, processing and interpretation during the building service life. Nevertheless, taking into account that the assessment and control are frequently needed for endangered structures with restricted accessibility, the proposed tools are meant to minimize the "onsite" permanence of operators and maximize the reliability of the acquired information for "on-desk" elaboration. To this end, easy-to-use devices and expeditious procedures for survey, including 4K videos and 360° panoramas, provide with a reliable baseline for objective and remote examination. In fact, 4K videos enable the collection of images, with reduced recording times, very dense overlaps between frames and often a larger depth of field compared to pictures, leading to 3D models where alteration patterns are automatically mapped and quantitatively measured. Furthermore, 360° panoramas, still processed through supervised machine learning systems for decay mapping. are used for qualitative comparison of decay evolution over time, based on a quantitative reference scenario.