DOCUMENTATION OF CULTURAL HERITAGE THROUGH THE FUSION OF GEOMATIC TECHNIQUES. CASE STUDY OF THE CLOISTER OF "SANTO DOMINGO" (JAÉN, SPAIN)

: This study describes the methodology and results obtained for the modelling of the cloister of “Santo Domingo” located in the city of Jaén (Spain). The main objective of the application of geomatics techniques to this case was the graphic documentation of the site, the development of HBIM and the diffusion of the results. For this purpose, we developed a methodology based on the fusion of data obtained using several sensors, acquisition techniques and scales. In this sense, we used TLS, RPAS aerial photogrammetry and terrestrial photogrammetry using a spherical camera. As results, we obtained a 3D model with real texture and from this we developed several applications focused on the documentation and diffusion of the results, such as the BIM (HBIM), an interactive virtual tour and a 360 virtual tour available on a website. The proposed methodology has demonstrated its feasibility for performing this type of study due to the results and accuracy achieved with this application. We highlight the potential of current geomatic techniques for documenting this type of cultural heritage regardless of its structure and dimensions, selecting in each case the most appropriate methodology in relation to the scale of the products demanded. In addition, we also emphasize the need to use several techniques and the fusion of their partial products in order to obtain a complete product of the whole scene, taking advantage of the specificities of each one.


INTRODUCTION
3D heritage documentation has undergone a great evolution in recent years thanks to the development of geomatic techniques. The current availability of high-speed data acquisition and accurate sensors, mounted on remotely-controlled platforms, along with the improvements in hardware and software capacities, among other factors, has provided an enormous advantage for documentation purposes with respect to classic techniques. Parallel to these developments new necessities and purposes of this documentation have appeared such as virtual tours based on augmented reality (AR), which allow users to visit and interact with virtual scenarios from anywhere in the world.
Geomatic techniques applied to heritage documentation, and more specifically to heritage buildings at local scale (Lambers and Remondino, 2007), can be grouped into active and passive sensors (Remondino and El-Hakim, 2006). In the first case, we include Terrestrial Laser Scanning (TLS) and as passive sensors we highlight the use of low-cost cameras to develop photogrammetric surveys and more specifically based on Close Range Photogrammetry (CRP). On the other hand, this acquisition can be developed from the ground (e.g. terrestrial photogrammetry) using a platform to lift the sensor, such as masts (Mozas et al., 2012;Ortiz et al., 2013) and remotely piloted aircraft systems (RPAS) (Nex and Remondino, 2014;Colomina and Molina, 2014;Campana, 2017). Some studies (Remondino andEl-Hakim, 2006, Hassani et al., 2015) have analysed the advantages and disadvantages of these techniques taking into account the characteristics of the scene to be documented and the project requirements. Although these techniques can be applied independently, several studies have demonstrated the advantages of their integration (Kadobayashi et al., 2004;Alshawabkeh and Haala, 2004;Guarnieri et al., 2006;Grussenmeyer et al., 2008;Mozas-Calvache et al., 2020).
One of the common problems of heritage buildings is the existence of narrow spaces that make it difficult to apply traditional techniques with commonly-used sensors. In these cases some imaginative solutions have been successfully applied, such as the use of GoPro cameras (Fiorillo et al., 2016) and image capture from cameras supported on masts to survey inaccessible areas (Pérez-García et al., 2019). In the case of special sensors, we highlight the use of fisheye lens cameras (Covas et al., 2015;Barazzetti et al., 2017;Perfetti et al., 2017) and spherical cameras (Pérez-Ramos and Robleda-Prieto, 2016; Barazzetti et al, 2018;Fangi et al., 2018). On the other hand, some recent studies describe improvements in data acquisition efficiency using mobile mapping systems (MMS), such as those based on simultaneous localization and mapping (SLAM). These systems calculate their trajectory using point clouds or images (LiDAR SLAM and Visual SLAM), achieving accuracies of several centimetres (Di Stefano et al., 2021).
Recently, one of the main information systems related to heritage building documentation is Historic Building Information modelling (HBIM) (Murphy et al., 2009), which can achieve a great level of detail and accuracy depending on the level of geometry (LOG) and grade of accuracy (LOA) (Brumana et al., 2018;Banfi et al., 2019). In addition to the documentation of historic buildings, scan-to-BIM applications allow supporting other purposes such as virtual reconstructions, structural analysis, etc.
In this sense, one of the recent objectives of 3D documentation is related to virtual reality and more specifically to the development of interactive applications. Unity 3D is a platform for the creation of games, which allows the development of interactive content that can include 3D documentation of heritage. In this context, several studies have described the possibilities of this game engine and other platforms to immerse users (virtually) inside historical buildings (Kontogianni and Georgopoulus, 2015;Ferdani et al, 2020), both in the current conditions and using virtual reconstructions.

The cloister of Santo Domingo
The cloister of "Santo Domingo", located in the centre of the city of Jaén (Spain) (Figure 1a), was built in the second half of the 16th century, although it underwent some minor modifications later. The structure consists of a square floor plan of about 30 meters and an interior courtyard of about 20 meters. The cloister is composed of two floors reaching a height of about 10.5 meters with an additional sloped roof. There are some adjacent buildings, such as a church, which reach a total height of about 21.5 meters. The arches of the four galleries that limit them with the courtyard are semi-circular and are placed on paired columns of Tuscan order although in the four corners of the galleries these sets of columns are composed of three instead of two ( Figure 1b). The cloister is composed of 28 arches and, consequently, 60 columns. The galleries are covered by barrel vaults with lunettes. The upper floor contains alternating balconies and windows. These elements are surrounding by vegetal and allegorical decorations sculpted into the stone. In total, the cloister is composed of 12 balconies and 16 windows (Figure 1c). The courtyard contains four cypress trees of about 7 meters distributed in a diamond shape enclosing a decorative fountain. It should be noted that there were several difficulties for the acquisition of data that constituted a challenge to overcome in this study. Firstly, the galleries of the cloister contain vaults with lunettes separated by pairs of semi-circular arches that protrude from vaults and complicate the total coverage of the scene by the presence of occlusions. In addition, the ceiling and walls are plastered in white colour without a clear texture. This supposes an additional difficulty for photogrammetric orientation procedures considering the obtaining of orthoimages and photorealistic textures. The columns that compose the pairs and triplets that limit the galleries are quite close to each other leaving narrow spaces, which condition the acquisition from several points of view to guarantee a complete coverage. Secondly, the great abundance and high detail of the stone decorations located on columns, arches and around balconies and windows make documentation of these elements very difficult due to the need to capture them from several surrounding points of view in order to survey a full scene. Finally, the four cypress trees situated in the central area of the courtyard make it difficult to acquire data from distant points of view due to the occlusions generated by these obstacles. In addition, the adjacent church, which is higher than the eastern part of the cloister, complicates the execution of aerial surveys from this side.

Objectives
The main objective of this study is the development of a methodology in order to obtain a 3D documentation of complex buildings considering the integration of geomatic techniques to overcome the difficulties of the scene. As a secondary goal, we consider the development of products that facilitate the diffusion of results and the interaction of users using virtual models of the building.

Methodology
The geomatic techniques applied in this study have been conditioned by the conditions of the scene, the requirements of the project and the products demanded to document the cloister.
The methodology developed in this study, adapted from previous studies (Perez et al., 2021;Mozas et al., 2023), has been based on the fusion of data obtained through various sensors, acquisition techniques and scales ( Figure 2). To summarize, the methodology includes the use of TLS supported by RPAS photogrammetry and spherical photogrammetry. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-M-2-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy The methodology, summarized in Figure 2, is divided into three main blocks: data acquisition and fusion, products and applications focused on decision making and diffusion of the graphic documentation of the cloister: 1. Data acquisition and fusion: This stage is divided into two main steps. Firstly, the scene analysis for planning the data acquisition, the distribution and measurement of some surveying points in the courtyard to establish a common reference system to be applied for georeferencing all products and the materialization of some permanent targets distributed over the scene and used both to georeference TLS (control points) and to check the fusion results (checkpoints). Secondly, data acquisition using three techniques: TLS, RPAS photogrammetry and a spherical photogrammetry. The capture procedure, including the number and positions of the stations (scanning and photographs) is conditioned by two main aspects: the products demanded and the need for data fusion during the processing stage. In terms of products, the main demand was to obtain a photo-realistic 3D model of the cloister using TLS in order to obtain the scene geometry and the application of several photogrammetric techniques to complete this geometry, texturize the model and generate orthoimages.
Regarding data fusion, this process is developed from the beginning of data processing. This aspect involves the development of a previous planning stage for the acquisition of TLS and photogrammetry to ensure the successful orientation and fusion of all data. Data fusion ( Figure 2) is based on the extraction of keypoints from all datasets. In this sense, the data acquisition procedure is very important to facilitating the automation of the process, guaranteeing adequate results in subsequent processes such as TLS registering, image orientation, camera calibration and data fusion. Therefore, the procedure is not only focused on capturing all scenes with a certain level of quality, but it must also guarantee the data fusion considering the possible increase of stations (TLS and photographs) needed to cover the scene. Our priority is to reduce the processing time even if it means a slight increase in acquisition time.
After capture, the next stage is related to data orientation, which is also divided into two steps. Firstly, the registering and georeferencing of TLS data. This procedure is based on a minimum number of well-distributed targets obtaining a complete point cloud. From this point cloud we obtain a set of checkpoints (remaining targets) that will be used to assess the results of the orientation and data fusion processes. Secondly, we implement the orientation and fusion of the photogrammetric blocks composed of both sets of images, obtained from RPAS photogrammetry and spherical photogrammetry, with respect to the TLS point cloud. The strategy is based on considering the TLS point cloud as a spherical panorama associated to the scanning intensity and, on the other hand, the images obtained from the RPAS (frame photographs) and the images from the spherical camera (spherical panoramas). The intensity spherical panorama (TLS) and the spherical panoramas (spherical camera) are used jointly in the orientation process using the SfM algorithm (Ullman, 1979) by considering the positions of the intensity spherical panorama as fixed. The orientation is based on the extraction of keypoints and tie points between the images using the SIFT algorithm (Lowe, 1999). The fixed positions and orientations of the TLS data in the autocalibration process include similar constraints to those usually defined by control points. In this way, we obtain the absolute orientation of all photographs with a simple alignment of all images. After that, we measure a set of checkpoints (defined by targets) and compare their coordinates to those obtained from the TLS to assess the quality of the orientation, calibration and data fusion processes. 2. Products: After the orientation of the data, the methodology includes the fusion of the point clouds obtained from TLS and RPAS photogrammetry obtaining a definitive point cloud. Using this point cloud, the associated normal vectors and the oriented images we obtain several final products such as the textured 3D model and the orthoimages by projecting data to the planes of interest.
3. Applications for decision-making and diffusion: Finally, the methodology includes a final stage developed in order to implement applications that facilitate decision making related to the conservation of this building, such as the development of a BIM (HBIM) and others related to the diffusion of the documentation based on an interactive virtual tour and a 360-degree virtual tour.

Materials and application
The proposed methodology was applied to the cloister described in section 1.1. The project required a 3D documentation of the entire cloister and high resolution orthoimages of the floor and some walls. In addition, an analysis of the state of the columns, a simulation of different scenarios with different possible interventions to be implemented in the courtyard and the development of different tools to facilitate the online diffusion of the documentation carried out in the study were required. The location and characteristics of the site limited the use of GNSS in the courtyard although two surveying points were measured using this positioning technique, obtaining their coordinates with reference to the official Spanish coordinate reference system. From these points we obtained the coordinates of 4 well-distributed targets using a total station. These 4 targets were sufficient to geo-reference the point cloud obtained by TLS. In addition, we also materialized 31 targets used as checkpoints (see distribution of targets in Figure 3a). Regarding the planning and data acquisition processes, the cloister was divided into three zones (Figure 3b Data acquisition was developed simultaneously using the three techniques described in the methodology. The TLS survey was developed using a Faro Focus X130 laser scanner. The acquisition was configured to obtain an average spacing range of about 1-2 centimetres and the number of scanning stations was planned following the criteria described previously. In this way we performed 52 scans distributed over the scene following the scheme shown in Figure 4a. The RPAS photogrammetric survey was carried out using a DJI PH4-PRO V2 system. The positioning of the RPAS using GNSS-RTK was very complicated due to the characteristics of the cloister and the presence of trees. This caused the impossibility of developing a planned automatic flight. Consequently, we executed several manual flights following a pre-configured scheme. In zones A and B we developed conventional flights at different heights capturing vertical photographs following a rectangular scheme and several oblique flights, obtaining both inclined and convergent photographs and pairs of stereoscopic photographs. In this regard, we followed the CIPA recommendations (Waldhäusl et al.,2013) for the simple photogrammetric documentation of architecture. In the case of zone C, we developed a linear flight by obtaining vertical photographs at a certain height calculated in order to cover the width of the corridor with a single strip and considering the purpose of obtaining high-resolution orthoimages. In this sense, we obtained 832 images using this method (framed images). To complete the coverage of zone C, we developed a survey based on spherical photogrammetry using a low-cost 360-degree camera (RICOH Theta SC2).
Considering the limitations of these cameras (Herban et al., 2022), their use implies two main advantages in narrow and complex spaces, with reduced distances between the camera and the object. Firstly, the reduction of images with respect to those required using a conventional camera to cover the scene completely. This means a reduction in acquisition and processing time. On the other hand, this system allows the fusion of image and TLS data because the images from both sources have the same projection and are captured from similar positions. This facilitates the automatic identification and measurement of keypoints between both systems, improving the orientation of the block. Furthermore, considering that zone C was almost covered by TLS, spherical photogrammetry was only used to texture the 3D model. Therefore, the use of this type of camera was clearly justified. In this case, we obtained 68 spherical images from the 360-degree camera and more than 900 images in total (Figure 4b). The processing stage started with the TLS data using Maptek Point Studio software. This processing included a basic registration of point clouds that was automatically developed and a fine registration using the Iterative Closest Point (ICP) algorithm. The 52 scans were registered with a mean error of about 1.5 centimetres and a final residual after georeferencing of about 1.3 centimetres. After obtaining the final point cloud, we identified and measured all targets manually. Subsequently, the processing included the fusion and orientation of all images related to the three techniques applied (TLS, RPAS and spherical photogrammetry) following the procedure described in section 2.1. The orientation was developed using the SfM algorithm following a similar strategy to that proposed by di Filippo et al. (2022). We used Agisoft Metashape Pro 1.8 software to implement this procedure. After the orientation, the RMS of checkpoints measured from the RPAS and spherical photographs was about 1.5 centimetres. This value reflected the correct data fusion achieved in this study. Following the procedure described in section 2.1, after orientation we obtained several products using Agisoft Metashape software. Firstly, we generated the point clouds separately: one from the TLS survey ( Figure 5a) and another from the RPAS images ( Figure 5b). This second point cloud was limited to zones not covered by the TLS survey. After filtering and removal of gross errors (performed manually), we selected a set of points based on a certain confidence level (considering the number of images where each point is found). The final step consisted of merging both point clouds ( Figure  5c) where each point was associated with a normal vector (Figure 5d). In addition, we performed a filtering based on minimum distance between adjacent points of 2 centimetres. Using this point cloud and the oriented images we generated the required products, such as the 3D model and orthoimages. Finally, we developed the applications mentioned above. Firstly, a BIM-based application (HBIM) to document the cloister, analyse the state of the columns and perform several simulations on possible interventions and previous states of the building in the past. The inputs of this application were the 3D model, the point cloud and semantic information based on manuscripts and historic documents related to the construction stages and the conservation state based on the knowledge of experts. In addition, we also used several historic photographs of the cloister to analyse its configuration in the past. Using these data we developed the BIM using Autodesk Revit software. Firstly, we modelled all architectural components, saving them as parametric objects in a specific library and adding some properties based on the information collected for each element (Figure 6). For example, each object related to a column (supported on a pedestal) was enriched with information about its state based on three levels. With this parameter, we could simulate hypothetical interventions in the cloister once the BIM was finished, adding and removing modelled elements ( Figure 6). Secondly, we also developed two applications focused on the diffusion of the documentation. The first one was based on the use of a multiplatform game engine, more specifically Unity (REF), to develop a scenario where the user can interact with the scene in first person using the photorealistic 3D model. In this sense we simplified the model, added the environment and configured the illumination to simulate similar conditions to those of data acquisition. The aim was to generate a more realistic sensation. The application was compiled to be loaded in a web browser in order to facilitate the diffusion of this study. The second application was based on the images captured using the 360-degree camera. In this case, we created a virtual tour. In this type of application knowledge is needed about the image positions. Therefore, the calculated external orientation parameters (EOP) were used to establish the stations of this tour.

RESULTS AND DISCUSSION
The main results obtained in this study were the 3D model and the orthoimages. In this context, Figure 7a shows two views of the 3D model and Figure 7b show two orthoimages with five millimetres of spatial resolution of the floor (courtyard and corridor). To obtain the texture and orthoimages, we selected some images of the photogrammetric block and created masks to avoid overexposed or underexposed areas. The 3D model and the point cloud are reliable products used to develop the applications included in this study but they can also be used to provide other applications, such as those based on digital twins. The final model of the cloister included in the BIM is shown in Figure 8. In this application, we included part of the 3D model in the courtyard façades and the corridor walls as wall families in order to obtain a more realistic visualization of the BIM (Figure 8a and Figure 8b). In addition, Figure 8c shows the analysis of the condition of the columns and pedestals. Each column is represented by a colour (red, yellow and green) considering the qualifying value assigned about its state. Finally, based on the current state we have simulated the state at various dates in the history of the cloister, but also some possible interventions to be carried out in future (Figure 8d). Therefore, the BIM application allows us to manage a tool that facilitates decision-making when defining the strategy to follow, for example, for the restoration of columns. In addition, we can simulate the appearance of the cloister if we modify some non-historic elements, such as the removal of trees (Figure 8d).  Finally, Figure 9 shows some screenshots of the application focused on diffusion, such as the virtual interactive tour (https://coello.ujaen.es/proyectos/sft/stodomingoint/) ( Figure  9a) and the 360 virtual tour based on the 360-degree camera (https://coello.ujaen.es/proyectos/sft/stodomingo/) (Figure 9b). User feedback has demonstrated the potential of these applications to provide a complete and immersive experience to users regardless of their location thanks to these online tools.

CONCLUSIONS
The main conclusions of this study are related to the potential of current geomatic techniques for documenting this type of cultural heritage regardless of its structure and dimensions, selecting in each case the most appropriate methodology in relation to the scale of the products demanded. Furthermore, we also highlight the need to use several techniques and the fusion of their partial products to obtain a complete product of the whole scene, taking advantage of the specificities of each one. In addition, this study has shown that the use of TLS and photogrammetry (conventional and spherical) allows us to reduce and almost eliminate the need for surveying to obtain control points due to the complete and accurate orientation of images based on TLS. The combined processing of all data has also allowed us to perform a checking process of the results without using surveying because checkpoints were obtained from TLS. Spherical photogrammetry has revealed its feasibility in complex and narrow scenes for providing textures to generate the 3D model. In addition, the use of 360-degree cameras implies a reduction of acquisition time. In this sense, we suggest the location of acquisition stations in positions similar to those implemented in the TLS survey. The results obtained in this study suggest the need for this type of 3D documentation to cover all possible projects on heritage buildings such as simple documentation, diffusion, restoration projects, simulation of previous states or future interventions. In this sense, BIM has revealed its potential for covering these and other applications. On the other hand the online diffusion of graphic documentation based on interactive applications has shown great potential for the possibility of immersive tools being available to users regardless of their location all over the world. Future work will focus on the inclusion of handheld devices based on mobile mapping systems (MMS) in order to improve the efficiency of data acquisition and processing, but guaranteeing the accuracy of the results.