BRANCACCI CHAPEL IN FLORENCE: SURVEYING AND REAL-TIME 3D SIMULATION FOR CONSERVATION AND COMMUNICATION PURPOSES

: In 2020 to diagnose and monitor the condition of the frescoes in the Brancacci Chapel in Florence, the CNR initiated an important diagnostic and three-dimensional survey of the monument. These restoration preparatory activities have provided the opportunity and data for the creation of a real-time 3D simulation model that will not only help conservators and art historians better understand and manage the artwork, but will also enable the planning of improvement measures and the communication of the fresco cycle and its state of preservation to the public via the Brancacci POV application. Brancacci POV is a hybrid multi-user experience that allows a group of tourists to visit the chapel under the guidance of an expert, either on-site or off-site. The Web3D application is based on ATON, a framework developed by CNR ISPC that integrates 3D simulation of the monument and the discovery of invisible layers (e.g., ultraviolet data) processed and optimised for real-time rendering. The article illustrates all the steps performed from the 3D survey to the Web3D implementation to obtain the 3D simulation and achieve the expected interactive result. In particular, we focus on the optimization workflow of the reality-based model developed for this case study and on the processing of high-resolution maps (visible and invisible layers) for real-time web interaction.


INTRODUCTION
Surveying the shape and appearance of buildings, especially those of high historical-artistic value, in addition to the typical characteristic purposes of geometric description of architecture, can also serve the dissemination of Cultural Heritage, especially when carried out with newer methods. Considering the high implementation costs and the documentary value of the collected information, the metric and diagnostic data obtained in survey campaigns for restoration projects should be at the same time the object of effective processes, possibly already in the research planning phase, that make them useful to the public both in the results and in the languages. This ideal-type assumption collides with organisational/design and technical difficulties. The answer to the former may come from the more widespread practice of viewing data themselves not only as specialised but also as narrative tools. The response to the technical difficulties, on the other hand, necessarily has a more articulated response related to the nature of data, its resolution, and the still "artisanal" development of the practices that enable its full unfolding. The first aspect to consider concerns shape's survey. If in architectural language this is determined by scale factors and the resulting graphical errors, which in turn determine the choice of instruments and resolution, the use of measurements for dissemination purposes on the one hand allows for more informal decisions about the density of data, also through computer graphics "artifice" (i.e., normal mapping, roughness), but on the other hand forces the use of slender geometries as a counterpart to allow for ease of use, for example, via the web. The second issue to be solved, in order to achieve full reuse of diagnostic data in Cultural Heritage dissemination projects, concerns the extreme heterogeneity of data and their use to create UV maps. While it is true that diagnostic imaging techniques are now widely used, their use to generate textures on the one hand * Corresponding author and their dialog with three-dimensional surveying, on the other, is an unexplored field that still leaves room for experimentation and hypothetical formulations, both technical and methodological.

CASE STUDY RELEVANCE
In the first decades of the 1200s, the Carmelites settled in Florence in the area of Oltrarno. In 1268 they began building their church, Santa Maria del Carmine (Giovannini and Vitolo 1981). Agnese, the widow of Cione di Tifa, decided to donate a sum of florins and land for the construction of the church and the monastery. The construction work continued through the 1300s and 1400s until April 1422, when the church of Santa Maria del Carmine was consecrated. In the right transept of this church is located a masterpiece of art of all times, with which begins the period of the first Florentine Renaissance in painting: the Brancacci Chapel (Procacci andBaldini 1984, Rocca andRossin 1991). It was painted by Masaccio and Masolino and completed in the early 1480s by Filippino Lippi. Dedicated to the Madonna del Popolo (of the people), whose image is venerated above the altar, it remained under the patronage of the Brancacci family from the second half of the 14th century until 1780, when it passed to the Ricciardi family. In 1367, Piero Brancacci bequeathed 5 gold florins per year for the construction of the family chapel in the church, even if the building was erected only about 20 years later. In 1423, the merchant Felice Brancacci commissioned the frescoes of the chapel. The cycle of frescoes represents Historia Salutis, the story of the Redemption. It is introduced on the two large access pillars by the scenes of Original Sin and the Fall, and continues inside the chapel with the stories of St. Peter. After Masolino left for Hungary and Masaccio for Rome, where he died in 1427, the frescoes in the chapel remained unfinished. It was not until the 1480s that the fresco cycle was completed by Filippino Lippi.
The Brancacci Chapel was in danger of being lost several times. Finally, in the fire of 1771 (Procacci 1932) the interior of the church was almost destroyed. On the other hand the Brancacci Chapel was preserved, although it underwent colour changes due to the heat of the flames. The restoration of the Brancacci Chapel began in the second half of the sixteenth century, with a sequence of minimal interventions until in 1981 the Opificio delle Pietre Dure was commissioned by the Consiglio Superiore di Antichità e Belle Arti e della Soprintendenza per i Beni Artistici e Storici of Florence to carry out preliminary tests for restoration. A nine year (1981)(1982)(1983)(1984)(1985)(1986)(1987)(1988)(1989)(1990)) restoration campaign began (Zorzi 1992). On this occasion, many interventions were done, including the removal of the Baroque altar, the detachment of the sinopias of two lost semi-lunettes by Masolino and Masaccio, now preserved in the Room of the Column (Sala della Colonna), and the removal of superficial deposits that dulled the colours. To integrate and support the restoration work, many scientific analyses were run with cutting-edge techniques and instruments and called upon to collaborate on the conservation of this human genius and creativity example. In the autumn of 2020, the fall of fragments of pictorial plaster on the floor of the Chapel was reported by a custodian. This event highlighted a problem with the painting layer of the right wall realised by Filippino Lippi. Scaffolding was assembled right away to assess the situation. Critical issues related to the detachment between the plaster and the masonry were reported immediately. Thirty years after the last restoration intervention, it is today necessary to review the state of conservation of the frescoes, using innovative techniques capable of adding information on the state of conservation and on the executive techniques (figure 1).

AIMS
During the diagnostic campaign, led by CNR ISPC in collaboration with Opificio and the Superintendency of Florence, various metric and diagnostic acquisitions were made using numerous technologies and methods (photogrammetry, TLS, ultraviolet, Raman spectroscopy, etc.). The result was a large 3D digital dataset that helped both conservators and art historians to better understand and manage the artwork and, subsequently, the visitors to discover the monument, especially the inaccessible parts, and see invisible layers (e.g. ultraviolet data) This paper presents the solutions to the challenges encountered in creating such a 3D simulation. Specifically, it presents the followed workflow, from surveying to 3D modelling for a realtime simulation environment and implementation in a web3D application: Brancacci Point Of View (POV).
Brancacci POV is a hybrid multi-user experience that allows a group of tourists to visit the chapel under the guidance of an expert, either on-site or off-site. The application is designed to enhance the perception of authenticity of the virtual visit and to create a sense of wonder through the exchange of ideas and comments among participants. Users can participate in the experience using smartphones, tablets or PCs, or become part of the event in a more immersive way using an HMD (Meta Quest). The use of the immersive VR experience was also proved to be of high interest for the professionals working in the monitoring and restauration campaign. They in fact can immediately compare the 3D simulation of the monument with its visualisation in Ultraviolet. The following paragraphs describe the pipeline used to create the digital asset, while the Brancacci POV experience is not the object of this work.

MATERIALS AND METHODS
The following is a detailed description of the workflow as summarised in figure 2, including all steps, tools, software, and workflows used to achieve the expected results.

3D Survey
According to recent positive experiences (Bruno et al 2022, Luhmann 2020, it was decided to perform a survey by integrating active sensors, terrestrial laser scanners (TLS) and passive RGB photogrammetry (IBM), in order to mitigate the limitations of the two techniques while enhancing their respective strengths. The second methodological decision was to survey not only the Brancacci Chapel, but the entire Santa Maria del Carmine. Although the project involved only one part, the chapel, it was deemed prudent not to separate the chapel from its architectural context with minimal effort, hopefully allowing other parts of the monument to be included in the real time experience in the future. Accordingly, the initial documentation effort at Santa Maria del Carmine, as shown in figure 3, included photogrammetric and TLS surveys of the entire interior of the complex. The combination and integration of different sensors and techniques was considered the ideal solution for an efficient capture and 3D representation of the chapel (Remondino 2011, Fiorillo et al 2021. This rapid acquisition activity for the TLS part was carried out with a Leica BLK 360, with which 50 scans were taken, while for the photogrammetric acquisition a Canon EOS 6D with optical 35 mm was used, with which 600 images were taken from the ground at 5000x4000 pixels.

Figure 2. Pipeline graphical flowchart
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-M-2-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy Laser scan registration was performed using the C2C method within Leica Register 360, resulting in an average RMS of 6 mm. Integration between image-based and range-based data was then computed with Reality Capturing Software using a camera alignment approach based on synthetic images generated from the registered TLS point clouds (Grassi et al 2022). Matching the cube maps produced by the laser scans with the photogrammetric survey of the chapel proved to be much more problematic than in the rest of the church because of scaffolding that had not yet been erected during the photogrammetric survey of the chapel. However, by using the decorated surfaces in the intrados of the vault, it was possible to achieve a solid match between the earlier photogrammetric survey and the new TLS acquisition. The result of the processing was a very high-resolution model (average edge length 4 mm, texture texel size 2.5 mm), consisting of over 350 million of triangles, that required intensive postprocessing to improve its appearance and to be used interactively for an immersive experience in a web browser using the ATON framework.

Model processing and optimization
Once completing survey and 3D data processing, the next steps were related to the optimisation of geometry and textures. Firstly, the model was decimated to 1 million point in RC and exported in local coordinates as *.obj file. Given the presence of topological issues in the mesh, like small holes, non-manifold edges and overlapping faces, a first optimisation was performed in Meshlab and Blender software, using automatic filters and some manual intervention, especially for filling holes with complex edges or to model geometries not reconstructed by the digitising process, due to lack of data caused by shadow areas or occluding geometries. Subsequently, the model was processed using Instant Mesh. Instant Mesh is a software that uses unified local smoothing operator to remesh the surface optimising both the edge orientations and vertex positions in the output mesh (Jakob et al 2015). This allowed to compute a new low-polygon model, while maintaining the original shape and border and regularise the topology, obtaining a pure quad mesh. This new mesh obtained was easier to edit manually in computer graphics software than the previous triangular meshes, generated by the photogrammetric software and used to process the survey data. The following step was performed in Blender. The model was segmented into several parts, according to the sections of the frescoes, to better control the portion of the model assigned to a single texture and, above all, the creation of the UV coordinates for the texture mapping, the process in which the software applies 2D images to a 3D model to simulate the colours and make it photorealistic (A-A1 fig. 4). The process was carried out manually by selecting the edges as seam lines and organising the UV islands in the UV space In this way, it was possible to control the texture space allocated to the different parts of the model, fill up the UV space as much as possible, and to give higher priority to the areas of greater interest than others, such as the frescoes (weighted detail process). This texture mapping was performed with both RGB and Ultraviolet texture, which share the same UV space, to achieve a perfect match between the two layers, which are experienced interactively within the application using a special tool (see infra). In order to generate the textures, two different approaches were chosen according to the photographic data (RGB or Ultraviolet) (figure 6): • RGB: once the UV coordinates were created, the models were exported in .obj format and re-imported in RC for texture building by keeping the created UV coordinates. During this process it was possible to combine the different cameras (selected by the user according by coverage) to automatically generate the final textures. • Ultraviolet: Since a photogrammetric model for the ultraviolet images was not available, the texture building was carried out manually in Photoshop using the RGB texture as reference for the matching.
Finally, the separated meshes have been joined in one single mesh (500.000 polygons) with 20 materials (one for each 8K texture) at the end of the process and exported in gltf format (Robinet et al 2014) for the final interactive presentation in the Web3D application ( figure 6).

Web3D application
The final goal of the workflow was to interactively experience the 3D model of the chapel through a Web3D application (Brancacci POV) -accessible on mobile, desktop and immersive devices. This web application was based on the open-source ATON framework (Fanini et al 2021), developed by CNR ISPC. The framework is designed around robust web standards (such as The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-M-2-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy glTF (Robinet et al 2014) and large open-source ecosystems (such as Three.js: https://threejs.org/ and Node.js: https://nodejs.org/). ATON offers institutions, museums, labs, researchers, and developers a scalable and modular solution to craft and deploy cross-device (mobile, desktop, immersive 3-DoF/6-DoF devices) Web3D/WebXR (González-Zúñiga and O'Shaughnessy 2019) applications targeting the Cultural Heritage field (Barsanti et al 2018, Turco et al 2019, Fanini et al 2022. It provides advanced features in terms of 3D presentation, annotation, immersive interaction and real-time collaboration. Its modularity and architecture allow to quickly develop and then deploy custom interactive experiences (web-apps) with specific requirements, without any installation needed for final users. The Web3D application had to face several challenges, including the immersive VR experience of high-detailed frescoes and realtime inspection on mobile devices with limited resources. This was possible thanks to the development of on-demand loading and unloading of higher resolution maps at runtime, depending on both user proximity and orientation. The spatial segmentation was based on the texture UV-mapping of frescoes scenes, described in previous sections. Furthermore, for higher resolution maps (closer inspection) a second texture with ultraviolet data is dynamically requested. This enables the interactive discovery of invisible layers (Ultraviolet in this case) following the interactive lens mechanics (Tominski et al 2017), with the adaptation of a previously developed model (Fanini et al 2021b) (figure 5). Past works demonstrated the elegance, flexibility, and simplicity of interactive lenses employed in several fields and case studies. Such developed mechanics allow final users to interactively discover hidden Ultraviolet information on high-res maps, from mobile devices, up to 3-DoF (e.g. cardboards) or 6-DoF immersive headsets (e.g. Meta Quest).
The Web3D prototype was presented during TourismA 2022 and during an E-RHIS international event at Manifattura Tabacchi in Florence (figure 7) for the first time and it will be soon available on the official website of the project (www.brancaccipov.cnr.it).

CONCLUSIONS
The paper has presented in detail how the 3D simulation of the Brancacci chapel was obtained. The various steps of digitising the monument and processing the 3D models have been described, with the aim of re-using them in real-time Web3D platforms, such as Aton, for content analysis and dissemination. The working and optimization pipeline for the 3D models was developed for this case study specifically to solve problems related to the management and use of assets in real-time web environments, characterised by the high visual detail of the subjects represented in the frescoes and by the presence of multiple layers of information (RGB textures and ultraviolet textures). It successfully integrates typical approaches of geomatics with those of computer graphics, as well as automatic and manual processes, to create a flexible and usable product for both specialised audiences and ordinary visitors. This product allows extremely complex data to be managed and presented in a simple and interactive way, while maintaining the consistency of the scientific data even after the simplifications and optimizations required by the fruition tools. The described workflow can also be easily adapted to similar case studies, involving the simulation and interactive consumption of frescoes, paintings or decorated surfaces via Web3D applications with multi-layered data. As mentioned in the introduction, the ability for both professionals and the general public to use data derived from specific investigations is consolidating in technology and, consequently, in practice. Nevertheless, theoretically and methodologically, there is still much to be studied and compared As might be expected, it is not possible to cover all aspects of the project in detail in a single article. In future works, we plan to present in more detail the different multi-resolution approaches for 3D datasets created in the project and to address specific topics, such as the design and user experience (UX) of the Brancacci POV project and the development of the specific features for the interactive Web3D app for managing 3D/2D content.

Figure 7.
A session of Brancacci POV with visitors using a tablet and an Oculus HMD and the guide in the background.