STRATEGIES AND EXPERIMENTS FOR MASSIVE 3D DIGITALIZATION OF THE REMAINS AFTER THE NOTRE DAME DE PARIS’ FIRE

: After the catastrophic fire at Notre Dame de Paris, a significant challenge was presented by the numerous lead-contaminated remnants. To address this, a detailed digitization strategy was devised and executed, tailored to the unique needs of this extensive and diverse corpus. This strategy involved the development and customization of both hardware and software tools, ensuring their effectiveness throughout the digitization process – from initial data acquisition to data dissemination. Central to our approach was the alignment of our methods with the distinct characteristics of each artifact, facilitating their effective preservation and future utility. Our strategy's adaptability was key, allowing us to incorporate advanced deep learning techniques into various aspects of our workflow. Notably, this included the implementation of the Segment Anything Model for automatic image segmentation, enhancing our image-based modeling capabilities. We also ventured into pioneering methods like 3D Gaussian Splatting and the exploration of radiance field methods for visualization. Moreover, the project has been mindful of data responsibility, aiming to make all digital data openly accessible beyond 2025. We have placed a strong emphasis on harmonizing and managing data, minimizing redundancies, and ensuring efficient storage, all while maintaining transparency about the limitations and errors in our methodologies. This holistic approach to digitization, balancing technological innovation with responsible data management, aims to preserve and make accessible the digital heritage of Notre Dame de Paris for future generations.


Context
Following the fire of April 15, 2019, which destroyed part of the cathedral Notre Dame de Paris, several working groups have been set up by the National Center for Scientific Research (CNRS) to help and organize the restoration work (Ball, 2020).This scientific action involves more than 175 researchers specialized in materials (stone, wood, metal, stained-glass), decoration, structural acoustics behaviors, and heritage emotions.Among them, the Digital Data Working Group (Groupe de Travail Données Numériques) is coordinating the development of a digital ecosystem aiming at managing the complete data lifecycle produced within the framework of the general scientific site (De Luca, 2024).The core technological framework of the digital ecosystem consists of a digital platform for the data centralisation (Néroulidis et al., 2024) including dedicated web services opened to more than 100 registered users for the ingestion, indexing and thematic categorisation of multimedia content, as well as for interactive analysis of 2D/3D/4D representations of architectural elements and remains.Since the beginning of the project, several temporal states of the cathedral (before-during-after the restoration) are digitized and integrated and semantically annotated with Aïoli (Abergel et al., 2023).Within this framework, image-based digitization holds a socle role as a vector of interaction between on-site realities, methods, analyses, and interpretations, through 3D models but also (and most importantly), source images (high definition photographs) oriented in the correspondent 3D space.This ad-hoc framework allows us to enrich the 3D digitisation with a set of localized scientific observations that build a structure of overlapping and interconnected layers of thematic analyses.Since 2019, Notre Dame's cathedral has been at the forefront of attention, but several thousand remains of the fire have been preserved (or recovered during archaeological and renovation operations), including timbers, vaults, metal artifacts, and polychrome sculptures.This paper focuses on the comprehensive strategy we designed and implemented for the extensive image-based 3D digitization of these artifacts.The entire corpus consists of about ten thousand pieces of timber, over six hundred pallets of stone elements, and three hundred pallets of metal objects, documented on-site and stored in warehouses adapted to work safely in the context of lead pollution.(Zimmer et al., 2024).

Thinking on the overall strategy
The digitization endeavor serves several complementary goals.Firstly, the digital preservation of artifacts is crucial, especially since most items in this corpus are contaminated with lead and/or too delicate to handle without risk.Secondly, ensuring access to these digital collections for the diverse stakeholders involved (both in research and civil society) is a significant challenge in Digital Humanities, a field continuously evolving with technological advancements.Lastly, it's imperative that these collections serve as valuable research resources, meaning they should be comprehensively comparable in all their complexities and nuances.The goals outlined are offset by certain "bottleneck" constraints intrinsic to this corpus, as highlighted in the previous section.Firstly, the sheer size of the corpus poses a challenge, encompassing several thousand artifacts that require digitization.Secondly, each artifact possesses unique characteristics, including material composition, surface treatment, functional scope, and conservation requirements.Finally, it's crucial that each digitization process is directly beneficial to researchers, with the Digital Data Working Group operating in close collaboration with other members of the Notre Dame scientific community.Our operational approach is situated at the intersection of these objectives and the aforementioned constraints.The distinct nature of the Notre Dame corpus required the development of a comprehensive digitization strategy from an early stage.This strategy was designed to accommodate specific approaches for each group of items, also taking into account unique scientific and heritage requirements.Our overarching strategy is anchored in the following key concepts (as illustrated in Figure 1), which allow us to navigate through constraints while steadfastly adhering to our primary objectives : • Optimizing Data Acquisition and Processing: This aspect was meticulously planned to minimize delays in scientific studies.Our approach involves working synergistically with other teams operating concurrently, ensuring that our activities complement rather than hinder the broader research efforts.• Accommodating a Wide Dissemination: The diversity in potential applications, ranging from model analysis to public visualization, is aligned with the evolving realities of practice and technical advancements.This broad scope necessitates a flexible approach to cater to various requirements.• Harmonizing and Managing Heterogeneous Data: The strategy for dealing with the inherently diverse nature of the data revolves around ontological principles.This includes the use of enhanced metadata and paradata, ensuring consistency and accessibility across the dataset.

Figure 1. Overall strategy diagram
Our overarching strategy was adapted to align with the specific expertise of our team.For instance, in advocating for consistent data management, we opted for a singular digitization approach.Image-based modeling was selected, not only for its long-established use in our team's research (as detailed in works by De Luca et al., 2022;De Luca, 2023;Pamart et al., 2020), but also due to its recognized benefits, which we will discuss further below.Moreover, as previously noted, our strategy required customization to accommodate the diverse nature of the corpus.This adaptation was based on several intrinsic characteristics of the artifacts, including: • Morphology: Considering material, size, weight, and shape; • Conservation Conditions: Assessing factors such as pollution or surface treatments like traces of polychromy.• Access Conditions: Evaluating aspects like responsibility, storage location, and pollution levels of the area.• Intended Scope of Use: Determining if our acquisition solutions are optimally calibrated, ensuring they are neither over-nor under-utilized.
A research endeavor like ours is subject to various evolutions, a fact we anticipated from the beginning.This foresight allowed for flexibility, fostering innovation in several areas, particularly in the integration of deep learning solutions into our digitization strategy.This approach facilitates the exploration of diverse alternatives, enhancing both the optimization and dissemination of our protocols and data.The exploration of deep learning serves a dual purpose: • Integration with Current Data Production: We aim to discover how these advanced tools can effectively complement and enhance our existing data production processes.• Testing with Real Heritage Data: Unlike relying solely on academic benchmarks or synthetic data, we are testing these tools with actual heritage data.This approach provides a more authentic and practical understanding of their capabilities and limitations in real-world scenarios.

OPTIMIZATION STRATEGIES
The term "optimization" in our context refers to identifying solutions and methods that enhance the efficiency, preservation, and management of our corpus.This also includes facilitating communication with other members of Notre Dame's scientific group, while aligning with the concurrent timing of scientific studies and digitization efforts.We have categorized this optimization endeavor into several key segments.These are primarily adaptations of standard photogrammetry digitization protocols: Thus, we can divide this optimization effort into several segments, which are, for the most part, derived from standard image-based digitization protocols: (1) optimization of data acquisition and calibration; (2) optimization of dense matching processing and export of intended models (via Agisoft Metashape's python API), optimization of long-term archiving and data management; (3) experimentation with optimized processing protocol through automatic mask generation for each image collection using deep learning (via Segmentation Anything Model).

Acquisition and calibration challenges
From the outset, the task of acquiring photogrammetric data for the Notre-Dame corpus presented significant complexities.The lead pollution coating the objects has been a notable issue.Moreover, the majority of the artifacts can be characterized as both fragile and cumbersome.An additional challenge lies in the time constraints, necessitating a high level of efficiency in terms of manpower per data acquisition campaign.For instance, we undertook a rapid and urgent data acquisition campaign for the 40 remaining pieces of the oculus.Each piece required handling by multiple operators, emphasizing the need for speed and efficiency.This was made feasible within a narrow time frame through the deployment of a lightweight photogrammetric studio, as illustrated in Figure 2 (top-left).This approach exemplifies the need for agility and precision in our acquisition and calibration processes, particularly under challenging conditions.
To optimize the acquisition process, we established a partnership with Mercurio Imaging, a specialist company based in Marseille.This collaboration led to the creation of three  Our photogrammetric protocol is practically implemented through a method we term "half-shell" acquisition.This approach involves rotating the object to capture all its surfaces through multiple complementary acquisitions, termed as "half-shells."These half-shells are then meticulously merged to form a "complete shell" at the conclusion of the photogrammetric processing.For each half-shell acquisition, we employ coded markers in the form of horizontal squares.These markers serve a dual purpose: they establish the scale and define the horizontal plane of the object.This setup is crucial for automating the subsequent processing stages, ensuring accuracy and efficiency in the digitization process.The integration of these markers into our workflow exemplifies the attention to detail and precision required in high-quality heritage digitization.

Photogrammetric processing enhancement
Once we had successfully acquired and organized the source image collections, it became evident that software processing required similar optimization.We opted for Agisoft Metashape, a proprietary software, which is popular in Cultural Heritage digital documentation.The software was adapted early for intensive GPU use, enhancing its speed, and it features an interface integrated with a Python API that covers all its processes.This integration allowed us to create specific commands for the automatic processing of all acquisitions in our corpus, addressing the unique aspects of the image collections.We focused on color calibration, as outlined in Gaiani and Bellabeni (2018), and format conversion.This involved establishing a connection between our Python commands and external tools such as Shaft2-CLI and Rawtherapee-CLI.In addition to these steps, our process included the use of coded markers for retrieving the scale and establishing the horizontal plane, modeling each half-shell, and ultimately merging all the half-shells to produce the final output, as shown in Figure 3.In practice, processing each half-shell is a meticulous task.It not only involves accurately recording the intrinsic parameters and camera orientations but also entails generating masks from the horizontal plane, which are marked with coded markers.These masks play a crucial role in the process, as they allow for the isolation of the object from its support.This isolation is key for correctly aligning the half-shells with each other during the merging process.It helps to identify the relevant correspondences while avoiding false correspondences that might arise from the support, which could be redundantly present in each image.After completing the final mesh, we can export it from Metashape in various formats.This versatility in exporting is crucial for the next phase , which we will discuss in more detail later.The export formats range from standard ones like .obj and .ply to more specific or software-oriented formats such as potree, 3DHOP, and Aioli.In cases of these specific formats, we integrate our Python Metashape commands with external converters provided by their creators (NexusConverter.exe,PotreeConverter.exe,MS2AIOLI.py 1 ).

SAM
The Segment Anything Model (SAM) is a cutting-edge image segmentation foundation model developed by Meta, as outlined by Kirillov et al. in 2023.As a foundation model, SAM is a large deep neural network trained on an extensive dataset, typically through unsupervised learning.This training enables it to be adapted for a wide array of tasks, as discussed by Bommasani et al. in 2021.SAM's training involved the massive SA-1B dataset, which comprises over 1 billion instance masks across 11 million images.This vast and varied dataset endows SAM with exceptional generalization capabilities, allowing it to perform zero-shot segmentation.This means SAM can segment objects and regions in images without needing prior training on specific objects or categories.SAM operates by producing a segmentation mask in response to a prompt, which could be a set of points, bounding boxes, or text.In our photogrammetry pipeline, we utilize images with bounding boxes as inputs for SAM to generate binary segmentation masks, which are then integrated into our pipeline.SAM's architecture consists of three core modules: This key finding from our previous research highlights the effectiveness of SAM, particularly when using bounding boxes as prompts.This method leverages SAM's advanced capabilities, demonstrating its adaptability and precision in complex image segmentation tasks.The use of bounding boxes provides clear, defined areas for SAM to focus on, enhancing the accuracy of the segmentation process (Figure 4).

HARMONIZATION PROCESS
In our digitization project, each artifact presents unique characteristics, such as shape, material, and specific treatments, often detailed in scientific studies.Similarly, their virtual counterparts also possess distinctive features related to their creation or topological aspects.Beyond these, the digital models hold contextual specifics: the location and date of image acquisition, the software used, and the operator involved.Digitization, as a scientific endeavor, generates a collection of resources (data) following a technical protocol (metadata), while also preserving a spatio-temporal uniqueness (paradata).
The final model, a product of this combination, should accurately represent these elements and be capable of being queried in its full complexity.To achieve this, a harmonization of the data/metadata/paradata package is essential.This process ensures that the data is not only comparable within our project but also interoperable with external systems and projects.
Harmonization plays a critical role in maintaining consistency, reliability, and usability of the digitized data, making it a pivotal aspect of our work in digital preservation and research.

Data, Metadata and Paradata integration
In our effort to harmonize and ensure the interoperability of our data, we have integrated a paradata container into each stage of our workflow, drawing on the MEMoS system (Pamart et al., 2022).MEMoS, based on the W7 system (Liu and Ram, 2017), facilitates the embedding of paradata within the images via a QR code.This code encapsulates contextual information across seven categories: What, When, Where, Why, Who, Which, and hoW.We have adopted this classification to systematically document the digitization activity's paradata in several complementary files, each dedicated to different contexts: acquisition, processing, and data management.Moreover, we diligently maintain the metadata for both the source and calibrated images, as well as preserving the Agisoft Metashape projects.This approach includes defining the boundaries of our work by exporting various indicators, particularly those related to spatial orientation uncertainties.These indicators, such as RMS error per camera, SEUW error, and homothety uncertainty, are crucial.They not only allow us to assess the performance level of our operational workflow for each digitization campaign but also aid in identifying any projects that encountered issues.

Data management
Our workflow is designed to manage a vast array of complementary data efficiently, aiming to organize them systematically from the outset to prevent the creation of unmanageable digital bulk.Our approach to data management is straightforward yet effective, with each digitization project encapsulated in a structured folder hierarchy: • Acquisition: This contains the source images.Within it, a MEMoS file in JSON format documents the paradata from the acquisition phase.• Processing: This includes the Metashape folder, converted and calibrated images (in JPG or TIF formats), and a processing MEMoS file describing the workflow.This folder also stores data on digitization uncertainties.• Export: This folder houses each of the virtual model formats generated during the digitization process.• Long-Term Archive: It contains the Submission Information Package (SIP) for long-term archiving purposes.At the pinnacle of this hierarchy is a general index that provides an overview of each folder.Crucially, this index establishes a link between the virtual model and the physical artifact, referencing the perpetual inventory number assigned by the scientific site, as illustrated in Figure 5.This artifact-specific data structure is generated automatically, at the same time as each MEMoS, based on information pre-filled by the various operators.In this way, we try to minimize the risk of error in our data management.

DISSEMINATION STRATEGIES
The dissemination of Notre Dame's digitized artifacts is envisioned to encompass a broad spectrum of applications.These range from long-term digital preservation to the practical utilization of virtual models, and also include making these resources accessible to the public.Public engagement could be achieved through online open-source visualization platforms or through immersive experiences like Virtual Reality solutions.Furthermore, an essential aspect of this dissemination process is to ensure that these digitizations contribute to the AIOLI platform.This platform plays a pivotal role in analyzing and preserving the complex scientific discourse of Notre Dame's scientific group.
• Complexity and Scientific Value: The digitization process addresses the need to capture the complexity of the artifacts and the associated scientific discourse.This is particularly crucial in the heritage discipline, where data is inherently fragmentary and heterogeneous.• Responsibility and Reliability: In the field of scientific heritage, the responsibility of data providers extends to ensuring traceability and reliability.This is vital due to the fragmentary and diverse nature of heritage data.• Technological Innovation and Experimentation: The dissemination process is also an opportunity for technological experimentation and advancement.One such area is the use of Radiance Field methods.These methods can be useful in documentation, visualization, and communication, offering new ways to present 3D models of cultural heritage sites and objects.This multi-faceted approach to dissemination not only preserves the heritage value of the artifacts but also opens up new avenues for research, education, and public engagement.

Visualizations, measurements, annotations and preservation techniques
For our project, we have carefully selected file formats for exporting digitized artifacts to ensure compatibility with a wide range of processing, analysis, and visualization tools.
• Dense Point Cloud export: We export the dense point cloud in the Stanford Triangle Format (.PLY).This format is recognized by both processing software like CloudCompare and Meshlab, and visualization software through plugins for Blender and Unity.The .PLY format is versatile, allowing for the storage and conversion of various properties within the point cloud into other formats, such as .LAZ and E57.Its widespread use in the Digital Heritage community further validates our choice.• Mesh export: For meshes, we have chosen the Wavefront OBJ format, which is commonly used in processing and analysis (CloudCompare, Meshlab) as well as in modeling software (Blender, Unity).This format has evolved to include Physically Based Rendering (PBR) specifications within the .MTL parameters, making it compatible with modern rendering engines.• Online Visualization: We generate two types of online visualizations, each as a self-sufficient webpage containing all necessary libraries.The dense cloud is accessible in the Potree format (version 2.1, selected to avoid the file multiplicity of version 1.7), while the mesh is provided in the nexus compressed format (.NXZ) for the 3DHOP viewer.This approach ensures that each artifact's visualization is autonomous, allowing for the creation of specific collections either by linking to our generated .htmlpages or, for more integrated solutions, directly to the potree/nexus model itself.
Our export workflow extends beyond simple 3D asset generation; it includes creating a ready-to-use AIOLI project, particularly noteworthy for its functionalities : • Visualization of the Dense Cloud: It provides an interactive display of the dense cloud, facilitating detailed examination and analysis.• Spatially Oriented Image Visualization: More importantly, it allows for the visualization of each spatially oriented image.• Multi-layered 2D/3D annotation: This feature is crucial for complex and precise annotations of the virtual models directly through the images.These annotations are reprojected onto the complete collection, utilizing photogrammetry's intrinsic parameters and camera positions (computed by an internal integration of the MicMac photogrammetric runtime).
In line with the French national digital preservation policy, our project is evolving within the TGIR Huma-Num and the Archeogrid platform (Tournon et al., 2020) for data storage, referencing, and long-term archiving.This collaboration includes creating a bridge between our solutions and the aLTAG3D software.This enables us to generate a Submission Information Package (SIP) for each item, comprising a collection of archives formatted according to a specific archival profile.These packages are then deposited on the national digital archiving service provided by the CINES (Centre Informatique National de l'Enseignement Supérieur).

Enhanced visual reconstruction by Gaussian Splatting
While photogrammetry requires many images to accurately reconstruct a scene and produce detailed and accurate 3D models, there are several alternative techniques like Neural Radiance Fields (NeRF) (Mildenhall et al., 2021) or 3D gaussian splatting (Kerbl et al., 2023) that can be an effective solution when quick content production is needed or when realistic 360°imaging is required for example.In the context of cultural heritage, several NeRF methods have been used on different cultural heritage datasets: smartphone videos, touristic approaches, or reflex cameras.Among them, Instant-NGP (Muller et al., 2022) and Nerfacto (Tancik et al., 2023) methods achieved the best outcomes (Mazzaca et al., 2023).Several studies have showed that NeRFs could be used for rendering objects (sculptures, archaeological remains, sites, paintings etc.) that are challenging for photogrammetry (Condorelli et al., 2021;Croce et al., 2023).NeRF has been successfully used for the documentation of cultural heritage sites.A study comparing NeRF and Multi-View Stereo Structure From Motion (MVS-SfM) found that while NeRF requires fewer images to produce accurate models, MVS-SfM provides more precise reconstructions of the structure of sites (Balloni et al., 2023).
As far as our knowledge, GSPLAT has never been used on cultural heritage dataset.Therefore, in this work, we decided to used GSPLAT on our data.
3D Gaussian splatting is a volume rendering technique that has emerged as a promising alternative to NeRFs for real-time rendering of 3D scenes.Unlike NeRFs, which rely on neural networks for scene representation, 3D Gaussian splatting utilizes a sparse set of 3D Gaussian functions, also known as "splats," to approximate the scene's radiance field.This approach offers several advantages, including improved rendering speed, reduced memory usage, and the ability for direct editing of the scene geometry.3DGS represent complex 3D surfaces using a collection of discrete splats.Each splat is defined by its center position, scale, and orientation, along with additional parameters that capture the scene's radiance information.By distributing these splats throughout the scene, the overall radiance field can be approximated with a high degree of accuracy.The rendering process in 3D Gaussian splatting involves evaluating the radiance contribution from each splat that is visible from the current viewing point.To determine the visibility of a splat, the technique utilizes a visibility-aware splatting algorithm that efficiently assesses the occlusion relationships between splats.Once the visible splats are identified, their radiance contributions are combined to generate the final image at the current viewing point (Figure 6).These techniques offer different advantages and trade-offs compared to photogrammetry.3D Gaussian Splatting achieve high-quality rendering with faster training and real-time performance, particularly for complex scenes and high-resolution outputs.However, 3D Gaussian Splatting may struggle with scenes that have many occlusions or areas with little texture.Our results on Notre-Dame objects show that there are still some limitations.We were able to observe transparency problems on some of our objects.In addition, this method was not designed to process large quantities of high-definition images from photogrammetry, making calculation times particularly long.

CONCLUSION
In this paper we presented a full acquisition-processing-publication workflow that can be adapted to different object typologies and volumes, and takes into account the traceability of information.From acquisition to dissemination, we have put in place solutions to overcome most of the obstacles inherent in a corpus such as that of Notre Dame remains.So far, we have been able to confirm its effectiveness on several different groups of artifacts : 80 timbers and 20 metallic staples acquired over 5 campaigns spread over 2023, but also 120 stone fragments acquired between 2021 and 2022.More than 1000 items are planned to be digitized en 2024.In addition, we are experimenting with its reinforcement through calibrated and strategic interventions of AI-based methods to streamline the workflow.As we aim to make all digital data from the Notre Dame de Paris project available in an open-access format post-2025, we fully acknowledge the responsibility inherent in creating such a comprehensive digital corpus.Recognizing that no solution is perfect and digitizations are influenced by various priorities, transparency becomes a key aspect of our approach.It's essential to articulate our data collection process as clearly as possible, facilitating its broad dissemination.Our ambition to harmonize and manage data is intrinsically tied to this sense of responsibility.That's why we offer complementary models to minimize the need for users to transform them for use.Each modification in the data should be accompanied by a corresponding update in the data information, maintaining integrity and clarity.Additionally, it's our duty to clearly delineate the limitations of our methods, such as by making processing errors accessible and explaining our choices transparently.Another aspect of our responsibility is data efficiency.Given that our corpus will ultimately encompass several terabytes of data, it's crucial to avoid unnecessary redundancy.This means not only ensuring that saved data doesn't duplicate, but also defining and targeting the optimal use of the data we produce.We facilitate this by allowing users to engage with every stage of our workflow -from calibration and processing to exports and archiving.By upholding these standards, we not only fulfill our responsibility towards the digital heritage community but also contribute to the creation of a rich, accessible, and valuable digital resource that can be utilized and built upon well into the future.
Patrimoine (FSP) and the European Research Council (ERC) Advanced Grant nDame_Heritage project funded by the European Union.
innovative devices, each tailored to specific needs of our digitization project: Dome-Shaped Rig (Figure 2 -top-right): This rig was developed to facilitate data acquisition on the complex remains of the transept crossing's double-arched keystones, as detailed in Guillem et al., 2023.Its dome shape is particularly suited for capturing detailed images from multiple angles.Multi-Camera Studio (Figure 2 -bottom-left): Designed as an alternative to the dome, this studio is particularly effective for digitizing smaller artifacts, such as metal staples.Its multi-camera setup allows for comprehensive coverage, ensuring detailed and accurate digitization.Step-Over, an Arch-Shape In-Line Photogrammetric Structure (Figure2-bottom-right): This structure is specially designed for efficiently digitizing artifacts that are elongated along a specific axis, like timbers.Its arch shape and in-line configuration allow for quick and thorough data capture.

Figure 3 .
Figure 3. Automatic image-based digitization workflow used for Notre-Dame corpus, from acquisition specificities to deep learning improvements.

Figure 4 .
Figure 4. Overview of SAM for automatic segmentation.

Figure 5 .
Figure 5. Data structure for each item

•
Image Encoder: Utilizing a MaskedAutoEncoder (MAE) based on a Vision Transformer (ViT) model, as proposed by He et al. in 2022 and Dosovitskiy et al. in 2020, this module is responsible for image feature extraction.It processes an image to create an image embedding.• Prompt Encoder: This module can process positional information from the prompt input, such as a set of points and/or bounding boxes.These inputs provide additional contextual information for the mask decoding process.• Mask Decoder: A two-layer transformer-based model, this module is tasked with generating the final segmentation mask predictions.