Milan Cathedral Digitized: A Stone-by-Stone Segmentation Approach Using SAM2

Abstract. High-resolution architectural documentation goes beyond geometry—it requires a deep understanding of the building’s structure, materials, and historical layers. This often means interpreting hidden construction logic and identifying even the smallest components, such as individual stones or bricks, to produce meaningful data for conservation, analysis, and interpretation. Identifying and describing all the individual components that constitute the building, such as the type, arrangement, and state of preservation of stones, bricks, mortars, or decorative materials embedded in the walls, is a real challenge due to the large quantity and the complex spatial distribution of each element. Recent advances in AI, particularly foundational models and zero-shot models, offer potential solutions to speed up the documentation process. Taking the gothic complex of Milan Cathedral as the monument object of study, the research hereby presented implements a SAM2 (Segment Anything Model) based stone-by-stone segmentation, leveraging object detector for semantic interpretation. The proposed framework integrates 2D stone block segmentation with photogrammetric 3D reconstruction, enabling accurate projection of semantic labels and geometric data from images to 3D point cloud, allowing a detailed 3D segmentation in all the components of the structure.