COMBINING PATTERN PROJECTION AND CROSS POLARIZATION TO ENHANCE 3D RECONSTRUCTION OF FEATURELESS REFLECTIVE SURFACES.

: 3D digitisation is essential to enhance knowledge and conservation processes for Cultural Heritage (CH). However, movable heritage collected in museums, e.g., archeological finds, statues, coins, and musical instruments, often consists of highly reflective surfaces and/or featureless textures. The aim of this work is to stress the Structure from Motion Dense MultiView Reconstruction (SfM-DMVR) of objects, whose optical properties are challenging to this image-based technique. To improve the results obtained by SfM-DMVR standard acquisition workflow, this study exploited the potential of combining Noise Function Pattern (NFP) projection and light Cross Polarization (CP). Moreover, High Dynamic Range (HDR) image reconstruction was tested on images acquired according to different CP angles. The SfM-DMVR of a texture-less shiny ceramic object was then compared to the one obtained by a structured light triangulation scanner, the most reliable tool for such surfaces, but unfortunately quite expensive. The advantages and limitations of the presented method are discussed.


INTRODUCTION
3D digitization applications in the field of cultural heritage are directly linked to research and study, with the main objective of producing diverse material that can be used for documentation, restoration, and valorisation purposes (Pavlidis and Koutsoudis, 2022). In the 2019 Declaration of Cooperation on advancing the digitisation of cultural heritage, 27 European countries have called also on the European Commission's Expert Group on Digital Cultural Heritage and Europeana (DCHE Expert Group) to contribute to the development of common guidelines for comprehensive, holistic documentation of European 3D cultural heritage assets. In the "Basic principles and tips for 3D digitisation of cultural heritage" we read "Photogrammetry is suitable for materials such as stone, wood, concrete, textile, plastic, or metal (matte surface), but not for shiny, transparent or highly glossy objects, nor for objects with loose/movable parts" (Expert Group on Digital Cultural Heritage and Europeana, 2021) The fully automated workflows based on Structure from Motion (SfM) and Dense Multi-View Reconstruction (DMVR) algorithms are widely used in the field of Cultural Heritage (CH). This method takes full advantage of low-cost tools to generate accurate 3D digital replicas aimed at the study, classification, preservation, communication, and restoration of different kinds of artefacts (Apollonio et al., 2021;Ch'ng et al., 2019;Marín-Buzón et al., 2021;Rahaman and Champion, 2019).
SFM-DMVR is a non-contact measuring technique that is used to find a set of feature points that appear in different images. The feature points can be used in order to retrieve the captured scene and estimate the position and orientation of the camera stations. * Corresponding author By utilizing the orientation parameters, the three-dimensional (3D) coordinates of the camera stations and sparse point cloud can be estimated in 3D space.
The reconstruction pipeline typically includes numerous processing steps when passive techniques are selected to deal with object digitisation (Farella et al., 2022). In particular, the SFM phase relies on feature extraction for the subsequent image matching process. As the starting point, the feature detectors and descriptors are key tools for obtaining a good reconstruction. Unfortunately, it demonstrated limitations due to the optical properties of the investigated object (Koutsoudis et al., 2013), such as in the presence of challenging light transport effects including partial occlusion, low contrast, and reflective or refractive surfaces; or in the absence of a surface texture (uniform, monotone, repetitive textures, etc.).
Therefore, texture analysis has been widely investigated in computer vision and pattern recognition applications because of its ability to extract discriminative features (Engler and Randle, 2009). This is such a common and widespread problem that some studies have been undertaken to formulate feature detection algorithms capable of overcoming these limitations. (Dansereau et al., 2019).
Indeed, tangible cultural heritage, especially movable cultural heritage, very often presents these challenging characteristics. Consider for instance ceramics or gold in archaeological museums. On the other hand, scanning systems capable of overcoming these limitations are still very expensive. For this reason, experimenting with tools and techniques in the field of photogrammetry is a challenging task.

AIM OF THE RESEARCH
The aim of the work presented here is to digitize cultural artefacts with challenging surface optical properties using passive techniques. The case study chosen for the test is a ceramic statuette typical of the Sicilian tradition, a testa di moro, with a very shiny surface, monochromatic and a homogeneous texture that is practically non-existent (Fig. 1).
In the literature, the use of Cross Polarization (CP) is specifically intended to avoid direct reflections that can compromise the photogrammetric process and produce a not faithful texture. Excellent results have been demonstrated using cold lamps and circular polarized filters applied to camera lens and lamps. Cross polarization indeed reduces the amount of light hitting the sensor, requiring lower aperture number and/or higher exposure time. With matted surfaces, cross-polarization removes veiling glare and increases saturation and contrast (Nicolae et al., 2014). In other work the feasibility of creating high dynamic range images from a camera with on-chip multi-directional polarization filters is studied. By exploiting the fact that in environments with polarized lighting, the effect of the polarizers is analogous to that of imaging with multiple exposures. This gives rise to the possibility of reconstructing the HDR image of an environment from the polarization images (Wu et al., 2020).
In addition, surfaces with poor visual texture (repetitive, monotonous, etc.) challenge the feature extraction and matching stages and affect the quality of the reconstruction. The projection of image patterns using a video projector during image acquisition is a well-known technique that has proven successful for such surfaces. In (Hafeez et al., 2020), the authors evaluate the performance of different feature extraction methods on texture-less surfaces with the application of synthetically generated noise patterns (images). Previously, other works applied the Noise Function-based Pattern (NFP) projection in the data capturing phase, demonstrating its effectiveness in performance enhancement of SfM-DMVR applied to featureless surfaces (Guidi et al., 2014;Koutsoudis et al., 2015;Santoši et al., 2019).
In this paper, we challenged the SfM-DMVR method by exploiting the potential of CP and pattern projection on highreflective and featureless surfaces. The SfM-DMVR result obtained combining CP and pattern projection was then compared with the one generated by a structured-light triangulation scanner, the most reliable tool, but unfortunately also quite expensive. Advantages and limitations of the presented methodology are discussed.

PHOTOGRAMMETRIC 3D RECONSTRUCTION OF FEATURELESS REFLECTIVE SURFACES
The methodology proposed for this work mainly relies on two different techniques which can be applied in image acquisition: CP and NFP projection.
Concerning the first, polarizing different light sources implies multi-image capturing from a position, shooting the same frame according to different rotation angle settings on the CP lens filter. Subsequently, images are pre-processed with HDR algorithms to return a blended one. With regard to the NFP projection, this study does not investigate how different NFPs affect the SfM-DMVR process, exploiting the use of a Wavelet NFP, which (Koutsoudis et al., 2015) proved to be the most performing for these purposes.

Data capturing
The SfM algorithm reconstructs a sparse point cloud of a surface after recognizing corresponding features visible on different pictures depicting the same areas. Therefore, its success mainly depends on image quality and on the characteristics of the portrayed surface, this being the reason why not all objects are suitable for being reconstructed using this method. Particularly, monochromatic, or high-reflective surfaces may result in image matching errors and in extremely noisy point clouds.
As already stated, we attempted to overcome this shortcoming by integrating different solutions in the image acquisition and preprocessing phases. So, tests were carried out on a featureless high-reflective ceramic artefact, that represents the worst-case scenario for SfM-DMVR. A Sony α9 mirrorless full-frame camera (24 Mpx) mounting a macro-lens (105 mm) was used at an approximately object-to-camera distance of 70 cm. Three video projectors (resolution of 1024 × 768 px) lighted the object with a NFP to provide recognizable features on its surface. Since the shiny ceramic generated specular reflections, polarizing filters were mounted on the camera lens and projector lamps. Highlights were thus eliminated by adjusting the rotation angle of the filter mounted on the camera lens. Due to the object geometry and projectors positioning, two different settings were needed to remove all the highlights alternately. So, two images referred to a different CP angle were captured from each viewpoint, for a total amount of 72 pictures from 36 viewpoints (Fig. 2).
To verify the effectiveness of combining CP and pattern projection (Case 4), the data capturing process was also repeated without pattern projection and CP (Case 1), and alternately using CP (Case 2) and NFP projection (Case 3) Finally, the 3D reconstruction accuracy was assessed by comparing the SfM-DMVR result to the model processed using a structured light 3D scanner (Case 5). Particularly, it was used a Creaform Go!SCAN 3D, setting accuracy to 0.2 mm, and disabling noise reduction.

Data processing
The raw images were pre-processed in Adobe Camera Raw. Data sets related to the use of CP were furtherly handled, by blending each couple of images captured from the same viewpoint in a single HDR image, not based on multi-exposure but on different light cross-polarization angles.

Figure 1
The ceramic testa di moro used as a case study.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-M-2-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy  The use of CP became even much more relevant when combined to the NFP projection. By strongly reducing reflections, it thus increases the number of matchable details, rising the number of matched points. This was clearly shown by SfM-DMVR results, as reported in (Tab. 1) and represented in (Fig. 3). Tie points obtained using NFP projection (Case 3) were increased from 32886 to 45438 by adding CP (Case 4), corresponding to a remarkable +38%. To assess the accuracy of the SfM-DMVR carried out by integrating NFP projection and CP (Case 4), the point cloud was compared with the 3D reconstruction automatically obtained through the structured-light 3D scanner using the software VXelements (Fig. 4).
Both the 3D models were imported in CloudCompare, where the photogrammetric point cloud was roughly aligned to the 3D mesh, and then finely registered obtaining a final RMS of 3.19581e-05.

Figure 3
Tie points and point clouds related to CP HDR image processing (Case 2), NFP projection (Case 3), and NFP projection and CP HDR image processing combined (Case 4).
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-M-2-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy The model (1.77 mln faces) processed by scanning the object has a higher quality than the image-based 3D reconstruction. This can be easily verified even through subjective methods, e.g., visual comparison of the two models in terms of geometrical details and lack of noise. However, it must be underlined once again that the proposed case study was chosen among the most challenging artefacts for image-based methods due to its optical surface properties. Nevertheless, the produced image-based 3D model can be used for several purposes, enabling different uses for CH documentation and experience. (Fig. 5) shows the front view of the two 3D models, the image-based point-cloud coloured according to the absolute distances between the scanner 3D model and the image-based point cloud frequency of occurrence, also shown in the graph.

Overall performance evaluation
As expected, the artefact reconstruction without use of NFP projection and CP proved to be problematic. The image alignment failed due to lack of features. Thus, the data capturing phase was repeated by including additional Agisoft Metashape standard black and white targets to provide more features. Thus, the alignment succeeded, but the reconstruction was of poor quality due to noise presence and wide areas not fully reconstructed.
The obtained point clouds demonstrated that both NFP projection and CP improve the SfM-DMVR 3D reconstruction. Particularly, the first provided more features, so more tie points were generated. Meanwhile, CP, by reducing reflections, also reduced the point cloud noise presence.

CONCLUSIONS
The paper showed the integration of image capturing and preprocessing solutions applied to texture-less and highly reflective surfaces. Particularly, the presented workflow aims at improving the image matching within the SFM-DMVR pipeline.
The proposed tools were: i) at optical level the complete attenuation of specular reflections according to different CP angles and the projection of a NFP on the object surface. Reflection can affect image matching algorithms resulting in fake matchings, originating inaccurate point clouds, whereas NFP projection increase the number of features that could be recognized by those algorithms, improving the output quality; ii) at digital level the use of HDR processing applied to images acquired from the same shooting position but according to different CP angles. This solution was applied to obtain details in those image areas affected by spotlights, whose matching are intrinsically inaccurate due to the absence of information to be matched. Such tools have been discussed and applied to a specific and critical CH artefact: a white texture-less object made of ceramic. Both NFP pattern projection and CP HDR processing demonstrated their effectiveness in improving image quality for SfM DMVR processes.
The main achievement of the presented study is the evidence of complementarity of the two solutions, which if integrated can further enhance the image alignment process. However, this approach has negative aspects, too, i.e., it reduces the level of automatism both in data capturing and processing. Therefore, it must be considered as an effective solution only for artefacts not suitable for standard SfM-DMVR pipelines, when highgeometric accuracy is needed.