OPTIMIZATION OF VR APPLICATION IN TEXTURING CULTURAL HERITAGE

: The research question underlying this short essay refers to the possibility of realizing through a well-established workflow a high level of immersivity in the VR representation of Cultural Heritage to be used as a working tool by architects and renovators as well as heritage scholars. At present active or passive 3D scanning techniques are a known reality where the choice of data acquisition system depends both on the final purpose and characteristics of the object to be surveyed. With the maturity achieved in point cloud acquisition, post processing phases and the development of BIM-based systems, the model has given the possibility to become a repository of information related to the existing, to be used for maintenance or renovation processes. If the model can represent the existing with a certain level of detail, it can be assumed that a differentiated Level of Immersivity can be organized depending on the specific needs it intends to fulfil. In the context of renovation or preservation of Cultural Heritage, the potential offered by VR becomes more interesting when it can provide a realistic portrait not only of the geometry, but also of the materiality and state of preservation of the buildings. This research opens new possibilities to develop tools to aid designers and renovators in degradation analysis, intervention projects, and scheduled maintenance. The research question starts analyzing the possibility of creating photorealistic immersive environments that are easy to access, organized based on surveys of existing heritage buildings using specific datasets.


INTRODUCTION
The case study originates from the surveying and modelling of one of the Sanctuaries along the path of Saint Augustine (Ornaghi, 2015), located in the Lariano Triangle, specifically the Sanctuary of San Martino in Valmadrera.The Sanctuary of San Martino (Figure 1), also known as the Sanctuary of Our Lady of Milk (Piefermi, 2014), is one of the 50 Marian sanctuaries that make up the path of the Rose of the Celebratory Journey, marking the traces of the saint.This route is distinctive for recalling the stylized figure of a rose, with its corolla encompassing the Brianza comasca, Lecco, and Monza regions.The sanctuaries along this route represent places of worship from different epochs and styles, often built on the preexisting sites of ancient religious nuclei.They are scattered across a territory between Monza and Lecco, in locations of significant landscape importance.The Sanctuary of San Martino is situated in Valmadrera on a promontory to the east of the city, in a commanding position over the surrounding territory.This location suggests that, in its initial establishment, the sanctuary may have had military functions due to its control over the road.The sanctuary has a predominantly Romanesque typological definition, undergoing some modifications around the mid-1400s, new constructions in the mid-1500s, and renovations around the 1700s.While the sanctuary was initially dedicated to Saint Martin, the church preserved a fifteenth-century image of Our Lady of Milk, highly venerated by the local population, along with other frescoed works such as a large Crucifixion, the four Evangelists, and other pieces reminiscent of contemporary works by Bergognone, Leonardo, and Bramantino.To document this and other Heritage Buildings an in-depth digital survey campaign has been organized.The objective of this campaign is to create an accurate digital 3D model whose 3D immersive experience in photorealistic mode becomes the essential element to enable various operators to work remotely in simulation processes also.For this reason, when surveying and modelling are proven to be accurate, the same level of accuracy must be ensured in the immersive environment, where navigation requires a perfect match with the original geometries and meticulous attention to the details of the visualized models, including colorimetric and material aspects.The proposed workflow explores an immersivity solution that, while considering the limitations imposed by hardware, does not relinquish any of the specificities derived from the artifact itself and the scientifically conducted survey (Trizio, 2019).

The path to accurate immersive environments of Cultural Heritage
In recent years, the application of digital tools and techniques has facilitated the study of Cultural Heritage for various interest groups, fostering a strong interest in integrating multiple survey and modelling practices dealing with interoperability themes among platforms very often.Noteworthy developments have seen a rising trend since 2010 with the advent of advanced Laser instruments and algorithmic modelling software; an integrated approach for creating a 3D digital reconstruction can be traced back to archaeological reconstruction writings (Guidi et al., 2013).The need for integrating multiple survey methodologies remains a pervasive theme in many studies (Grussenmeyer et al., 2012); at the same time achieving increasingly accurate documentation of Cultural Heritage requires specific accuracy studies (Nocerino et al., 2014).With the emergence of Historic Building Information Modelling (HBIM), attempts to reconstruct 3D models from survey data in BIM platforms have multiplied, aiming to enrich the model semantic information of greater accuracy (Lopez et al., 2017).The wealth of data derived from point clouds is not sufficient to satisfy the requirements of an informative HBIM model and on the threshold of the immersive visualization era, both the systems are still under development (Quattrini et al., 2018).In the age of immersive reality, the learning path is influenced more by entertainment-related domains than the technical needs of immersivity, with all that implies (Di Paola, 2019).Virtual Reality (VR) as a tool for professionals in restoration and recovery still hangs with issues left unresolved by survey and modeling studies still undergoing under a profound development (Del Pozzo, 2022).The aim of this research is focused on the generation of geometric and photorealistic 3D models for accurate reconstruction to be easily shared in immersive visualization purposes.The selected approaches are based on the combination of photogrammetric dense matching Terrestrial Laser Scanning (TLS) techniques and modelling in immersive environments.

Immersive environments
The 3D immersive experience of the Heritage asset in photorealistic mode could become an essential element to enable various operators to co work remotely in simulation processes.When surveying and modelling are proven to be accurate, the same level of accuracy should be ensured in an immersive environment, where navigation requires a perfect match with the original geometries and meticulous attention to the details of the visualized models, including colorimetric and material aspects.The case presented here was developed within the limits of the equipment represented by two Desktop computers and Headsets: an Intel i9 9900KF as CPU, Nvidia RTX 2080 Ti as GPU and 64 Gb DDR4 of RAM installed, a Portable Computer with an Intel i7-10750H as CPU, Nvidia GeForce 1650 Ti as GPU and 16 Gb DDR4 of RAM installed and two VR headsets, Meta Quest 2 equipped with a Qualcomm Snapdragon XR2 processor and GPU and 6 Gb LPDDR4X of RAM and the supported resolution 1832x1920 per eye and Meta Quest Pro which has a Qualcomm Snapdragon XR2+ as CPU and GPU, 12 Gb LPDDR5 of RAM installed and 1800x1920 per eye's resolution.Most of the meticulous details of the VR photorealistic environment struggle to find an effective result in current hardware instrumentation, often necessitating compromises for smoother architecture visualization, which can compromise the realistic experience.The proposed workflow explores an immersiveness solution that, while considering the limitations imposed by hardware, does not relinquish any of the specificities derived from the artifact itself and the scientifically conducted survey.

The survey
Due to the scale and complexity of the sanctuary, a 3D survey was organized using a TLS with time of flight, an RTC 360, and a mobile one, a BLKtoGO; the parts with frescoes have been captured with a photogrammetric campaign using a Canon Eos 6D Mark II.The whole point cloud has a consistency of 1,2 B points, with a maximum error of 0,002 m (Figure 2).Subsequently, the modelling process was carried out in a Building Information Modelling (BIM) environment for most of the complex, with the assistance of NURBS-based modelling software for the more shape complex parts.The geometrical accuracy considered the maximum tolerance accepted 30 mm.Following this, the entire model was prepared for export in IFC 4 (Industry Foundation Classes) to allow integration into a compatible modelling environment.Additionally, an export in FBX format was prepared to facilitate subsequent implementation into the visualization engine and the preparation of photogrammetric textures.The texturing process for the immersive visualization focused on two specific areas of the church: the chapel to the south with a series of frescoes depicting the great Crucifixion, the Evangelists of the vault, the stories of Christ, the St. Martin, and a high late Gothic relief known as the Madonna del Latte.

Mesh creation and unwrapping on different surfaces
Reality Capture (Epic Games, 2023) software has been used to create meshes from the photogrammetric survey; the software allows to create 3D point cloud models aligning digital pictures  The clean-up process utilizes Boolean operations to cut and delete parts that are not useful for the goal; for a more deep and precise clean operations is possible to enter in "Edit mode" and select vertex, edges and faces singularly and modify or delete them.Once the clean process is finished is possible to manage and edit the mesh, in "Sculpt mode", to add parts that, in the process of photogrammetric interpolation, had been lost or smooth the high noises that are the consequence of the lights inside the church.An important feature that the software offers is the possibility to manage the texture and modify it by paint it on the surface cleaning possible errors that may occur during the previous process.The next step is to duplicate the mesh to have one that has all the geometry details and high-resolution textures and the other one editable.The duplicated model is organized by parts by using the selecting tool; this process is meant to group the vertex that comprehend the wall that the high relief is attached to, geometrical parts as the columns, tympanum, and fountain and then the subject of the high relief: the Madonna with child.The chapel walls and roof had been divided in clusters: the south wall, east wall, west wall, and the vaulted roof (Figure 5).This process is useful to organize and proceed by a nondestructive workflow with each cluster of vertices and modify, bake, and simplify one by one.Blender offers a specific tool that allows to decimate the geometry by using a ratio; by the previous step is possible to decimate the mesh by parts, so in this way, the user can decide how and how much the geometry should be decimate for each vertices cluster (Figure 6).The wall has a highly dense mesh, with no important information, and it has been strongly decimated to keep only important geometrical parts.After the simplification tasks is now necessary to proceed by a retopology process; this process allows to prepare a mesh easier for the software and hardware to be read and processed by reorganizing how the faces of the mesh are.The two possibilities the mesh model will be easier and faster to be managed are by triangulate or make them pure quads.This process uses software's built in modifier that reconstruct the geometry automatically but after that a manual intervention is necessary to adjust the mesh faces.After the retopology Blender keeps the vertex that we previously divided in parts and by selecting them is possible to divide those vertices creating clusters of the high relief and the chapel, working on them singularly.This step is necessary to bake the high-poly visual mesh's data like diffuse maps, depth and normal maps by compressing them from 16k/8k to a lower resolution up to 6 times lower, in this case a 2k resolution each into the low poly mesh; by clustering the meshes is possible to separate the baking process in multiple tasks and images so, when the mesh is unwrapped it would occupy all the available space on the texture's table, 2k resolution, and it will improve the texture resolution of the cluster by having almost the same size, geometrically inscribed inside the texture table, of an 8k resolution.Once the mesh is separated in different objects, those meshes need to be unwrapped and UV mapped to be geometrically inscribed into image texture space and have the necessary coordinates that can communicate with the high poly mesh.This process bridges the coordinates from the first mesh to second one, allowing geometrical and texture communications between them and caging the first mesh and texture information around the second.The textures map that has been baked from the high poly mesh to the low poly mesh in this case are the normal map and the diffuse map, they are necessary to understand and visualize the model.In the "shading tab" has been created two new image texture slots for baking process having the resolution of 8192x8192 each.The baking process consists in a simulation of the high poly mesh model's characteristics, as geometrical, depth, color, and texture characteristics into the low poly mesh model (Figure 7).Thanks to this process the baked model will appear like the photogrammetric one, with the only difference that the first one has very low dense mesh and optimized textures which means less hardware workload.
Once the baking is completed the multiple meshes clusters are rejoined together and are ready to be exported to different software, including the new textures and maps.
Table 1.High-poly and Low-poly mesh comparison tables.

Texturing and mesh implementation for VR
After the export from Blender is completed, the next step involves organizing the folder structure for importing into the visualization software.At the beginning of the process certain features are set to enhance CPU and GPU utilization with the aim of reducing computational and rendering costs.Unreal Engine 5.3 that has been used for VR project needs a project folder; it should include one directory for the meshes and another for the textures obtained from the baking process.The selected format should be.fbx, which includes meshes and texture.Subsequently, the mesh skeleton is recalculated in "Nanite" mode to enable triangulation subdivision based on distance, thus optimizing the computational process associated with meshes according to the user's observation.This entails scalability of the meshes based on the viewpoint or project type.
In the LOD (Level of Detail) parameter tab for the meshes, one can specify the reduction ratios of the meshes in relation to the viewing distance by creating different mesh model "LODs" so the application can decide which one to show depending by the user distance or purpose.In the subsequent texturing phase, (Figure 8) textures extracted from the .fbxfile are associated with the meshes using "Master Materials" featuring palettes representing parameters for optimal rendering.During this phase, maps derived from photogrammetry's baking can be edited within the software, adjusting characteristics such as roughness or depth for particular uses or to achieve realistic results in every possible scenario.This process is not mandatory, inside Blender are generated all the necessary texture maps for a PBR rendering engine and are imported with the mesh.The texture imported are ready to use.Inside the software is possible to use Blueprints to visually coding transitions from a rendered display to specific map-based visualizations, ensuring a precise alignment with reality.In a subsequent step, physically realistic lighting environments are defined in accordance with PBR (Physically Based Rendering) standards so the model can respond accurately and realistically to the lights and the environment.

Visualization in the VR environment
Inside Unreal Engine 5.3, is possible to create VR ready applications, VR environments and similar.The main issue with VR's headsets is the lack of computational performance for interactive, immersive, and photorealistic VR applications due to the hardware specifications which are improving but still need very optimized 3D objects.
Previous processes prepared the meshes to be visualized inside the virtual environment, by optimizing them and be VR ready.
The official Meta developer website states that usually the Meta Quest 2 works well until the target of 750k to 1M triangles per application level visualized.So, our main target of usage to have a fluid application and maintain from 60 to 72 fps was set to 750k triangles for each application level.Before starting to design the Unreal Engine 5.3 VR application project some settings to optimize the VR project had been tweaked to achieve the optimal balance between visualization and smoothness.The settings consist in how the software should compute the graphical part of the application changing the rendering shader from "Forward Renderer" to "Forward Shading" which has faster rendering passes that means a gain in overall hardware performance; the anti-aliasing method was switched to "Multisampling Anti-Aliasing" which is the best choice for VR applications reducing the blurriness caused by the constant VR headtracking.It has been also activated the "Instanced stereo" functionality which optimize the CPU and GPU utilization, thanks to this option the render engine will not render the scene two time, one per eye, so it will not create loops in the rendering process.Some other settings utilized to enhance the global quality and fluidity of the application were done from the command console inside the software and inside the UE 5.3 ".ini" file.
UE 5.3 also worked as a bridge to connect the IFC model of the "San Martino's sanctuary" designed inside a BIM software and the photogrammetric optimized models.Inside the software is possible to import the IFC model with the "DATASMITH Importer" that keeps the georeferenced coordinates, so the alignment process of the models will be automatic.Thanks to the software's coding capabilities has been created an interactive switching visualization that goes from specific maps to others.The final VR application resulted stable and smooth fitting all, the previously set, usability parameters (Figure 9).

CONCLUSION AND FUTURE RESEARCH
The workflow briefly outlined here is a process that combines photogrammetric techniques and mesh manipulation to foster photorealistic immersive view.The objective is to merge detailed photorealistic information with immersive smart environments.The workflow is analyzed in relation to the tools available for the creation of a stream that can interface in the future with all VR equipment, from standalone equipment to streaming platforms.The achieved goal of photorealistic representation shows that it is possible to create easily accessible 3D immersive environments without sacrificing the quality of the view.However, the workflow was used for a single modelling environment, without experiencing the fluidity of models and textures in immersive use with other possible immersive visualization (streaming from the platform for example).On these two fronts, a methodical review of case studies and different workflows is therefore necessary to proceed in the useful discussion of professionals, edu-game and forthcoming metaverse uses.

Figure 1 .
Figure 1.A historical postcard of the sanctuary.

Figure 2 .
Figure 2. Above the map of the point station of the survey campaign with TLS.Below the photogrammetry chapel in Reality Capture platform.

Figure 4 .
Figure 4. Shaded view, vertex view and solid view in Blender.

Figure 5 .
Figure 5. Solid vertex and shaded view in Blender.

Figure 6 .
Figure 6.Comparison between High-poly and Low-poly models in Blender.

Figure 7 .
Figure 7.Comparison between High-poly and Low-poly rendered model in Blender.

Figure 8 .
Figure 8. Texture maps viewable inside the application.

Figure 9 .
Figure 9. Example of interaction between the user and the model inside of the VR application