EVOLUTION OF RENDERING BASED ON RADIANCE FIELDS. THE PALERMO CASE STUDY FOR A COMPARISON BETWEEN NERF AND GAUSSIAN SPLATTING

: In recent years there has been a rapid diffusion of new digitization methodologies based on radiance fields and the implementation of new rendering processes and learning systems based on neural networks. The article focuses on these new tools and how they can be used for the knowledge and dissemination of Cultural Heritage. A case study is then described regarding the video acquisition of a noble chapel of the Cemetery of Santa Maria dei Rotoli in Palermo to promote knowledge of ‘fragile’ artefacts, exposed to the risk of radical transformation or degradation, and thus protecting their conservation. The research aims to compare the first results obtained through the NeRF and Gaussian Splatting methodology which constitute the current state of the art of this type of processing; both the source algorithms (Nerfacto and 3D Gaussian Splatting) and the Luma AI web app were used, and data management was studied using third-party software such as Blender 3D, Unreal Engine 5.0 and the playcanvas game engine. The results obtained with this case study are of particular interest, above all for the processing of data useful for the visualization of heritage starting from unconventional acquisitions.


INTRODUCTION
The constant development of digital technologies greatly influences the possibilities offered today in the field of Cultural Heritage.In addition, the great heterogeneity of the architecture present on our territory allows new methodologies to be tested on complex case studies that are far from the ideal ones to verify their advantages and critical issues in a concrete environment.Each survey must be designed according to the purpose and the object to be investigated and often involves the integration of different methodologies and techniques (Parrinello et al., 2023).Furthermore, the situation surrounding the spaces and the working conditions (limited space around, a lot of light contrast, presence of trees or objects that obstruct the view, reflected surface etc.) often could contribute to making the work more difficult.Innovation and research in documentation and digitalisation methodologies are therefore increasingly supportive and make processes more easier and efficient (Morena 2023;Bolognesi and Fiorillo 2023;Lin et al. 2020).In recent times, in addition, the new frontier of applied artificial intelligence has influenced and changed the methodological approach in numerous fields, as well as in the field of architecture and its representation.In just few years, complex algorithms and systems have developed that have contributed significantly to the improvement of threedimensional visual representations and the definition of more accurate Digital Twins (DT).This work focuses on the documentation of a chapel in the Cemetery of Santa Maria dei Rotoli in Palermo, using various new methodologies.The cemetery is one of the largest in the city and contains particularly valuable noble chapels with interventions by well-known architects and sculptors from Palermo.As was often the case, the cemetery represented areas of experimentation of new architectural forms and styles that only later found their way into the city.It was built around 1837 and, following the various needs over the years, underwent a series of interventions and extensions until it assumed its current conformation.The cemetery of Santa Maria dei Rotoli is organised in sections along the main streets and internal roads to reach the various burial grounds.The chapel under study, G.ppe Locascio Chapel, is located near the main axis that separates the exedra, which characterises the layout of the plant (Figure 1).In its surroundings, there are mainly several inhumation and obelisk burials, which make it difficult to move around freely, as well as additional noble chapels and vegetation.The Locascio chapel dates back around the 20th century and currently falls within the landscape and monumental area of the cemetery.It is built in calcarenite and externally is composed of a central volume of greater height and characterized by a tripartite door-window and two lateral volumes on which 'sarcophagi' are placed.In its perimeter, there are flowers and barries decorations typical of the Palermo style of the time.
The following work aims to provide an initial qualitative analysis of the results obtained with these two innovative methods still in the development and testing phase.The advantages and criticalities of the methodology are analysed, as well as the effectiveness and possible future developments of the digitisation method for the restitution of models faithful to reality, which can be used for representation, visulization and popularization of architecture.

RELATED WORKS AND POTENTIAL IN THE CULTURAL HERITAGE FIELD
In the field of three-dimensional digitization and computer graphics, among the various research conducted in recent years, of particular interest is the investigation of the development of NeRF (Mildenhall et al., 2020).Although in the beginning it was not designed for the generation of common 3D objects (point clouds or meshes) but for 3D rendering, over the years the development of graphics techniques has guaranteed the conversion and widespread use in various fields of investigation (Murtiyoso and Grussenmeyer, 2023).Current research are different and some of them have been directed towards deepening various aspects of the methodology or improving some of them (Gao et al., 2022, Müller et al. 2022;Xu et al. 2022).The advantages of Neural Radiance Field (NeRF) are also being tested in the field of Cultural Heritage.Some research compares and analyses the advantages of using NeRF over the photogrammetric technique or laser scanning, especially in critical conditions such as low number of photos or low image resolution (Croce et al., 2024;Palestini et al., 2021Balloni et al. 2023), in the reconstruction of buildings with very difficult conditions from typical ones (Condorelli et al. 2021), or also explore the potential of NeRF in the cases of unconventional acquisition conditions (Mazzacca et al. 2023).
Recently, 3D Gaussian Splatting (GS) represents a further competitive alternative for 3D render reconstruction (Kerbl et al. 2023) Mip-NeRF360, the system that seems to be one of the most advanced to date to function in the rendering of large architectural spaces (Barron et al. 2023).However, the timing for training and, generally, rendering remains extremely high.
Recently, these technologies merged to create a new algorithm called GANeRF, which is currently in the experimental beta phase.GANeRF aims to define excellent quality results, setting new standards for automatically generating complex threedimensional spaces from image data while keeping computation times relatively low.This collaboration between Machine Learning and GAN aims to combine the highly time and resource-consuming rendering capability of the NeRF system, capable of producing detailed and realistic results, with the twodimensional optimization offered by GAN systems, based on the interaction of two discriminators working together to correct any imperfections and digital disturbances in the images generated by NeRF, all in real-time and with exceptional results.However, presently, the system providing the best visual results is known as GS, an evolution of NeRF that does not directly employ neural networks but relies on the photogrammetric process of input image data and utilizes Structure from Motion (SfM) algorithms to define the initial three-dimensional representation of space.In this article, we specifically focus on experimenting with this system.It specifically uses three-dimensional Gaussian functions, which are mathematical functions describing the distribution of light and object shape, to realistically render the scene.Unlike NeRF, which requires significant resources and time for training data according to applied models and subsequent cleaning of image data, GS avoids the direct use of neural networks, directly utilizing the sparse SfM point cloud produced during camera calibration, thereby reducing computational time.This approach leverages the acceleration provided by graphics processing units (CUDA) and is inspired by the raster tiles transformer, enhancing the efficiency of the rendering phase without depending on fixed-function GPU rasterization hardware.A key aspect of GS is the use of 'Splats', essentially small disks or ellipsoids distributed to generate volumetric lines.Together, based on their position and chromatic qualities, they generate a composite spatial representation, creating a three-dimensional digital environment.These 'Splats' can be perceived as a 'cloud of blobs in space'.Each particle within this cloud has opacity and chromatic representation that can vary depending on the viewing direction.During the rendering phase, the particles are projected as two-dimensional Gaussians onto the screen to accelerate the real-time process.Thus, three-dimensional Gaussians prove to be a flexible and expressive representation of real scenes.The data acquisition process uses cameras calibrated via SfM, enabling the structuring of the threedimensional Gaussian set using a sparse point cloud generated during the SfM process.Unlike many other point-based solutions requiring Multi-View Stereo (MVS) data, the GS system produces high-quality results using only SfM points as input.Three-dimensional Gaussians are an excellent choice as they represent a differentiable volume shape and can be rasterized very efficiently by projecting them in 2D and applying an α blending, similar to NeRF (Kerbl et al.,2023).
In summary, the virtual digitization technique of GS proves to be a rather expedient method for creating realistic three-dimensional images of previously captured scenes through photos or videos, even without following a rigorous data acquisition methodology (this is one of the strengths of this method).Three-dimensional Gaussians have the advantage of preserving the properties of continuous radiance fields, which are models describing how light can propagate in a real scene, while avoiding unnecessary calculations to define empty spaces.The virtual digitization technique of GS involves three main steps: -The first step is the generation of three-dimensional Gaussians from sparse points, using a clustering algorithm that groups points based on their proximity and direction.
-The second step is the optimization of three-dimensional Gaussians, involving the modification of Gaussian parameters, such as mean, covariance, and weight, to better fit the input data and control the density of Gaussians in the scene to define the volumetric representation of objects.
-The third step is the rendering of three-dimensional Gaussians, involving projecting the Gaussians onto the image plane using a technique called anisotropic splatting, which accounts for the visibility and previously structured shape of the Gaussians.The virtual digitization technique of GS, therefore, allows for obtaining high-quality three-dimensional images, maintaining competitive training and rendering times, and enabling real-time synthesis of new views.This technique is useful for creating virtual and augmented reality applications that require a faithful and interactive representation of real scenes, as well as numerous other uses compatible with 2.0 three-dimensional digital storage, architectural and territorial analysis, and design.

Data acquisition
The workflow implemented in this paper starts with on-site acquisitions.The aim is to investigate the results that can be obtained with unconventional acquisitions, using inexpensive instruments that allow easy and replicable methods.In this regard, the survey of the chapel was realised by recording a video with the lightweight action camera Insta360 ONE R.This is a low-cost device equipped with a 1/2.3"sensor with a resolution of 12 Megapixels and the lens used was a wide-angle, with an aperture of F/2.8.The video was shot in 4K mode and 30 fps, the action camera was moved slowly, looping around the chapel.The shot was first taken at a greater distance, to cover the entire structure, and then it was brought closer to the object, always rotating around it but at three different heights and angles.In the survey phase, it was decided to operate with the action pole to conduct an agile acquisition by increasing the shooting height.The global illumination at the time of the survey was quite good, this is always a positive factor, even if the GS method still guarantees good results even with reduced brightness.The main problems encountered during the survey phase were the absence of free space to move with constant distances around the object and, consequently, also the challenge to guarantee reduced tilting and avoiding panning and zooming as much as possible; the presence of vegetation which hinders the complete view of the case study; and the height of the chapel which makes the acquisition of the upper parts difficult.
As written previously, the source data was used for both NeRF and GS processing.They have been processed with different applications that operate with various methodologies and modalities: Nerfacto and 3D Gaussian Splatting, which are open-source algorithms, to export NeRF and GS results respectively and Luma AI free and easy-to-use applications to process and export NeRF and GS data.

Data processing
As a first processing step, the data acquired as just described were processed with open source algorithms, then directly using the source code of GS as released by the authors (https://github.com/graphdeco-inria/gaussian-splatting).As for NeRF-based algorithms, among those recently released, we selected the Nerfacto model for processing our data (https://docs.nerf.studio/nerfology/methods/nerfacto.html).This is a model based on a set of techniques from different studies and put together to achieve higher reconstruction performance.
In particular, this model works well with videos and images acquired with distortions due to wide-angle viewing such as our case study because it implements pose refinement by correcting errors due to unclear images.For the implementation of Nerfacto, the Nerfstudio framework was used to train the neural network and export the final data.Implementations of both methods require high performance calculations and high processing times that vary depending on the method used and the dataset (obviously reconstructing large spaces in detail requires higher computational powers).In the present case, a specific building, the Locascio Chapel, was reconstructed without high detail in the surrounding environment, so an analysis of the results will be performed.The workflow followed for the processing is as follows.The acquired data were first processed using the SfM algorithms found in COLMAP (Schönberger et al., 2016) to extract camera parameters and poses.This first step is the starting point for both reconstructions with Nerfacto and 3D Gaussian Splatting.
In terms of processing, the difference between the two algorithms lies in how the dataset is prepared, which must meet certain compatibility requirements to be recognized (Kerbl et al., 2023).Once the dataset has been prepared, training can begin, which can be followed with a real-time viewer that allows all processing steps to be monitored (Figure 2).The visualizer for Nerfacto (Figure 3) was used for Nerfstudio (https://viewer.nerf.studio).The same was used for 3D Gaussian Splatting along with the SIBR viewer (https://sibr.gitlabpages.inria.fr/?page=index.html&version=0.9. 6).Nerfacto, and NeRF in general, require training a neural network on observed data to learn the radiance values of 3D scenes.The main advantage of NeRF is the high level of detail and precision it offers in point cloud or mesh modeling scenes.However, this precision can come at the expense of computational intensity, making real-time rendering difficult.Although 3D Gaussian splatting can provide real-time performance, it can provide a coarser representation than NeRF and may sacrifice some degree of accuracy.In this case, it was interesting to analyze the results obtained by exporting the point clouds and meshes from Nerfacto and 3D Gaussian Splatting respectively.Despite some critical conditions (such as the type of acquisition, difficulty in moving freely around the object and the presence of visual obstacles), it provided an enough satisfactory result.As hypothesized, the result comes with a sparse cloud and not-so-fine details, however, the main problems occurred at the top of the model due to the information gap from the video captures (Figure 4).An interesting test was also conducted with a user-friendly and economical application (app or web browser) such as Luma AI.Differently from previous algorithms, the actions to be performed are very limited and no hardware or software requirements are needed.It is becoming very popular due to the simplicity of transforming real objects into photorealistic threedimensional models even for non-experts people.Once an account is created, simply upload the captured videos and wait for the data to be processed.Once the processing is complete, the data can be exported in various 3D formats, including Luma Field and Gaussian Splat, which correspond to NeRF and GS models that can only be viewed with the appropriate plugin or add-on and can then be managed in supported software such as Unreal, Unity and Blender.

VISUALIZATION AND DATA MANAGEMENT
Some user-friendly software has guaranteed easier opening of the obtained files; they allowed both the visualization and management of the data: thanks to the plug-in developed by Luma AI for Unreal Engine 5.x it was possible to display both the .ply of the GS and the NeRF file acquired by Luma (with a proprietary extension .luma);with the help of Blender 3D and playcanvas.com it was possible to both view and manage the files derived from 3D Gaussian Splatting.For managing the .plyfiles generated by the 3D Gaussian Splatting algorithms, the Blender 3D add-on named "ReshotAI" created by Alex Carlier (an independent developer) is considered valid.Being still in an optimization phase, it turns out to be a good tool for modifying the point cloud at the base of the GS, but not very exciting from a rendering point of view.Through the add-on it is still possible to access the file, both via the standard editing commands and via the geometry node editor.The latter provides the modification of some parameters already present within the node group such as the size of the ellipsoids, their opacity and establishes, through a slider, the density of points that must be visible within the scene.There are also two display modes: one in the form of point cloud -which in addition to the points, shows a sort of circular halo around each point, very similar to Gaussian ellipsoids, but certainly easier and quicker to manage by the rendering engine -and one in the form of splatter.Through the geometry node editor, it is therefore possible to access information regarding the conformation of the particular type of .plywhich, being derived from a SfM process, is bound to a precise geometry: the alignment point cloud of the dataset of images provided in the training phase.To prove this, it is possible to connect the node referring to the input geometry (the point cloud) to the output node, displaying the source point cloud on the screen, however without the Gaussian information.It is important to underline how the .plyfile is readable even without a particular add-on.If you import the .plynormally into a software (for example Blender 3D or Rhinoceros) it will consist of a sparse point cloud with vertex color; however, the Gaussians attributed by the algorithm to each point of the cloud will be missing (Figure 5).An alternative method that appears to be a valid solution for managing and editing this type of file is the Playcanvas web platform.Born on June 4, 2014 and developed by Will Eastcott, Dave Evans, Vaios Kalpias Ilias, Kevin Rooney and Maksims Mihejevs and owned by Snap Inc., it is an open-source game engine that allows the creation of interactive 3D applications.The engine is accessible through the browser (which supports WebGL), in which a proprietary platform installed in the cloud can be used.Through this platform, it is possible to manage different types of files, animations and is equipped with a decent physics engine.The platform allows access to various configurators, of which the main one concerns the editing of the project and the assets that compose it.A new configurator is recently available, called 'super splat', which allows the management of .plysderived from GS. Through the latter it is possible to access various editing commands capable of scaling, translating, and rotating the Gaussian cloud, as well as modifying its origin, cleaning it from excess points not necessary for the survey, exporting it in three different formats (.ply cleaned; .plycompressed; .splat).Particularly useful is the session dedicated to point selection which presents the rectangular selector, the brush selector and some parameters that allow you to select specific points based on the density of the Gaussians or their size.This platform turns out to be much more valid from a rendering point of view than the ReshotAI add-on; objects appear very defined and clear and retain shading properties such as reflections, emission and refraction of light, as well as the colorimetric property and density of splats.(Figure 6).The characteristic that the two methods just described have in common refers to the shader construction method: in both Blender and Playcanvas, in fact, the Gaussian surveys appear to shine with their own light; this is because the shader -which can be view through Blender 3D shader editor (Figure 5)despite being very complex and made up of many nodes, is not composed of a real diffuse channel, but is composed of an emission channel and an alpha channel and therefore uses the emission for displaying the color and the alpha channel for displaying the transparencies.From this consideration, it is possible to argue that the photorealistic effects of the reflections, the characteristic bloom effect of the light points, are the result of different light emission factors depending on the position of the point of view.Thus, systems for visualizing and managing the threedimensional elements generated by these new methods are multiplying, being progressively easier to use and compatible across multiple platforms.The case study example allowed to evaluate a number of platforms, mostly free and online, suitable for the fast management and modification of 3D assets generated by NeRF and GS methods.

RESULTS
The methodologies implemented and the results obtained allow us to expose some initial considerations.The acquisition part is certainly fundamental for every investigation and has a significant influence on the analysis of the results.It should be pointed out that the work aims to evaluate and analyse the results of simple captures made under very different conditions than those designed ad hoc, to test the results that can be obtained by using even amateur videos as starting data.Working in this way, it is possible to evaluate also eventual advantages of using tourist videos of buildings that no longer exist or are partially destroyed to restore, albeit virtually, the conformation of the past.In this comparison, as previously written, we limit ourselves to a qualitative analysis of the visualization results, based mainly on texture and detail reproduction quality, and not a metric comparison of the possible extractable models.The first substantial difference to highlight is the implementation of different applications for data processing: some in open-source code to install on our computer machine (Nerfato and 3D Gaussian Splatting) and the other in remote (Luma AI).In the case of Nerfacto e 3D Gaussian Splatting, the installation and data processing times are quite long and specific hardware and software requirements, as well as technical knowledge, are necessary to obtain satisfactory data.The Luma AI application, on the other side, makes it possible to obtain photorealistic models quickly and in simple ways without necessarily having technical knowledge or highperformance computers because data processing takes place remotely.This last aspect represents both an advantage, for the simplicity of the workflow, but also a limitation as it does not allow personalized management of the data and the possibility of intervening during the various phases of the process.However, to some extent, in our case, this problem has been overcome by using third-party software that allows one to act on the final data and enables better management of data visualisation, as mentioned in the previous paragraph.Among the different applications used, the best results that we obtained with this case study were with Luma AI, so below we will limit ourselves to making a comparison between the NeRF and GS results obtained with it (Figure 6).The NeRF data have the advantage of being able to export, through various dedicated algorithms, point clouds and meshes.In addition, the models obtained show better handling of the reconstruction of information gaps resulting from missing data in the survey phase.GS, on the other hand, made it possible to obtain a model with a better texture and superior management of brightness.The images show a more realistic and refined view of the materials that make up the chapel, as well as better management of light and shadows.Furthermore, although in both models the detail is not perfectly reconstructed, in the case of the GS the depth effect is better and, although to a limited extent, the floral decoration is a little bit more evident (Figure 7).

CONCLUSION
The paper through the case study presents a preliminary analysis of the results from the application of several innovative surveying methods, based on AI training and the innovative Gaussian render, with a focus on the evaluation of methodological advantages and criticalities.The effectiveness and potential future developments of such digitization methods for the creation of true-to-life models are examined, with specific applications in representation, degradation assessment, and architectural analysis.The use of techniques such as NeRF and GS reveals considerable potential for improving threedimensional representation without the need to resort directly to point clouds, instead exploiting three-dimensional representations of scenarios.This enables the creation of scalable and proportionate models that can be used in various applications such as games, animations, augmented reality and animated movie scenes.In addition, such methodologies reduce dependence on a large number of images to obtain accurate representations, overcoming the challenges inherent in imagebased methods.These developments represent a significant advance in supporting representation in multiple contexts, opening up new application perspectives in future scenarios.Add-ons and independent online applications, such as Blender plugins, facilitate handling in multiple formats, including Gaussian .plyfiles, while keeping the colorimetric and structural properties of the original model intact.However, remember that these methods are based on visualization rendering and not on the direct creation of point clouds that can then be edited later with tools such as Cloud Compare; the integration of NeRF and Gaussian systems, therefore, with traditional surveying methodologies, such as the use of photoplanes for scale redrawing, still has critical issues.In particular, these systems show inaccuracies in reprocessing features located at considerable distances and in foreshortened perspective relative to camera viewpoints, inducing errors in the upper portions of representations.On the other hand, the use of exported 3D objects in video game editing platforms, such as Unreal Engine 5, shows considerable potential, although they remain in the maturing phase with regard to extremely close visualization (which is grainy and ill-defined) and the definition of specific features, such as nanite self-optimization or simplified collisions with interactive game elements.However, such methodologies prove effective for the creation of general purpose, choral representations capable of realistically rendering the atmosphere and form of real elements.In conclusion, these approaches represent a significant advance over digital sensing methodologies Range based and Image-based, although some shortcomings persist, especially in relation to error handling, while still guaranteeing appreciable photorealistic rendering when employed in directly three-dimensional contexts, as in the case of virtual platforms such as Unreal Engine and Unity, supported by appropriate internal plugins (e.g., luma_ai plugin reader).

Figure 1 .
Figure 1.Locascio Chapel in the Cemetery of Santa Maria dei Rotoli in Palermo

Figure 2 .Figure 4 .
Figure 2. Nerfacto interface and NeRF general data obtained from the video of the Locascio Chapel

Figure 6 .
Figure 6.Playcanvas Supersplat, visualization and management of the Gaussian .plyand various possibilities for selecting the points of the sparse point cloud

Figure 7 .
Figure 7. Gaussian Splatting (left) and NeRF (right) data obtained from Luma AI of the Locascio Chapel