DEVELOPING ENHANCED AND ENRICHED 3D MODELS OF CULTURAL HERITAGE ASSETS AS PART OF THE EU CO-FUNDED 5DCULTURE PROJECT

: 5Dculture is a 24-month collaborative project co-funded by the European Union. It includes twelve partners from diverse sectors of cultural heritage, including archaeology, museums and fashion. It aims to enrich the offer of European 3D digital cultural heritage assets in the data space and fosters their reuse in important domains such as education, tourism and the wider cultural and creative sectors towards socially and economically sustainable outcomes. In particular, the project will deliver high-quality 3D content by identifying and engaging existing datasets from partners’ collections focusing on topics of fashion, archaeology and architecture, all of which occupy a central place in the vast cultural heritage of Europe. It will also develop several reuse scenarios, which will experiment with the aforementioned assets in their complexity (from high-quality to derivatives). The Discovery Programme has a long research interest in 3D digital documentation and has participated in collaborative projects in the past, notably 3D-ICONS. However, opportunities and challenges in the reuse of 3D archaeological content exist, including the reuse of 3D content to enhance visitors experience at archaeological sites and the ability to exploit 3D surrogates for different sectors (tourism, education, gaming, the arts, conservation and heritage management). This paper will present the research that has been undertaken in this context, through the development of an enhanced processing pipeline to improve the quality of 3D models of cultural heritage assets, resulting in their improved re-use. It will present the approaches to improving model geometry, generating and applying enhanced textures, creating rich and functional metadata and strategies for persistence and archiving. A core component


INTRODUCTION 1.1 Challenge
The Discovery Programme has been digitally documenting cultural heritage assets in Ireland for nearly twenty years - (Shaw, 2018).Since early adoption of terrestrial laser scanning in the late 1990's (Jacobs, 2021), other technologies such as structured light scanning, Structure-from-Motion (SfM) and dense Multi-View Stereo (MVS), lidar and UAV mapping have been added to the range of 3D digital documentation options employed by The Discovery Programme.These technologies were originally supporting The Discovery Programme's own long-term archaeological research projects but are now increasingly used for collaborative work with other national heritage agencies, for example, the Office of Public Works (OPW), National Monuments Service and the Heritage Council, who are involved in cultural heritage conservation and public outreach activities.The challenge of presenting the results of such surveys to researchers, archaeologists and conservation architects has long been a significant obstacle to their full value being appreciated.Generally, these audiences are more familiar and comfortable working with 2D plans, elevations, and sections (Li et al., 2023).Attempts to engage in the 3D environment floundered due to lack of user-friendly interfaces, a lack of experience and training, and a desire for continuity of familiar 2D outputs for inclusion in reports and documents.As a result, orthometric extractions from the 3D data set had become the most common deliverable from our surveys.

Additional Audiences, Additional Opportunities
In recent years there has been a growth of new audiences and consumers of 3D cultural heritage data in the form of 3D models.Education, tourism and the wider cultural and creative sectors are all developing interesting and dynamic uses for 3D models (Medicini, 2020;Little et al., 2020;and Stolilas, 2022).These 3D models are generally lower resolution, simplified, but aesthetically pleasing and are often of cultural heritage sites or objects which have been, or are planned to be, documented for conservation and research purposes.How to re-use existing data and ensure that new data surveyed can maximise the potential of these new audiences has been an impetus to the development of a new processing pipeline.The aim is to satisfy the demands for accessible 3D models, alongside conventional and practical 2D outputs.

Proprietorial Data Viewers
3D point clouds are the regular product of the terrestrial laser scan survey, following processing stages including registration, colourization, georeferencing, segmentation and cleaning.The 3D point clouds can be viewed and interrogated in the proprietorial software of the instrument manufacturer (in this case Trimble Realworks) and distributed to clients to be viewed in 3D viewer applications (Trimble Realworks Viewer), which are often licenced by the manufacturer and therefore come with a cost to the client.These proprietorial data viewers have limited functionality but generally include navigation (i.e.rotate and zoom) and measurement tools (Trimble RealWorks Development Team, 2023).

Orthometric outputs from 3D Terrestrial Laser Scans
In addition, the point cloud processing software allows the user to generate and export high resolution orthometric views from segmented elements of the point cloud as 2D images.Rendering the 3D point cloud using true colour with ambient shading, the orthometric views are given a reasonable colour representation (Campbell et al., 2018).They can be viewed as standalone products or as the basis for vectorisation.It is important to observe that these output images are point cloud derived and are not based on surface models.While many terrestrial laser scanners capture RGB imagery, unfortunately, the quality of images captured does not allow for photographic textures to be generated.Our new pipeline will demonstrate integrating SfM-MVS photogrammetry and then extracting orthometric views from textured surface models to significantly enhance this type of output.

DIGITAL DOCUMENTATION PIPELINE
The process of 3D digital documentation of cultural heritage objects developed by the Discovery Programme involves several steps and methodologies.The details of this specific pipeline may vary for other institutions depending on the tools, software and technologies available for use.A general overview of the pipeline for 3D digital documentation is provided in Figure 1 and explained in more detail in Sections 4-7.

Combining Survey Techniques
Prior to the development of this new approach, data capture was normally based on a singular technology, selected on the scale of the asset to be surveyed.For example, carved stone monuments were surveyed with a structured light scanner or by terrestrial SfM-MVS photogrammetry; historic buildings were documented using a terrestrial laser scanner; and earthwork monuments using UAV SfM-MVS photogrammetry or lidar (Historic England, 2017;Historic England 2018;and Remondino, 2011).These technologies are now mainstream in archaeological and architectural survey, but each has advantages and disadvantages when applied as a standalone solution.Terrestrial laser scanning offers rigorous geometric accuracy, with quality assurance reports to backup accuracy statements (El-Din Fawzy, 2019).but, as discussed, the outputs lack the colour and texture quality provided by SfM-MVS photogrammetry.
Conversely, while, SfM-MVS photogrammetry provides more optimal colour and texture quality, the SfM-MVS photogrammetry outputs lack the geometric accuracy of terrestrial laser scanning (Guarnieri et al., 2006).An additional problem is that in many architectural environments neither terrestrial laser scanning nor SfM-MVS photogrammetry will result in a complete survey, with roof lines and wall tops excluded as they are not visible from a ground position.In many cases, however, this can be rectified by employing a UAV photographic survey optimised for SfM-MVS processing (Tysiac et al., 2023).What has become apparent is the huge potential for 3D documentation that combines survey data from multiple sources, achieving complete and geometrically accurate models with the highest quality textures.

Optimising Data Quality
Regarding terrestrial laser scanning data capture and processing, best practices as developed and implemented with the support of instrument suppliers over several years, such as ensuring appropriate control networks, point density and registration (Historic England, 2018;and Wilford, 2020), were adhered to.However, for SfM-MVS photogrammetry significant improvements were introduced to the Discovery Programme workflow.Firstly, high quality DSLR cameras were purchased with fixed focus lens, (Canon EOS 6D II, 24 or 50 mm lens).Training was undertaken in operating the cameras in manual mode, ensuring optimal camera settings were selected for capturing SfM-MVS optimised images, which considered ambient site conditions, complexity of object etc.When large cultural heritage structures such as historic buildings, were surveyed, additional elevation was provided to capture the images through the employment of a camera pole, and if required a UAV with imaging sensor (Phantom 4 RTK).Secondly, the use of Macbeth colour calibration charts was implemented, with a colour calibration image taken for each camera set-up.Thirdly, images were shot in raw mode.In doing so, these two steps gave the opportunity for pre-processing image manipulation, to generate images optimal for colourmetric texturing and improved geometric matching with the generation of high contrast images (Carnevali et al., 2021).Finally, sets of metrically accurate calibrated scale bars were purchased.These were placed on the ground or against the wall within the area of interest and captured by the SfM-MVS photographic survey.This ensured the best geometric accuracy of the SfM-MVS photogrammetry outputs in addition to the overall geometric accuracy provided through the integration of TLS data.

Archiving of primary field data
Given the value now being placed on the legacy data created by the Discovery Programme, an important final stage of our workflow is the digital preservation through archiving and deposition with the Digital Repository of Ireland (DRI).In doing so, the FAIR data principles (Wilkinson et al., 2016) are adhered to.Making our data FAIR facilitates and encourages its discovery and reuse.The appropriate data, in the appropriate file format, with robust metadata, as advised by Richards-Risseto and Schwerin (2017), are deposited with the data repository and made available through a creative commons licence for public consumption and reuse.To improve the data's findability, controlled vocabularies are used to describe and index the data.These vocabularies include the Dublin Core (DublinCore, 2024) terms to describe key metadata, the Getty Art & Architecture Thesaurus® (AAT) (The Getty Art & Architecture Thesaurus, 2024) for survey activities and heritage asset descriptors, logainm (logainm.ie,2024) for placenames, and Perio.do(PeriodO, 2024) for temporal metadata.Terrestrial laser scanning point cloud data is archived in the E57 file format.The E57 file format is a compact and vendor-neutral format, which also stores the images and metadata.This file format was selected as it is in accordance with specifications established by the ASTM (ASTM E2807) an international standards organization (Payne and Niven, 2023).For the photography, images are batch migrated to the Digital Negative (DNG) format.DNG is a type of raw file format developed on the TIFF 6.0 format and enables the full bit depth of the RAW format to be preserved for potential future processing enhancements.Other important elements such as GNSS control, site plans and sketches are also archived to enable researchers to repeat the pipeline proposed by the Discovery Programme or to undertake their own bespoke processing workflows.

Primary Processing & Mesh Cleaning
Following initial preprocessing of the terrestrial laser scanning data in Trimble Realworks (geo-registration) the bulk of the data geoprocessing is carried out in RealityCapture software.Firstly, geo-referenced terrestrial laser scanning data is imported as separate scan stations in E57 format and aligned, proving a highly accurate overall geometric framework for the survey.Once alignment has been achieved this data is exported as a component.Subsequently, the SfM-MVS photogrammetric data is imported with two versions of each image provided: a colour calibrated version for the final texturing and a high contrast version for the alignment and mesh extraction processes.Once aligned the resulting SfM-MVS component is exported.If the survey required several discrete photographic setups such as the use of different lenses, highly variable lighting conditions or the capture of images from a UAV, separate alignments and resulting components are created and exported.Once all components have been created, they are imported into a new project and are aligned to each other with the terrestrial laser scanning component position locked to ensure overall geometric accuracy to the survey.Bringing together all the discrete components is normally achieved through automatic alignment, however the use of control points between components is often used to ensure good quality fit between the data and clean meshes, reducing ghosting effects of misalignments.After data alignment has been satisfactorily achieved, mesh generation is carried out.Depending on the size and complexity of the object, mesh processing in RealityCapture is undertaken using either the normal or high setting, with the final polygon count size being between 1 billion to 100 million polygons.Mesh cleaning routines are then carried out to remove rough points, terrestrial laser scanning targets and areas outside of the area of interest.Once completed the subsequent model is simplified down slightly to a more manageable size (250 -50 million polygons) to ensure the high-poly model can be used within any texture baking procedures (Section 6).Topologic checks are also completed to ensure holes in the mesh are filled and non-manifold geometry issues are eliminated, which can cause problems further down the processing pipeline.

Mesh refinement
Through the application of digital documentation and processing methods outlined in Sections 3, 4 and 5.1, the resulting polygon mesh created of the cultural heritage asset is normally made up of millions of polygonal facets, referred to as 'high poly' models.The full display of this 3D data is challenging due to very high loading and rendering times because of the limiting polygon draw rates of GPUs.A solution to this is the derivation of simplified low resolution 3D models, referred to as 'low-poly' models (Cohen et al., 1998;Stathopoulou et al., 2011;and Chopine, 2011).Figure 2 shows a high cross documented by the Discovery Programme.The high cross is displayed as a high-poly and a low-poly model demonstrating simplification.Utilising a combination of simplification and smoothing processing polygon counts are optimised to a point where the overall geometry is retained whilst the detail captured such as fine decoration or evidence of micro erosion is smoothed out of the model.The resulting lowpoly models are assessed through observation of their silhouette to ensure overall geometric form has not been downgraded.Final low-poly model sizes will vary depending on the complexity of the object and normally range from between 1 million to 50,000 polygons in total.

3D MODELLING
In the field of digital documentation of tangible cultural heritage objects, effective visualisation and rendering of final products is critical to the final presentation of 3D data and perceptions of quality by the user (Guo et al., 2016).Critical decisions on appropriate methodologies within the 3D modelling pipelines can affect the user's visual quality assessment when observing and interacting with the 3D model.

Texture maps
Through the process of UV baking, details from a high-poly 3D model, can be transferred to a 2D texture map that can be applied to a low-poly version of the model for optimised and real-time rendering.This enables efficient visualisation and viewing of the 3D asset.In order to carry out the baking process, a UV map which is associated with the low-poly model must be created.The creation of UV maps for 3D models can be achieved automatically by photogrammetric software (Metashape, Reality Capture), however, the resulting texture map is often made up of hundreds of separate UV islands.This results in large amounts of seamlines and inefficient use of the texture space due to large amounts of pixel padding required to minimise the effects of seam edges.Within our developed pipeline a combination of semiautomatic and manual UV map creation is carried out in order to reduce the number of seamlines.Initially utilising the Unwrella Plugin (Unwrella Development Team, 2023) within Autodesk 3dsMax (Autodesk Development Team, 2024) the low-poly model is unwrapped.The subsequent UV map is visually assessed against draped on the low-poly model to identify where seamlines could be welded to form larger UV mapped areas with fewer seamlines as shown in Figure 3.The final 3D model may contain areas which have been cut through either by sectioning the low-poly model or by removing unrequired detail from the surrounding environment (box cutting the model).These areas are shaded with a flat colour grey to indicate that they do not represent the surface of the cultural asset.As there is no textural detail in these areas the associated UV islands are reduced in size and overlayed on top of each other preserving texel density for the surface of the cultural asset.Before finalising the UV map creation, the resulting texture is assessed for both angle and area distortion within the UV editor.This is to ensure that the final model will generate high visual quality models that retain intricate details of the cultural heritage assets surface.To produce effective visualisation of the 3D texture and ensure optimal display performance it is important to consider the optimal size of your UV map.For photogenerated texture this should be as high as possible with 8 k (8192 x 8192 px) to ensure all photographic detail of the surface is preserved.For any textures automatically derived through the analysis of the high-poly and low-poly models against each other (normals, height, convex, position, curvature, concavity, convexity and ambient occlusion), limitations on the detail which can be effectively represented and displayed in the texture map is controlled by the overall polygon count of the high-poly model.For these maps the total texel resolution of the UV map should be around the same size as the total polygon count of the highpoly model as each polygon surface can be represented by only one texel (see Table 1).The export of UV maps can be limited by texturing software such as Adobe Substance Painter to 8 k (Adobe Development Team, 2023).Although UDIM functions are available in most texturing and rendering software which enable the use of multiple texture tiles, they are not currently employed within this pipeline as they are not supported by Sketchfab.1. Optimal UV texture map sizes depending upon the total size of the high-poly model.

Synthetic texture generation
When generating textures for the cultural heritage 3D asset, Physically Based Rendering (PBR) also known as physically based shading (PBS) is employed which is a method of rendering a 3D model where the material properties simulate the physical way light interacts with surfaces enabling realistic visualisation under different and dynamic lighting conditions (Pharr, 2023).The surface behaviour considers several factors including the albedo (diffuse colour), roughness, metalness, reflectivity, in addition to the environmental lighting conditions.Surfaces are described through the use of separate texture maps in addition to a normal map which provides additional surface detail by encoding variations in surface normal between the high-poly and low-poly models, allowing for the simulation of fine details without additional geometry.At a minimum an albedo and a normal map are used to provide the texture of 3D heritage assets, with the photorealistic albedo texture being derived from the original SfM-MVS photogrammetry imagery.
The use of photo generated albedo textures has the ability to generate highly realistic reality models of cultural heritage objects.Many of the cultural heritage assets currently being digitally documented by the Discovery Programme include a large proportion decorated stone features such as decorated high crosses, or megalithic art or the remains of large stone structures and monuments.For these type of cultural heritage assets, the ability to inspect form and geometric shape is critical in identifying and interpretating carved detail or evaluating conservation issues due to surface deterioration.By simply employing a photo realistic texture map the geometric aspects of the surface of objects with highly variable surface relief, can be masked through colour variations present on the surface for example, due to lichen growth or differences in stone colouration.Efforts to enhance geometric visualisation through the use of ambient occlusion (AO) textures (Pharr and Green, 2004;and Rolland et al., 2021) have been employed which produce shadowing technique used to make 3D objects look more realistic by simulating the soft shadows that should naturally occur when indirect or ambient lighting is cast out onto your scene.While AO textures provide the ability to enhance surface geometry, the appearance can look somewhat artificial and removed from the original object.Within Adobe Substance 3D Painter these baked texture maps can be imported together with normal and AO textures and combined as a series of stacked texture layers, which can be adjusted to produce a synthetic texture (i.e.Smart Material) surface that both emphasises the geometric features of the cultural heritage assets whilst retaining a visually pleasing and realistic appearance.Once finalised, the Smart Material can be saved and applied to additional 3D models taking the geometry of the targeted 3D asset into account and adjusting the textures automatically.Within the Discovery Programme pipeline, both a final photo texture and synthetically derived Smart Material textures are derived in conjunction with a final normal map.
Where required an additional diffuse texture can be derived by combining the two derived layers in Adobe Photoshop to produce a new hyper realistic texture which illustrates both the colour texture of an object but also the geometric details. Figure 5 displays 3D models of a high cross rendered with photo derived texture map and with synthetically derived Smart Material texture.

Orthoimage generation
Although the ability to generate interactive 3D models exists, conservation experts often require a series of formal plans, elevations and sections when presenting potential condition interpretation and intervention planning.Normally this is delivered in the form of a photo derived orthoimage through photogrammetric software (Metashape, RealityCapture).However, the geometric nature of the object's surface is not present within the output.As described above, the new synthetically derived Smart Material and hyper realistic textures can be used in conjunction with the normal map to effectively visualise the heritage but a method to derive orthoimages from this model was required.Utilising Marmoset Toolbag's ability to provide orthographic cameras, an orthographic studio rendering setup was created (Figure 6).Six orthographic cameras orientated in primary directions (front, back, top, bottom, left and right) were placed in order to generate a series of corresponding orthoimages.For the photographically textured orthoimages the scene is lit with a plain white High Dynamic Range Image (HDRI) which ensure full and consistent lighting across the model.When generating orthoimages based on the synthetically derived textures each camera orientation has an associated 3point lighting setup (key, fill and rim) which provides the maximum visualisation enhancement for each orthoimage.To provide scale to the series of exported orthoimages a regularly sized cube (e.g.1m 3 , 2m 3 , 5m 3 ) is included within an additional renders and acts as a reference scale to proportionally size the final outputs.Once all the orthoimages have been rendered in Marmoset Toolbag, they are imported into Adobe Illustrator where they are scaled and arranged to form the final series of orthoimage plans, sections and elevations (Figure 7).Where required the creation of sections through the 3D heritage assets can be generated, however this requires use of the slice plane modifier in Autodesk 3DS max and subsequent editing of the UV map where new polygons are created at the point of the section cut.

Online access
The Discovery Programme's pipeline to deliver high quality 2D extractions from the models satisfies the professional user, but there is still a need to disseminate 3D models in a format and on a platform that is accessible, without the need for bespoke software.The solution chosen was Sketchfab, a 3D modelling platform developed to publish, share, discover, and sell 3D content.It provides a viewer based on the WebGL and WebXR technologies that allows users to display 3D models on the web, to be viewed on any mobile or desktop browser.The platform has proved hugely popular for cultural heritage content, with over 100,000 3D models in Sketchfab's Cultural Heritage & History category.Europeana has its own collection to encourage exploration and sharing of models from many of Europe's most important heritage institutions (Murphy, 2017).Low poly models are uploaded, (obj, max file size 2GB) along with the material file textures generated in the processing described in section 5 to Sketchfab.Adding relevant titles, descriptions, tags and categories is crucial to make the 3D model discoverable.The Discovery Programme collection contains over 500 models (some private), structured into collections to help the user find associated models.(https://sketchfab.com/discoveryprogramme/models). Figure 8 is one such 3D model created by the Discovery Programme that has been uploaded to Sketchfab for public viewing.

Alternative Approaches
Autodesk Revit is the leading software for Building Information Modelling (BIM), but it has yet to be fully adopted for Historic BIM (HBIM) (O'Rielly et al., 2023).This is because the software was developed primarily as a modelling tool for parametric and geometric shapes (O'Rielly et al., 2023).Archaeological and Heritage sites do not generally conform to these shapes as they often contain complexities and irregularities that present challenges for modelling in software such as Revit (O'Rielly et al., 2023).For sites that may be more suited to 3D modelling, mainly buildings, there is a lack of modelled parametric families available for elements that can be found in multiple sites (O'Rielly et al., 2023).While Revit can be used as a HBIM tool, at present it is generally used for individual projects.To be employed on a wider scale it would require a large amount of time to set up a comprehensive library of parametric families (O'Rielly et al., 2023).

DISCSSION
This pipeline has been well by researchers at the Discovery Programme, and by clients in heritage agencies who have previously struggled to access and engage with 3D survey data.Providing outputs that can further research and conservation agendas justifies the investment this approach to survey.It also provides a clear pathway to increase the audience for such models.The Discovery Programme Sketchfab site has 9,500 followers, with over 60,000 views of a 3D model of the passage at Newgrange.A consequence of the raised profile and awareness of our collection of models had led to innovative and novel uses; from 3D games, film and TV shows to use in education and art.Issues to consider when thinking about applying this workflow are resources, time, and technical skills.The workflow for producing models with such high-quality textures is computationally intensive and therefore requires high computational power as well as personnel time and expertise.For modelling relatively small objects such as high crosses, these three issues did not present themselves as big problems.Modelling complex historic buildings using multiple data sources, however, was found to be significantly challenging.The editing required in mesh unwrapping currently requires skills beyond those from a geomatics background.This will likely become more automated in time but there may be situation where a more modest set of textures, based on point cloud data will need to suffice.Sketchfab is a commercial company, and in July 2021 it was acquired by Epic Games.This emphasises the value of 3D models to game developers, but also highlights the need be cautious in relying on a single commercial platform for online access.The components of our models are versatile, robust, and will be appropriately archived, so they will be fully adaptable to the needs of dissemination platforms in the future.

Figure 1 .
Figure 1.Illustration of the modelling pipeline developed by the Discovery Programme for the digital documentation of cultural heritage structures.

Figure 2 .
Figure 2. Simplification of high-poly mesh of 128 million polygons (left) to low-poly mesh (left) consisting of 250,000 polygons and the reduction in detail in the decorated surface.

Figure 3 .
Figure 3. Reduction of seamlines (green) in two 3D models of Ahenny High Cross (North), County Tipperary with texture map derived from the distance between the low-poly and high -poly models.Position x, y and z position of your mesh represented in the RGB channels of the image Curvature greyscale map representing concave (bright) and convex (dark) shapes within a single map Concavity greyscale map of the concave nature of the object surface, representing small, indented features such as pits are highlighted Convexity greyscale map of the convex nature of the object surface, surfaces representing sharp edges are highlighted Table 2. List of additionally baked textures used to enhance visualisation of detailed surfaces.

Figure 4 .
Figure 4.An example of the baked texture components used to generate synthetic Smart Material textures.Recent developments in procedurally based texturing tools such as Adobe Substance Painter (Adobe Development Team, 2023) and Marmoset Toolbag (Marmoset Development Team, 2023) now enable the development of a new enhanced geometric texture which can generate several further geometrically baked texture representations of the surface in addition to AO and normal textures which are summarised in Table 2 and illustrated in Figure 4.

Figure 5 .
Figure 5.An example of a photo derived texture map (left) and synthetically derived smart material texture (right) of the 3D model of the decorated Northern High Cross at Ahenny, County Tipperary.

Figure 6 .
Figure 6.View of Marmoset Toolbag software setup with three-point lighting and orthographic cameras.

Figure 7 .
Figure 7.An example of a scaled orthoimage elevation and section plots of Strade Abbey, County Mayo, utilising combined photographic and synthetic textures arranged and scaled in Adobe Illustrator.