INTEGRATION OF TERRESTRIAL LASER SCANNING AND SMARTPHONE LIDAR: THE CASE STUDY OF LIDZBARK CASTLE

: In many cases, archaeological research allows to explain the history and evolution of heritage buildings. To support these studies, the availability of reliable 3D representations of such buildings has already been shown to be of remarkable importance. Thanks to the development of Terrestrial Laser Scanning (TLS) and close-range photogrammetric techniques, the generation of such 3D representations by means of remote sensing techniques has become quite common nowadays. While TLS can still be considered as the state-of-the-art method for generating accurate building geometric models, acquisition methods based on the use of mobile devices is becoming a quite attractive solution in order to speed up the survey while also enabling an easier data acquisition on certain narrow places. Despite a plethora of mobile mapping tools is currently available on the market, smartphones are clearly the most widely used smart mobile device in the world, being an ideal solution in terms of usability, portability, and spread. Motivated by this observation and by the recent availability of smartphones provided with Light Detection and Ranging (LiDAR) sensors, this paper aims at investigating the effectiveness of the integration of traditional TLS with smartphone LiDAR on the survey of the castle in Lidzbark Warminski, which dates back to the mid-fourteenth century. TLS, with the scanner set on a standard tripod, was used to scan easily accessible places. TLS mounted on a dedicated tripod, patented by the co-authors of the project was used for less easy-to-reach places.


INTRODUCION
Geometric documentation is one of the most basic and important tasks in the fields of conservation policy and management of cultural heritage objects (Markiewicz et al., 2015).3D documentation is a prerequisite for both conservation and restoration interventions on historical objects and sites.It usually includes: (a) coordinates (x, y, z) of a set of selected points representing key locations needed to recreate projections of the object outline (e.g.building corners), (b) object model in digital form, e.g.HBIM (Heritage Building Information Modeling) models (Abbate et al., 2022), (c) 3D models in the form of printouts, e.g.display models in museums in appropriate scales.In order to prepare such documentation, first of all, it is necessary to accurately reconstruct the shape of the considered object.This operation is often performed by using Terrestrial Laser Scanning (TLS), which can be considered the state-of-the-art technology for this task.TLS is a very convenient 3D data acquisition tool, as it rapidly collects even millions of 3D point positions per second, and it is a non-invasive surveying technique (Błaszczak-Bąk et al., 2019;Suchocki & Katzer, 2016), hence it avoids accidental damages to the surveyed objects.Nevertheless, despite TLS scanners are portable, and certain of moderately small size are now available on the market, their usability in narrow places is still quite low.Hence, alternative acquisition methods shall be considered for hard-to-reach places, or to increase the level of detail on certain characteristic areas of the considered object: in these cases, the use of a mobile device (e.g.smartphone or tablet) provided with a LiDAR sensor could be a viable choice (Murtiyoso & Grussenmeyer, 2021).Narrow places, where it is difficult to set up a geodetic instrument, can hardly be measured with TLS.Despite some actions can be undertaken in order to ease a laser scanner acquisition in these cases, e.g. using a non-standard tripod, they are not always sufficient to obtain an effective result.For instance, due to the size of terrestrial laser scanners, even the use of non-standard tripods may not give the expected measurement results.In addition, the integration of TLS surveys with other methods is motivated by the need of achieving the required point cloud density on all the object areas.Among the possible additional tools to be integrated with TLS, this work considers the use of a smartphone with LiDAR (SwL).Given its small size, it can be effectively used to collect measurements even where a large terrestrial scanner cannot be set up and where a TLS laser beam cannot reach all the object surface due to obstructions.Two scanners can be available in SwL devices: mounted on the front and rear of the device.Their characteristics are usually different: when using an iPhone 12 Pro, the rear scanner minimum spatial resolution is 5 mm, while for the scanner on the front is 0.5 mm.The LiDAR sensor in the iPhone 12 Pro' most significant characteristics are: (1) Range: The LiDAR sensor in the iPhone 12 Pro has a range of up to 5 meters (16 feet), which means it can accurately measure distances up to 5 meters away.
(2) Resolution and speed: The LiDAR sensor in the iPhone 12 Pro has a resolution of up to 1 million points per second, which means it can create a highly detailed 3D map of the surrounding environment and its very fast and can capture 3D data in realtime.
(3) Power: The LiDAR sensor in the iPhone 12 Pro is low-power and has minimal impact on the device's battery life.SwL scanners have already been tested for inventory and documentation in several kind of scenarios, such as in forests (Gollob et al., 2021;Mokroš et al., 2021) and cliffs (Luetzenburg et al., 2021).In the authors of the mentioned papers found that the LiDAR sensor, introduced by Apple Inc. in 2020 for the iPad Pro and iPhone Pro models, represents a novel, cost effective and time efficient alternative to established methods of topographic land surveying, like TLS and Structure from Motion/Multi-View Stereo (SfM/MVS).
The paper aims at showing the advantages of using a smartphone with LiDAR when completing the TLS survey on a portion of the walls of the ruins of the Lidzbark Castle (Lidzbark Warmiński, Poland).In particular, smartphone with LiDAR has been used on those locations, hard-to-reach, where TLS either cannot be used or the obtained TLS point cloud was incomplete.

Methodology
An accurate documentation of culture heritage objects, sometimes even of the remaining object fragments, often requires detailed measurements and a dedicated approach.Despite TLS scanning spatial resolution can usually be considered as high, it is still insufficient in certain cases, in particular when the scanner cannot be positioned sufficiently close to the object.Hence, an integrated approach is considered here, where Figure 1 summarizes the integration workflow implemented in this paper, in order to properly merge TLS point clouds with SwL ones.

Figure 1. Methodology
Step 1 deals with the survey planning and execution, properly taking into account of the object and sensor characteristics.For what concerns the TLS survey, the TLS network (i.e. the number of scans and their positions) should be determined so as to ensure full coverage of the object, to meet the required precision, while minimizing the survey time and scan locations (Jia & Lichti, 2019).Increasing the viewpoints, i.e. the number of scan locations, clearly increases the coverage and the details, but also redundancy in the acquired point clouds and processing time, i.e. increasing the time required for point cloud registration.The optimal scanner-object distance should be chosen taking into account of the required precision and of physical/geometric constraints in the considered area.Increasing the number of scan stations in certain cases may not be so convenient, for instance if the goal is that of covering areas associated to unavoidable occlusions for the TLS survey.Integrating the TLS survey with information acquired by means of other instruments (e.g.smartphone with LiDAR) can be convenient in these operating conditions, as done in the case study considered in this work.In particular, after step 1 two kinds of point clouds were obtained: TLS (TLS_PC, coloured, thanks to the RGB data acquired by the scanner's built-in digital camera) and SwL point cloud (SwL_PC).
Step 2 is the pre-processing phase, during which the point cloud is denoised and trimmed to the area of interest.CloudCompare software was used to combine the TLS_PC point clouds obtained from different workstations, and then for the integration between TLS_PC and SwL_PC.
Step 3 aims at analysing the obtained results in order to assess the quality and usefulness of the integrated point cloud.

Research object
The Lidzbark Castle is a historical place, well-preserved despite its quite old age.Its history includes the centuries-old relations between Warmia and the Crown of the Kingdom of Poland.The location of this historic castle is shown in Figure 2. The castle is open to visitors, but not all its parts are restored and safe.Indeed, a portion of the castle is mostly composed by ruins that have to be constantly monitored.In these destroyed parts of the castle, measurements, archaeological and architectural research are carried out.TLS measurements in this area of the castle are very difficult due to the narrow gaps, collapsed walls and short passages, which make almost impossible to set up a TLS station there.
A hardly accessible portion of the ruins and a fragment of an object of archaeological interest are shown in Figure 3.

Results
As previously mentioned, data acquisition campaign has been carried by using two different instruments (TLS and Smartphone with LiDAR), with TLS working either on a standard or a dedicated patented tripod.TLS point cloud, namely TLS_PC, has been obtained by the combination of all the TLS scans.

DISCUSION AND RESULTS ANALYSIS
This Section aims at providing some insights on the results obtained with the proposed method, which, in particular, will be done by comparing the scan fragments visually (subsection 3.1) and statistically (subsection 3.2).Finally, the effects of using either the merged point cloud on mesh generation are shown in subsection 3.3.

Visual comparison
Visual comparison of some portions of the overall 3D point clouds are presented in Figures 7 and 8. Figure 7 and 8 visually shows that conducting a multi-sensor campaign allowed to improve the coverage of the object surfaces, hence significantly reducing the gaps present in the final point cloud.

Statistical comparison
This subsection at providing some numerical results in order obtain a more objective comparison between the data acquired by means of TLS and Smartphone LiDAR.To such aim, some areas of the object have been considered, i.e. those shown in Figure 9 and The number of points acquired with TLS and Smartphone LiDAR are shown in Table 1 for the two areas shown in Figure 9 and 10 and for the overall survey dataset (last line in Table 1).
It is worth to notice that, despite the Smartphone LiDAR-based point cloud do actually is much smaller (cardinality one order of magnitude lower than the TLS one), it allows to properly cover the gaps in the TLS cloud.This is clearly due to the use of the Smartphone LiDAR on the most critical areas.To be more specific, the numerical comparison of Table 1 shows that in the areas considered in Figure 9 and 10 the Smartphone point cloud is 15-25% smaller than the TLS one, hence the proportion among the average density of the Smartphone and TLS clouds is slightly higher on these fragments: this is clearly due to the fact that the Smartphone cloud covers also those areas not properly covered by the TLS cloud, hence compensating the deficiencies of the TLS data acquisition.In accordance with what mentioned above, the TLS-Smartphone LiDAR merged point cloud can be considered as much more adequate with respect to the TLS one in order to properly describe all the object areas, e.g.significantly reducing the blind spots.

Mesh generation
Mesh model generation clearly cannot succeed in properly describing all of the object areas in case of gaps in the point cloud used to generate the mesh.This subsection aims at showing that the point cloud obtained merging TLS and Smarthone LiDAR ones is well suited for mesh generation, thanks to the compensation of TLS cloud gaps with the Smartphone data.To be more specific, two additional portions, Fragment 5 and 6, are shown in Figure 11a and 11b, respectively.Fragment 5 represents a stony, irregular part of the wall, while fragment 6 is a portion of a wall made of regular bricks.The bottom of Figure 11 shows a magnification of the mesh generated for such two fragments, making the TIN structure visible.From the magnified views in Figure 11 it is quite clear that the point cloud obtained properly merging TLS and Smartphone data allowed the generation of a TIN properly modelling the considered areas, where blind spots are not visible anymore.

CONCLUSION
This article shows some advantages of a multi-sensor survey campaign, where TLS data have been merged with those obtained with the LiDAR of a smartphone iPhone 12 Pro.Both the obtained datasets are point clouds stored with the data file type and they can easily be exported in several formats (e.g.e57, .pts,.las),which can be used in many laser data processing software.
In accordance with what shown in this paper, integrating data from TLS and smartphone with LiDAR have several advantages, including: 1. Improved coverage of the object surfaces: combining point clouds from multiple sources can result in higher coverage, which in certain cases may even be associated to a higher data quality, as each technology has its strengths and weaknesses.TLS is ideal for capturing high-resolution data of easily visible/reachable objects, while SwL can be a good solution to capture data on hard-to-reach areas.By combining these data sources, it is possible to fill the gaps in the respective datasets and hence improve the overall coverage/completeness of the obtained model.2. Enhanced visibility of certain object geometric features, which can be exploited not only for the mere generation of a geometric model of the object, but also for implementing more engaging and informative data visualizations.Overall, integrating data from Terrestrial Laser Scanner and smartphone with LiDAR data can lead to a more detailed, accurate, and informative representation of historic buildings.This can support documenting, determining the need of maintenance operations (Suchocki et al., 2019), restoring and preserving cultural heritage sites, enabling a higher level of data interpretation, and it can be potentially useful also for increasing the public engagement.

Figure 2 .
Figure 2. View of the Lidzbark Castle and its localisation in Lidzbark Warminski, Poland

Figure 3 .
Figure 3. Fragment of research object

Figure 4 .
Figure 4. Tripod No. P.430066: (1) three-arm cast iron base, (2) adjustable legs, (3) telescopic leg, (4) vial, (5) head Finally, iPhone 12 Pro smartphone, provided with a LiDAR sensor, enabled 3D data acquisition on very hard-to-reach locations.The smartphone was at a distance of about 20-30 cm from the object during data acquisition.Scanning with the iPhone 12 Pro LiDAR can be carried out using a number of applications available in the App Store.The authors used the 3d Scanner App [Laan Labs].
Similarly, SwL_PC, the Smartphone with LiDAR point cloud, results from the all Smartphone data acquisition.Two examples of the data acquired by TLS and Smartphone are shown in the Figure 5. Registration of TLS_PC and SwL_PC has been performed by means of the Leica Cyclone software.

Figure 6 .
Figure 6.Point clouds of Figure 5 after fine registration (colour intensity)

Figure 10 .
In both the cases, the merged point cloud is shown on the top of the figure, whereas details of the TLS and Smartphone point cloud are shown in subfigures (a) and (b) on the bottom.