HYBRID ADJUSTMENT OF UAS-BASED LiDAR AND IMAGE DATA

: Several advancements are going with Unmanned Aerial Systems (UAS) with the addition of multiple sensors and simultaneous data acquisition to obtain detailed geo-data for various applications. However, simultaneous data acquisition with multiple sensors, namely camera, and LiDAR, will also result in possible discrepancies associated with them, and they need to be solved to use a reliable and accurate final product. Several errors can be associated with both camera and LiDAR datasets due to the different characteristics of the sensors and terrain conditions. This research paper aimed to minimize the errors between LiDAR and the image datasets simultaneously acquired with an Unmanned Aerial System (UAS) by implementing a hybrid adjustment approach with a criterion for the roughness and threshold angle between surface normals. The initial trajectory of the UAS, raw LiDAR measurements, and image observations were the inputs used for the hybrid adjustment. The hybrid adjustment workflow minimizes the discrepancies with a least-squares-based simultaneous adjustment for both LiDAR and image datasets. For the hybrid adjustment process, three types of correspondences were established, namely: between image pairs, overlapping LiDAR strips, and between Image tie points and LiDAR strips. For quality control, mean Cloud-to-Cloud distances (C2C) were compared between both LiDAR and camera point clouds before and after hybrid adjustment. The surface-level analysis of the results was also carried out to analyze the errors before and after hybrid adjustment at a surface level for different types of surfaces. The results showed that the alignment between the point clouds has significantly improved from the range of meters to a centimeter-level after implementing the hybrid adjustment process. The proposed hybrid adjustment workflow can be used in mapping applications where a centimeter-level accuracy is requested.


INTRODUCTION
In the last decade, there have been significant advances in Unmanned Aerial Systems (UAS) based data acquisition.It has been an efficient acquisition method these days to simultaneously collect multi-sensor data onboard a UAS platform.The technical progressions in UAS-based sensor technology and data acquisition have led to the emergence of high-resolution data availability.The multi-sensor platform-based applications are highly flexible for data acquisition (Baltsavias, 1999).The different sensors, namely LiDAR, and camera, are emerging technologies for 3D topographic mapping, especially when used onboard low-cost UAS platforms.The LiDAR point cloud provides accurate 3D surface information in the form of the scattered point cloud, whereas aerial photogrammetry provides information through stereo vision directly in the form of spectral imagery (Yang and Chen, 2015).Optical imagery has been known for providing high-quality details along the object boundaries with variations in elevation (Kim et al., 2006).Numerous studies have been carried out for the reconstruction of 3D surfaces using LiDAR data and aerial photogrammetry simultaneously due to their complementary characteristics (Habib et al., 2005).The main advantage of using the LiDAR system is the direct acquisition of the 3D coordinates from the ground objects and the better vertical positional accuracy when using an airborne platform.In contrast, camera images provide dense spatial information with better horizontal accuracy (Choi et al., 2011).The datasets acquired from the UAS-based LiDAR system have low-cost, denser point clouds and shorter response times than traditional aircraft or helicopter-based LiDAR data acquisition (Nex et al., 2022).In the case of multi-sensor data acquisition, the advantage of the complementary characteristics of different sensors can be utilized and hence combining the data from multiple techniques would give us accurate, high-quality, and detailed surface information (Baltsavias, 1999).The use of multi-sensor platforms with a LiDAR unit and a single/multiview camera for synchronized data acquisition has represented a market trend for the 3D reconstruction of the earth's surface (Toschi et al., 2018).The accurate coregistration of hybrid UASbased camera images and LiDAR data can positively influence several applications in the sector of mapping, 3D city modeling, the AEC sector (Alsadik et al., 2022).The LiDAR point cloud and the derived point cloud from camera images have some misalignments during their coregistration due to systematic errors in the multi-sensor UAS system (Glira et al., 2019).The primary source of these systematic errors is GNSS/INS navigation systems and the boresight alignment from mounting calibration.These misalignments and distortions compromise the coregistration quality of the datasets.The possible errors between LiDAR and camera point clouds can be attributed to the different data acquisition sensors and their characteristics.It is not easy to get a highly accurate coregistration between LiDAR point clouds and camera point clouds using only hardware synchronization and bore-sight calibration (Skaloud and Lichti, 2006).The initial UAS trajectory data also needs to be considered for the precise coregistration of both datasets.To solve this geometric state of art problem, many strategies exist to minimize the discrepancies between the LiDAR and camera datasets, but very few focus on the simultaneous adjustment of both LiDAR and camera datasets.The combined advantage of LiDAR data and images can be fully utilized after eliminating geometric inconsistency between both datasets, i.e., geometric registration, which arises due to the systematic errors of a multi-sensor system (Habib et al., 2005) There are numerous supporting reasons for the applications of multi-sensor data, including the complementary characteristics of two sensors and the detailed multispectral information offered in combination with photogrammetry (Toschi et al., 2021).Their use is limited to concurrent flights and finds its place in exploring the fusion of data collected at different timestamps.
The two primary approaches for registering multi-sensor datasets, especially LiDAR, and camera, can be classified as area-based registration and feature-based registration (Palenichka and Zaremba, 2010).Area-based registration methods focus on optimizing exterior orientation parameters of the images by maximizing the statistical or grayscale comparations of the similar area in both datasets.On the other hand, data-driven feature-based fusion approaches perform the integration by fetching out the features from LiDAR and images to generate correspondences to estimate the camera poses.The area-based registration approaches are based on the statistical dependence of LiDAR and imagery and are mainly dependent on the quality and correctness of image intensity which is interpreted with intensity calibration effectiveness (Yang and Chen, 2015).However, the feature-based method extracts geometric features from a scene to register the datasets (Zhou et al., 2021).Generally, the methods used for extracting the features from the datasets depend on the characteristics of the individual source data, which can vary if different datasets are used.Built upon Glira et al. (2019), the proposed hybrid adjustment methodology includes additional constraints on parameters like surface roughness and threshold on the surface normal of the corresponding points from LiDAR point cloud and image tie points, which affect the overall performance of the hybrid adjustment.In our hybrid adjustment approach, the LiDAR strips were adjusted along with the camera images with an estimation of camera orientation parameters.In the methodology, a uniform sampling was also considered to select the corresponding points more effectively for the hybrid adjustment of LiDAR and camera data.In summary, the main contributions of the paper are:

•
A workflow for the hybrid adjustment of UAS-based LiDAR and camera data without the use of any ground truth data; • Use of uniform data sampling LiDAR and camera to establish the correspondences between LiDAR and image data from the entire dataset area; • Involving the criteria for the roughness and threshold angles between the surface normal for the hybrid adjustment.

RELATED WORKS
Many potential research works have been carried out for the combined processing of LiDAR and imagery to get a detailed and accurate end product, e.g., an integrated approach to generate the orthophotos, LiDAR, and image data integration for building modeling (Brenner, 2005), combining image and LiDAR data for automatic reconstruction of railroad centreline (Beger et al., 2011), and fusion of image and LiDAR point clouds for the derivation of Digital Surface Model (DSM) (Mandlburger et al., 2017).Toschi et al. (2021) has formulated a process based on aggregating the features and evaluating all 3D points sensorspecific and pointwise.Their integration approach has proven to work well with georeferenced point clouds without any flight trajectories, but it has some limitations with point density variation, misalignment within a point cloud, or point clouds acquired from different platforms.Acquisition of data from different UAS or terrestrial platforms would lead to point clouds with diverse quality and requires the selection of the most suitable quality features (Toschi et al., 2021).
Most of the research in this field is focused on the independent adjustment of LiDAR block adjustment and aerial triangulation for camera images.As per the integration framework by Abayowa et al. (2015), the Iterative Closest Point algorithm (ICP) was implemented to optimize the discrepancies between the Digital Surface Models (DSMs) obtained from LiDAR and image data.This approach was based on relative orientation by matching invariant and salient features in DSMs from LiDAR and imagery point clouds.
The adjustments of the LiDAR strips (Strip Adjustment, SA) and the Bundle Block Adjustment (BBA) for image rays with the same GNSS/INS trajectory have also led to an acceptable coregistration of the multi-sensor datasets.Ground Control Points (GCPs) enable the refinement of the internal camera parameters, boresight calibration, and image orientations.(Yang and Chen, 2015) presented a sophisticated approach to integrating the image sequences and LiDAR data from a UAV, minimizing the discrepancies between two-point clouds.The approach was based on matching building outlines without any rigorous modelling of the measurements.In this approach, only a rigid body transformation was applied to match the images and LiDAR block, resulting in a moderate georeferencing accuracy.
The method of integrating point clouds from LiDAR with the camera images point cloud on the criterion of bias detection after adjustments had been a compelling method with the optimal need of GCPs collection for BBA (Toschi et al., 2018).The similar geometric features (points, surfaces, and edges) create a transformation between different LiDAR reference frames and camera coordinate systems (Habib et al., 2005;Peng et al., 2019;Yang and Chen, 2015).
The GNSS/INS-assisted LIDAR integration approach by (Zhou et al., 2021)initiates with the point cloud generation with point positioning equation, and LiDAR/GNSS-assisted SfM followed by iterative identification of correspondences from both point clouds and integrated bundle adjustment.SIFT algorithm-based tie points are also established in this process to derive the sparse image point cloud after refining system calibration parameters from bundle adjustment.It also considers the planar constraint to the seed point to identify the corresponding patches from LiDAR data.Another target-based image and LiDAR data integration approach was conducted by (Pentek et al., 2020), employing LiDAR strip adjustment (LSA) in the initial step, followed by using LiDAR point cloud from the initial step as a reference for the camera system calibration.3D coordinates were estimated with the intersection between light rays from images and LiDAR points, minimizing the distance between these correspondences for every corresponding pair of image points from image matching.However, this integration approach did not consider the probable errors in the trajectory of data acquisition.Another efficient approach is the hybrid adjustment which considers the simultaneous adjustment of both LiDAR and camera datasets.A unified process inclusive of the strip adjustment and bundle block adjustment was found to be more robust and efficient than the existing multi-sensor data integration approaches.Glira et al. (2019) framed a method for hybrid orientation with a rigorous and iterative determination of the correspondences between the image tie points and LiDAR points.However, this hybrid adjustment method optimizes the multi-sensor block stability and integrates the sensors with the possible inclusion of the UAS trajectory and ground truth data.
In (Haala et al. (2020) authors have also developed an approach for generating ultra-high accurate LiDAR point clouds from UAVs by combining image measurements from block adjustments and trajectory corrections at the different photogrammetric processing steps.In this hybrid adjustment approach, the flight trajectory was adjusted in a LiDAR strip adjustment with the addition of observations from the photogrammetry.The authors aimed to enhance the accuracy of LiDAR strips with the use of imagery, and the accuracies with this approach were achieved to the range of Ground Sampling Distance (GSD) of the imagery without using any ground-based inputs.In Haala et al. (2022), the hybrid georeferencing of UAVbased LiDAR and camera images was implemented using image space tie points, checkerboard targets, and LiDAR control planes as additional inputs to improvise the accuracy of the adjustment.

METHODOLOGY
The hybrid adjustment has been implemented in the stripAdjust module in Orientation and Processing of Airborne Laser Scanning data (OPALS) software (Pfeifer et al., 2014).The hybrid adjustment methodology involves the initial preprocessing of raw LiDAR data, camera images, and initial UAS trajectory to be used as inputs for stripAdjust.The UTM zone, hemisphere, time lag between the platform and LiDAR sensor, and orientation of the sensors on the platform are also used as inputs in the hybrid adjustment along with the constraints on the roughness and surface normal of the corresponding points from LiDAR and camera datasets.After the implementation of the hybrid adjustment, the quality check has been done to evaluate the performance by computing the mean cloud-to-cloud distances (c2c) between LiDAR and the camera point cloud.

Data Pre-Processing
For the hybrid adjustment, raw LiDAR measurements (.rdbx) in Scanner Coordinate System (SCS) were used as inputs.The camera images were processed in Agisoft Metashape with the initial orientation parameters.Further, the processed Metashape project was exported to the exterior orientation of images, tie object points, and image point observations as inputs to the hybrid adjustment using a Python script.The sensor orientation of the scanner on the UAS platform and the time lag between the UAS platform and sensor observations were also determined using visualization and functions in MATLAB.The data preprocessing workflow has been represented in Figure 1.

Implementation of Hybrid Adjustment
The hybrid adjustment was implemented with the stripAdjust module in OPALS software.The hybrid adjustment of UASbased camera data and LiDAR strips can be possible with two types of image inputs: one is loose images, and another is coupled images.In the case of hybrid adjustment with loose images, their relation to flight trajectory cannot be established because of the unavailability of timestamps for images.In this case, the exterior orientation parameters are directly estimated by the adjustment.
The coupled images are tied to the flight trajectory, and their exterior orientation can be estimated through the direct georeferencing equation as a function of UAS trajectory and camera mounting calibration parameters.The positional and rotational parameters obtained through the direct georeferencing equation can be inaccurate if the timestamps of images are inaccurate or residuals show systematic errors.Therefore, the exterior orientation is corrected by the three coordinate correction parameters and three rotational angle correction parameters in case of hybrid adjustment with coupled images.In Figure 2, it is evident that for the loose images, the mounting of the camera is not coupled to the trajectory whereas in the case of coupled images, the camera is coupled to the trajectory.Figure 2 shows the difference between loose and coupled images in relation to UAS trajectory.In the adopted hybrid adjustment, the least squares adjustment minimizes the weighted square sum for the following mentioned types of correspondences: Further, the overlap between pairs of the LiDAR strips (STR-STR) and between image tie points and LiDAR strips (IMG-STR) are determined for the correspondence.After finding overlap, the query points are selected from STR-STR and IMG-STR correspondences with a uniform sampling technique, which selects the points from both datasets in the object space as consistently as possible.The uniform sampling technique ensures that the uniform distribution of points in the correspondences and equal-area regions are weighted equally within the hybrid adjustment.The primary iteration loop in the hybrid adjustment starts with the direct georeferencing of the LiDAR strips (with the initial parameters in the first loop and estimated parameters from the hybrid adjustment in the subsequent loops).The potential correspondences are matched, i.e., the nearest neighbour of a query point in the overlapping point cloud.The false correspondences are rejected and removed in the subsequent step based on roughness criteria, the distance between the corresponding points, and the threshold angle between the normals of the corresponding points.The correspondences that remained after the rejection step are weighted based on their surface roughness and angle between respective surface normals.It is worth mentioning that the correspondences are also newly established in each iteration of hybrid adjustment.After a given number of iterations are completed, the LiDAR strips are georeferenced with the estimated parameters in the final iteration loop of the hybrid adjustment.Figure 5 represents the main hybrid adjustment workflow after performing data-processing.The data outputs from the hybrid adjustment are as follows: • Calibrated LiDAR strips  The undistorted images and estimated orientation parameters from the hybrid adjustment are used for the post-processing of the camera images.The purpose of this step is to generate Dense Image Matching (DIM) point cloud from the distortion-free images and parameters post hybrid adjustment.The main factor considered here is that calibration was restricted to the accurate geolocations estimated by hybrid adjustment The hybrid adjustment in Opals stripAdjust module was implemented with different parameters as it was originally designed for the adjustment of ALS-based datasets.The difference in the case of the UAS-based datasets is that the correspondences/planes extracted for the matching of the LiDAR and camera datasets would be larger as compared to ALS-based datasets.

Quality Check (QC)
To assess the results from the implementation of the hybrid adjustment, mean cloud-to-cloud (C2C) distances were treated as a measure of the misalignment between LiDAR and camera point clouds.Figure 6 represents the workflow for the Quality Check (QC) of the hybrid adjustment methodology.The distortion free images from the hybrid adjustment were processed with Dense Image Matching (DIM) with adjusted orientations to obtain a point cloud.The mean C2C distances between point clouds from adjusted camera images and adjusted LiDAR strips were computed in CloudCompare using a 'Compute Cloud/Cloud distance' tool with a "quadric" model to measure the misalignment between the two-point clouds.This C2C distance method gives the distance differences between two-point clouds based on the nearest neighbor distance between two-point clouds.
For the computation, one point cloud is to be defined as a "reference" point cloud and the other as "compared" for which the distances have to be computed.For every point in the "compared" point cloud, the algorithm search for its nearest point in the "reference" point cloud, and the Euclidean distances between the nearest neighbor points are computed ((Ahmad Fuad et al., 2018;Yadav et al., 2022)).Figure 7 represents the basic idea of C2C distances computation.The "Quadric" local surface model was used in C2C distance computations to get a better approximation of the distances between the point clouds.The basic concept of C2C distances computation with local surface modelling has been represented in Figure 8.The mean C2C distances have been computed between the LiDAR point cloud after hybrid adjustment and the camera point cloud generated from the undistorted images and orientation parameters estimated from the hybrid adjustment.For mean C2C distances calculation LiDAR point cloud has been considered as 'reference' entity and camera point cloud as 'compared' entity.This C2C distance method gives the distance differences between two-point clouds based on the nearest neighbor distance between two-point clouds.

TEST AREA AND DATASET ACQUISITION SYSTEMS
Three datasets were collected over an area that comprises features like buildings, bare land, roads, and vegetation surfaces.
The ground coverage of the three datasets is ca 0.2 km 2 , ca 0.2 km 2, and ca 0.15 km 2 respectively.The datasets were acquired with a hybrid UAS-based camera and LiDAR sensors as shown in Figure 5   Image exposure stations were recorded precisely with integrated GNSS/IMU systems at an interval of 5 milliseconds (frequency = 200 Hz) for all the images acquired with the camera.The characteristics of the three datasets acquired are listed in Table 1.

RESULTS
The initial camera point cloud was generated from the images and initial orientations to compare with LiDAR point cloud to check the extent of misalignment before the adjustment.After the hybrid adjustment, the point cloud was generated from the undistorted images and estimated orientations for comparison with the adjusted LiDAR point clouds.

Mean cloud-to-cloud distances between LiDAR and Camera point clouds at the dataset level
In this research, the hybrid adjustment was tested with loose and coupled images using bias and linear trajectory correction models.For the quality checks of the performance of the hybrid adjustment, the mean cloud-to-cloud (C2C) distances were computed between the LiDAR point cloud and the camera point cloud before and after the hybrid adjustment.The c2c distances were computed with the LiDAR point cloud as "reference" and the camera point cloud as a "compared" entity for all the comparisons due to the reliability and higher number of points in LiDAR point clouds.The local modeling strategy with a versatile "quadric" model was used to compute the C2C distances along the smooth and curvy edges in the point clouds.The higher standard deviation in the statistics is due to the higher number of points for the surfaces in the LiDAR point cloud for which there are very few or no corresponding points in the camera point cloud for the computation of the C2C distances.Table 2 summarizes the mean C2C distances before and after hybrid adjustment with loose and coupled images using a BTCM.The results from the hybrid adjustment with a LTCM in combination with loose and coupled images has been compiled in Table 3.Initially, the mean C2C distances between LiDAR and camera point cloud were 1.172 m, and after hybrid implementation with loose and coupled images using bias and linear trajectory correction models, the mean C2C distances came down to the range of few centimeters range.Out of all the hybrid adjustment experimentations, we found that the hybrid adjustment with coupled images and bias trajectory correction model resulted in the least mean C2C distances, i.e., the orientation of LiDAR and camera dataset was adjusted with the least errors by hybrid adjustment with coupled images using a bias trajectory correction model.The higher standard deviation in the mean c2c distances can be attributed to different point densities of LiDAR and camera point clouds, noise in the point clouds, and possible differences in the exact coverage of the point clouds.The other reason can be given by the penetration capability of LiDAR through some surfaces like vegetation and transparent surfaces where LiDAR would have points, and the camera point cloud would not have any points.So, the absence of corresponding points would lead to a higher standard deviation in c2c distances.
Figure 10 represents the improvements in the alignment of the camera and LiDAR point cloud before and after hybrid adjustment.It is evident that there are more green regions with lower misalignments post-hybrid adjustment as compared to prehybrid adjustment implementation.
The hybrid approach was also tested on another H3D benchmark dataset acquired with different sensors and UAS platform (Kölle et al., 2021).The results from this experimentation were also found to be in similar range of centimeters as from the dataset used in this research.From Figure 11, it is evident that very few points or regions with C2C distances are within 10 cm, and after hybrid adjustment, there were relatively higher points with C2C distances within the 10 cm range.10 cm range implies the C2C distances between 0 and 10 cm.The value of 10 cm was chosen for the comparison because the average mean C2C distances after hybrid adjustment were ~ 10 cm, and this value would give a better visual interpretation of the performance of the hybrid adjustment approach.The regions or the points in RGB are those where mean C2C distances between LiDAR and DIM point clouds are less than 10 cm, whereas for the regions in grey, mean C2C distances are higher than 10 cm.

Mean cloud-to-cloud distances between LiDAR and camera point clouds at surface level
At the whole study area or dataset level, there can be some uncertainties in the computations of the mean C2C distances between LiDAR and camera point clouds due to the different sensor characteristics of LiDAR and camera sensors.These uncertainties in the mean C2C distances can also be due to different point densities of LiDAR and camera point clouds.The primary reason for this is the higher number of points in LiDAR and its penetration capability through the surfaces.LiDAR sensors with their multiple pulse returns capabilities can also include points through the surfaces like grass, and transparent glass, whereas camera point clouds only cover the top part of the surfaces.So, the surface-level analysis can better interpret the mean C2C distances before and after hybrid adjustment.For this analysis, five different types of surfaces from the study area were identified from all the datasets.The identified surfaces were the flat roof, slant roof, road surface, bare land, and road surface.The mean C2C distances before and after hybrid adjustment were computed between the different types of surfaces from both the LiDAR and camera point cloud datasets for the surface-level analysis.The planar roof, slant roof, bare land, road, and vegetation surfaces represented in Figure 7 were segmented using the "segment" tool in Cloud Compare software.It is worth mentioning that similar surfaces with the same extent and coverage were compared here for the computation of the mean C2C distances before and after hybrid adjustment.From the surface level analysis results for different types of surfaces in the study area, the errors between LiDAR and camera point cloud for different surfaces were also found to be at sub-centimeter levels.It was observed that the mean C2C distances were smaller at the surface levels except for the vegetation surfaces as compared to the mean C2C distances for the complete dataset.The higher mean C2C distances for vegetation surfaces can be attributed to the penetration capability of LiDAR through the vegetation surfaces.Figure 12 represents the mean C2C distances computation for a selected vegetation surface after hybrid adjustment.
Figure 12: Surface level mean C2C distances for a selected vegetation surface after hybrid adjustment.

CONCLUSIONS
This research contribution aimed to realize a hybrid adjustment approach for UAS-based LiDAR and image data.The purpose of the hybrid adjustment was to minimize the errors between the LiDAR point cloud and the camera-based point clouds.In the hybrid adjustment approach, the LiDAR strips were adjusted along with the camera images with an estimation of camera orientation parameters.The hybrid adjustment approach was implemented within OPALS software with two cases for image data inputs: the loose and the couple images.In the case of loose images, the exterior orientations of the camera are estimated in the adjustment itself.In contrast, in the case of adjustment with coupled images, the images are indexed to a strip, and their orientations are derived from mounting calibration parameters and UAS trajectory.Three datasets have been collected with the same UAS platform and sensors.The implementation time was almost similar for hybrid adjustment with loose and coupled images.For the initial experimentation with dataset 1, the hybrid adjustment approach was implemented with a bias and linear trajectory correction model to see their effect on the results of the hybrid adjustment.Based on the hybrid adjustment approach results, it was found that the hybrid adjustment with coupled images and bias trajectory correction model resulted in the most accurate orientation between the LiDAR and camera point clouds.For the quality check, the mean C2C distance between LiDAR and camera point clouds was chosen as a measure of the errors where a lower mean C2C distances between point clouds indicate the better relative orientation of point clouds.The initial C2C distances between LiDAR and camera point clouds were in the range of meters (1.172 m), whereas, after hybrid adjustment, the mean C2C distances were reduced to a centimetre's range (0.088 m/8.8 cm).Results from this experimentation indicated that the hybrid adjustment with a bias trajectory correction model gives a more accurate orientation of the point clouds.The mean C2C distances at the complete dataset level can give an inaccurate interpretation of the errors between the point clouds due to their different point density, penetration capability of LiDAR, different sensor characteristics, and different features present in the point cloud scene.So, the surface level analysis of mean C2C distance was done for five different surfaces, namely flat roofs, slant roofs, bare land, road, and vegetation for all three datasets.It was observed that the mean C2C distances were smaller at the surface levels except for the vegetation surfaces as compared to the mean C2C distances for the complete dataset.The results of the hybrid adjustment workflow in this research article indicate that it can minimize the errors between the LiDAR and image data from the range of meters to the centimeter range without using any ground truth inputs.This adjustment workflow can be used in the mapping applications where a centimeter level accuracy is acceptable.

ACKNOWLEDGMENT
The authors would like to acknowledge and thank team OPALS, TU Wien, Austria for providing us a scientific license key for carrying out this research.We would also like to recognize Alto Drones for their datasets which were used in the research.We would also like to appreciate and express our gratitude to IFP, University of Stuttgart for helping us with H3D benchmark datasets which were used to validate the results from the mentioned approach on other datasets as well.

Figure 1 :
Figure 1: Data pre-processing for the hybrid adjustment.

Figure 2 :
Figure 2: Hybrid adjustment with the loose and coupled images.As the trajectory correction also has a significant role in the hybrid adjustment, the tests were carried out with the bias and linear trajectory correction models in combination with the adjustment using loose and coupled images.The original trajectory of the UAS system establishes the basis for the direct georeferencing of UAS-based LiDAR strips and camera images.However, in(Skaloud et al., 2010) it was concluded that GNSS and INS trajectory measurements are strongly affected by external influences like flight maneuvers and satellite constellations.The accuracy in the measurements cannot be treated as constant w.r.t.time, leading to the time-dependent errors of the estimated trajectory, to be corrected by adjustment.The six trajectory elements of the original position [ 0  (), 0  (), 0  ()] and original orientation [ 0 (), 0(), 0()] are corrected in the hybrid adjustment for each LiDAR strip by a correction function ∆(.)[]().The simplest trajectory correction model is the Bias Trajectory Correction Model (BTCM) and Linear Trajectory Correction Model (LTCM),which correct a bias by a 0-degree polynomial and corrects the trajectory with a 1-degree polynomial respectively for each of the six trajectory elements, individually for each strip.Figure3and Figure4represent the corrections from BTCM and LTCM used in the hybrid adjustment.The correction coefficients for the respective trajectory correction model have been estimated in the hybrid adjustment process.Figure3represents the concept of a BTCM where a constant is added for bias correction whereas Figure4shows for LTCM in which corrects the trajectory with a single degree polynomial.
• IMG-IMG: reprojection errors of image tie points in the image space • STR-STR: signed perpendicular point-to-plane distances between overlapping LiDAR strips • IMG-STR: signed perpendicular point-to-plane distances between image tie points and overlapping LiDAR strips The implementation of the hybrid adjustment process starts with the image-based processing step.Initially, correspondences between the image pairs (IMG-IMG) are established in the first step.Then, aerial triangulation is implemented in the subsequent step to estimate the 3D coordinates of the image tie points from the initial step.The IMG-IMG correspondences established in this step are used in the next steps of the hybrid adjustment.

Figure 6 :
Figure 6: Quality check (QC) of the point clouds before and after hybrid adjustment.

Figure 8 :
Figure 8: Principle of C2C distances computation with the local surface modeling.
flying at ca 100 m, 90 m, and 85 m, respectively.A SONY ILCE-7RM3 camera.With a resolution of 7952 × 5304 pixels, a pixel size of 4.5 micrometers, and a focal length of 21 mm was used to acquire camera images with a GSD of around 2 cm, For the laser scanning data acquisition, a RIEGL miniVUX-3UAV scanner was used to collect the laser point cloud of 51.39 pts/m 2 , with a scanning rate of up to 100 scans per sec and up to 360 degrees of field of view.

Figure 10 :
Figure 10: Mean cloud-to-cloud (C2C) distances between LiDAR and camera point cloud before (a) and after (b) hybrid adjustment.

Table 1 :
Description of the datasets used.

Table 2 :
Comparison of mean C2C distances after hybrid

Table 3 :
Comparison of mean C2C distances after hybrid adjustment implementation with a linear trajectory correction model.