REFERENCE LIDAR SURFACES FOR ENHANCED AERIAL TRIANGULATION AND CAMERA CALIBRATION

Due to the complementary characteristics of lidar and photogrammetry, the integration of data derived from these techniques continues to receive attention from the relevant research communities. The research presented in this paper draws on this by adopting lidar data as a control surface from which aerial triangulation and camera system calibration can be performed. The research methodology implements automatic registration between the reference lidar DTM and dense photogrammetric point clouds which are derived using Integrated Sensing Orientation (ISO). This utilises a robust least squares surface matching algorithm, which is iterated to improve results by increasing the photogrammetric point quality through self-calibrating bundle adjustment. After a successful registration, well distributed lidar control points (LCPs) are automatically extracted from the transformed photogrammetric point clouds using predefined criteria. Finally, self-calibrating bundle block adjustment using different configurations of LCPs is performed to refine camera interior orientation (IO) parameters. The methodology has been assessed using imagery from a Vexcel UltraCamX large format camera. Analysis and the performance of the camera and its impact on the registration accuracy was performed. Furthermore, refinement of camera IO parameters was also applied using the derived LCPs. Tests also included investigations into the influence of the number and weight of LCPs in the accuracy of the bundle adjustment. Results from the UltraCamX block were compared with reference calibration results using ground control points in the test area, with good agreement found between the two approaches. * Corresponding author. A. S. Gneeniss, Cassie Building, Newcastle University, UK, NE1 7RU


INTRODUCTION 1.1 Background
Despite advances in surveying instrumentation and methodology, the collection of ground control points (GCPs) remains an expensive and time consuming task.Sometimes it is even more costly than image acquisition and block adjustment (Jacobsen, 2004).Therefore, in photogrammetry, the use of GNSS/IMU for direct measurement of camera stations during flight has greatly reduced mapping costs by limiting the number of GCPs needed for aerial triangulation.This integrated sensor orientation (ISO) approach is feasible for many applications where introducing tie points in the solution minimises the RMS of y-parallaxes and improves the point quality without the need for GCPs in the bundle block adjustment.Moreover, adopting large amounts of image overlap provides strong block geometry, improving point height accuracy by increasing the number of rays per observed point on the ground (Ladstädter and Gruber, 2008).Increasing the number of rays also aids the removal of the effects of any remaining errors in the orientation parameters (Cramer and Stallmann, 2001).However, the main concern in ISO is that systematic errors in the GNSS/IMU measurements, or changes in the system calibration parameters between calibration and actual flight, may not be detected without the presence of GCPs (Heipke et al., 2002) which necessitates camera re-calibration.
Camera calibration parameters are usually provided by the manufacturer, however under flight conditions the camera geometry may change relative to the situation in the laboratory (Kruck, 2006;Jacobsen, 2007).Therefore, tests to simultaneously determine the camera calibration parameters are usually performed in a permanent test field of control points (Honkavaara, 2003).Environmental conditions between the test field and mapping areas may still differ, however.Establishing a new test field for every mapping area is expensive and largely impractical.Therefore, the possibility of extracting reference control targets from lidar data has received attention from researchers.
Aerial lidar systems deliver vertical and horizontal information at high density and vertical accuracies (Lim et al., 2003).Moreover, continued improvements in the accuracy of lidar systems in recent years have enabled the use of lidar data as a source of photogrammetric control (Habib et al., 2005).The methodology for integrating lidar and photogrammetric data usually includes defining common features, establishing a mathematical model and undertaking a similarity assessment (Habib et al., 2005).Current registration methods are mainly based on identification and extraction of common spatial features such as points, lines and planes.This is followed by determining the parameters of the transformation required to align the two datasets -usually based on the 3D conformal transformation (Armenakis et al., 2012).
Feature extraction from lidar data is usually performed using segmentation or classification of the lidar point clouds.Habib et al. (2004) identified and used straight lines as conjugate features in the registration process.Straight lines were either extracted by intersecting two planes or through direct manual observation.Other methods have used planes as common features (e.g.Sampath and Shan, 2006;Brenner et al., 2008).Surface-to-surface registration is also possible by interpolating both datasets into regular or irregular surfaces, where the lidarderived surface is matched to the photogrammetric surface and the registration is accomplished by minimizing either vertical or International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-1/W1, ISPRS Hannover Workshop 2013, 21 -24 May 2013, Hannover, Germany Euclidean distances between the two surfaces (Akca, 2007).The quality of the registration is highly dependent on the registration process which can be classified into three categories: manual, semi-automatic and automatic.
Aerial triangulation is a point based process, which allows estimation of the exterior orientation parameters (EOPs) of camera positions.A number of different methods have been used to extract reference control points from lidar data for subsequent use in aerial triangulation.Mitishita et al. (2008) used centroids of rectangular building roofs as a single control point in the aerial triangulation process.Yastikli and Toth (2007) and Habib et al. (2005) used manual extraction of control points from lidar point cloud, intensity and range data.James et al. (2006) used high resolution shaded lidar DEMs to manually extract reference control points for the use in establishing a photogrammetric model.Linear features have also been used as reference targets.Habib et al. (2005) directly incorporated linear features as a source of control in the photogrammetric bundle adjustment.However, more linear features are needed to reach the accuracy achieved by conventional control point patterns in the photogrammetric block (Mitishita et al., 2008).Moreover, for large photogrammetric blocks, significant numbers and good configurations of these reference targets may not be readily available.Deriving point based control points from lidar data is also hindered by the difficulty in finding the corresponding point in the lidar dataset (Baltsavias, 1999;Habib et al., 2006;Mistishita et al., 2008).This research presents the development of a methodology to automatically register dense blocks of photogrammetric point clouds derived by ISO to corresponding lidar surfaces using a least squares surface matching algorithm.This is followed by automatic extraction of suitable lidar derived control points to perform aerial triangulation and camera calibration.

METHODOLOGY
Aerial imagery was acquired on 11 th September 2008 with a Vexcel UltraCamX camera.The data was acquired over the Vaihingen/Enz test field in Germany as part of the DGPF camera evaluation project (Cramer, 2010).The selected subblock used herein consisted of four parallel strips of 40 images with 8 cm resolution.Two strips were flown in an east-west direction and two in a west-east direction.The lidar dataset was captured on August 21 st 2008 using a Leica ALS50 laser scanner.Flying height was approximately 500 meters.Ten lidar data strips were captured with mean lateral overlap of 30% and a median point density of 6.7 points/m 2 .
The research methodology implements an automatic registration between a dense network of photogrammetric tie points obtained by ISO and a reference lidar DTM, and is performed using a 3D least squares surface matching algorithm.The principal advantages of this methodology are:  Reference tie points are measured using an automatic image matching technique which provides measurement accuracy of 0.1 pixels (Alamús and Kornus, 2008);  Extracted lidar control points (LCPs) are in a point form which can immediately be introduced into the bundle block adjustment;  The registration methodology is based on an automatic surface matching method which optimises the transformation.Repeated matching allows for improvement of the registration results through the introduction of additional parameters during the ISO process.The adopted additional parameters are a standard 12 parameter set, designed to model physically justified effects such as radial lens distortion.These parameters can be introduced in the bundle adjustment even without control points.Figure 1 shows the main steps of the research methodology.
Figure 1: Main steps of the research methodology.

Photogrammetric Point Cloud Processing
Automatic dense tie point measurement was performed using BAE Systems SocetSet 5.4.1.Automatic image measurement provides higher measurement accuracy than manual observation, which helps improve the height accuracy degraded by the smaller B/H value of digital cameras (Alamús and Kornus, 2008).It also permits measurement of dense photogrammetric point data, thereby providing a strong surface description for the matching.Combined aerial triangulation with GNSS/IMU data and tie points (ISO) was then performed using the BLUH software (Leibniz Universität Hannover).The accuracy assigned to GNSS coordinates was 10 cm in X, Y and Z. Due to the large number of tie points used in the bundle adjustment, blunder detection and elimination was performed in two steps: firstly using data snooping during the approximate image orientation calculations; secondly using robust estimators in the first run of bundle block adjustment.

Registration Method
Registration of the photogrammetric points to the lidar reference DTM was performed using the in-house, robust surface matching software LS3D (Miller et al., 2008).Initially, the software was developed to assess coastal change (Mills et al., 2005) and was later improved by introducing automatic down-weighting of outliers using iteratively re-weighted least squares (Miller et al., 2008).The software minimises vertical or Euclidean distances, using a point-to-surface approach, to obtain the seven parameters of a 3D conformal transformation (Tx, Ty, Tz, ω, φ, κ, s) (Kunz et al., 2012).In this research, TerraScan software was used to classify the lidar data.A lidar DTM reference surface was then produced.To generate a photogrammetric DTM, the tie points were processed to remove all points located over buildings and woodland.Leaving these points in the matching process would deteriorate the matching alignment by introducing anomalous regions with associated large residuals.A thematic image for areas with more than 2.5 m height obtained using a normalised DSM (nDSM) (Eq. 1) was used to omit these points.
International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-1/W1, ISPRS Hannover Workshop 2013, 21 -24 May 2013, Hannover, Germany The matching procedure is repeated to allow further registration improvement by introducing self-calibration in the bundle adjustment.At the first iteration, ISO was performed without self-calibration.At the second run, the BLUH 12 additional parameters were introduced to compensate for distortions in the aerial imagery.

Control Point Selection Strategy
Software was developed in Matlab to bridge between the BLUH bundle adjustment and the least squares surface matching algorithm.This software provides a semi-automated approach for data handling and conversion.It also includes additional procedures for automatic extraction of reliable lidar derived control points.The algorithm performs pre-match and postmatch tasks.In the pre-match task, ground coordinates of the photogrammetric point clouds computed by the BLUH bundle adjustment software are filtered using the thematic image mask obtained using Eq. 1. Points determined to be located over buildings or woodland areas are automatically removed.Finally, points are sorted, indexed and saved into two separate files; X, Y and Z format for the matching algorithm and ID, X, Y and Z as the point list file.In the post-matching step, the output files of the surface matching software are read and re-indexed using the point list file.At this stage, if the matching was not successful or the accuracy is low, a new surface matching run will commence after introducing a self-calibrating bundle adjustment in the BLUH software and repeating the pre-match task for the new set of photogrammetric point clouds.On the other hand, if the matching was successful and accurate, the algorithm will start the LCP selection procedure.This procedure includes different tests, as summarised below: Euclidean Distance: Points located in the 'best' match areas with the smallest residuals are selected.A threshold of ± 2 cm was set as a maximum distance.
Plane Test: Of the selected points with smallest Euclidean distance, only points located over planar areas were reselected.Planarity was measured by finding the best-fit plane of a 3 x 3 pixel (0.25m) window from the DSM file surrounding each point (Figure 2).Plane parameters were determined using the plane equation (Eq.2).
A threshold of 3 cm was set as the maximum distance between every window point and the computed best fit plane.This threshold represents the average of the maximum distance from the best fit plane for 17 selected points located over flat surfaces (asphalt and concrete).
Slope Test: This test is introduced to avoid points located on steep surfaces.A slope threshold represents the average slope angles for the same 17 points used in the plane threshold test.The average angle of 10.3° was set as a maximum surface angle.
Reliability Test: After applying the previous test to each of the selected points, the possibility of including blunders still exists.Therefore, this test is introduced to remove any points listed as blunders detected by robust estimators during the bundle adjustment.Also, all remaining points must be observed in at least four images.This condition is set to avoid any undetected blunders, since in "two ray points" errors cannot be detected, and in "three ray points" errors may be detected but cannot be corrected.LCP Configurations: Reference point distribution plays a critical role in achieving high triangulation accuracy.Accordingly, the algorithm was developed to maintain good point distribution based on the number of selected LCPs.In the four control point option, the block is divided into four subblocks.The nearest LCP to the centre of each sub block is selected.A similar approach is adopted if nine control points are selected, with the block divided into nine sub-blocks.The same procedure is also applied for 15 and 20 control points.However, for large numbers of points (from 50 to 300) the LCPs will be randomly selected from the control point lists.Finally, all selected points are saved in separate files in a format accepted by BLUH.

Reference Lidar Data Processing
The Vaihingen/Enz lidar data was pre-processed and corrected as a part of DGPF camera evaluation project (Haala et al., 2010).Quality assessment of overlapping lidar strips over flat surfaces did not reveal any apparent horizontal or vertical offsets.To produce a reference DTM, the data was firstly classified into different classes using TerraScan software (ground, buildings and vegetation).To reduce the data size, the ground class was cleaned and thinned.A 'key points' class was extracted from the ground class and used to produce the reference DTM.(Figure 3).

Results of Lidar and Photogrammetric Data Registration
Surface matching was performed using the lidar DTM as a fixed reference surface (Figure 4), while the photogrammetric point clouds were treated as floating matching points.The seven transformation parameters were recovered and used to align the two datasets.Accuracies and precisions of these parameters were estimated from the matching statistics.Table 1 shows the final transformation parameters and difference statistics for the matching results.Table 1: Transformation parameters determined with and without self-calibration.
As expected, the UltraCamX imagery displayed relatively small systematic errors.Therefore, introducing the general distortion parameters in the bundle adjustment has limited influence in the surface matching results.This is reflected through the small changes in the mean and standard deviation of the point residuals.Although any changes in the camera paramters have a direct influence on the absolute accuracy of the photogrammetric point clouds before matching, the derived LCPs are not affected as shifts caused by these changes will be recovered by the matching transformation parameters.

AERIAL TRIANGULATION USING LCPS
After surface registration and selection of reference control points, aerial triangulation was performed for the UltraCamX block using LCPs.Since the accuracy of the derived LCPs is highly dependent on the matching results, tests included investigating the influence of the accuracy and number of LCPs on the resulting block accuracy.To check the accuracy of aerial triangulation, 50 LCPs were randomly selected from the LCP list and used as independent check points.Since the accuracy of the extracted lidar reference targets relies on the absolute accuracy of the lidar dataset, tests included assigning different accuracies to the input LCPs, ranging from 5 cm to 15 cm for the horizontal and vertical components.
As expected, the accuracy of aerial triangulation was low when a small number of LCPs was used with lower point accuracies.
The block accuracy was high when the input point accuracy was high.When the point accuracy was lower, block accuracy was improved by increasing the number of LCPs.Figures 5, 6 and 7 show the RMS values at check points for the three components.
It can be seen that when the number of LCPs exceeded 80, the accuracy of the aerial triangulation stabilised at 2 cm in all components for the different point accuracies.Similarly, introducing the BLUH 12 additional parameters only improved the accuracy when a small number of LCPs were used.No improvement was observed when the number of LCPs exceeded 80.
Figure 5: Easting RMS at 50 independent check points.Due to the availability of field surveyed ground control points in the test area, an independent calibration was performed using these field surveyed control points.This was used as a benchmark to validate the calibration results obtained using differing numbers of derived LCPs.Table 2 shows reference results of camera parameters refined using 22 reference ground control points and results obtained using lidar derived control points in increasing number.

CONCLUSION
A method has been proposed to extract reference control points through automatic registration of photogrammetric point clouds with a reference lidar surface using least squares surface matching.This approach allows a dense network of reference points to be extracted.It has been demonstrated that using increasing numbers of these is able to improve block accuracy, potentially overcoming the need for conventional ground control.Effects of changes in the camera parameters or errors in the GNSS data will appear in a form of shifts or rotations in the transformation parameters.Many of these effects are recovered by the surface matching algorithm.Self-calibration using the BLUH 12 additional parameters appears to have limited influence in either the UltracamX matching or the final triangulation results.Future work will focus on applying the research methodology to other systems, such as small format and non-metric aerial cameras.