ONE STEP MOBILE MAPPING LASER AND CAMERA DATA ORIENTATION AND CALIBRATION

Modern mobile mapping systems include one or several laser scanners and cameras. The main outputs of these systems are oriented camera images and 3D point clouds. These point clouds can be derived from pairs of overlapping images or using the laser raw data together with platform trajectory. A mobile mapping campaign may include several overlapping areas, generally, the derived point clouds of the same area are not properly registered, due to partial or total GNSS occlusions, multipath and inertial drift and noise. Nowadays, the standard procedure for co-registration between laser and laser and between camera and laser, includes several steps. The first ones are the system calibration where the lever arm and boresight between laser and IMU, and between camera and IMU must be determined. After the calibration steps, a camera and LiDAR point cloud can be derived. Then, a co-registration between LIDAR points clouds and between camera point cloud and LiDAR point cloud are computed. In contrast to the standard approach, in this paper we propose to solve the orientation and calibration of laser and camera data in a single, combined adjustment. Solving the orientation and calibration allows us to implicitly deal with the co-registration problem. The proposed method is based on the identification of common tie features between images and point clouds and their use in a combined adjustment. This common tie features are straight line segments. The preliminary results indicate the feasibility and the potential of the approach.


INTRODUCTION
Nowadays, 3D georeferenced data are widely used or the primary data, for many applications such as 3D city modelling, cadastrial mapping, cultural heritage, facility management, traffic accident investigation, to mention a few examples.Dense 3D point clouds are the directly output of a mobile mapping system (MMS).A MMS is a particular case of terrestrial laser scanner, where the sensor is mounted on a moving platform.The point clouds are generated kinematically, with a variable scanner position and orientation during the scanning time (Kutterer, 2010).
High-end mobile mapping systems integrate several laser scanners and several individual cameras or 360 • cameras.A mobile mapping campaign may include several overlapping areas, leading to overlapping point clouds.Partial or total GNSS occlusions, or multipath, may occur, specially in urban areas.This causes an error to the platform or vehicle trajectory determination.Other sources of error can be Inertial Measurement Unit (IMU) modelling errors and eventually system calibration errors.All these errors produce that overlapping point clouds are not properly registered (Figure 1).To solve this, a laser-laser registration must be performed.Also, some applications might require having coloured point clouds.Then, the coloured point cloud is usually derived from another registration process between camera and laser scanner.
The standard procedure for laser-to-laser and camera-to-laser coregistration includes several steps.The first ones are the system calibration where the lever arm and boresight between laser and IMU, and between camera and IMU must be determined.With current INS, GNSS and image matching technologies, the best results are obtained with the well known Integrated Sensor Orientation (ISO) method.In ISO, tie points measurements are used in combination with ground and platform control observation.The ISO method has been proven to be feasible and efficient for IMUcamera boresight calibration of mobile mapping systems (Kerst-ing et al., 2012).For aerial laser and mobile mapping data, ISO has also been proven to be effective for the IMU-laser boresight calibration with single and multiple laser scanners (Skaloud and Lichti, 2006), (Chan et al., 2013).
After the calibration steps, a camera and LiDAR point cloud can be derived.Then, a co-registration between LIDAR points clouds and between camera point cloud and LiDAR point cloud are computed.Alternatively the LiDAR point cloud can be projected onto 2D image space, and then several image-to-image registration algorithms can be applied.The Iterative Closest Point algorithm (ICP ) (Chen and Medioni, 1992), is widely used for registration of point clouds, enabling many improved variants (Novák and Schindler, 2013).
The ICP has been used for registration of pairs of point clouds generated from laser, from pairs of overlapping images, or a combination of them.The ICP has been proven to be effective with point clouds that were initially aligned (Novák and Schindler, 2013).(Gressin et al., 2012) propose a method where uses ICP as initial step for trajectory improvement.The overall registration is performed improving the original platform trajectory.(Elseberg et al., 2013) deals also with laser-to-laser registration by improving the platform trajectory.The trajectory is improved using a semi-rigid Simultaneous Localization and Mapping.
In contrast to the standard approach, in this paper we propose to solve the orientation and calibration of laser and camera data in a single, combined adjustment.Solving the orientation and calibration allows us to implicitly deal with the co-registration problem, because in our approach, we do not model the error symptoms but the error sources.The proposed method is based on the identification of common tie features between images and point clouds and their use in a combined adjustment.This common tie features are straight line segments.This approach presents several advantages.The use of laser raw and photogrammetric measurements, related through common tie The paper is organized as follows.Firstly, the main ideas behind the simultaneous network adjustment are introduced.Then, the proposed models as well as some mathematical concepts behind the use of tie lines are explained in detail.The concept validation section presents the preliminary results of an adjustment obtained with real data.The last section summarizes the conclusions of the proposed approach and discusses future improvements.

PROPOSED APPROACH
We propose a method where camera, laser and camera-laser ISO are performed in a single, combined adjustment.In it, the imaging observations are image coordinates measured on camera data (the digital camera images) and selected measurements of the laser images (ranges and scan-angles).The other observations are surveyed ground control points and platform time-Position-Attitude (tPA).In state-of-the-art mobile mapping systems, the tPA is previously estimated using INS, GNSS and odometer observations using a Kalman Filter approach.Ground control points can be used to derive indirect control observations like ground control lines and planes.The tie features are points (camera ISO), planar surfaces (laser ISO) and straight line segments (cameralaser ISO).Points, planes and lines are defined by 3, 3 and 4 independent parameters, respectively.
The classical camera ISO approach is well known.From the observations mentioned above, estimates for the tie points (TP), for the exterior orientation parameters of the camera images (EO), for the self-calibration parameters of the camera, for the tPA shifts and possibly for other system calibration parameters are computed.In laser ISO, the unknown parameters of the tie planar bounded surfaces -tie planes (TPL)-are estimated together with tPA shifts, laser self-calibration parameters and possibly other system calibration parameters.
In the proposed camera-laser ISO method, as mentioned, straight line segments -Tie Lines (TL)-are used as tie features between camera and laser images.TLs tie camera and laser images as follows.Assume that s is a TL that can be recognized in the camera image c and in the point cloud of a laser image a, and also assume that s is the intersection of two planar surfaces p1, p2 determined by points of the laser image a.Given s, c, a, p1 and p2 we will then use three types of observation equations.The first type of observation equation (a coplanarity equation) relates image measurements of the s TL on the c camera image with s TL parameters, with the EO and self-calibration parameters of c.At least two of these observation equations are needed.The second type of observation equation (a line-in-plane relationship) compels s to lie on planes.In general, there will be two observation equations of this type, for p1 and p2, although it is also possible to use just one TPL.The third type of observation equation relates TPLs -in this case p1 and p2to the measured laser points, i.e., relate the laser range and scan-angle measurements, to the laser self-calibration parameters, the derived tPA, the tPA shift parameters and possibly other system calibration ones.There will be, for each p1 and p2, as many of these observation equations as laser points that define the plane.Details of the models are given in the next section.
In our combined camera-laser ISO concept, laser raw data instead of processed point clouds are used.3D coordinates of point-cloud points are not explicitly computed: the laser points that define the TPLs are parameterized with raw range and scan-angle measurements.This is a key aspect of the concept as it allows for self-calibration of laser measurements and system calibration refinement.Our approach include also a platform trajectory shift to model trajectory errors due to GNSS occlusions, multipath or inertial noise and drift.Self-calibration of camera and laser image observations, system calibration refinement and trajectory error improvement, are the key to correct and consistent results.A differential characteristic of the combined camera-laser ISO concept is that can be used with camera and laser data acquired in different period of time and even with independent acquisition platforms.
It is out of the scope of this paper to discuss the measuring (TLs, TPs and TPLs extraction) and matching techniques that precede the ISO network adjustment stage.We summarize them for the sake of completeness.TLs are extracted from camera images by combining edge detectors and the Hough transform (Vosselman et al., 2004).Once they are extracted, a matching procedure is performed to identify homologous TLs in different images.TPs are extracted and matched from camera images using a BRISK operator (Leutenegger et al., 2011).The outliers are filtered then, using relative orientation between photographs, derived from platform trajectory.In the laser ISO case, in this research, only planar surfaces, belonging to building walls and street pavement, are extracted.For the purpose, a region growing algorithm has been used.

MATHEMATICAL MODELS
The mathematical models needed to perform a single, combined camera-laser ISO were introduced for the first time in (Angelats et al., 2012).In this paper, airborne laser scanner and photogrammetric camera data were simultaneously adjusted using straight lines as tie segments.In order to expand this concept to mobile mapping systems, the presented models are still valid without modifications except for the MMS plane observations equations.In this section, we present in detail this model and the remaining models are briefly introduced.The interested reader may refer to (Angelats et al., 2012) for extended details of the mathematical modelling.
We begin by presenting some notation conventions and then develop the observation equations.

Some naming and notation conventions
The Coordinate Reference Frames (CRFs) involved in the mathematical functional models of the paper are detailed in Table 1.
The CRF l of a variable X is denoted by a superscript symbol like X l .For a rotation matrix R l b , the subscript indicates the origin CRF while the superscript indicates the final CRF so it can be written For the sake of simplicity, X e = (x, y, z) e is used instead of the formally correct X e = [(x, y, z) e ] T .

Camera point collinearity observation equations
In camera ISO, the known collinearity observation equations are used to relate the camera image coordinate observations with the EO, tie point and self-calibration parameters.

MMS plane observation equation
The model extends the one proposed by (Angelats et al., 2012) to be used in mobile mapping systems.The observations are the two MMS laser measurements (range and scan-angle) and the tPA (position and attitude) ones.The MMS plane observation equation is where and where The model includes an additional scale factor self-calibration parameter for the scan-angle observation because our experience reveals its significance.The Hessian form of a plane is used to parametrize the planar surface.The plane is then characterized by a unit normal vector n l = (nx, ny, nz) l (equation 5) and d, the orthogonal distance between the plane and the CRF origin.λ0 and φ0 are auxiliary vectors that brings an orthogonal vector n l 0 close to the nominal normal vector.Then, the parameters of the plane are the orthogonal distance d and two rotation angles δl,δp.These rotation angles use to be quite small by construction.
The model have additional parameters to model tPA errors.These parameters are tPA position and orientation linear shifts.Table 2 summarizes the mathematical symbols of equations 1 to 5.

3D straight line parameterization
Straight lines are elementary mathematical objects with many possible parameterizations.We propose yet another one which is convenient for numerical computations and whose four universal parameters (p, q, α, β) can be used regardless of the line location and direction.
Table 2: Symbols in the MMS plane observation equation.

FC line coplanarity observation equation
This model relates the parameters (p, q, α, β) of a 3D line, the EO parameters of an image, and the image observations (x, y) of a point of the line.It has been derived from the collinearity condition and the 3D straight line parameterization.

Line-in-plane observation equations
This model relates lines and planes in a 3D space.From a geometrical point of view, a line belongs to plane if two conditions are satisfied.The first condition is related to the orthogonality, that is, plane's normal vector and line's director vector must be orthogonal.The second condition is related to the distance between line and plane.This distance must be 0 and it is equivalent that the line's point belongs also to the plane.

CONCEPT VALIDATION
The proposed concept was tested and validated using real data from a high-end mobile mapping system.The data originate from a mobile mapping campaign over a controlled area of Dortmund (Germany).The mobile mapping system was an Optech Lynx system, from TopScan GmbH, that includes two laser scanner and two cameras.As for the software, an experimental "model toolbox" with the models discussed before was developed at CTTC that runs on the generic network adjustment platform GENA (Colomina et al., 2012).
The mobile mapping system was mounted on a van and it was driven over a controlled area around three hours, resulting in 11 overlapping strips.The controlled area was additionally surveyed to provide ground control points.The points clouds where generated, combining the laser raw measurements, the system calibration and the platform trajectory.The platform trajectory was computed, using a tightly coupled approach, combining differential GNSS, IMU and odometer measurements.The points clouds, for both laser scanners within the same strip, and also between strips, where not co-registered.Photographs for each of the cameras, were also acquired.The platform trajectory was also used to provide an initial orientation of the photographs.The camera was previously calibrated and the lens distortion was removed from the photographs.
Since the aim of this paper is not an exhaustive study, a small subset of the original data has been used to validate the feasibility of the one step orientation and calibration of laser and camera data from mobile mapping systems.For that reason, only data from one laser scanner and one camera have been used.The main characteristics of the camera and laser scanner systems as well as the test configurations are detailed in Table 3.The precision of the observations can be found in Table 4. Notice that from the camera derived data, 4 tie lines have been extracted.These tie lines correspond to some straight elements located on building walls such as cables, window edges, i.e.On the other hand, from the MMS-derived data, 12 planes have been used.11 of them are present in 3 strips while the remaining plane only in 1. 4 planes of the first strip are used as a ground control planes while the remaining planes are used as a tie planes.Tie planes are vertical planes, corresponding to building walls, or horizontal planes, corresponding to street pavement.In this paper, the straight lines used as a common feature, belong to the tie planes, but are not breaklines, that is, the ones resulting from intersection of two tie planes.
With the given data, the configurations described in Table 5 have been processed.Block configuration cam TP performs a classical camera ISO using all available tie points and a ground control point (GCPs) and EOs.Test cam TL adds 4 straight lines as tie lines.This configuration allows to validate the coplanarity equation model with tie lines.As for the MMS tests, an MMS orientation and calibration adjustment without camera data is performed using planes as tie features with four ground control planes (GC-PLs).The ALL configuration combines camera data with MMS data using tie points, ground control point for the camera ISO, tie planes and ground control planes for the MMS ISO and tie lines for the camera-ALS ISO.This test allows to test simultaneously all developed models.
The EO parameters are estimated for all proposed test.The proposed approach includes additional parameters, such as a tPA shift, corrections in the boresight matrix between the IMU and the camera (Table 6) and self-calibration parameters.However, in this preliminary study, they are not estimated due to the small number of photographs and a weak camera geometry, with only photographs from one strip and parallel to the vehicle movement direction.
For the MMS subblock, a tPA shift for each strip, can be estimated.Vertical tie planes allow to estimate the e,n component while using horizontal planes allow to estimate the h component.
Self-calibration parameters of the MMS, that is ∆r, ∆θ and S, cannot be estimated, neither the boresight matrix between IMU and laser scanner.These parameters cannot be estimated because they are strongly correlated between them and the geometry of the sub-block does not allow to decorrelate them.
The test results are shown in Table 7 and Table 8.As it was expected, due to the small data set and GCP configuration, cam TP and cam TL results show a similar performance in terms of precision (Table 7) for the EO parameters and tie points.The co-registration of point clouds is implicity done by improving the orientation and calibration.In order to evaluate the quality of the co-registration, a new point cloud for each strip has been generated, adding the tPA shift to the original tPA.The point cloud differences, between pairs of strips, before and after the registration has been computed using the point cloud distance tool of Cloud Compare SW (Giradeau-Montaut et al., 2014).The differences, before and after the registration, are shown in Figure 2. The differences are also computed for each component.We use this SW, only to show the potential of the approach and give a coarse point cloud difference.Further research must be done to compute accurately the distance between point clouds, eventually leading to better results.

CONCLUSIONS AND FURTHER RESEARCH
A method where camera and MMS ISO were performed in one step by using straight as tie features, has been presented.It consist on a combined adjustment where points (camera), planes (MMS) and lines (camera-MMS) are used as a tie features.The mathematical models: 3D straight line parameterization, a coplanarity observation equation and a line-belongs-to-plane, presented in (Angelats et al., 2012) has been proved to be effective, with real data from a high-end mobile mapping system.A new MMS model has been successfully developed and tested.In the MMS model, laser raw measurements are used to allow for system and sensor calibration.In addition, includes also a tPA shift to model platform tPA errors due to GNSS occlusions, multipath.
The results of this paper are only preliminary because of the small size of the data sets.Photogrammetry results show that with this combined camera, MMS, and camera-MMS, concept, the performance in terms of precision, is at least maintained.The results suggest also that the platform trajectory can be improved with a linear shift.With this trajectory improvement, the co-registration between overlapping point clouds, is improved.The MMS selfcalibration has not been performed due to a weak geometry and a small data-set.Further research must be done to explore how the combination with photogrammetric data can benefit the MMS the system calibration, and so, the co-registration.
With this approach, a single or several camera and laser data from different systems or from the same system with different acquisition, can be integrated and so, co-registered.The next step is to validate the presented concept and models with larger data sets and to understand the behavior of camera and MMS selfcalibration parameters in combined adjustments.For larger data sets, alternative models than a linear shift to tPA correction, must be explored.Additionally, not only the precision, but also the accuracy of the one step approach must be measured.Further research must be done to explore the number of required ground control points and planes, since the previous experience suggest that the combined adjustment could reduce the required ground control information.

Figure 1 :
Figure 1: Point cloud of 2 different strips from overlapping area

Table 1 :
Reference frames and coordinate systems.

Table 3 :
Dortmund block geometric configuration.The result of tests case ALL shows that the simultaneous block adjustment works, although this resultin terms of precision is inconclusive given the small differences and the small size of the experiment.The results indicate that it is possible to estimate a tPA shift for each strip (Table8), for MMS and ALL tests.From the results, it can be observed that the h component has a higher Cxx than e,n component.This result was expected because more vertical planes than horizontal ones were used as tie planes.

Table 4 :
Precision of the observations.

Table 5 :
Test camera ic GCPs TPs camera TLs MMS rs TPLs GCPLs MMS TLs Observations of block configurations.

Table 6 :
Estimated parameters of block configurations.

Table 8 :
MMS test resultsFigure2: Point cloud differences from overlapping stripsThe International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-3/W1, 2014