INTEGRATED GEOREFERENCING OF STEREO IMAGE SEQUENCES CAPTURED WITH A STEREOVISION MOBILE MAPPING SYSTEM – APPROACHES AND PRACTICAL RESULTS

Stereovision based mobile mapping systems enable the efficient capturing of directly georeferenced stereo pairs. With today's camera and onboard storage technologies imagery can be captured at high data rates resulting in dense stereo sequences. These georeferenced stereo sequences provide a highly detailed and accurate digital representation of the roadside environment which builds the foundation for a wide range of 3d mapping applications and image-based geo web-services. Georeferenced stereo images are ideally suited for the 3d mapping of street furniture and visible infrastructure objects, pavement inspection, asset management tasks or image based change detection. As in most mobile mapping systems, the georeferencing of the mapping sensors and observations – in our case of the imaging sensors – normally relies on direct georeferencing based on INS/GNSS navigation sensors. However, in urban canyons the achievable direct georeferencing accuracy of the dynamically captured stereo image sequences is often insufficient or at least degraded. Furthermore, many of the mentioned application scenarios require homogeneous georeferencing accuracy within a local reference frame over the entire mapping perimeter. To achieve these demands georeferencing approaches are presented and cost efficient workflows are discussed which allows validating and updating the INS/GNSS based trajectory with independently estimated positions in cases of prolonged GNSS signal outages in order to increase the georeferencing accuracy up to the project requirements.


INTRODUCTION 1.1 Motivation
First developments in vision based mobile mapping data back around two decades (Schwarz 1993).Recent developments, e.g. in the development of new dense stereo matching algorithms or in imaging sensors and computing performance, are starting to open up new interesting and innovative application scenarios.However, mobile mapping data and deduced information are only usable for a wide range of applications, if the georeferencing accuracy of the images is well known and within bounds of a predefined accuracy level across the entire project perimeter.The great benefits of digitised street or rail corridors with mobile mapping systems are the manifold use for different mapping tasks, information extraction applications or spatial analyses.Many application scenarios like street or rail furniture mapping, asset management or surveys for planned construction work require a homogeneous georeferencing accuracy better than 10-15 cm with regards to a local geodetic reference frame and 2-3 cm locally.Guaranteeing a georeferencing accuracy within predefined specifications across the entire project is one of the key factors in ensuring the return on investment The widely used INS/GNSS based direct georeferencing approaches, e.g. by Schwarz (1995) or Mostafa and Hutton (2005), for position and attitude estimation of mobile mapping sensors lead to degraded accuracies in the following two cases: a) during GNSS signal outages especially in urban canyons or on roads leading through forests and b) precise georegistration of the trajectory to a local, possibly distorted, geodetic reference frame.The wide spectrum of applications makes it desirable to increase the georeferencing accuracy to a predefined accuracy level, either on demand for specific parts of the project perimeter, e.g. for road sections with planned construction work, or for an entire project perimeter in a cost efficient manner.This paper discusses integrated georeferencing approaches and workflows to meet this requirements in a cost efficient way.

Related Work
The following known and established approaches are used for increasing the georeferencing accuracy of directly georeferenced sensor observations with respect to a local geodetic reference frame.Many workflows nowadays used in practice are based on a 3d helmert transformation approach.The 3d coordinates determined based on the directly georeferenced trajectory are subsequently transformed as a whole or in sections based on known control points.The helmert transform based georegistration approach is relatively simple and can be applied to 3d coordinates derived by any kind of sensor.However, it assumes the absence of a number of potential error sources such as drifts in sensor orientation parameters or local distortions in the reference framework.Further, in many cases discontinuities arise between two transform parameter sections.
A more advanced and more powerful georegistration is offered by integrated georeferencing.In integrated georeferencing the exterior orientation parameters of mobile mapping sensors are determined in a combination of INS/GNSS-based direct georeferencing and indirect georeferencing often incorporating a bundle adjustment (Eugster 2011).In airborne photogrammetry different concepts can be distinguished.Haala (2005) and Cramer (2001) propose the stripewise integration approach and the orientation image approach, which allow the estimation of biases of the directly measured INS/GNSS trajectory, in relation to a local reference frame with the aid of additional image observations to known ground control points.Further, many approaches use the exterior orientation parameters measured by INS/GNSS to support the aerotriangulation (AT) with additional observations (Heipke 2002).A general integrated georeferencing concept which is derived from simultaneous localisation and mapping (SLAM) integrates the INS, GNSS and vision observations in the estimation of an optimal position and attitude trajectory (Durrant-Whyte and Bailey 2006).Ellum (2006) or Bayoud (2006) show implementations of that concept for photogrammetric mobile mapping applications.Further approach is proposed and investigated by Hassan et.al. (2006( ) or Forlani et. al. (2005) ) using a photogrammetric adjustment to bridging land based mobile mapping image sequences during GNSS denied track sections.The following section introduces the stereo vision mobile mapping system together with the imagery processing pipeline and client components for exploiting the georeferenced stereo imagery sequences.The main section describes the proposed integrated georeferencing approach combining stereo vision, INS and GNSS which allows increasing the georeferencing accuracy during GNSS signal outages on the one hand, and which provides precise georegistration of the trajectory regarding to a local geodetic reference frame on the other.In the following section, the performance of the algorithm is evaluated and discussed based on several test sequences captured in real urban canyons.The paper is concluded with some cost-benefit considerations by putting the number and spacing of ground control points in relation to the obtainable accuracies and with an outlook on on-going and future development steps in optimal georeferencing of vision-based mobile mapping data.

STEREO VISION BASED MOBILE MAPPING
Stereo vision based mobile mapping allows the flexible, highly efficient mapping and the subsequent easy interpretation of digitised street or rail corridors.The support for multiple, freely configurable stereo imagery systems on varying mobile mounting platforms provide accurate 3d position estimation in combination with the possibility to extract rich semantic information of arbitrary geoobjects.

Stereo image acquisition system
The stereo vision acquisition system used in our investigations consists of the following components: a. multiple configurable stereo systems, b.INS/GNSS based navigation sensors together with a navigation controller and c. imagery acquisition, synchronisation and triggering soft-and hardware components.Figure 1 shows the stereo vision acquisition system mounted on a mobile mapping motor vehicle 1 with three free configurable stereo systems on the roof top.One important advantage of stereo systems is that the relative orientation of both cameras can be estimated with high accuracy.This not only allows the determination of highly accurate 3d coordinates and relative measures inside the same stereo frame (Luhmann 2003).The stereo image acquisition geometry also enables an exterior orientation estimation with regards to the local reference frame for each measuring epochindependent from any INS/GNSS observations.

Data pipeline and processing workflow
In order to support a flexible and distributed use of the stereo imagery for a number of visualisation, inspection or measurement tasks, a powerful data processing pipeline and suitable exploitation software components are required.The processing and exploitation workflow is shown in Figure 2. The stereo image acquisition system, introduced above, acquires raw stereo image sequences up to 25 fps in full HD or 11 megapixel quality with 12 bit radiometric resolution together with INS/GNSS observables.The subsequent preprocessing step includes: a. calibration procedures for interior orientation, relative orientation, misalignment and lever arm parameter estimation, b. direct or integrated georeferencing of the captured stereo sequences and c. the stereo image processing to radiometrically corrected, distortion free and normalised high resolution stereo image sequences.Depending on the following data utilisation a semi-global block matching algorithm is used for extracting dense depth maps or dense rgb 3d point clouds from the stereo imagery.These products (stereo imagery and depth maps) constitute a highly detailed digitised real world for a flexible and distributed use.Based on the application scenario a stereo or a mono image use is possible (see Fig. 2).The stereo client supports stereoscopic 3d vision and 3d measuring functionality while the mono client provides accurate 3d measuring functionality by 3d monoplotting exploiting mono imagery and the corresponding dense depth maps.The introduced stereo vision based mobile mapping enables absolute 3d point accuracy of 3-4 cm (1 sigma) under reasonably good GNSS conditions (Burkhard et. al. 2012).Relative 3d measurements within a stereo frame or within neighbouring frames of the sequence are better than 1 cm.

Basic algorithms
Our proposed integrated georeferencing approaches are based on the following algorithms.For mobile mapping stereo sequences an optimized least squares multi ray matching algorithm (Huber et al. 2011a) is used which allows the semiautomatic measurement of control and tie point.These automatically generated multi-image observations are repeatedly used in later processing steps.For the independent estimation of exterior orientation parameters in a local geodetic reference frame over neighbouring stereo frames (Huber 2011b) a constrained stereo bundle adjustment is used.Pre-calibrated relative orientation parameters can be introduces as constraints for stabilising the bundle geometry.Further a loosely coupled Kalman Filter (Angrisano 2010) is applied to fuse pre-processed GNSS and vision position updates with the INS based navigation solution.

INS/GNSS/Vision fusion
For increasing the georeferencing accuracyespecially during GNSS signal outageswe propose additional stereo vision based position updates to support the directly measured INS/GNSS trajectory.For that the vision-based position updates can be fed as additional observations into a loosely coupled Kalman Filter based trajectory estimation.Depending on the desired mapping accuracy and the quality of the INS integrated on the mobile mapping system, control points (CP) or groups of them (CPG) in suitable locations along the trajectory are required.These control points serve as local datum information for the stereo vision based position updates.

Bundle adjustment based update:
In this case a stereo image sequence based bundle adjustment is used which allows the estimation of independent position and attitude updates.The mentioned stereo sequence matching algorithm for semiautomated tie and control points (Huber et al. 2011a) is used for creating the required image observables.The advantages of the bundle adjustment updates are: the possibility to validate the local datum information (geometry of the control point group) with the bundle adjustment geometry.Furthermore, this approach allows the creation of additional heading updates.

Single control point update:
This approach assumes that mainly the position term of the directly georeferenced trajectory is degraded and not the attitude term.Investigations show that this assumption holds true in particular for the trajectory problems caused by short-term GNSS outages (Huber 2011b).In that case only one control point is sufficient to determine an independent vision position update.The position update coordinates can be derived by correcting the directly georeferenced trajectory with the position difference between a) the intersected 3d position derived from the directly georeferenced stereo image sequence and b) the known control point coordinates.To measure precise 3d point coordinates, we use the mentioned semi-automatic multi ray stereo matching approach followed by a multi ray intersection.If the directly georeferenced attitude solution does not fit the accuracy requirements, an iterative reprocessing using this concept can be employed until the trajectory solution converges.

Registration to a local geodetic reference frame
In some cases systematic errors between the directly measured INS/GNSS trajectory and a local geodetic reference frame are caused by local net deformations, reference frame scale differences or insufficient transformation models between ellipsoidal and physical heights.Most of the practical applications require a specified mapping accuracy with respect to a local geodetic reference frame and not to a correctly processed INS/GNSS trajectory.To satisfy this requirement we propose an integrated georeferencing concept which is derived from the mentioned orientation image approach.With the same approaches as presented in sections 3.2.1 and 3.2.2 the systematic error of the trajectory can be determined for each control point group respective a single control point.With this information a correction based on a 3 rd order spline function can be determined for each coordinate component (east, north and height).The directly georeferenced positions of the original trajectory can be corrected with the estimated correction spline.
Afterwards the trajectory can be reprocessed with the corrected position updates in a loosely coupled Kalman Filter.

Test configuration an validation test sites
The proposed INS/GNSS/Vision georeferencing approach was validated in the following two independent test projects:

A B
Test site City of Lugano

Integrated georeferencing results
The following Table 1 summarises the integrated georeferencing (IG) accuracy for the two test projects A and B in comparison with the degraded direct georeferencing (DG) solution.These investigations were carried out with the proposed INS/GNSS/Vision fusion approach described in section 3.2.Figure 5 shows the residual vectors in horizontal position and height for the control and check points based on the directly georeferenced trajectory (left) and the integrated solution (rigth) for test site A. The residuals of the direct georeferenced solution show clear systematic error behaviour in horizontal position and height caused by GNSS signal outages.The integrated georeferencing approach show a significant improvement of the georeferencing accuracy compared to the direct georeferencing solution.The residuals are nearly free of systematic effects.The proposed INS/GNSS/Vision fusion allows preserving the georeferencing accuracy during degenerated trajectory sections at the same accuracy level as in unobstructed parts of the project with an optimal direct georeferencing solution.Further, the obtainable absolute georeferencing accuracy is also depending on the control point quality with respect to the local geodetic reference frame.This becomes evident if empirical accuracies of test sites A and B are compared (see Section 4.1).The selectable CP or CPG spacing depends on the required georeferencing accuracy, the quality of the integrated INS as well as on the acquisition speed of the mobile mapping system.

COST-BENEFIT CONSIDERATIONS AND RECOMMENDATIONS
If the direct georeferenced trajectory is degraded due to GNSS signal outages or due to systematic errors such as multipath effects, incorrectly fixed GNSS carrier phase ambiguities or cycle slips the proposed integrated georeferencing approach based on INS/GNSS/Vision fusion should be preferred.For trajectory parts with (expected) systematic GNSS errors and with sufficient and well-distributed control points, the available GNSS updates can be fully substituted by vision updates.For correctly registering the directly georeferenced trajectory to a local geodetic reference frame which is affected by a network scale or local network distortions both integrated georeferencing approaches presented in section 3.2 and 3.3 are well suited.Compared to the single control point update concept, the bundle adjustment based coordinate approach permits the fully independent determination of vision based coordinate and heading updates without any initial trajectory attitude solution.
Further, the involved control points inside a CPG can be checked with the stereo bundle adjustment results.On the other hand the single control point update approach is more flexibly deployable.Especially if existing geoobjects are available in the project area, their coordinates can easily be used for single point updates in order to support the trajectory determination locally.
If a project perimeter is to be mapped repeatedly, e.g. for infrastructure asset management web services, it is sensible to create a control point cadastre with CP or CPG which can be used for the georeferencing process of each mapping campaign.
If a trajectory part has to be improved, control points can be defined and signalised in advance.Further control points can be defined and measured afterwards on an 'as needed basis' depending on the quality assessment of the directly georeferenced trajectory.Especially image data are well suited to measure appropriated points to support the directly georeferenced trajectory afterwards.For this purpose traffic sign corners or street marks are well suited.

CONCLUSIONS AND OUTLOOK
The proposed integrated georeferencing approach shows that degraded direct georeferencing trajectories especially in urban canyons can be improved by additional vision based updates using tie points and control points.This possibility is very important in order to guarantee a specified accuracy level of mobile mapping data across a whole project perimeter also with unfavourable GNSS conditions.As shown in this paper, stereovision-based mobile mapping offers several powerful options for improving the georeferencing accuracy even under adverse GNSS conditions and in locally distorted reference frameworks.Furthermore, the proposed integrated georeferencing approaches are capable of providing high quality results without a need for the ultimate high-end IMU sensors.
Further work will proceed with fusion of INS, GNSS and vision on raw observables e.g. in a tightly coupled Kalman Filter or information filter.Interesting concepts and approaches are discussed in the SLAM community and need to be adapted to geodetic application scenarios.Furthermore, stereo vision is well suited as additional independent sensor for navigation data processing.A stereo image sequence bundle adjustment allows bridging and supporting the INS observations during GNSS outages.It provides independent position and attitude information, thus increasing and stabilising the continuous trajectory estimation.In this way the trajectory can directly be processed in an arbitrary, potentially distorted, geodetic reference frame.

Figure 1 .
Figure 1.Stereo vision based mobile mapping system 1.3 Structure and goal of the paper

Figure 2 :
Figure 2: Stereo vision data pipeline and distributed use

Figure 3 .
Figure 3. Stereo bundle adjustment network geometry with integrated relative orientation conditions

Figure 5 .
Figure 5. Residual vectors in check and control points

Table 1 .
Direct vs. integrated georeferencing accuracyresidual coordinate differences in control and check points 2 Manufacturer's data from http://www.applanix.comFigure 4: Urban canyon in test site A (top) and buildings and trees in test site B (down)