CHANGING THE PRODUCTION PIPELINE – USE OF OBLIQUE AERIAL CAMERAS FOR MAPPING PURPOSES

This paper discusses the potential of current photogrammetric multi-head oblique cameras, such as UltraCam Osprey, to improve the efficiency of standard photogrammetric methods for surveying applications like inventory surveys and topographic mapping for public administrations or private customers. In 2015, Terra Messflug (TM), a subsidiary of Vermessung AVT ZT GmbH (Imst, Austria), has flown a number of urban areas in Austria, Czech Republic and Hungary with an UltraCam Osprey Prime multi-head camera system from Vexcel Imaging. In collaboration with FBK Trento (Italy), the data acquired at Imst (a small town in Tyrol, Austria) were analysed and processed to extract precise 3D topographic information. The Imst block comprises 780 images and covers an area of approx. 4.5 km by 1.5 km. Ground truth data is provided in the form of 6 GCPs and several check points surveyed with RTK GNSS. Besides, 3D building data obtained by photogrammetric stereo plotting from a 5 cm nadir flight and a LiDAR point cloud with 10 to 20 measurements per m2 are available as reference data or for comparison. The photogrammetric workflow, from flight planning to Dense Image Matching (DIM) and 3D building extraction, is described together with the achieved accuracy. For each step, the differences and innovation with respect to standard photogrammetric procedures based on nadir images are shown, including high overlaps, improved vertical accuracy, and visibility of areas masked in the standard vertical views. Finally the advantages of using oblique images for inventory surveys are demonstrated.


INTRODUCTION
In recent years, capturing and processing of oblique imagery has become a mature technology for 3D information extraction based on photogrammetric principles (Remondino and Gerke, 2015).When the first commercial oblique camera systems appeared on the market in the early 2000s, users looked at this new technology with curiosity but considered it a tool for supplementary functionalities rather than for photogrammetric deployment.In fact, the main applications of oblique images so far are visualization of buildings roofs and facades from a 360degree perspective (Gerke and Kerle, 2011), realistic texturing of 3D city models (Jurisch and Mountain, 2008) and measurement of distances in monoplotting mode -the latter depending on the availability of an existing digital terrain model.Applications for cartography and accurate mapping were not an option.Since then, significant developments took place at sensor level and in the processing chain.The number of manufacturers of oblique camera systems has increased, thus opening market competition.With respect to the design, more recent oblique cameras are a combination of two cameras (fan configuration) or four cameras (Maltese-cross architecture).The Maltese-cross family dominates the market.The most common design uses one photogrammetric nadir viewing camera combined with four tilted cameras with forward-, backward-, left-and right-viewing directions.With respect to their spectral characteristics, the oblique sensor heads acquire the visible (RGB) bands.Some recent models also include the NIR channel for the nadirlooking sensor head (Gruber, 2015).Parallel to the progress driven by the manufacturers, these new sensors caught the interest of researchers and software providers.At scientific level, investigations have focused on the quality of the images, on acquisition geometry, on block triangulation accuracy and Dense Image Matching (DIM).Research topics include optimal viewing angles of the oblique sensor heads (in relation to surface characteristics), optimal overlaps between images and strips, tie point extraction approaches, and image orientation strategies, and have been reported in a number of publications (Jacobsen, 2008;Gerke and Nyaruhuma, 2009;Wiedemann and More, 2012;Rupnik et al., 2013;Rupnik et al., 2015).Given the increased number of oblique cameras on the market and the increased use of oblique imagery, software providers have updated the standard algorithms designed for vertical imagery, e.g.including oblique images in the aerial triangulation (AT).Since the first application of oblique imagery, their potential to see areas masked in nadir views was obvious.Other than that, the higher degree of image overlap that is typical for oblique projects favour dense point cloud generation and orthophoto production in urban areas.At least theoretically, dense matching of oblique images has the potential to surpass LiDAR acquisition in the modelling of building facades.Recent results seem to confirm this idea (Haala and Rothermel, 2015).While these developments are very promising in theory, their real-life capabilities for the commercial market still need to be fully demonstrated.At Terra Messflug GmbH (TM), a subsidiary of Vermessung AVT ZT GmbH (Imst, Austria), there is a strong belief that oblique imaging can significantly improve the efficiency of photogrammetry for precise mapping applications.AVT is a well-established surveying company, offering various services ranging from cadastral surveying to topographic mapping, engineering geodesy and aerial photogrammetry.One important branch is cadastral surveying and topographic mapping for public administrations or private customers.So far, such projects involved a nadir photo flight along with stereo plotting complemented by terrestrial on-site surveying.By extending the photogrammetric process with oblique flights, the efficiency, and thus the market position, can be improved substantially.Based on these considerations, TM has flown a number of urban areas in 2015 with an UltraCam Osprey Prime multi-head camera system from Vexcel Imaging.In collaboration with FBK Trento (Italy), the joint research project "GEOBLy" was initiated to optimize the photogrammetric processing chain of oblique imagery and to develop a software solution with specific functionalities for mapping projects at AVT.In this paper, the achievements obtained from the oblique images acquired over Imst (a small town in Tyrol, Austria, with 10.000 inhabitants) are presented and commented.After an outline of the Imst dataset in Section2, the geometric image processing is presented.The specifics of aerial triangulation (AT) are described in Section 3, followed by DIM, 3D pointcloud generation and automatic LOD2 building extraction in Section4.Finally, the application of oblique images for building mapping is presented in Section 5.In each step, the differences and innovation compared to standard procedures based on nadir images are commented, paying attention not only to accuracy but also to productivity aspects.

IMST DATA SET
The Imst data set was recorded with an UltraCam Osprey Prime.Nadir images are obtained from two PAN cones with focal length 80 mm stitched together yielding a total image size of 11674 pixels across and 7514 pixels along flight direction, respectively.Colour is provided by an RGB cone (Bayer pattern) and an NIR cone with pan-sharpening ratio 2. Oblique images are taken in RGB mode (Bayer pattern) in the four cardinal directions (forward, backward, left, right) at an inclination of 45°.The oblique focal length is 120 mm and the image size is 8900 pixels across and 6650 pixels along flight direction, respectively.The physical pixel size of all eight sensors is 6 µm.The camera was flown on a fixed-wing aircraft modified for aerial shooting, suspended in a gyro-stabilized mount, and supported by GNSS/INS equipment from IGI (Aerocontrol IMU II-d 256 Hz with drift better than 0.1°/h, NovAtel OEM V-3).The project area is the central part of Imst, located in the Austrian Alps, with approximate size 4.5 km by 1.5 km and altitude between 700 m and 1.000 m (Figure 1).During flight planning of oblique projects, special care has to be placed on expected ground-sampling distance (GSD) in the nadir and oblique images as well as spatial coverage, according to the camera parameters and terrain characteristics (Rupnik et al., 2015).The aim is to minimize the presence of occlusions in the images and to guarantee that the project area is fully covered by all five viewing directions.In case of Imst, the flight plan was designed using an average nadir GSD of 6 cm, and alongand across overlaps of 75% and 60%, respectively.It resulted in 5 strips and a total of 780 images (Figure 2, right-hand side), i.e. 156 images for each camera, with mean scale ca 1:7400.The flight was executed on 1 st October 2015 with favourable weather conditions.Ground truth data was provided in the form of 6 ground control points (GCPs) and 14 independent check points (CPs) surveyed with RTK GNSS with a mean 3D accuracy of 5 cm (Figure 2, left-hand side).Moreover 3D building data generated from an earlier photogrammetric (nadir) campaign and a LiDAR point cloud with 10 to 20 measurements per m² were available.

AERIAL TRIANGULATION
The main strategy followed during direct georeferencing and aerial triangulation (AT) was to consider the Osprey system as a rigid assembly of cameras, rather than a system of five independent cameras.The standard image-to-ground transformation for nadir images was extended to include the four additional oblique views by means of relative orientation parameters (offsets, misalignments) available in the camera calibration certificate.Prior to the AT, a self-developed (FBK) forward ray intersection tool was used, to analyse the quality of the GPS/IMU observations and calibration parameters.The AT was then run with up-to-date versions of three commercial software packages: Match-AT (Match-AT, 2016), UltraMap AT (Gruber, 2015) and Pix4D (Pix4D, 2016).The apriori standard deviations were chosen according to empirical results from earlier projects: for the GCPs 5 cm in planimetry and 7 cm for the height; for the GNSS/INS observations 10 cm in planimetry, 15 cm for the height, 0.002° for omega and phi, and 0.005° for kappa; and for the tie points 3 µm for automatic and 5 µm for manual image measurements.Image correspondences were automatically extracted across the multiple aerial views.In all tests, the initial values of IO parameters (focal length, principal point) and additional parameters of all sensors were estimated by self-calibration.Furthermore, both forward intersection and AT were carried out with and without the oblique images.Despite their different design concepts, software packages delivered similar results for AT.For this reason, only results obtained from Pix4D are shown here.
In Table 1, the root mean square errors (RMSE) on CPs are reported as measures of adjustment accuracy in object space.The horizontal accuracy from (pre-AT) forward intersection is around 20 cm using nadir only or nadir + oblique images.In contrast, the vertical accuracy improves from 40 cm (nadir only) to some 23cm (nadir + oblique).Using the AT results, the horizontal accuracy reaches 3 to 4 cm in both variants (the numerical differences are not significant).Similar to the pre-AT scenario, the vertical accuracy improves from 9 cm (nadir only) to 6 cm (nadir + oblique).Clearly, the height benefits from the addition of oblique images which is mainly due to the improved intersection geometry of the rays.One interesting issue is the number of matches between camera views, as shown in Table 2. Similar considerations are presented in Gerke et al. (2016).As expected, image pairs acquired with different viewing directions get fewer correspondences (orange and blue cells), if compared to pairs pointing to the same direction (green and yellow cells).It has to be noticed that images generated by right (R) and left (L) cameras, as well as backward (B) and forward (F) cameras, in adjacent strips, have similar aspect, due to the symmetric design of the camera system and the parallel flying directions (Figure 3).If the pairs consist of images from R (or L) camera and B (or F) camera (orange cells), the feature information is clearly different due to the perpendicular incidence angles and aspect of the objects.The aspect in the images is displayed in Figure 4 for a point of interest, where the similarity between B / F and L / R images in two adjacent flight strips is evident.In the F / L and B / R pairs, on the other hand, there is little feature similarity.To investigate the impact of automatic tie point extraction on computational costs, the AT was run on the same work station at half and quarter image size and the results compared to those achieved with the full-size images.Table 3 compares, in percent, the processing times.Despite a bigger GSD, no significant changes (in terms of accuracy of object coordinates) are found when reduced image sizes (half and quarter) are selected for the tie point extraction.

Dense point cloud extraction
Starting from the AT results, the 3D reconstruction process was aimed at providing an accurate and easy-to-handle representation of the urban environment.In this context, mapping applications require clear information for the building facades and footprints.Since such features are difficult to be extracted from traditional nadir imagery, DIM from multi-view aerial blocks can be an effective solution to overcome the problem of viewpoint restrictions.From an algorithmic point of view, this means coping with the well-known potential difficulties of DIM in oblique scenarios, i.e. large scale variations, multiple occlusions and severe illumination changes (Rupnik et al., 2014).Furthermore, the traditional 2.5D processing for DSM raster production from nadir images should be replaced by a more compelling modelling in "real" 3D space.
State-of-the-art commercial solutions allow for dense pointcloud generation from both nadir and oblique aerial imagery (Haala and Rothermel, 2015).Such results can be used as input for automatic 3D building reconstruction (Tutzauer and Haala, 2015;Remondino et al., 2016).
The dense image matching algorithm implemented in Pix4D was adopted in this study to derive dense point clouds from the Imst data set.This was carried out using images reduced at one quarter of the original geometric resolution with a multiscale approach and a matching window size of 9 x 9 pixels.To investigate how the quality of the final output is altered by the introduction of oblique views, both image block scenarios were tested, i.e. nadir only and nadir + oblique.The derived dense point clouds feature an average point density of 8.93 pts/m³ (nadir only) and 10.88 pts/m³ (nadir + oblique).Figure 5 shows the result of dense matching from nadir + oblique images on a subset of the project area, with close up views in the nadir and nadir + oblique clouds.In addition, some height profiles were generated in the nadir, nadir + oblique and LiDAR point clouds in correspondence of a selection of buildings for comparisons (Figure 4).The qualitative comparison of the point clouds highlights the significant improvement on the reconstruction of the building façades when the oblique images are added to the dense matching procedure.

3D building reconstruction
The high amount of data obtained by state-of-the-art pixelbased dense matching algorithms, although suitable for simple point-based visualization purposes, is not suited for GIS packages and topologies need to be extracted.With focus on buildings, the development of tools for automatic 3D city modelling has been facing this challenge for almost three decades.The most popular approaches can be categorized as follows (Haala and Kada, 2010): (i) reconstruction with parametric shapes, (ii) reconstruction based on point-cloud segmentation and (iii) reconstruction based on simplifying a meshed DSM to the required abstraction level.For the Imst data set, 3D building models were automatically generated using the tridicon/Hexagon suite of tools (tridicon, 2016) which adopts the second aforementioned strategy to reconstruct the building shapes through best fitting the input elevation data with geometric primitives.The two modules "BuildingFinder" and "CityModeller" were tested.The latter requires building footprints as input which often do not exist in a complete form or are not up-to-dated.This former issue was faced in the project, thus the BuildingFinder tool was adopted as it automatically detects buildings from point clouds without the need for any auxiliary information.The point cloud was first segmented, thus deriving a partition of the points into segmented planar regions.The detection of building objects was then performed by searching for selected types of internally defined roof types.To support the detection of building ground heights, an available DTM (1 m resolution) was used in the processing.Furthermore, points belonging to vegetation were filtered out and excluded from the building search.The resulting output (Figure 7) consists in building models that feature flat façades (details like windows or doors are not currently modelled by the tool) and distinctive roof structures, i.e. consistent with level of detail 2 (LOD2) within the OGC standard CityGML (Gröger and Plümer, 2012).The same building extraction procedure was tested on both dense point clouds, i.e. with and without the inclusion of oblique imagery.
To evaluate the achieved results, 3D building models from a previous stereo restitution project (2013 -approximate LOD2 standard) were used as ground truth data.Due to the difference in age, some differences in the building mass have taken place.The changed buildings have been excluded from our analysis of the results.

Gain from oblique imagery for automated building model extraction
A twofold approach was chosen to determine whether and to what extent automatic building detection benefits from oblique imagery: (i) a geometrical error evaluation and (ii) an independent visual assessment by a human operator.A sample set of 22 buildings was selected randomly from the central area of Imst.The following three data sets were used in the evaluation process: 1. Reference model: 3D building models from the manual reconstruction based on photogrammetric stereo measurements and editing in UVM Systems ' CityGRID Modeler (CityGRIG Modeler, 2016).2. Model nadir: 3D buildings generated with the tridicon suite based on the dense point cloud derived from nadir imagery only.3. Model oblique: 3D buildings generated with the tridicon suite based on the dense point cloud derived from a combination of nadir and oblique imagery.With CloudCompare 2.6.2 (CloudCompare, 2016) absolute distances between the reference model and the two comparison models were calculated.From the 22 compared buildings, 15 show smaller distances for the nadir + oblique model.For the other remaining 7 buildings, the nadir-only model shows smaller distances.The geometrical analysis is based on the 3D faces of the building models that were converted into point clouds utilizing a random point sampling with an average pointto-point distance of 20 cm.Point-wise for every building in the sample set, the absolute distances between the reference model and the comparison models were calculated using a Hausdorff distance algorithm (Girardeau-Montaut et al., 2005) as well as least-squares local plane fitting resulting in different distance values but in similar outcomes in the comparison process.The human operator was instructed to visually analyse the 3D buildings (shaded faces) and determine which of the comparison modelsoblique or nadirbest fit the reference data regarding topological resemblance and completeness.Buildings were classified as "nadir + oblique better", "nadironly better" and "same quality".The results from this analysis were consistent with the outcome of the geometrical discrepancies.All 13 buildings that were manually classified as "nadir + oblique better" had smaller geometric discrepancies in the oblique model and all 5 buildings classified as "nadir-only better" had better geometric results in the nadir model.In four cases the operator chose "same quality", two of which had superior geometric outcomes in the nadir + oblique version and vice-versa.An increase in model's detail could be observed in the nadir + oblique comparison.The difference is apparent in Figure 8 which depicts the model variants used in the process as well as the orthophoto for one of the sample buildings.The comparison model nadir + oblique discriminates correctly the two parts of the building, while the nadir-only model combines them into one entity.The small tower on the roof is omitted in the comparison models as they are limited to LOD2.

MAPPING APPLICATIONS
In AVT the main aim of oblique image acquisition is the improvement of the current performance for mapping purposes (i.e.cadastral building update, surface mapping of water and canalization and other detailed restitution of objects).The main advantage of oblique imagery is the improved visibility of objects partly occluded in the nadir view, such as facades, building footprints, road edges, walls, man-holes or roofs with special overhanging features (Figure 7).In cadastral surveys, such information has traditionally been collected through expensive and time-consuming ground surveys.Our aim is to reduce the burden of field surveying for such projects.Clearly, not all occlusions are eliminated by adding oblique views, but the share of objects measured by photogrammetry can be increased significantly.With oblique imagery providing a level of accuracy (after AT) similar to or even better than nadir-only flights, the only remaining hurdle is the lack of an effective measurement and inspection tool.Within the framework of the research project GEOBLy, AVT and FBK have developed a very useful tool to aid operators in this task, combining precisely oriented imagery and several 3D measurements techniques (Figure 7).A new pipeline for AVT's surveys has been set up in the GEOBLy tool where oblique imagery and feature measurements are an integrated part.As existing data can also be imported into the measurement tool, it is also very well fitted for map update projects.

CONCLUSIONS
AVT, TM and FBK have conducted some investigations on the potential of photogrammetric techniques applied on oblique aerial images for mapping purposes.The tests performed on the Imst data set, acquired with an UltraCam Osprey Prime and processed with FBK-developed and commercial software solutions, have shown a quality in the orientation comparable, and even better, to that achieved with only nadir image blocks, if the calibration parameters of all cameras are properly introduced in the geometric sensor model.Beside AT outperformances, oblique views offer important benefits also in the dense point cloud generation, as they allow to reconstruct, e.g.buildings façades or other features generally not visible in the nadir views, thus overpassing the capability of LiDAR acquisitions in urban areas.The gained value is also evident in the performance of the subsequent automatic building reconstruction.Results achieved with tridicon software package over a small test area proved significant advantages from including well-oriented oblique imagery into automatic building reconstruction.Also for inventory survey and other mapping purposes, the advantages of oblique imagery for 3D restitution have been demonstrated.To compensate the lack of a suitable tool for digitalization in oblique imagery, an ad-hoc software was developed to satisfy operational needs.

Figure 3 .
Figure 3. Similarity of B/F and L/R Osprey images from different strips.Zoom on AVT building in Imst.Visualization in the GEOBLy tool.

Figure 4 .
Figure 4. Image footprints of five Osprey Prime cameras on AVT building (indicated in red) in adjacent flight strips.

Figure 3 .
Figure 3. 3D point cloud of Imst extracted with Pix4D using nadir + oblique images (top).Close up views (bottom) in 3D point cloud extracted using only nadir (left) and nadir + oblique images (right).

Figure 4 .
Figure 4. Height profiles in LiDAR (top), nadir (middle) and nadir + oblique (bottom) point clouds.The profile section is indicated in yellow in the orthophoto.

Figure 7 .
Figure 7. Image measurements in the GEOBLy tool: "Traditional" roof edge measurements (top) and measurements of complete building footprints (middle and bottom).

Table 1 .
Results of forward intersection and AT of Imst nadir (N) and oblique (O) images.The RMSE values refer to 14 independent CPs.

Table 2 .
Median / maximal number of matches between camera views.

Table 3 .
Processing time of AT, when tie point extraction is performed at 0.5 and 0.25 image scales.Time is represented in % with respect to full-size images (image scale 1, as reported in Table1).