DETERMINATION OF THE UAV POSITION BY AUTOMATIC PROCESSING OF THERMAL IMAGES

If images acquired from Unmanned Aerial Vehicles (UAVs) need to be accurately geo-referenced, the method of choice is classical aerotriangulation, since on-board sensors are usually not accurate enough for direct geo-referencing. For several different applications it has recently been proposed to mount thermal cameras on UAVs. Compared to optical images, thermal ones pose a number of challenges, in particular low resolution and weak local contrast. In this work we investigate the automatic orientation of thermal image blocks acquired from a UAV, using artificial ground control points. To that end we adapt the photogrammetric processing pipeline to thermal imagery. The pipeline achieves accuracies of about ± 1 cm in planimetry and ± 3 cm in height for the object points, respectively ± 10 cm or better for the camera positions, compared to ± 100 cm or worse for direct geo-referencing using on-board single-frequency GPS.


INTRODUCTION
The application of micro Unmanned Aerial Vehicles (UAVs) in geomatics is increasing since they facilitate the rapid and flexible acquisition of areas and objects at a medium scale.According to (van Blyenburgh, 2011), a micro UAV is defined as a small unmanned aircraft with a maximum payload of 5 kg and a flight altitude up to 250 m.
Mostly, micro UAVs are used in mining, agriculture, urban and architectural mapping, as well as archaeology (Eisenbeiss, 2009).In agriculture, UAVs can be used e.g. to detect fawns before the harvest like in (Israel, 2011), or to measure the nitrogen status of sunflowers, as shown in (Agüera et al., 2011).3D applications include different forms of topographic mapping.In (Neitzel and Klonowski, 2011) different methods for the generation of dense 3D point clouds are compared.An example application, which is presented in (Neitzel and Klonowski, 2011) is the mapping of a landfill.The mapping of landslides near road embankments is presented in (Carvajal et al., 2011).
In contrast to manned aerial vehicles, micro UAVs are equipped only with low-cost GNSS receivers and IMU sensors due to payload limitations.The accuracy of the sensors is not sufficient to use their measurements for direct geo-referencing, which is why UAV imagery is normally oriented through aero-triangulation.The present paper investigates whether thermal images in combination with artificial ground control points (GCPs) may be used for direct geo-referencing.Thermal images have lower resolution than RGB images and at the same time more blur and more distortion, and usually only few, if any, crisp local features.Figure 1 shows an example image.

Objectives
The goal of the project is to implement and evaluate a purely image-based approach for automatic geo-referencing of thermal images, i.e. the determination of the camera positions and orientations.The images are acquired by a thermal camera which is mounted on a UAV.

Paper structure
Chapter 2 is dedicated to the data acquisition.In the first two sections, the employed UAV system (2.1) and the geometric calibration of the thermal camera (2.2) is described.The flight planning and the distribution of the ground control points (GCPs) is shown in section (2.3).The recording itself is briefly described in (2.4).The processing workflow is presented in chapter 3, including synchronization issues (3.1), automatic image measurements of the GCPs (3.2), and bundle triangulation (3.3).In chapter 4 the experimental results are presented, and the influence of different matching strategies (4.1) and number of GCPs (4.2) is evaluated.In (4.3) the use of GNSS and IMU measurements is discussed.Finally, the paper ends with conclusions and outlook in (5).

DATA ACQUISITION
The UAV system used for data acquisition as well as the thermal camera are introduced in this section.Furthermore, we discuss flight preparation, including the flight planning and the choice of artificial GCPs.

Video recording UAV control
Figure 2: UAV system.
The employed UAV system (Falcon 8 from Ascending Technologies), the remote control and the two required notebooks are illustrated in (Fig. 2).The Falcon 8 performs vertical take off and landing (VTOL).During the flight it can be controlled by either a human operator who uses the remote control, or by control software which runs on the first computer.A second computer is required to record the video signal from the thermal camera, which is down-linked during the flight.
2.2 Geometric camera calibration 2.2.1 Calibration method Before the image acquisition, the interior orientation of the thermal camera needs to be calibrated.The camera can be modeled just like a conventional camera, as central projection plus (considerable) lens distortion.Different calibration workflows for thermal cameras have been presented in the literature.In (Luhmann et al., 2011) the calibration field is made of aluminium and has adhesive coded targets.Aluminium reflects almost 90% of the thermal radiation in the atmosphere (Ostermann, 2007).The absorbed intensity is low, therefore the aluminium appears dark in the thermal image.The coded targets absorb more energy and appear brighter than their aluminium base.This is illustrated in (Fig. 3).
Figure 3: Aluminium calibration field with adhesive coded targets, including a few elevated targets for a better geometric configuration.
A calibration field for indoor environments is presented in (Buyuksalih and Pétrie, 1999).The calibration field has a two-layered structure with a glass plate and an overlaid steel plate.Cross shaped openings in the steel plate serve as target features.In (Lagüela et al., 2011) a wooden board is equipped with a symmetrical pattern of lamps.In (Simmler, 2009) the indoor calibration field is a plate with embedded lamps that can be used as calibration targets.Because of their temperature gradient, the application of lamps in (Lagüela et al., 2011) and(Simmler, 2009) is not optimal: the lamps' temperature decreases gradually with increasing distance to its centre and leads to fuzzy boundaries.This problem does not apply to the approaches of (Buyuksalih and Pétrie, 1999) and (Luhmann et al., 2011).In both approaches, the target appear sharp in the thermal images, with well-defined boundaries.Among the two, the calibration field in (Luhmann et al., 2011) does not need a separate power supply, while the one of (Buyuksalih and Pétrie, 1999) does.The passive calibration field is more flexible for both indoor and outdoor applications.Moreover, it is easy to build.Based on the review we have chosen to use the calibration field of (Luhmann et al., 2011) in this project.

Calibration results
In our experiment, an uncooled thermal camera FLIR Tau 640 with a relatively high geometric resolution of 640 × 512 pixel has been used, as shown in (Fig. 4).The vanadium oxide microbolometer detector is sensitive to wavelengths between 7.5 µm and 13.5 µm.Following (Brown, 1971), lens distortion is modelled by radial symmetric distortion coefficients (A1, A2, A3), tangential distortion coefficients (B1, B2) and two parameters (C1,C2) for aspect ratio and shear.The calibration results are listed in Table 1.The parameter A3 deviates from 0 by less than 2 standard deviations and is not required, but listed for completeness.Figure 5 illustrates the radial symmetric distortion behaviour with increasing distance to the principal point.The test area for the flight is situated at the Science City Campus, ETH Zurich, and covers an area of 55 m × 40 m.In Figure 6) the waypoints are visualized.

Ground Control Points
In order to geo-reference UAV imagery in the absence of precise navigation data, GCPs are required.For optical images, natural GCPs are the norm.In contrast, it is in many cases not possible to find natural GCPs in thermal images which appear sufficiently crisp and can be located accurately thus we decided to use artificial GCPs.Based on the experience gained from camera calibration, we have chosen to use aluminium as the material for the GCPs.Aluminium sheets have a sharp boundary in a thermal image (see Fig. 3), are rugged, and do not require a power supply.The aluminium sheets used as GCPs for our specific project are circles with a diameter of 30 cm.The size was chosen based on the ground sampling distance, to yield a dimeter of approximately 6 pixels in the image.
The GCPs are distributed evenly in the test area.The configuration was designed such that in each image at least three GCPs should be visible, so that in case of difficulties during block orientation one could resort to calculating the absolute orientation of each image with spatial resection.The configuration is thus overly dense, and as will be seen (and could be expected) fewer GCPs will suffice (see 4.2).GCPs not used for geo-referencing were used as ground truth in the evaluation.

Acquisition of thermal images
The images were recorded according to the flight plan described in (2.3.1).Before the UAV starts the flight, the GCPs were laid out on the test field, and their coordinates determined with differ-

DATA PROCESSING
Data processing starts with the synchronization between the video and the position log of the UAV.Based on the synchronization, thermal images are selected from the video stream.As preparation for bundle adjustment the GCP locations are detected automatically in the thermal images.The bundle adjustment itself was performed with the photogrammetric software INPHO.

Synchronization
Since continuous video is recorded, the first step is to extract suitable images from the video file -in our case the frames corresponding to the waypoints of the flight planning.To that end the video sequence and the position log of the UAV navigation system need to be synchronized.Both recordings have independent time stamps.Therefore synchronization is based upon the UAV take off and landing events.In total, the extracted image block consists of 45 images.The detection of the GCPs and the measurement of their centres in each thermal image are the next processing step before one can proceed to bundle adjustment.To maximize automation, an algorithm has been developed and implemented in MATLAB to facilitate automatic GCP detection and measurement.

Automatic image measurements of GCPs
The first step is to threshold the image in order to detect dark pixels.Each connected component of such "dark" pixels (according to the 8-neighborhood) is regarded as a potential GCP region (Fig. 8).The set of candidate regions is then pruned by taking into account the area, circular shape and maximum intensity of a region.For our block this procedure was sufficient, although in general it may be advisable to use coded GCPs to rule out miss-detections.Finally, for the remaining regions the image coordinates of their centres are determined, by fitting a circle to the region boundary.The center coordinates of the circle are the (sub-pixel precise) GCP coordinates in the thermal image (Fig. 7).
Figure 8: Boundaries from thresholding (yellow); Input image as background.

Bundle triangulation
The aero-triangulation (tie-point matching and bundle adjustment) is performed with the commercial photogrammetric software IN-PHO by Trimble, as are the subsequent steps of DTM generation through dense matching, and ortho-rectification.
The tie point extraction and the following bundle adjustment is done with INPHOs "Match AT" module.Its automatic tie point extraction uses the feature based matching approach, followed by least squares matching.The matching accuracy can reach ± 1 10 pixel in good imaging conditions (Trimble Germany GmbH, 2010), although we find that this is usually not reached in practice, even for good optical images.
The numerical results from bundle adjustment are presented in chapter 4. Figure 9 shows the GCP locations, the four flight strips and the extracted tie points.

RESULTS
Overall, indirect geo-referencing of thermal images proved to work well, reaching similar reprojection errors as for optical images.The results of the bundle adjustment are affected by the number of GCPs and the applied image matching strategy.To investigate the impacts of both factors, different configurations with a varying number of GCPs and different matching strategies have been tested.

Adjust the image matching strategies
Match-AT allows one to vary the image matching parameters, which influence the number and quality of tie-points and thus the block orientation.The available tuning parameters are the search window size, the permissible disparity range and the minimal distance between two tie points.
Three different versions of matching were run, followed by bundle adjustment.The first version uses the default image matching strategy.In the second version the search region for feature points is decreased from 100 × 100 pixels to 40 × 40 pixels.Because there is no extreme height difference and large overlap, the parallax bound is decreased from 30 pixels to 15 pixels.In the third version, we additionally reduce the minimum tie point spacing from 50 pixel to 30 pixel and change the tie point density from the default value to "extreme" to obtain a maximal number of tie points.Figure 10 illustrates the evaluation results.The second version improves over the first one, seemingly the default options lead to a significant amount of incorrect matches, which can be prevented by the reduced search region and parallax bound.The third version reaches the lowest standard deviations for all coordinate directions.It seems that the "extreme" tie point density pays off, meaning that although potentially weaker tie points are added, the increased redundancy improves the robustness against false matches and the block stability.The third version is used in the subsequent experiments.

Adjust the number of GCPs
Not only the number and distribution of tie points have an influence on the accuracy but also number and distribution of GCPs.Using less GCPs, indirect geo-referencing gets more efficient, due to the reduced effort for GCP surveying, and when using artificial GCP targets, as we do, also for preparing the targets.This time the first tested version uses all 16 visible GCPs during bundle adjustment.They are visualized in (Fig. 9) as white and yellow GCPs.In the second version the 10 GCPs at the boarder of the block and a single GCP in the block center are used.These are only the white GCPs in (Fig. 9).Bundle adjustment with less than these 11 GCPs did not lead to satisfactory accuracy, particularly in the height.The results in terms of root mean square error (RMSE) are shown in (Fig. 11 ).With only 11 GCPs the bundle adjustment yields almost the same RMSE as with all 16.The differences in X,Y and Z are in the order of 1cm.The height of the control points has a lot lower accuracy than the planimetry, possibly due to the flat topography, although the statistics based on 5 control points is rather weak.Furthermore, we also show the standard deviations of tie points for the two variants.As can be seen in Fig. 12), going from 11 to 16 GCPs does not have much effect on the object point accuracy.For both cases, the achieved accuracy is about ± 1 cm for the horizontal components and about ± 3 − 4 cm in the vertical direction.The distribution of 10 GCPs at the boarder of the bundle block and one GCP in the middle appears to be sufficient for the block tested here.

Discussion of GNSS and IMU measurements
It is a natural question to ask whether the GNSS and IMU observations from the navigation system on the UAV should be used during bundle adjustment.While they can certainly serve as initial values (UAVs have less stable flight dynamics than large aircraft and the assumption of nadir views is sometimes not good enough to initialize aero-triangulation), the navigation data does not have any effect on the bundle adjustment.If used with the correct weights (which are exceedingly low because of the low navigation accuracy) it had no influence at all on the results.If used with higher weights it even degraded performance, supporting the claim that indirect geo-referencing is to be preferred if sensors on micro UAVs are to be oriented accurately.A promising approach to improve the navigation data is to use differential GNSS, as proposed in (Bláha et al., 2011).

CONCLUSIONS AND OUTLOOK
We have implemented and evaluated an automated photogrammetric orientation pipeline for thermal imagery acquired from UAVs, from sensor calibration to aero-triangulation (and orthorectification).It has been shown that in spite of the blurry and low-contrast nature of thermal imagery, existing automatic tiepoint matching delivers satisfactory results, and reaches imagespace accuracies comparable to optical imagery.We have advocated the use of artificial GCPs -on one hand well-defined natural points are hard to guarantee in thermal images, on the other hand the extra effort for the GCP targets is offset by the advantage that GCP measurement can be automated.
The overall result of our study is that for thermal mapping an object point accuracy of ± 1 cm = ± 1 5 pixel in horizontal direction and ± 3 − 4 cm in vertical direction is reachable without additional sensors (such as external trackers, high-precision GNSS receivers, or an additional optical camera).Thus, the evaluated standard approach should be suitable for most applications where geo-referenced thermal images are required.Furthermore the UAV position could be determined with an accuracy of less than ± 10 cm, a good order of magnitude better than with singlefrequency GNSS and cheap inertial navigation.If the orientation procedure could be made to run in real-time (which appears to be within reach with the next generation of mobile computing devices), thermal photogrammetry might also be suitable for outdoor UAV navigation.
The only manual step in our study was to assign the correct ID to the automatically detected GCPs.In future work this process shall also be automated.One possible solution is to use coded GCPs.Another, possibly more flexible solution is to triangulate the GCPs after relative orientation and compare their 3D configuration to the known configuration in the world coordinate system.Unless the configuration is symmetric, there will be only one matching solution.

Figure 1 :
Figure 1: Example for a thermal image acquired by a thermal camera FLIR Tau 640.

2. 3
Flight planning 2.3.1 Flight path A conventional flight planning was done to achieve the desired image scale and overlap.In our experiment we chose an image scale of 1:3000.Given the camera's pixel size of 0.017 mm this results in a ground sampling distance of 5 cm and flying altitude of 38 m over ground.The image overlap was set to 90 % in both longitudinal and lateral direction.Such high overlaps are common with UAVs, in order to increase the redundancy at virtually no additional costs (as long as the processing is performed automatically).

Figure 5 :
Figure 5: Radial symmetric distortion of the thermal camera FLIR Tau 640.

Figure 7 :
Figure 7: Automatic measurement of a GCP in a thermal image.
search region and parallax bound Adjust tie point spacing and density

Figure 10 :
Figure 10: Mean standard deviations of tie points in object space regarding three different bundle block adjustments.The matching strategies are different in each method.

Figure 9 :
Figure 9: Visualization of the bundle block with tie points (plus signs), GCP locations (triangles) and flight strips (blue lines).

Figure 11 :Figure 12 :
Figure 11: Root mean square error of GCPs and control points using either 16 GCPs (and thus no control points) or 11 GCPs and 5 control points.