A NOVEL MOSAIC QUALITY MEASUREMENT METHOD FOR UAV SURVEILLANCE AND REMOTE SENSING

A novel hardware independent real-time aerial image stabilization and mosaicing system is developed for mini UAV surveillance and remote sensing operations. In order to measure the quality of the constructed mosaics, several in-door and flight tests were performed. A novel mosaic quality measurement method utilizing 5 axis CNC for 3D positioning of the camera and printed high resolution aerial images for ground truth information is described. Results of the path following tests employing several state of art registration algorithms are provided. Mosaics constructed in real-time during flight tests are presented.


INTRODUCTION
Mini UAVs provide cost efficient, fast, flexible surveillance and mapping and can be deployed easily by using a mobile Ground Control Station with fewer crew requirements compared to conventional methods.On the other hand size and weight constraints of mini UAVs create several distortion effects such as severe vibrations and sudden movements.Several authors investigated image mosaics in order to address these issues and provide a natural way of real-time mapping (Morse et al. 2008, Kumar et al. 2001, Buyukyazi et al. 2013).Specialized hardware can provide real-time processing speeds but it may not be suitable for installation in small size mini UAVs (Kumar et al. 2001).Constraining the camera orientation and UAV movements can restrict the flexibility required for better examination of the objects of interests (Morse et al. 2008).Brief description of a novel hardware independent, real-time mosaicing and image stabilization system can be found at (Buyukyazi et al. 2013) where real-time processing speeds were achieved by using Ground Control Station CPU and optimized algorithms adjusted for surveillance conditions.
While image mosaics were investigated by many authors, there is a lack of unified metrics for measuring quality of resultant mosaics.Several authors addressed this issue by employing artificially created images and defining various metrics (Azzari et al. 2008, Paalanen et al. 2009, Zou et al. 2011).On the other hand in order to realistically measure the performance of the algorithms for surveillance and remote sensing conditions, a testing methodology utilizing hardware used during flights is required.
In this paper a novel mosaic quality measurement approach developed for performance tests of the system described in (Buyukyazi et al. 2013), by calculating amount of total drift resulting from following predetermined paths is presented.Instead of using artificially created images from a base image, method utilizes a set up consists of a 5 axis CNC for path following of the camera, printed hardcopy of high resolution aerial images for providing aerial scenery and ground truth information and camera used in mini UAVs for simulating operation of the actual system.Main contribution of this paper is the novel Mosaic Quality Measurement method using 3D positioning and printed images.On the other hand, this method is developed as a part of larger project, and uses components of system described in (Buyukyazi et al. 2013).System described in (Buyukyazi et al. 2013) was tested by utilizing several state of art image registration methods.Also a brief overview of novel aerial mosaicing and image stabilization system, achieving real-time processing speeds by optimized algorithms, working on standard ground station without requiring any additional hardware in mini UAVs is provided.System described at (Buyukyazi et al. 2013) is able to work on a wide variety of illumination and terrain conditions with both day light and infrared cameras and were tested on real world working situations by using an actual on-service UAV.In addition to indoor tests, results of the flight test that were conducted in order to investigate the mosaic quality of the real-time mosaicing system are discussed.

Related Works
A good example mosaicing and stabilization of aerial images acquired by using a mini UAV can be found at (Morse et al. 2008).In (Kumar et al. 2001) an integrated aerial surveillance system which uses a video processing hardware installed on UAV is presented.A good study on real-time mosaic mapping using autonomous vehicles can be found at (Richmond 2009) where author uses an Autonomous Underwater Vehicle (AUV) to obtain mosaic maps of sea floor.More general surveys on constructing mosaic images from a sequence of frames can be found at (Irani et al. 1998) and (Szeliski 2006).
Although image mosaics are widely investigated by many authors, there is a lack of unified methods to examine the quality of the results.Several authors addressed this issue by proposing various evaluation criteria and methods using artificially created images (Azzari et al. 2008, Paalanen et al. 2009, Zou et al. 2011).
A good survey and comparison of various image registration algorithms can be found at (Tuytelaars et al. 2007).Also detailed descriptions of the algorithms that were investigated in this study can be found at their respective references (Harris et al. 1988, Rosten et al. 2006, Lowe 2004, Bay et al. 2008, Calonder et al. 2010).

System Overview
In order to overcome size and weight constraints of the mini UAVs, system developed for mini UAV surveillance is designed to be as much hardware independent as possible.For this purpose, basic UAV system configuration consisting of one UAV and one Ground Control PC was considered as default setup.Any additional hardware requirement for computation such as a GPU or on-board video processor card was intentionally avoided.All processing was done on software and real-time processing speeds and operational level accuracy are aimed to be achieved by optimizations done on algorithm side.To further increase the flexibility, processing software was designed to be single threaded so that tested performance was also independent from CPU architecture.Schematic of the system overview is shown in Figure 1 where an additional processing station is optional.
Mini UAV used in this study is Bayraktar mini IHA.It has a wingspan of 2 meters, length of 1.2 meters, and weights 4.8 kg.Its standard operational speed is 60 km/hour at an altitude of 1000 meters and has a maximum altitude of 4000 meters.Its range is 15 km and can carry day light and infrared cameras as primary payload.UAV is controlled from a mobile Ground Control Station via a tracking antenna.Ground Control Station weights 11 kg and designed to be easily deployable to mountain areas.It has Intel i5 CPU having two 2.67 GHz cores and 4 GB RAM.All image processing is done by using a single core of CPU.Camera used in this study provides 576 x 720 pixel frames at 25 Hz.
For path following tests a 5 axis CNC having 3x6 meters base area were utilized.Camera used by mini UAV mounted to CNC head for 3D positioning.Hardcopies of high resolution aerial images are printed in various dimensions ranging from 1x1 meter to 2x3 meter.Camera were connected to an Intel i7 3.40 GHz desktop PC.Printed aerial images placed to the CNC base and CNC head utilized for following various paths.Frames received by camera were similar to images received in actual flight tests.Since printed images are placed on a flat surface, this set up did not simulate effects introduced by the height of the objects.These effects are observed to be apparent in low altitude flights and with excessive zooming.On the other hand in many surveillance conditions height of the objects is small compared to surveillance altitude which creates image sequences similar to the ones obtained at in-door CNC tests.Furthermore developed surveillance system utilizes RANSAC algorithm as described in section 2.2.3 that filters out outlier feature couples representing a different image motion then majority of the feature matches.Because of this factor, system observed to track background instead of tip of tall objects such as tree tops and roofs of tall buildings.So images acquired by the CNC test set up considered as a valid representation of actual surveillance conditions for evaluation of the performance of the developed methods.

Processing Methods
Detailed descriptions of the methods developed for real-time CPU based processing are beyond the scope of this paper and can be found at related reference (Buyukyazi et al. 2013).Instead a brief overview is provided in the consecutive sections.In-door mosaic quality tests were aimed to use same methods as much as possible for better simulation of the actual working conditions.

Image Acquisition and
Preprocessing: For UAV surveillance system, two most important factors affecting the quality of the images received before processing were motion blur and communication interference.Motion blur effects were prevented by applying appropriate shutter speed adjustments and any increase in the noise levels due to decreased amount of light received by camera sensor were aimed to be handled by choosing registration algorithms robust against noise.Frames corrupted by communication interference were handled by rejecting frames failing registration and continuing with the next one.If registration failure continues for an extended period of time algorithm is designed to reset itself.Because of the concerns about computational speed, no additional filtering applied apart from deinterlacing filter.A region of interest and scale factor selection was employed as a speed and accuracy control.
For in-door mosaic quality tests, same configuration of the surveillance system is used and only condition that was not realized was interference errors.On the other hand, since excessive communication interference to degrade acquired images, making image registration impractical for every method that is tested, in-door set up still considered as a valid indicator of relative algorithm performance.

Image Registration:
Main aim for the image registration part was to find registration methods that will best suit the mini UAV surveillance conditions.After surveying existing state of art algorithms, Harris corner detector (Harris et al. 1988), FAST detector (Rosten et al. 2006), BRIEF descriptor (Calonder et al. 2010), SIFT detector and descriptor (Lowe 2004) and SURF detector and descriptor (Bay et al. 2008) were selected for further investigation.On the other hand speed of the SURF in its original version was not enough for real-time processing in this test set up.In order to increase processing speed with a minimum loss in accuracy, a Modified Algorithm that utilizes a quick search over a few layers where feature points were clustered was developed.Results of the experiments showed that with the modifications, it is possible to attain real-time processing speeds with an insignificant loss in accuracy.In order to further increase robustness against scene changes and low gradient aerial images, an adaptive Hessian Threshold selection approach was developed.Details of this method can be found at (Buyukyazi et al. 2013).

Transform Estimation and Constructing Mosaics:
For estimation of the frame to frame transformation matrix, RANSAC algorithm was used (Fischler et al. 1981).Since performance of RANSAC algorithm is affected by the number of outliers in a matching set, a method for filtering of point correspondences were applied.Every feature in the based set was coupled with two features in the target set.Later, resultant set of correspondences were refined by comparing matching scores.For modeling image motions, similarity transformation was selected over Euclidean and homography because of its optimum properties concerning computational burden and number of degrees of freedom.For presenting user with the most recent information, replacing old pixels in the mosaic image with the newer ones in the frame was selected as blending strategy.Both flight tests and in-door path following testes used same configuration for consistency.

Measuring Mosaic Quality
Although there is an extended literature on image mosaics for a wide variety of applications, most of the evaluation methods based of subjective examination and method specific metrics.Main problem with objective evaluation of the mosaic quality is obtaining accurate "Ground Truth" data (Azzari et al. 2008).Several authors employed artificially created realistic camera frames by using a base image and reference image is used as ground truth information.(Azzari et al. 2008, Paalanen et al. 2009, Zou et al. 2011).
Although artificial frames based of real images were used in early simulations, this approach was intentionally avoided for later comparison tests.Instead methods that enable testing during the operation of the UAV and setups simulating real world conditions including camera and optical effects were developed.Path following method that is described in this paper was developed in order to measure drifting due to registration errors.In ideal registration when camera frames follow a path and come back to the starting point of the first frame, total transformation matrix should be equal to identity matrix.Any difference between the identity matrix and the final total transformation matrix is considered due to drifting registration errors (1). (1) Several camera paths shown in Figure 3 having pure translations and combinations of translation and rotations were tested.Camera followed these paths at a fixed amount of time so effects of  missing frames due to computation were also included.Difference between the original central point ( 2) and final central point (2) were calculated by using ( 3) and ( 4) in order to provide a drifting value in terms of number of pixels. (3)

Operation Modes
Detailed descriptions of four operating modes are beyond the scope of this study and can be found at related references (Buyukyazi et al. 2013).Since mosaic quality measurement methods are designed to evaluate the performance of various methodologies developed for real-time UAV surveillance system, it is considered beneficial to present actual operation of the system.It should be noted that path flowing testes used only Classical Mosaicing mode.On the other hand, since all modes used same registration and feature filtering approaches, results of path following tests can be considered as an indicator of the performance of other methods.
Mosaicing mode is used to provide user with a temporal local map of the scenery and natural image stabilization (Figure 4).First frame acquired is used as reference frame and consecutive frames  Rotating mosaics mode is developed in order to align the mosaic image to UAV orientation while filtering out small vibrations in rotational degree of freedom (Figure 5).Stabilization mode is developed for providing user with vibration smoothed camera frames while following the general camera motion (Figure 6).Hybrid mode is designed to provide user with background preserving stabilization by using mosaicing principle (Figure 7).

Path Following Tests
High resolution aerial images ranging from 3456 x 5184 pixels to 9624 x 9568 pixels were printed to hard copies ranging from 1 m x 1m to 5 m x 3 m in dimensions such as shown in Figure 8. Camera used in UAV is attached to the head of the CNC as shown in Figure 8 (2).Video stream was supplied to an Intel i7 3.40 GHz desktop PC as shown at Figure 8 (3).For path following tests, a 600 mm x 1000 mm rectangular translation path and an 822.7 mm long path having translational, rotational and scaling motions were  It should be noted that path following tests are designed to evaluate the performance of real-time mosaicing algorithm using various registration methods instead of evaluating these methods isolated from other factors.Factors effecting performance in actual surveillance such as feature filtering are also present in these tests.
On the other hand, these tests are considered as a good indicator of relative performance of various registration methods employed in proposed surveillance system.It is possible to extend the testing method in order to evaluate the isolated performance of registration algorithms to various other conditions but this is beyond the scope of this study.
Results of the translation and rotation-scale paths are shown at Table 1 and Table 2 respectively.0.5 scaling factor applied before the processing of the images.Both Modified algorithm and SURF utilized adaptive algorithm for threshold adjustment as in actual flight conditions while SIFT, Harris BRIEF and FAST BRIEF configurations used fix parameter sets selected for detection of similar number of features.For Modified Algorithm and SURF, adaptive algorithm was configured to hold the ideal number of detected points around 100. SIFT used with 4 octaves, 3 layers per octave and a threshold of 0.0797.For Harris, a quality factor of 0.18 is used.For FAST algorithm, a threshold of 42 to determine the status of a pixel with respect to central pixel is employed.BRIEF is used with standard configuration described in original paper.Processing speeds of SURF and Modified algorithm were 44 ms and 25 ms respectively.SIFT had the slowest computation speed with an average of 132 ms, fluctuating between 114 and 169 depending on the number of features detected.Harris points detected are also observed to fluctuate depending on the nature of the intensity distribution throughout the path.
Results of path tests showed that modified algorithm performed with lower drifting error compared to original SURF algorithm at a speed increase approximately with a factor of 2. This considered due to the fact that SURF algorithm detects feature at higher levels of the scale pyramid creating additional errors.In terms of drifting error it was only surpassed by SIFT algorithm.SIFT algorithm had the highest accuracy in translational path tests.This situation can be explained by SIFT having robust nature to changing scenery because of multi-scale detection and matching while other algorithms were more receptive to changes in intensity distribution nature.On the other hand Harris BRIEF configuration showed the largest results although it is known for its accuracy.Using a fixed parameter set instead and adaptive approach as in SURF and Modified algorithm may affected the results but on the other hand SIFT algorithm also having a fixed parameter set performed better than adaptive approaches in terms of drifting error.This may be due to the fact that Harris points were more affected by changing gradient nature throughout the path which was also observed by severe fluctuation in the number of detected points.Furthermore FAST BRIEF configuration performed better than Harris BRIEF but still poorly compared to multi-scale Hessian based detectors.

Flight Tests
Since surveillance system is developed to be used in actual operational conditions, performance of the mosaicing algorithm is also investigated at flight tests.On the other hand, evaluating quality of the mosaic is generally done by subjective visual inspection.A good way to obtain an idea of the mosaic quality is examining straight lines as shown in Figure 10.It should be Figure 10.Two mosaic maps created in real time with day light camera.noted that registration errors present in mosaic are cumulative results of various factor such as perspective effects, algorithm errors etc.
Mosaic image at Figure 4 (1) was composed of 1375 frames integrated over 55 second and Figure 4 (2) was composed of 325 frames integrated over 13 seconds.Although excessive rotational and scaling effects were present in both cases, resultant mosaics were relatively well formed.Average processing time was 44 ms.
Figure 10 (1) shows a mosaic image composed of 420 frames constructed by using a slower version of the developed algorithm with an average processing speed of 142 ms. Figure 10 (2) was composed of 213 frames and were processed at an average speed of 47 ms.Both cases demonstrated well-formed mosaic images with minor registration errors noticeable.
Tests utilizing infrared camera also were conducted.Figure 11 (1) demonstrates the effects of accumulated error when a previously constructed region is revisited.Mosaic was composed of 463 frames and processed at an average speed of 41 ms.Straight structures such as highways are also good features for examining errors.Figure 11 (2) demonstrates such a case where slight registration errors are noticeable through the way.It is composed of 488 frames processed at an average speed of 43 ms.

CONCLUSION
In this paper, a novel system for hardware independent, real-time surveillance and remote sensing system utilizing a basic mini UAV configuration was described.In order to measure mosaic quality, in-door and flight tests were conducted.In order to accurately measure effectiveness of state of art algorithms in operating conditions, a novel mosaic quality measurement method composed of 3D positioning and printed high resolution aerial images were developed.Results reveal optimum performance of Modified Algorithm in terms of speed and accuracy and developed system was able to create high quality mosaics at actual flight conditions in real-time.

Figure 1 .
Figure 1.Schematic of the system showing Bayraktar mini UAV, tracking antenna and Ground Control Station.Second station is optional and does not change processing performance.

Figure 2 .
Figure 2. Test set up that were used in in-door tests.

Figure 4 .
Figure 4. Two mosaic images constructed in flight tests.

Figure 6 .
Figure 6.A stabilization sequence recorded over approximately 2 seconds.

Figure 7 .
Figure 7. Approximately 1 second long Hybrid sequence processed at 18 frames per second.areplaced by calculating frame to frame correspondences and estimation of total frame transformation matrix.If newly acquired frame is transformed outside the boundaries of the mosaic, distance from frame location to mosaic boundaries is calculated and used to shift the mosaic image in order to fully enclose the newest image.

Figure 8 .
Figure 8. Test set up used in path following tests.

Figure 9 .
Figure 9.Translation path (1) and rotation path (2) drawn on actual aerial image used in path tests.Test image is available online MapKnitter.org with title "Lemon Fair Burn" and internet link is provided at references for repeatability of the tests.used.Path following speed was kept constant at 6000 millimeter / minute and a complete path time for translational was 40 seconds and for rotational path was 50 seconds.

Figure 11 .
Figure 11.Mosaics constructed by using IR camera.

Table 1 :
Harris BRIEF had 19 ms and FAST BRIEF had 12 ms average processing time.Number of features detected by FAST algorithm changed drastically resulting in a significant speed reduction.Number of Drifting errors in translation path

Table 2 :
Drifting errors in rotation-scale path