AUTOMATIC ISOLATION OF BLURRED IMAGES FROM UAV IMAGE SEQUENCES

: Unmanned aerial vehicles (UAV) have become an interesting and active research topic for photogrammetry. Current research is based on images acquired by an UAV, which have a high ground resolution and good spectral and radiometrical resolution, due to the low flight altitudes combined with a high resolution camera. UAV image flights are also cost effective and have become attractive for many applications including change detection in small scale areas. One of the main problems preventing full automation of data processing of UAV imagery is the degradation effect of blur caused by camera movement during image acquisition. This can be caused by the normal flight movement of the UAV as well as strong winds, turbulence or sudden operator inputs. This blur disturbs the visual analysis and interpretation of the data, causes errors and can degrade the accuracy in automatic photogrammetric processing algorithms. The detection and removal of these images is currently achieved manually, which is both time consuming and prone to error, particularly for large image-sets. To increase the quality of data processing an automated filtering process is necessary, which must be both reliable and quick. This paper describes the development of an automatic filtering process, which is based upon the quantification of blur in an image. A “shaking table” was used to create images with known blur during a series of laboratory tests. This platform can be moved in one direction by a mathematical function controlled by a defined frequency and amplitude. The shaking table was used to displace a Nikon D80 digital SLR camera with a user defined frequency and amplitude. The actual camera displacement was measured accurately and exposures were synchronized, which provided the opportunity to acquire images with a known blur effect. Acquired images were processed digitally to determine a quantifiable measure of image blur, which has been created by the actual shaking table function. Once determined for a sequence of images, a user defined threshold can be used to differentiate between “blurred” and


INTRODUCTION
Research involving UAV's is a current research topic (Aerometrex, 2012). The evolution from manned to unmanned aerial vehicles has progressed rapidly using advancing technology, faster and more powerful computers, increasing knowledge about aerodynamics and the development of newer, lighter and more robust materials. The removal of an on-board human makes a UAV both time and cost efficient. Pilot training, registration and airport fees are not required and the usage of SLR cameras instead of aerial imaging systems reducing costs significantly. UAV's are of particular interest for military use as human life is not endangered (Shane and Shanker, 2011).

Motivation
One of the main problems influencing image quality, beside the flight altitude and the camera model, is the blur of the image. Blur caused by the forward movement of the UAV can be computed using the following equations:  Image blur introduced by the forward motion of the UAV can degrade the quality of data, even if a very short exposure time and high flight altitude is used (Fig1 and 2) for example a flight altitude of 100m, a flight speed of 11m/s and an exposure of 1/100s generates blur equal to 4 pixel. However, turns and turbulences cause a more significant blur effect on image quality. These blurred images cannot be processed using automatic image processing software because the software can fail to identify control targets or tie points ( Fig  3). Manual processing therefor becomes necessary, which is time consuming and prone to error.
If an automated procedure could be developed to filter out blurred images, data processing efficacy would be improved.

Aims and objectives
The main aim of this study is to filter out blurred images from datasets automatically prior to image processing using a developed software algorithm. As an initial step, images with a defined blur have been captured, which have been used to develop and test a new algorithm. This algorithm is capable of quantifying image blur, which is used to compute a threshold value. This threshold value can be used for detecting and excluding blurred images from flight datasets. It is recognised that the on-board autopilot of a UAV can be used to improve flight quality and additional data derived from Global Navigation Satellite System (GNSS) and Inertial Navigation System (INS) could be used in identifying blurred images. However, the strength of the approach developed here is that only images are required.

Current research
This paper describes the current development of an algorithm to detect motion blur in images. Existing blur detection algorithms often require additional information associated with both the image and image acquisition. Many of these are using information of INSs (Shah, 2012), whilst others include a video camera (Ben-Ezra and Nayar, 2004, Tai et. al., 2008, Agrawal et. al., 2012. An INS which is included in the UAV autopilot does normally not have a measuring frequency that is high enough to calculate a blur kernel. An additional video camera is often too heavy and requires additional modification and calibration for the calculation of a blur kernel. A blur kernel represents precisely the three dimensional movement of a camera during image acquisition. It can be used to separate images with a small blur, from images with a larger movement (big blur). Also other hardware modifications like "Forward Motion Compensation" (Pacey and Fricker, 2005) or image stabilisation are difficult to include in a light weight UAV or a low cost camera. In the case of a "Forward Motion Compensation" the whole camera, the film or the sensor is moved in the direction of travel during image acquisition to compensate blur effects. Image stabilisation uses a mathematical model to compensate the motion blur effect. Both methods are only efficient for small blurs but cannot compensate significant image blur caused by UAV turns or turbulences. Other detection algorithms modify the exposure time to detect blur effects, which is difficult to realise for low cost high resolution cameras. A modification can be done for each of the RGB channels, which results in a different amount of blur for each channel (Lelégard et. al., 2010). Another approach is the usage of an unblurred reference image (Paramanand andRajagopalan, 2012, Charlmond, 1991), which cannot be used for moving UAVs. Other algorithms which are not based on additional information or hardware modifications work often with edge detection (Ong et. al., 2003, Joshi et. al., 2008, Narvekan and Karam, 2009) and frequency analyses (Clark Jones, 1958, Liu et. al., 2008. Blurred edges are not well defined in an image, which can also be seen in the frequency space by a decreased number of high frequencies. The number of high frequencies is reduced and the number of low frequencies is increased. These algorithms apply higher control using an existing dataset of known camera movement and the resulting blur, which is difficult to achieve for UAV imagery.

METHODOLOGY
The main problem of existing algorithms is that the blur in the images cannot be quantified, which prevents defining a threshold value to automatically exclude blurred images from datasets. An operator is required to manually identify blurred images, which is subjective. The subsequent sections describe the development of a new automated method to quantify image blur.

Image blurring using a shaking-table
To acquire images with a defined blur the movement of the camera and the point of time of the image exposure have to be known, which can be achieved using a shaking By fixing the camera on the table and measuring both the displacement and the acceleration of the platform, images with a known blur can be generated (Fig 4). A test field comprised of fixed photogrammetric targets was established on a horizontal ceiling located approximately 1.7m above the shaking table.
The camera was equipped with a 24mm lens providing an image scale of 1:70. A horizontal camera displacement of a single pixel (6.1μm) produced by the shaking table equates to a blur of 0.4mm in the object space. Additionally, a G-Sensor is attached to the camera body in direction of the movement of the table. A second one is attached to the camera body to ensure that the camera is only moving in one direction and a third sensor is attached to the platform. These sensors are primary used to gather additional information and control of the movement of the table and the camera. The movement of the table is based on a user defined input function describing the velocity (amplitude) of the platform. It is synchronised with the camera shutter to acquire images at a time of constant velocity (Fig 5), which generates homogenous blurred images. Additionally, measurements acquired using the displacement sensor can be compared to observations determined using the G-Sensor. It is noticeable that the G-Sensor measures a high acceleration immediately after the camera trigger signal, due to open and closing of the mirror. No additional camera movement was registered by the G-Sensor, showing that the camera is not affected by vibrations during image exposure (Fig 6). The image exposure is between the two signals in green (positive, start of exposure, negative, stop of exposure), delayed after the real triggering signal but during a period of constant displacement. After the shaking process, the signals can be evaluated and the movement of the camera during the exposure can be calculated. The producible blur is limited by both, the minimal movement speed (0.001m/s) and maximum displacement (0.15m) of the platform and needs to be combined with the time of image exposure. A longer exposure time requires a slower movement of the table and consequently a shorter exposure time requires a faster movement of the table platform. This provides the opportunity to generate images with both, different blur and different brightness, which is based on the exposure time.
Additionally, background textures were projected onto the ceiling to vary the image content (Fig 7). The complete dataset involved capturing nearly 2000, images consisting of 7 different exposure times and 10 different background textures. To visualise the blur a special test pattern was used. This pattern consists of lines of different width and orientation as well as a Siemens-Star (Fig 7). The advantage of the star is characterised by the possibility to increase visualization of the direction of movement in contrast to using a simple circular target.

Image blurring by image processing
Another method used for this paper to generate blurred images is the use of image processing software with image filters, for example a Gaussian filter. The aim of these filters is to "smooth" the image by blurring it. This results in an image distortion, which can appear similar to true optical blur. MATLAB also provides a specific filter, which simulates motion blur. The filter represents a vector, which is defined in both size and direction of movement (MathWorks 2013). This filter has been applied on images without any a-priori movement, which assures that no additional blur is introduced by the movement of the camera (Fig 8).

Blur detection
Images with a known blur, generated using the methods described in section 2.1 and 2.2 were processed to compute a "blur value" using a newly developed mathematical algorithm. However, it is necessary to investigate and exclude unwanted systematic effects.
To ensure that the movement of the mirror does not influence the computed blur in the images, the G-Sensor measurements of the camera body were analysed. Integrating twice over the time, acceleration was converted to distance and the camera shake visualised. This generated a movement of just 0.7•10 -5 μm, in comparison to a pixel size of 6μm. It would therefore be concluded that the movement caused Figure 9. Camera shake due to mirror movement.
by the mirror is insignificant and the computed blur for each image was based on the actual value measured by the laser displacement sensor (Fig 9). After ensuring that the mirror movement is not significantly influencing the image blur, it is possible to analyse the images and calculate a value which is related to the blur using the developed mathematical method. This blur related value is called "sieds". Figure 10 represents the sieds and demonstrates that the calculated sieds depends on the movement of the camera. Figure 10. Camera displacement in comparison to sieds.
The distribution of the results indicates that the calculated sieds is related to image blur via a modified damping function (Eq 2).
where b = blur δ = damping ratio x 0 = start position ω = angular frequency n = y-offset from zero w = sieds, derived from image To set the unknown parameters for the damping function, which are damping ratio (δ), start position (x 0 ), angular frequency (ω) and offset from the y-axis (n), it is required to include other information provided in the image, which are independent of the amount of blur. A blur independent value is the average grey-value of an image. The calculation of the sieds is also possible for the images blurred by image processing. But this graph represents a logarithmic function and not a damping function (Fig 11). Figure 11. Evaluation of simulated motion blur with MATLAB.
The intercept on the y-axis is much higher, except for 1/10s exposure, than in the images blurred with the shaking table. An explanation can be found through closer examination of the blurred results. It is possible to see clear differences in the centre of the images. In the real blurred image (Fig 12a) the centre is the darkest part of the smeared dot. In the MATLAB blurred image it is (Fig 12c) not the centre of the dot which is darkest, but there are two shadows which appear around the centre. It is known that the shaking table image (12a) was acquired during a constant velocity, whilst the still image (Fig  12b) was captured without any movement. By comparing this to the MATLAB processed image (12c), it appears that the image was not blurred with a constant velocity but with acceleration. Figure 12. Difference between real blur with constant velocity (a), still image (b) and simulated MATLAB blur with acceleration (c).

Discussion
By calculating the blur independent parameters and including these in the modified damping function together with the computed blur related sieds, it is possible to calculate the blur for an image. A first test of this process was conducted using a mixed dataset of images with a known blur, which demonstrates that it is possible to calculate blur for images. However, these results demonstrated this is not as accurate for short exposures as for images with a longer exposure time ( Fig  13).  This is perhaps expected as images with a short exposure do only contain small blurs. Closer examinations of Figure 10 reveals why it is more difficult to calculate small blur effects precisely. In the area of small blur the gradient of the damping function appears very flat. Small variations in the computed sieds have a large impact on the final estimated blur. A closer examination of the absolute values reveals that the images with an exposure of 1/200s are set to zero but all the other calculated blur values are close to the real blur (Fig 14).
In the 1/10s exposure, the calculated blur fits closely to the real blur of the images. Although the mean error of 7.65μm (compared with 6.1μm pixel size) shows that the calculation based on the images is very precise.
The algorithm was also applied on a real UAV image dataset. This demonstrates that the worst images are blurred to an equivalent of 95px (Fig 15).  The graph shows that it is possible to differentiate images with a different amount of blur. The examples from Figure 2 are classified with 34px and 0px blur. To give an impression which kind of blur can be differentiated a second example is shown in Figure 16. This demonstrates how effective the blur detection algorithm can be to detect blur in images. Even images with a very homogenous background, where a human cannot identify blur without a reference image, the algorithm is capable to detect blur. The calculation speed for nearly 200 images, with 3600px to 2700px is just three minutes using a laptop with 4 (2.5GHz) CPUs and 4GB RAM. This is clearly acceptable.

CONCLUSION
This paper has outlined the development of an algorithm to detect blurred images in UAV image datasets. This makes it possible to exclude these blurred images or use blur correction algorithms to improve further data processing. The algorithm does not contain any GNSS or INS data and does not include any information about the neighbouring images. This represents a very benefit of the developed algorithm and allows application beyond simple UAV image filtering.
The main problems which are not solved yet are the influence on the calculation due to the image size and the correlation of the detected blur size to a real movement of the camera. The algorithm does also not contain any blur kernel calculations which are normally required to correct blur in images. The algorithm is efficient, being fast and reliable and separates blurred images in a UAV image dataset, as long as the images have the same size and the same exposure time.

Future work
For the future is it planned to improve the accuracy of the blur calculation by taking account of the image size and other influencing factors and correlating the calculated movement to real world values. Additionally information of the low cost GNSS and INS sensors will be included to see if improve results justify the increased system complexity.