UAV Image Blur-Its Influence and Ways to Correct it

Unmanned aerial vehicles (UAVs) have become an interesting and active research topic in photogrammetry. Current research is based on image sequences acquired by UAVs which have a high ground resolution and good spectral resolution due to low flight altitudes combined with a high-resolution camera. One of the main problems preventing full automation of data processing of UAV imagery is the unknown degradation effect of blur caused by camera movement during image acquisition. The purpose of this paper is to analyse the influence of blur on photogrammetric image processing, the correction of blur and finally, the use of corrected images for coordinate measurements. It was found that blur influences image processing significantly and even prevents automatic photogrammetric analysis, hence the desire to exclude blurred images from the sequence using a novel filtering technique. If necessary, essential blurred images can be restored using information of overlapping images of the sequence or a blur kernel with the developed edge shifting technique. The corrected images can be then used for target identification, measurements and automated photogrammetric processing.


INTRODUCTION
A constraint enforced on the acquisition of early photographs used in photogrammetry was a stable camera position and a stationary object as the basic requirements for sharp images.Exposure times of many days were required to obtain an image (Maison Nicéphore Niépce, 2013}.Today, professional photographers use a tripod and remote shutter release to prevent camera movement during image acquisition.Unfortunately, unmanned aerial vehicles (UAVs) rarely provide a stable camera position.UAVs are affected by wind, turbulence, sudden input by the operator and also by the flight movement of the aircraft itself as well as vibrations of the engines.Hence, the acquired image sequence can contain blurred imagery which disturbs automatic post processing.Filtering these blurred images has to be carried out manually, which is exhausting for the eyes, prone to errors and time extensive.By filtering out blurred images the number of images is reduced and can negatively influence the result of post processing.This paper describes a series of experimental work: how a dataset of images was created with known blur characteristics; the effect of blurred imagery on photogrammetry and image processing; how to correct blurred images and subsequent image operations.

RELATED WORK
Photogrammetry is the science of reconstructing 'the position, orientation, shape and size of objects from pictures' (Kraus, 2007).To provide appropriate image geometry for 3D measurements a recommended along track overlap of 60% and a lateral overlap of 20% should be used (Kraus, 2004;Luhmann et al., 2006}.To calculate accurate 3D coordinates (X, Y, Z) for an object point, it is necessary to precisely measure the image coordinates (x, y) in at least two images.Today, fully automated data processing is demanded by an increasing number of users.Blur is expected to influence the accuracy and ability of image coordinate measurements, which is analysed in this study.
A series of methods have been developed in previous work to detect if an image is sharp or blurred.It is recognised that improving the image sharpness using deblurring algorithms is an important topic in computer vision and image processing.A widely used application for automatic blur detection is the 'auto-focus' system in cameras, which should prevent the user from taking optically blurred images caused by an inappropriate focal setting (Kim et al., 1998).As optical blur can be prevented using these systems, other methods are required to suppress blur due to motion, which should be carefully distinguished as a discrete type of blur.Motion blur is often caused by human hand movement (jitter).This jitter has frequencies of 2-10 Hz, with amplitudes of up to 1 mm (Stiles, 1976).Sachs et al. (2006) note that there is also a 'drift of the hand' of up to 5 mm/s and that commercial systems for shake reduction use gyroscopes to prevent motion blur.
Methods developed for aircraft include precise inertial measurement unit (IMU) and global navigation satellite system (GNSS) information, to reduce angular and forward motion blur (Pacey and Fricker, 2005).Both sensors can provide additional information for blur detection algorithms such as the approximate path followed when the image was blurred.Precise IMUs are also used to generate blur kernels which represent how an image is blurred.The kernel visualises how a point in an image would be blurred, which path it would have taken and which motion speed it would have had.IMUs used by most UAVs are too imprecise to establish a basic precise blur kernel.Assuming an image exposure time of 1/400 s would require an IMU of at least 800 Hz to measure a basic blur kernel for an image (Grenzdörffer et al., 2012).A blur kernel based on two measurements would be only a linear representation of the motion and would not represent more precise motion.However, even if linear representation of motion blur is sufficient to represent typical UAV flight motion, such systems are expensive and cannot always be used since these would exceed the payload of typical micro UAV platforms.Furthermore, the IMU and GNSS sensors incorporated for flight stabilisation are not accurate enough and lack sufficient acquisition frequency to determine if an image is blurred or not.This makes detection of motion blur during the post processing stage often necessary.
Blind blur detection algorithms normally use edge detection or frequency analysis.Edge detection methods focus on the spread and gradient of an edge (Joshi et al., 2008;Narvekar and Karam, 2009;Ong et al., 2003).In the case of a widely spread edge of low gradient it is assumed that the image is blurred.This edge smear due to blur is also used in image frequency analysis methods.When an image has extensively smeared edges in the spatial domain, it is characterised by the disappearance of high frequencies in the frequency domain (Liu et al., 2008;Rahtu et al., 2012).A problem with most blur detection methods is the presumption that the image is blurred and that these methods are developed using mathematically blurred images without any relation to geometrical motion blur or random hand held camera shake (Levin et al., 2009).In such circumstances the amount of blur is unknown and has to be evaluated subjectively.It is obvious that although subjective evaluation of blur maybe useful as fast and omnipresent method to evaluate images, it can be wrong.This explains the desire to develop a measure of blur, by generating images with a precisely known image blur which is not influenced by human perception.Using a blur kernel to represent blur is possible.However, comparing the blur between different images based on blur kernels is difficult due to the complexity, their shape and extent (Sieberth et al., 2013).
There are various methods for image deblurring which have been published over the last decades.Image deblurring approaches can be separated into blind and non-blind deconvolution.Non-blind image deconvolution uses a-priori knowledge about the blur in the image.Blind deconvolution has only the blurred image and no additional information but the task of deriving a sharp image remains.Wiener deconvolution (Wiener, 1950) and Richardson-Lucy deconvolution (Lucy, 1974;Richardson, 1972) are blind deconvolution methods proposed decades ago and remain popular because they are both simple and efficient.A significant problem that occurs using these methods are ringing artefacts at high contrast edges (Fig. 1).More advanced methods are often based on a probability calculation, which is also used by Wiener and Richardson-Lucy deconvolution such as for example (Sun et al., 2014).They aim to improve deblurring and the capability to work reliably, even with noise in the blurred image (Shan et al., 2008).There are also many other methods such as (Krishnan and Fergus, 2009) who does 'Fast Image Deconvolution using Hyper-Laplacian Priors' or (Michaeli and Irani, 2014) who uses recurrence of image patches.Non-blind image deconvolution methods can be carried out in various ways and requires additional knowledge.Additional information can be gained through a variety of methods including, other overlapping images (Agrawal et al., 2012), precise IMU measurements (Joshi et al., 2008), video cameras (Tai et al., 2010), fluttering shutters (Raskar et al., 2006) or colour channel dependent exposure times (Lelégard et al., 2012).The main aim of these methods is to establish an appropriate blur kernel, which can be used by established deblurring algorithms.

PRACTICAL WORK
The practical work described here is focused on three different problems associated with blur.First, how blurred images were created; second, how blur disturbs photogrammetic image processing; and finally, how blur may be corrected.

Generating blurred images
A mathematical low-pass filter could be used to simulate motion blur but would decrease noise and other hardware influences and return a non-realistic result (Sieberth et al., 2013).Instead of using image processing a shaker-table was used to acquire motion blurred images, which are influenced by typical degrading effects introduced by the camera hardware (Fig. 2).

Figure 2. Shaker table with camera mounted on it.
A shaker table is a platform used by construction engineers to investigate the likely response of a building to vibrating influences like earth quakes.It can be displaced with a known magnitude and frequency and movement of objects mounted on it can be monitored with displacement and acceleration sensors.
Here the device was used to mount a camera (Fig. 2).If the device is fitted with acceleration sensors, it is possible to determine precisely when the camera shutter opened and closed due to vibrations caused by the shutter movement.This can be combined with the displacement sensor allowing an accurate displacement of the camera to be calculated for each image (Fig. 3).However, the vibrations caused by the moving shutter remain small and do not significantly influence the camera (Sieberth et al., 2013).By using the displacement sensor, camera displacement can be measured precisely during image exposure.This precise measure provides a higher control for further processing.

Blur disturbs photogrammetry
A set of generated motion blurred images was used to analyse the effect of blur on photogrammetric data processing.This was achieved by processing blurred images, of high contrast targets, using photogrammetric and image processing procedures (Fig. 4).A range of typical photogrammetric procedures were examined and tested.The first concentrated on camera calibration used to determine the interior geometry of the camera (Sieberth et al., 2014b).The second procedure assessed the detection and registration of feature points in sharp and overlapping blurred images (Sieberth et al., 2014a).

Blur correction
After quantifying how much feature detection is influenced by blur and if image overlap can be calculated successfully, it can be decided which technique for deblurring can be used.One method developed for deblurring in this research uses information provided by an overlapping image determined after image registration.However, if an appropriate image overlap is unavailable, deblurring needs to be carried out using an independent approach.

Frequency transfer method using an overlapping image
The first method, using an image overlap is the frequency transfer method.This method uses a sharp overlapping image to deblur a blurred image.A perspective transformation is applied to the blurred image and both the sharp and the transformed blurred image are cropped to just the overlapping area.Then the images are transferred to the frequency domain using a Fourier transformation.It is well established that high frequencies are absent in blurred images (Lelégard et al., 2012).The absence of high frequencies can be compensated by enhancing the blurred image using high frequencies extracted from the sharp image.Afterwards, the enhanced image is transferred back to the spatial domain.As this frequency operation only works on single channel images, it is either possible to apply this method on each channel separately or to use just a grey scale image.If a grey scale image is used, the enhanced image can replace the intensity channel of the originally blurred image (Fig. 5).The approach relies upon an overlapping sharp image to correct the blurred image (Sieberth et al., 2014a) and if unavailable then an alternative approach is required.Figure 5. Deblurring using frequency transform method with 1.03 mm camera displacement.

Edge shift -an overlap independent method
If image registration is unsuccessful or no sharp image overlap is available then the frequency transfer method cannot be used and a different approach needs to be applied.The developed edge shifting approach focuses on high contrast edges.Suitable high contrast edges can include photogrammetric targets, which are often used for photogrammetric applications.They can normally be detected, identified and measured automatically in every image and the image registration can be conducted.Blur can prevent this automated process.These high contrast boundaries enable edge detection methods to find an edge in the image even if it is blurred.However, the blur causes a displacement of the edge (Fig. 6(b)), which is shifted by the size of the blur (Fig. 6).Shifting the displaced edges back to their original position provides the possibility to automatically detect the target and identify the target number based on the target code.To measure the target, the area of the target can be masked and the image moment (gravity centre) can be calculated, to represent the centre of the target (Fig. 6(c)).Using the target ID and coordinate measurement, allows the registration and orientation of the image.Unfortunately, the edge shifting approach returns just an edge image without colour information.However, after image registration it is possible to use the frequency transfer to generate a colour image when an overlapping image exists, which is sharper than the just processed image alone.

RESULTS AND DISCUSSION
The presented methods are effective and compensating for various stages of motion blur.All methods involved in this progress return novel and interesting results, which are explained further below.

Motion blurred images
Generating blurred images using a shaker table provides an accurate method to produce images with known characteristics.The fitted acceleration sensors are sensitive enough to recognise camera displacement and also the much smaller shake of the camera body caused by the shutter opening and closing.It was found that the vibration caused by the shutter is not significant, so that it can be neglected from further blur calculations (Sieberth et al., 2013).For further calculation of blur, the shaker table provides the opportunity to create many images with different blur characteristics under stable laboratory conditions.The same scene can be photographed with the same light conditions and with a displacement defined a-priori.However, one feature of the shaker table is that it is only able to simulate blur in one direction.This is appropriate because blurred images due to flight motion only contain one directional movement and not complicated blur paths.Forward motion as well as pitch and roll result in linear motion blur in the image.However, the combination of yaw rotation with forward motion would cause an almost linear blur.Only rotations around the yaw axis would cause a radial blur.Linear motion is also supported by the short exposure time used by UAVs and it can be assumed that complicated motions rarely influence the camera.A disadvantage is the limited displacement range of the table, which is less than 150 mm.In contrast to potential displacements created by UAVs, this displacement is very small.Although, real UAV images exhibit larger displacements, these occur at larger camera to object distances.Fortunately, these are equivalent to smaller displacements acquired at short object distances.Based on this assumption, the conducted laboratory experiments can be considered as representative.A major advantage of the methodology used is that the dataset is characterised by images with real motion blur, in contrast to other datasets which contain mathematically generated motion blur (e.g.'LIVE Database for image and Video Quality Assessment' (Sheikh et al., 2005)).The images created using the shaker table contain known blur defined by the camera displacement.This provides a higher control for following processing steps as calculated blur kernels can be compared to the true camera displacement.

Blur disturbs
After generating images with known motion blur, a range of tests were conducted.The first aimed to analyse whether and by how much blur disturbs normal photogrammetric processes (Sieberth et al., 2014b).It was found that it is difficult for the human eye to precisely quantify the amount of blur and assess whether it will disturb procedures adversely.This becomes even more difficult if there is no sharp image available for visual comparison.Even images of apparently high visual quality can cause problems, if a small amount of blur is present.The number of measured targets directly influences the accuracy and ability of subsequent calculations (Fig. 7).Measurements of blurred signalised targets become inaccurate and incorrect (Fig. 7(a)) therefore degrading subsequent procedures such as the calculation of the principal point (Fig. 7(b)).It can be assumed that detection and measurement of unsignalised targets is worse and again negatively influences the results.Additionally, feature detection algorithms are also affected, which can be identified in terms of both the number of detected features and correctly referenced features (Table 1).Figure 7. Negative influence of blur on target detection and camera calibration (Sieberth et al., 2014b).
The results supports the findings of Johnson and Casson (1995) who proved that human perception is influenced by blur.Furthermore, it shows also that blur influences photogrammetric image processing and in contrast to human perception, image processing procedures are more sensitive to blur.The detection of blurred simple structures such as round targets certainly causes problems for automatic procedures.Colombo et al. (1987) tested humans on the detection of structures in blurred images and found a decreasing ability to read text with increasing blur.However, the legibility of text is a much more complicated task than detection of round targets.
The results support the desirability to exclude blurred images from photogrammetric processing.Gülch (2012) recommends elimination of blurred images as a first step in UAV image processing.The findings also support work of Shah and Schickler (2012) who developed blur correction methods specifically for UAV applications.Lelégard et al. 2012 states that blur larger than 2 pixels is significant.Our findings would contradict this, suggesting that a blur of just 2 pixels is actually too small to influence detection, identification, referencing and measurement.

Blur correction
To prevent the negative influence of blurred images on post processing, blurred images should be detected and deblurred.
The frequency transfer method appears suitable to correct some blur, if overlaps are available and one image is sharp.If the image used for deblurring is itself blurred, then the enhanced result maybe better than the original blurred image but will contain at least as much blur as the 'sharper' image originally contained.Adding frequencies to a blurred image also introduces some noise after transforming from the frequency domain back to the spatial (Fig. 5).In the subsequent step of image transformation, which is necessary to regain the correct position, the rotation, shear and scale and the interpolation of the correct pixel intensities for the rectified image, can cause a 'blurring' effect.It could also happen that the transformation parameters of the images are unknown, which makes deblurring using an overlapping image, impossible.Overlapping images are often available in photogrammetric procedures, however, it can also happen that not enough overlapping images are available and the frequency transfer method cannot be applied.This frequency transfer is based on the idea of Jung et al. (2009).However, our method presented here uses images which are not highly overlapping or taken from the same camera point of view, which makes it applicable for aerial imagery.
The newly developed edge shifting approach is a method to deblur images containing high contrast edges.High contrast edges are often used for image registration, as these can be clearly detected and measured in image sequences.Standard deblurring methods like Wiener deconvolution or Lucy-Richardson deconvolution, produces ringing artefacts on high contrast edges making automatic detection of features significantly difficult (Fig. 1).In contrast, edge shifting does not produce these artefacts and makes automatic detection possible.It is also faster than standard deconvolution methods because it concentrates on edges only.However, the main problem is the loss of colour information, which needs to be restored afterwards by using a flood fill method.More significant problems are the many special cases and exceptions which occur.Handling these exceptions creates a range of challenges.It is possible that the shifting process generates holes in the shapes.This is caused by shifting two edges apart from one another, which gives the appearance of 'ripping' them apart.Furthermore, edges which have completely disappeared cannot be recovered.Deblurring targets which are smaller than the blur also need to be considered as special cases.Deblurring real images with different edge intensities, directions and blur length remains more challenging than in the high contrast test image.Despite these unsolved cases it is possible to achieve edge shifting using photogrammetric targets, to enable subpixel accurate measurement of target coordinates.Furthermore, it might be possible to incorporate different blur in different positions of an image by calculating localised blur kernels.This would be necessary for blurred objects close to the camera, which appear more blurred compared to objects further away.
The method developed here is completely different to previously published methods, because it does not concentrate on restoring colour information using probability theory (Lucy, 1974;Richardson, 1972;Shan et al., 2008;Wiener, 1950) but only edges.This provides the additional advantage of being much faster than methods previously published.

CONCLUSION
The study reported in this paper shows that it is difficult to generate blurred images that contain a known path and amount of camera displacement.However, a shaker table was found to be suited and preferred to simply simulating blur using a low pass filter.Precisely blurred images are required to analyse if and when blur degrades a photogrammetric processing.It was shown that even small camera displacements have a significant impact on the accuracy of subsequent photogrammetric image processing.A threshold can be defined, which can be used to filter blurred images and/or apply deblurring algorithms (Sieberth et al., 2014b).If deblurring is necessary or required, then overlapping images can be used to achieve reasonably sharp results for further processing.If overlapping images are unavailable then it is still possible to use the 'edge shifting' approach for image deblurring.This novel approach can restore images, provided that high contrast edges are available.

Figure 3 .
Figure 3. Image exposure with camera displacement.

Figure 4 .
Figure 4. Sharp and blurred image (1.03 mm camera displacement) of the dataset.
Figure 6.Distortion of targets and target code due to blur to elliptic shapes.

Table 1 .
Camera displacement while camera shutter is open vs. detected and correctly referenced feature points.Table 1 presents how many features were detected in images containing different camera displacements.With increasing displacement the number of detected features decreases.Furthermore, Table 1 does show how many of the detected features were referenced correctly to features detected in a sharp image.The number of correct referenced features decreases significantly with increasing camera displacement.