Automatic Geolocation of Targets Tracked by Aerial Imaging Platforms Using Satellite Imagery

Tracking of targets from aerial platforms is an important activity in several applications, especially surveillance. Knowledge of geolocation of these targets adds additional significant and useful information to the application. This paper determines the geolocation of a target being tracked from an aerial platform using the technique of image registration. Current approaches utilize a POS to determine the location of the aerial platform and then use the same for geolocation of the targets through the principles of photogrammetry. The constraints of cost and low-payload restrict the applicability of this approach from UAVs. This paper proposes a methodology for determining the geolocation of a target tracked from an aerial platform in partially GPS devoid environment. The method utilises automatic feature based registration technique of a georeferenced satellite image with an aerial image to retrieve the geolocation of the target. Since it is easier to register subsequent aerial images due to similar viewing parameters, the subsequent overlapping images are registered together sequentially resulting in the registration of each of the images with georeferenced satellite image. The result of this registration is overlapped with the initial registered image to obtain resulting registered image and thus location of tracked target in each of the frames. The proposed concept is verified experimentally and the results are found satisfactory. Using the proposed method, a user can obtain location of features on ground without any information for the position of the aerial platform. The proposed approach has applications in surveillance for target tracking, target geolocation as well as in disaster management projects like search and rescue operations.


INTRODUCTION
Surveillance of a hostile target by an aerial platform using data collected by inbuilt sensor is one of the recent developing fields in remote sensing.It involves tracking a target which aims to collect data using an appropriate sensor within a specific field of view containing the target of interest (Blackman and Popoli, 1999).It can also be used to obtain target velocity and to predict future position of the target.Aerial images obtained from aerial platforms have been widely utilized in this context.However, for real time tracking of a target and specifying its position from an airborne platform, knowledge of the geo-location of the target is essential.The term geolocation refers to estimation of the geographical location (latitude and longitude) of an object with reference to a given coordinate system.The coordinates of such targets can be obtained by various methods like by using geo-referenced images.The imagery for estimating the location of an object is usually obtained by a satellite or by an aerial platform.The proposed approach uses the registration of image obtained from UAV with the georeferenced satellite image which is already stored in UAV's database to specify the coordinates of the target via registered image.The key to this approach is a fast and robust algorithm for registration and which further leads to the target position using registered image.

STATE OF ART
The technique involves selection of best suitable algorithm for image registration for registering a georeferenced satellite image with an aerial image and subsequent target geolocation in subsequent frames of the aerial image.Both the images i.e. satellite image and aerial image vary in resolution, sensor specification, altitute of platform, climatic conditions, temporal changes and similar viewing parameters.A technique of extracting the common feature between image and map, named Diffusion Geodesic Path (DGP) with experimental results for image map registration using conventional features such as edge points or corner points signifies matching results of varying parameters (Tian, et al., 2006).Similarly, method of automatic registration of aerial oblique images and digital map by two-stage processing (Kawai, et al., 2007) namely global and local registration was found satisfactory.Matching two images with different resolutions which can be represented in scale-space and also described a one-to-many robust image matching strategy was successfully implemented (Dufournaud, et al., 2004).Similar results of image registration based on the combination of Feature Based Mapping (FBM) and Area Based Mapping (ABM) for two pairs of data sets was demonstrated experimentally (Hong, et al., 2008).The results established that the proposed algorithm can select enough control points to reduce the local distortions caused by terrain relief.A wavelet-based feature extraction technique, normalized crosscorrelation matching and relaxation-based image matching techniques were employed in this method.An efficient image registration between high resolution satellites and a UAV down-looking aerial image using composite deformable template matching has been proposed (Fan, et al., 2010).The problem brought by the environmental changes and different sensors was resolved by taking the advantage of edge and entropy of the image by experimental investigation.Another image registration technique was point based, corner based and contour based (Reji, et al., 2012).The proposed algorithm is fast and robust and can handle image with large rotation and translation, and image with scale difference if sufficient number of closed regions exists for satellite image registration.
Image captured from an aerial platform have been used and interpreted regularly.Eroglu, et al., 2013 implemented and tested a very fast and accurate terrain referenced navigation algorithm.They designed a pre-processing phase in which the terrain DEM (Digital Elevation Model) is prepared for lookups.The system was implemented only as simulations with assumptions to facilitate embedding on a real system.Experimental results showed that this algorithm produced realistic results on 15x15 km 2 territory.Further, they proposed that the performance of the existing system can be increased significantly with the help of parallel programming techniques.They latter emphasized on the localization of UAV using only the elevation data of the territory (Eroglu, et al., 2014).DEM with original 30 meter-resolution (Eroglu, et al., 2013) and synthetically generated 10 meter-resolution maps are utilized to represent the terrain data.The simulations on the synthetic terrain data showed that terrain data with high resolutions allows using longer profiles.Further, they also proposed the possible improvements on both success rate and localization.An area based matching for automatic georeferencing of archaeological photographs captured from a UAV with a preexisting Ortho Photo Map (OPM) and Digital Surface Model (DSM) taken from different point in time was presented by Karel, et al., 2014.The method uses Normalised Cross Correlation (NCC) coefficient of perspective transformed image patches with a specific threshold value.The control points were generated by interpolation of terrain heights in DSM to image points formed in OPM.Using the bundle block adjustment, the false matches were eliminated.

Proposed Concept
The present methods for target surveillance by an airborne imaging platform utilise its positional coordinates obtained from a POS and then use the same to compute the location of the target.However, such modules are widely dependent on the accuracy of the airborne GNSS position estimation and its efficient data processing to computes the relative ground coordinates.Thus, any small variation in its actual positional value and attitude of a UAS will result in larger error for locating coordinates of a target on ground.Similar or worse situations may also arise when working in a GNSS denied environment (e.g.jamming, interference, blocking).In such cases, though the user control is sharing the real time data, they are actually clueless of the location of the target under surveillance.To overcome such situations, a georeferenced satellite image of the general area of interest is used to determine the coordinates of the target being tracked.This needs a fast, robust and efficient algorithm and matching technique for image registration of the satellite image with the aerial image.Once a successful image registration is performed, the registered image computes the coordinates of the target w.r.t the georeferenced satellite image.Since both images vary in resolution, brightness threshold, viewing conditions, altitude and their viewing parameters, image enhancement is performed to make them compatible to each other.SURF detector and descriptor algorithm is used for image registration since it is a computationally fast, efficient, scale and rotation invariant (Bay, et al., 2006).Once the images are successfully registered, the next step is to retrieve the coordinates of the target tracked by the aerial vehicle (tracking module pre-incorporated).The same location is overlaid in the satellite image and generates the location coordinates in WGS-84/UTM.The proposed method is explained by a flow chart in Figure 1.

Image Registration
A high resolution satellite image with accuracy of a few meters of the general area provides better and accurate geolocation of the tracked target.Considering the fast and efficient results achieved after comparison with other methods/algorithm (Bay et al., 2008), SURF 64 has been selected for image registration of both the images.Since a satellite image has been taken from a much higher altitude with a small FOV, the features and images captured covers the two dimensional representation (top view) of the general area.However, the same features when taken from an aerial platform like a UAV are captured in images of comparatively smaller area with better resolution and have additional information related to relative heights of different features based on the obliqueness of the viewing angle of the area.Thus, the additional objective also involves matching similar features in two different images varying in dimensional representation and taken from different viewing angles.SURF computes integral image, detects blobs, interest points and orientation vector in both images separately.The orientation vector (64 bit) in satellite and aerial images are used to determine their scale-space representation, orientation and alignment.The automatic registration is carried out using nearest neighbourhood method.SURF detector ascertains interest point (space) and scale of feature identification (blobs)

Satellite Image
Aerial Image

Geo-location using registered image
Figure 1.Geo-location module using Feature based registration and the descriptor defines orientation and feature vectors.The sequence of steps is covered in Figure 2. RANSAC method (Fischler & Bolles, 1981)

Scale space representation
Scale space representation

Target Geolocation
The next step involves tracking a target in the aerial image.The inbuilt tracking module in the UAV tracks the target in different frames based on the frame rate of the installed camera.Thus, the location of the tracked target is known in a particular frame of the aerial image in terms of its row and column values.The aerial image is registered to the georeferenced satellite image which has geolocation of all the points and features in the image.The method involves locating the position of the tracked target in satellite image with reference to a frame of the aerial image.Hence the location of the target is determined in the registered image and its coordinates obtained using pre-existing data of the satellite imagery.The proposed model uses coordinate determination in latitude and longitude which are derived from the registered image between a satellite image and an aerial image.

Registration of Subsequent Frames
The registration of the two images locates the centre point of the blobs detected in the two images at a given scale and computes the descriptor vector for applying the transformation and generating the common image frame.For a real time tracking module, multiple frames per second of the moving target are captured.Hence, it is necessary to register subsequent frames to obtain target's position and direction of motion in nearly real time.In addition, adverse situation may arise in operating the aerial vehicle in locations experiencing denial of GNSS signals.In such cases, it is imperative to obtain the result of registering subsequent images with the initial aerial image using SURF algorithm and thus obtain a series of overlapping frame grid.The proposed method is faster than the registration of satellite image to each aerial frame since computation of scale-space and descriptor vector for a particular image is carried out only once.The obtained information is retained in the system memory and later matched with the computed details of the next frame.The method is also used to predict the coordinates of the target at a particular frame and at a particular time.The final result of this module gives a series of aerial images overlapped over the satellite image of the general area.

EXPERIMENT AND RESULTS
An experiment was conducted using an existing scaled model of a building at IIT Kanpur to ascertain the effectiveness and accuracy of the proposed method.The model is used as the actual landscape and satellite and aerial images are simulated using the given landscape.These satellite and aerial images are used as raw data in the proposed approach.

Parameters of the Model:
The methodology involves registration of imagery taken from two different sensors at varying altitude, viewing conditions, time variation and other climatic consideration.The existing model of Rajeev Motwani Building in IIT, Kanpur, India (Figure 3) was selected for the conduct of the experiment.The parameters considered for the model are listed in The sensed image at an oblique angle was taken using an Apple i4s rear camera from an altitude varying from 0.5 m to 1 m.This height was small to achieve the desired ratio of the relative height of the aerial platform and the satellite sensor.A moving target (toy car) was placed on the model for conduct of the experiment and the images were captured by both the cameras in different frames.The coordinates of the aerial platform is known when the first frame of the image was captured.The image registration was performed using SURF algorithm with RANSAC to remove outliers which generates the registered image with common reference coordinates for both the images.The next step involves transformation of the positional values of the target tracked in the aerial image to the registered image to define its location.The target location in the experiment is determined in terms of pixel values of the reference image.The third step of the algorithm covers the faster approach for registration of the subsequent frames using the preregistered aerial image in a GNSS denied zone.The initial frame of the sensed image is already registered with the reference image and hence the integral image has been computed and the blobs are already detected for its features.
The next frame of sensed image has common overlapping area with the first frame.Moreover, the viewing conditions, image size and sensor parameters for both these sensed image are similar in nature.Hence, to enhance the rate of registration, the integral image computation and blob detection for the subsequent sensed image is performed using SURF algorithm.The feature based matching for the two sensed image is carried out with RANSAC for outlier removal.The method is iterative for subsequent frames and the position of the target tracked by sensed image can be ascertained in any frame (Figure 4).

Results
The proposed methodology was implemented in three stages namely image registration, geolocation of a target and registration of subsequent frames of aerial images.

Registration of Satellite and Aerial images:
A set of free version of Google Earth image and aerial imagery from an aerial platform of IIT, Kanpur were obtained for analysis of image registration.The proposed algorithm was implemented for a varying set of images.The results obtained are listed in Figures 5-8.

Analysis of Results for Geolocation of
The results obtained from the experimental setup are based on the implementation of SURF algorithm for images captured from different sensors at varying altitude and viewing parameters (Figure 9).The images cover the same area of interest.Nikon D5100 SLR camera captured the image at an altitude of 20 m at Nadir and Apple i4s rear camera captured the image at an oblique angle from a distance of 1m.
Comparing the images, we find that the image captured in nadir direction gives us the top view of the features in the general area, whereas the image captured from near distance at an oblique angle gives us additional information relating to the relative heights of the features.In addition, the oblique image covers a smaller portion of the area of interest in greater detail (pixel size) whereas the complete area captured from nadir at a higher altitude has comparatively fewer feature details (Figure 10).The implementation of image registration algorithm was successful and matching result was obtained (Figure 11).The geolocation of the tracked target (toy car) in aerial image was found in terms of pixel values of the reference image.The concept can be visualized as a miniaturized form of implementation in real cases using a satellite image and an aerial image along with a pre-incorporated tracking algorithm.
The validation of the accuracy of the target location was performed using the actual location of target in reference image at the time it was captured by the sensed image.The result of validation shows a shift of actual position by 4 to 6 pixels along X and Y direction (Table 3).The target was represented in four frames in the registered image and three users were asked to note the location of a point in the target on these frames.The result was validated with the actual position as obtained by the Nikon D5100 camera at 20 m.

Registration of Subsequent Frames:
The next step of proposed method covers the registration of subsequent sets of aerial images over the satellite imagery.The results obtained after registering the subsequent frame along with the location of the target using proposed algorithm are listed in The second frame is overlapped over the initial frame and the geolocation of the target is obtained in the second frame Table 3. Accuracy analysis of the proposed method using the Original Target position captured by Nikon D5100 SLR at 20 m and the result obtained of target by the proposed method

Analysis of Result for Registration of
Frames of Sensed Image: Subsequent frames an aerial image in actuals consist of multiple overlapping frames with similar viewing parameters.The concept was replicated in the experimental setup with four overlapping sensed images to analyse the feasibility of implementation of fast algorithm using SURF and RANSAC.The overlapping sensed images are of nearly same dimension under similar conditions of image capture.The successful registration of the first sensed image with the reference image detects the blobs for feature matching in both the images.The subsequent frame of the sensed image is matched with the initial sensed image and the result generated is the combined output of both these images over the reference image (Figure 12).The algorithm is faster than registering a satellite image with an aerial image as it performs matching of overlapping areas of images with similar size and viewing conditions.The concept is used for the registration of subsequent frames over the reference image (Figure 13).The location of the tracked target can be performed in any frame as desired the user (Figure 12-14).This method generates faster and more accurate result as compared to registering each frame of an aerial image with the satellite image as the subsequent frames have common conditions for performing the feature matching.

CONCLUSION
The proposed method was successfully implemented for image registration for Google Earth image (satellite image) and an aerial image of IIT, Kanpur and experimental results have indicated that it is possible to geolocate a target in an image acquired from a moving UAV.Since we used the feature based matching technique, the method was found suitable to function even when navigation satellite signals were unavailable.The validation of this method was covered by an experimental setup.Based on the results obtained and their implementation we conclude the following.

Image Registration of Satellite and Aerial Image
The two types of images employed vary in resolution, sensor type, viewing conditions, altitude, climatic variation and temporal changes.Since SURF uses the concept of feature based matching, the image registration for both these images was carried out successfully.This facilitates identification and subsequent matching of features using dominant orientation.A larger scale may result in false matching as the feature vector describes the orientation within a radius of six times the detected scale.The algorithm is efficient in an area which has more significant features of varying geometry.The integral image computation reduces the computation time and the result is obtained in near real time domain.

Geolocation of Target
The experiment conducted for implementing the proposed algorithm using the concept of SURF is faster than many other techniques as it uses the concept of feature based matching with integral image computation.The registration of images and subsequent geolocation of the target with tracking module incorporated in the aerial image gives matching result with a slight shift due to the obliqueness of the captured aerial platform.However, the deviation is small as seen in experimental setup and thus the concept can be used in real case scenario where accuracy up to a meters in real time is acceptable.

Registration of Subsequent Frames of Sensed Image
Image registration of the first frame with the reference image using SURF locates blobs in terms of scale-space representation and descriptor vector along the dominant orientation in both the images.The subsequent overlapping images will always have a few common features with the initial image.Using SURF, we detect the blobs and the descriptor vector along the dominant orientation with scale space representation.The two sensed overlapping images are matched and the combined result is registered with the reference image.This process speeds up the registration and geolocation of the target in subsequent frames as we perform registration of images of similar sensor and viewing conditions.Thus, the computation cost is low and the method is time efficient.The tracking module in aerial platform is used to locate the target in any frame in real time.

Contributions made
This works is related to geolocation of a target tracked by an aerial platform using the coordinates of the corresponding satellite imagery.The experimental setup was conducted and some of the significant contributions made are as follows:- Developed a method for geolocation of a target tracked from an aerial platform using feature based image interpretation approach which can function even in GNSS denied environment.


Experimental verification of the method was determined.


Registration of subsequent frames of aerial image with the initial registered image.


Integration of the proposed concept together as a working module carried out experimentally.

Figure 4 .Figure 9 .
Figure 4. Work Flow diagram for conduct of experimentTable 2. Characteristics of Nikon D5100 SLR and Apple i4s

Figure 6 .
Figure 6.Automatic Image registration using SURF of Google Earth and Aerial Image of Department Office Building in IIT Kanpur

Figure 13 .
Figure 13.Geolocation of target in the fourth image frame.The image shows the four overlapping frames and the target is tracked in the fourth frame in near real time

Feature Matching using descriptor vectors Matrix transformation Superimposition of Aerial Image over Satellite Image
of model fitting with inliers is applied which uses smallest possible data set and enlarges it to estimate the model parameters.The final result is in the form of registration of aerial and satellite image.

Table 1 .
Factually, the satellite imagery are captured from a higher altitude (hundreds of kms) with a higher resolution camera and an aerial image from comparative lower altitude (few meters to few kms).Hence, a Nikon D5100 SLR camera was selected to replicate the requirement of depicting satellite imagery and the sensed image for aerial image representation was captured using the mobile camera of iPhone Apple i4s.The comparative characteristics of both the cameras used in the experiment are listed in Table2.
Figure 2. Image Registration sequence for Satellite and Aerial Images Table 1.Parameters of Selected Model for conduct of Experiment Figure 3. Google Earth Image (top) of IIT, Kanpur and the model of Rajeev Motwani Building selected (bottom) for conduct of experiment 4.1.2Selection of Sensors: Figure 12.Geolocation of target in the second image frame.