AN AUTOMATIC OPTICAL AND SAR IMAGE REGISTRATION METHOD USING ITERATIVE MULTI-LEVEL AND REFINEMENT MODEL

Automatic image registration is a vital yet challenging task, particularly for multi-sensor remote sensing images. Given the diversity of the data, it is unlikely that a single registration algorithm or a single image feature will work satisfactorily for all applications. Focusing on this issue, the mainly contribution of this paper is to propose an automatic optical-to-SAR image registration method using –level and refinement model: Firstly, a multi-level strategy of coarse-to-fine registration is presented, the visual saliency features is used to acquire coarse registration, and then specific area and line features are used to refine the registration result, after that, sub-pixel matching is applied using KNN Graph. Secondly, an iterative strategy that involves adaptive parameter adjustment for re-extracting and re-matching features is presented. Considering the fact that almost all feature-based registration methods rely on feature extraction results, the iterative strategy improve the robustness of feature matching. And all parameters can be automatically and adaptively adjusted in the iterative procedure. Thirdly, a uniform level set segmentation model for optical and SAR images is presented to segment conjugate features, and Voronoi diagram is introduced into Spectral Point Matching (VSPM) to further enhance the matching accuracy between two sets of matching points. Experimental results show that the proposed method can effectively and robustly generate sufficient, reliable point pairs and provide accurate registration. * Corresponding author


INTRODUCTION
Image registration is the process of transforming the different set of data into one coordinate system, also, may be said as the process of overlaying two or more images of the same scene taken at different times, from different viewpoints or from different sensors (Dawn et al., 20100).Since Synthetic Aperture Radar (SAR) is capable of large-area coverage, cloud penetration and all-weather acquisition, it is frequently combined with historical optical images for analyzing a selected area, especially in cases of natural disasters and military reconnaissance.For instance, in the event of an earthquake, generally, only SAR mages can be acquired, due to inclement weather.Naturally, registration of optical and SAR images has become a core problem.
Registration algorithms can be broadly classified as intensity or feature -based approaches.With intensity-based methods, pixel intensities are compared directly for a subregion of an image.In such a scenario, a similarity measure is coupled with an optimization algorithm in an attempt to identify the closest mapping of pixels (Bowen et al., 2015).The widely used intensity-based methods include mutual information (Siddique et al., 2012), the cluster reward algorithm (Inglada and Giros, 2004), and cross-cumulative residual entropy (Hasan et al., 2009).In contrast to intensity-based methods, the featurebased ones do not work directly with image intensity values.These approaches are based on the extraction of the salient structures and features of images.Significant regions (forests, lakes, fields), lines (region boundaries, coastlines, roads, rivers) or points (region corners, line intersections, points on curves with high curvature) are understood as features here.The features represent information on higher level.This property makes feature-based methods suitable for situations when illumination changes are expected or multi-sensor analysis is demanded (Zitova and Flusser, 2003).
Many different methods have been proposed for solving the problem of automatically locating tie points in multi-source images using feature-based matching (Gong et al., 2014;Wang et al., 2015).However, there still exist the following problems: (1) Feature-based registration methods rely on the algorithm for extracting the primitives to be matched; thus, matching results are largely dependent on the results of feature extraction, and weak feature extraction results would generate unsatisfactory matching results.(2) The common theme in all of these featurebased matching techniques is that each method relies on a single feature extraction algorithm for extracting the primitives to be matched.The obvious consequence is that the potential for finding common features in a pair of images is severely limited (Dare and Dowman, 2001).(3) Since the imaging geometry is different, corresponding features have different sizes and shapes.Furthermore, since images are acquired using different sensors, the grey values of common features are also quite different.Therefore, it is very difficult to recognize common features in both images.Based on the above problems, an automatic optical-to-SAR image registration method using multi-level, iterative and refinement model is proposed. To avoid failed registration caused by poor feature extraction, we propose a simultaneous feature extraction and matching method using an iterative and refinement model for optical and SAR images.When the extracted features are not sufficient for image registration, a re-extract mechanism will be triggered, until the features are matched successfully.
 A multi-level framework that provides coarse-to-fine registration behaved as local feature (visual salience feature) geometry feature (area and line features)  intensity feature is proposed to refine the result step by step.
Obviously, using the coarse-to-fine registration framework can improve the efficiency of the proposed method. The procedure incorporates numerous feature extraction and matching algorithms, rather than relying on just one.Effective integration of improved SIFT matching, shape curve matching and VSPM matching.The experiments show that by using multiple feature extraction and feature matching algorithms, the precision and reliability of matching results can be significantly increased.

PROPOSED METHOD
In this section, we present the framework of the proposed method (see Figure 1), and the detail steps of our algorithm are as follows: Step 1: Coarse registration between optical and SAR images using visual saliency feature (details can be seen in Section 2.1).If there are more than three tie points, then original SAR image can be rough registered.Otherwise, go to Step 4.
Step 2: Using level set segmentation method on original optical image, according to step1, original SAR image have been rough registered, and thus we can employ this optical segment result to initialize the level set function of rough registered SAR image, in order to make SAR image segmentation result more accurate.After that, sift matching method is applied on segmented optical image and rough registered SAR image.
Step 3: Using the geometric relationship obtained from the coarse registration result acquired by step1, to extract line features of both optical and SAR images.And then, fine registration result can obtained using KNN matching method (details can be seen in Section 2.3.3).Go to Step 5. Step 4: Applied iterative level set segmentation and matching method on original optical and SAR images (details can be seen in Section 2.2).Meanwhile, coarse-to-fine registration using line extraction and VSPM method is also used to original optical and SAR images (details can be seen in Section 2.3).
Step 5: Combined matching candidates acquired by the outlier removal method proposed by Gonçalves et al. (2011) was used to achieve sub-pixel accuracy.If sub-pixel accuracy cannot be achieved, we will keep the tie points which can make the accuracy maximum.

Coarse registration using visual saliency feature
Visual attention mechanism studies show that the human visual system will first focus on some of the visual saliency objectives, namely image local features, these local features can first be used as a mark for optical and SAR image registration, such as rivers, lakes and iconic buildings.Based on the above, this paper presents a fast coarse registration method based on visual saliency features.The main idea is to use notable regional characteristics and structural features to match images.For significant regional characteristics, this paper combines the visual saliency model and the level set segmentation method for object extraction, and a shape curve matching method was used for registration.
Firstly, Itti model is used to detect the notable regions for optical images (Itti et al., 2001).While for SAR images, some improvements were made to the Itti model, which mainly focused on the extraction of early visual features.Instead of color information, textures generated by the GLCM are applied to obtain feature maps for SAR images.Thus, below we mainly explain the improved Itti model (named TW-Itti) for SAR image, the classical Itti model for optical image will not described again.
Input is provided in the form of a static gray image; nine spatial scales are created using dyadic Gaussian pyramids, which progressively low-pass filter and subsample the input image.I is used to create a Gaussian pyramid ) ( I , where ) 8 ... 0 (   is the scale.Center-surround differences (  defined previously) between a "center" fine scale c and a "surround" coarser scale s yield the feature maps.
The first set of feature maps is concerned with intensity contrast, which, in mammals, is detected by neurons sensitive either to dark centers on bright surrounds or to bright centers on dark surrounds.Here, both types of sensitivities are simultaneously computed (using a rectification) in a set of six maps ) , ( Considering the dark grey value of water, a weight coefficient is introduced to calculate the center-surround differences, mainly to maintain the low reflective value of water.Then equation ( 2) can be rewritten as below: where  is the weight coefficient and A second set of maps was similarly constructed for the texture channels.A texture image T was generated by using GLCM and principal component analysis (PCA) algorithm.Here, T was comprised of five conventional textures (Angular, Contrast, Correlation, Inverse difference moment and Entropy).The window size was 7 7  , the distance value was 1 pixel and there were 16 grey levels.After acquiring these textures, the PCA method was used to obtain the first principal component to represent texture image T .Texture feature maps ) , ( s c T can be represent as follows: ) The third set of maps is the local orientation O , here O is obtained from I using oriented Gabor pyramids ) , , were encoded as a group, with a local orientation contrast between the center and surround scales: In total, 36 feature maps were computed: 6 for intensity, 6 for texture, and 24 for orientation.Then after processing of across-scale combinations, normalization and linear combinations mentioned in (Itti et al., 2001), the improved salience map is generated.
Finally, segment the salience map of both optical and SAR images into binary images using the level set method (see Section 2.2.1), and then match the objects in those binary image by shape curve matching algorithm, it can be described as below:

Automatic registration with iterative level set segmentation and matching
To avoid failed registration caused by poor image segmentation, we propose a simultaneous segmentation and registration method using an iterative level set segmentation and matching for optical and SAR images.When the segmented features are not sufficient for image registration, a re-segment mechanism will be triggered, until the features are matched successfully.

Area features acquired using level set segmentation
Due to the variation of illumination, different viewpoint and image quality etc., the idea of identifying the conjugate features for registration is more challenging.
Where z is an integer.
The evolution of opt  is governed by the following motion where t denotes the time step , ε  is a regularized version of the Dirac function and div represents the divergence.
A gamma model was used for high resolution SAR image segmentation.Suppose ) , ( u SAR y x is a SAR image, we model the image in each region i R by a Gamma distribution of mean intensity i u and number of looks L : Therefore, according to Equation ( 6) and ( 9), the level set functional for SAR images can be improved as follows: The evolution of SAR  is determined by the following motion partial differential equation:

Iteration strategy and matching
After segmented area features, SIFT descriptor was used to identify tie points and determine whether the image registration is successful, given that SIFT has been proven superior to others in terms of resisting common image deformations.More details for SIFT algorithm see (Lowe, 2004) and (Mikolajczyk and Schmid, 2005).However, one-time feature extraction algorithms may significantly limit the matching accuracy due to poor extraction results.Thus, a "re-extracting" and "re-matching" strategy is introduced to improve matching performance, which mainly involves adaptive parameter adjustment for image segmentation.
Assume reached the maximum number of iteration allowed.
Step 3: If the above process is unable to get satisfactory matching points at end of the inner loop, the value of opt S should be changed according to the floating up and down method.Therefore, t1 S is should be set to 70, assign the value of t1 S to opt S , and repeat step 2.

Coarse-to-fine registration using line extraction and VSPM
In this part, a multi-level framework that provides coarse-tofine registration is proposed.Searching the entire image for feature extraction would cause great challenge in feature matching since the details in the optical and SAR images differ a lot.Therefore, coarse registration using VSPM (Voronoi integrated spectral point matching) is implemented at a low resolution level to preserve the main spatial structures while eliminating the details, followed by feature extraction and matching.With the help of transformation obtained at the coarse registration, refined registration using KNN is implemented at the original images.Obviously, using the coarse-to-fine registration framework can improve the efficiency of the proposed method.

Line features extraction and pre-processing
Line Segment Detector (LSD) (Gioi et al., 2010), which aims to detect straight contours on images, has been applied to extract linear features from remote sensing image.It extracts line segments with relatively low false alarm rate.To this end, LSD is employed to extract line segments from optical images.
Taking the advantages of its flexible parameter selection, gradually widened spacing between two windows and good smoothness in local mean estimation, edge detector using Gaussian-Gamma-Shaped (GGS) bi-windows (shui and cheng, 2012) has shown better performance than traditional rectangle bi-windows for edge detection in SAR images.Then, after extracting edges using the GGS-based method, Hough transform is performed to obtain line segments in SAR images.
Because of image quality and down sampling, the terminals of extracted segments may have positioning errors, length and angel constraints are considered after line extraction.

Coarse scale image matching using VSPM
In our method, Voronoi polygons are introduced into spectral graph.Similar with the SPM method (Wen et al., 2008) measures point relationship by Euclidean distance, we measure polygon relationship by Hausdorff distance.Then, the proximity matrix H can be computed as: where r ij is the Gaussian weighted distance between two Voronoi polygons VP i and VQ j .hausdorff(A,B) is the hausdorff distance between polygon A and B, a and b are points of polygon A and B, respectively. is a parameter controlling the degree of interaction between points.
Next, the modal structure of point-set P associated with optical line intersection set and point-set Q associated with SAR line intersection set is found by performing eigenvalue decomposition on the proximity matrix HP and HQ.To avoid the problem of eigenvalue multiplicity [40], a perturbation matrix K is added to proximity matrix HP and HQ.
) δ and ζ are independent realizations from a uniform distribution over the interval (0,1).Since their value is very small, a slight perturbation operation still makes the eigenvalues distinct.
To solve the sign ambiguity when computing eigenvectors at the situation Ax=λx and A(-x)=λ(-x), a sign correction is performed.For each column v i in V, suppose that each column u i in U is given, then the corrected sign of v i is determined as follows: ( 1 m i n ( , ) ) According to Shapiro and Brady, the correspondence probabilities are assigned by the smallest modal Euclidean distance.In our method, the angle consistency constraint is used to guide the correspondence.Suppose that two line segments in the optical image generate the point P O and ) ， ( in the SAR image generate the point P S , and the two points are corresponding points.If the angle difference is too large between , it is impossible that the two points (P O , P S ) are corresponding points.Then, we quantize the angle consistency constraint as: A(P O ,P S ) is the possibility that P o and P S to be a pair of corresponding point from the angle consistency constraint concept.t is the threshold controlling sensitivity on deformations.For a good match,


. However, in practical aspects, if the deviation between the angles lies within the threshold t, then the angle consistency constraint is satisfied.It can be set to 5° according to experiments.Then, the final similarity Z between P and Q can be measured by computing the affinity between their corresponding spectral and the angle consistency constraint: ' ( ) (24) V' denotes the revised matrix of V.If Z ij is the greatest value in both row i and column j, then we considered the i th point in U and the j th point in V to be a match pair.
It should be noted, that the point matching process is performed on the coarse registration on low resolution image.In the proposed method, outliers come from isolated lines in the two images.In the low resolution image, extracted line features are salient structures in images.Thus, only a few isolated lines can be detected in the optical and SAR image.Then, few outliers exist in the point matching process.This can be one of the main reasons that we adopt the multi-scale registration strategy.

Original scale image matching using KNN
With the help of corresponding points obtained at low resolution level, the original SAR image can be transformed and then line segments extracted at high resolution level are matched based on a specific matching cost function and candidate tie points can be obtained.
In order to efficiently find conjugate line segments, a specific matching cost function which represents the matching quality between two line segments is developed.The matching cost function relies on the angular difference between the segments and on the distance from the center of one of them to the other straight line, and is defined as: Due to the significant differences between the optical and SAR images, there may exist distortions between the line intersections.In order to avoid wrong pairs in the line intersections, KNN graph matching is used to check and remove those wrong pairs, and the remaining corresponding points are used to produce the final registered images.Specific algorithm process can be seen in (Izadi and Saeedi, 2012).

EXPERIMENTS AND ANALYSIS
Here, two sets of experiments are designed to evaluate the proposed approach.The experiments are mainly aimed to demonstrate the necessity of a model using multi-features and multi-measures.Figure 2(a) shows the registration result with iterative level set segmentation and the matching method, three iterations is needed for dataset1 to adjust the segment parameter, performing feature extraction and matching only once would not be successful for this dataset, iterative feature extraction and matching is needed to provide robust matching.However, only 5 matching points can be obtained, these matching points are mainly distributed around the lake, and the accuracy of places far from lake is very low.Thus, line extraction and the VSPM method is applied follow.Figure 2 In order to qualitatively judge registration accuracy, the root mean squared error (RMSE) is used, and twenty uniform distribution checkpoints are selected manually.The value of RMSE for dataset1 using our method is 1.334 pixels.

Application on UAV images : using visual saliency and line features
This experiment simulates the matching process when unmanned aerial vehicle (UAV) is in flight.There are four experiment images, one is an optical reference image, the other three are the sequential SAR images.The study area is with Mianyang City, Sichuan Province, China.The tested images are high resolution UAV images in suburb area.The optical and SAR images have significant radiation differences.Furthermore, the quality of the optical image is degraded by cloud interference, which increases the registration difficulty.The optical image was acquired in April, 2012, with a resolution of 0.5m, the image size is 4140*3240.The sequential SAR images were also in April, 2012, with a resolution of 1m, the pixel seize of three sequential images are 2272*1636.
For Sequential SAR image1, 3 correct tie points can be obtained using visual salience features, see figure 3(c)(d).Therefore, a coarse corrected SAR image can be obtained.However, for this data set (the optical reference image and sequential SAR image1), level set segmentation and matching method, line extraction and VSPM method are all failed, no matching points can be found.
With sequential SAR image2, we first registered the image with sequential SAR image1, using level set segmentation and matching method, figure 4(a)(b) represent the image segment result and final set of tie points of the optical and SAR image.From figure 4(c), the region in SAR image with intensive tie points can be well superpositioned with optical image.Similarly, for this data set, no more matching points can be found using area-based or line-based registration methods.
Fortunately, for sequential SAR image3, line extraction and the VSPM method is effective, 219 tie points can be can be obtained for this data set, see figure 5(c).Figure 6 shows the corrected sequential SAR images overlay with the reference optical image.

CONCLUSIONS
In this paper, an automatic optical-to-SAR image registration method using multi-features and multi-measures is proposed.The main contribution is that the procedure incorporates numerous feature extraction and matching algorithms, rather than relying on just one.Experiments have shown the effectiveness of our method.
In future work, it is to be hoped that the present research can also be extended to other multi-sensor imagery registration.The main difference relies on the feature extraction techniques for extracting robust line segments.Also, in order to deal with the real-time processing of large data, parallel technology could be considered, it could make the proposed registration method effective when applied in military applications or disaster emergency response.

Figure 1 .
Figure 1.The basic principle of the proposed method.
5) where A,B represent the segment objects in optical and SAR image, A i d represent distance between the th i sample point on outer boundary of shape A to the centre point of shape A. from starting point j of shape B to the centre point of shape B. B j d is the mean of B ji d .
inside C and outside C , respectively, and  , v , function H , and the one-dimensional Dirac function  , respectively, by of E and D are eigenvalues of U and V in decreasing order, respectively.The columns of U (resp.V) are eigenvectors of HP (resp.HQ) corresponding to eigenvalues in E (resp.D).
Registration result using area and line features of dataset1.(a) Image segment result and final set of tie points of the optical image (left) and SAR image (right) using iterative level set segmentation and matching method.(b) Line segments extraction for optical image.(c) Line segments extraction for registered SAR image.(d) Tie points of optical image (left) and SAR image (right) using line extraction and VSPM.(e) (e) The chessboard overlay result using area and line features.Figure 3. Coarse registration result with visual saliency feature for sequential SAR image 1.(a) Visual saliency region obtained using Itti model for optical image.(b) Visual saliency region obtained using TW-Itti model for SAR image.(c) Set of tie points for optical image.(d) Set of tie points for SAR image.(e) The chessboard overlay result of coarse registration.Registration result using level set segmentation and matching method for sequence SAR image 2. (a) Image segment result and final set of tie points of optical image.(b) Image segment result and final set of tie points of SAR image.(c) Overlay result for sequence image Registration result using level set segmentation and matching method for sequence SAR image 3. (a) Line segments extraction for optical image.(b) Line segments extraction for SAR image.(c) Set of tie points for optical (left) and SAR image (right).

Figure 6 .
Figure 6.Sequential SAR registered images overlay with reference optical image.3.1 Application on Terra-SAR image : using area and line features For dataset1, both images have high spatial resolution and significant radiation differences.The study area is of the Changjiang River in central China.Optical image is acquired on Jun, 2013, with the resolution of 1m, the image size is 5632*5632.SAR image is acquired by TSX-1 in 2008 in descending VV polarization mode, with a resolution of 1 m and a size of 6202*7012 pixels.
Figure2(a) shows the registration result with iterative level set segmentation and the matching method, three iterations is needed for dataset1 to adjust the segment parameter, performing feature extraction and matching only once would not be successful for this dataset, iterative feature extraction and matching is needed to provide robust matching.However, only 5 matching points can be obtained, these matching points are mainly distributed around the lake, and the accuracy of places far from lake is very low.Thus, line extraction and the VSPM method is applied follow.Figure2(b) and (c) show the line segments detected in the optical and SAR image, respectively.And, 746 pairs of corresponding points are obtained in Figure 2(d).The chessboard overlay map for dataset 1 is shown in Figure 2(e).In order to qualitatively judge registration accuracy, the root mean squared error (RMSE) is used, and twenty uniform distribution checkpoints are selected manually.The value of RMSE for dataset1 using our method is 1.334 pixels.
The resulting normalized distance will be 0 if the i th line segment in optical image and j th line segment in SAR image is matching.If the distance is greater than 1, it is assumed that the segments do not match.