Line Matching Algorithm for Aerial Image Combining image and object space similarity constraints

A new straight line matching method for aerial images is proposed in this paper. Compared to previous works, similarity constraints combining radiometric information in image and geometry attributes in object plane are employed in these methods. Firstly, initial candidate lines and the elevation values of lines projection plane are determined by corresponding points in neighborhoods of reference lines. Secondly, project reference line and candidate lines back forward onto the plane, and then similarity measure constraints are enforced to reduce the number of candidates and to determine the finial corresponding lines in a hierarchical way. Thirdly, "one-to-many" and "many-to-one" matching results are transformed into "one-to-one" by merging many lines into the new one, and the errors are eliminated simultaneously. Finally, endpoints of corresponding lines are detected by line expansion process combing with "image-object-image" mapping mode. Experimental results show that the proposed algorithm can be able to obtain reliable line matching results for aerial images.


INTRODUCTION
Reliable extraction of corresponding straight lines in overlapping images can be used for different purpose such as 3D surface reconstruction, image registration, etc (Ok et al., 2012).Comparing to point matching, line matching is more challenging.The main reason is that: The extracted line in different images may be quite different due to the distortions caused by terrain relief and perspective projection; Line in one image may be occluded in other images and they may be broken into more than one piece due to image noise, light, and the deficiency of the line extraction algorithm (Zhang, 2005).The fragments in the line extraction increase the difficult of matching, and make matching results contain the phenomenon of "one-to-zero", "one-to-many", "many-to-one", and " many-to-many " relationships; Only epipolar constraint cannot provide powerful search range constraints in line matching processing.
Existing line matching approaches can be divided into two types: those that match individual line segments, those that match groups of line segments.Most of the approaches to match individual line segments are based on geometric properties similarities of the line segments (such as orientation, length, position, mid-point) and local appearance similarities of segments (Wang, et al., 2009;Juan, 2015).Additional strong constraints are used to reduce search space, such as epipolar constraint (Schmid et al., 2000;Zhang, 2005), corresponding point constraint (Tian et al., 2008;Fan et al., 2010;Fan et al., 2012), triangle constraint (Bulatov et al., 2011;Wu et al., 2012).Matching groups of line segments increases the geometric and radiometric information and reduces the uncertainty in the matching process (Elaksher, 2011).Line groups are established including the two or more lines from the same image, or from two different images.The approach in Kim et al. (2012), intersecting line pairs in 2D images that are coplanar in 3D are served as matching primitive by using Line Intersection Context Feature (LICF).The method carries out LICFs matching based on epipolar constraints and region descriptor of LICFs.In Ok et al. (2012), a pair-wise matching manner are proposed.The method only select pairs that are close to each other and that intersection angle smaller than a threshold at as reference line pairs, and apply seven pair-wise constraints to reduce matching candidate pairs, and the best matching pairs are selected based on a similarity score combining all the pairwise constraints.However, algorithm based on matching groups of line segments is more complex.It is a difficulty to determine the initial line group, conditions function is needed to establish, and matching process is more complicated.
Most individual line matching approaches are based on attributes similarity measure in image space.As the inclination of aerial images is bigger, the consistency of attribute in the same feature will be weaker in different images, because of this, matching result reliability is poorer based on image similarity constraints.In view of this, in this paper space plane as a matching foundation is introduced, it transform the traditional image similarity constraint to the object plane, combine with the image and space similarity constraints instead of traditional single image similarity constraint, ensure the reliability of the line matching.Collins(1995) proposed feature points matching algorithm in the object space through a sweep plane technique.In this paper, similarity to the Collins' algorithm, a space plane is used to constraint line matching.Different to sweeping plane from Zmax to Zmin, we determine the initial Z-position of the plane by using the corresponding points in the neighborhood of the matching straight line.Backproject reference line and candidate lines onto the plane Z = zi, similarity measure constraints are _____________________________ Corresponding author: Jingxue Wang E-Mail: xiaoxue1861@163.comenforced with projected lines in plane to determine corresponding lines.
The rest of this paper is organized as following.Section 2 gives an overview of the approach.The main method is explained in section 3. The experimental results and discussion are presented in section 4. The main conclusions are summarized in section 5, which also indicates some further work.

OVERVIEW OF THE APPROACH OPOSED ALGORITHM
An overview of our proposal for line matching is shown in Figure 1.There are some processing needs to be carried out.

Fig.1 Flowchart of line matching
Line extraction: Straight line detection is performed independently for each one of the input images by Line Segment Detector (LSD) algorithm (Rafael et al., 2010).
A multi-spatial resolution object plane pyramid building: The matching method is based on the space plane, as it is presented in (Collins, 1995).In the case of aerial images, we choose the horizontal plane Z=Zk partitioned into a grid of cells whose axis are chosen aligned with the X-Y axes of the scene ( Franck et al., 2012).The object pyramid plane is built in order to constraint the edge points and subsequent straight line matching.The cell size of the grid of the lower pyramid is the same as the test image resolution.The cell size of the grid of the upper pyramid is corresponding to the lower layer cell for 7 × 7. The upper plane is used to constraint the edge points matching, then the matching result is passed to the lower plane which is used to constraint line matching.
Corresponding points extraction: In order to ensure the existence of the matched points in the neighborhoods of line, the edge points detected by Canny operator are used as matching primitive.Then improving plane-sweep matching approach is adopted to obtain the initial corresponding points based on the upper pyramid plane.

Determine the matching candidates
A set of corresponding points is obtained by a improving planesweep matching method, denoted as {( ,( , ),( , ), ), 1,2, , } , where, i is the corresponding point index , and x y are the corresponding point coordinates of the reference and search images respectively, and Zi is the height value.
donate extracted line on reference image and search image respectively.For a straight line 1 m l on reference image, we will find the corresponding line from line set 2 L .Firstly initial corresponding points are used to constraint candidate lines, which will reduce the search space.Specific principles are shown in figure 2, neighborhood window is constructed which is around 1 m l on the reference image, the weight of window is 2w+1, in this paper w=6.And then the corresponding points in neighborhood window will be determined.Considering the error matching may be consisted in the initial matched points, therefore the corresponding points in the line neighborhood are checked respectively through elevation continuity constraint, enabling to eliminate error matching points.Then in the search image, according to the remaining corresponding points, the candidate line sets m C can be determined, that is to say on the search image, the straight lines whose neighborhood window contain the above corresponding points are the candidates.Finally, candidate set 1 m l on the search image is represented as , , , • The third measure is the distance between the mid-point of overlap segments in candidate line  ( , ) at the same time, the candidate lines is treated as the corresponding lines.The threshold of the parameters in this paper are set as:

Line merging
As the fragments are involved in the process of line extraction, the corresponding line features are extracted inconsistent in different images.Line fracture in the process of the line extraction is fully considered in matching processing, except " one-to-one " relationship in matching results, " one-tomany " , " many-to-one " , and " many-to-many " relationships are also in the matching result.As shown in figure 5, the third corresponding line in initial matching results, the number 3 line on the reference image is corresponded to the number 443 and 784 lines on the search image.Results are needed to integrate using the index corresponding relationship, which can get the one-to-one correspondence relationship between single and group lines on the reference and search images, results are shown in figure5.Next, "many" lines in the result are fitted by using the least square algorithm, the " one-to-one " corresponding lines are obtained, and these lines will be with the new index and endpoint coordinates.Fig. 5 Integrated results of "one-to-many" and "many-to-one"

Determine the endpoint of corresponding straight line
After line matching and integrated, "one-to-one" matching results are obtained.But the length and endpoints of corresponding line are quite different, so the corresponding endpoints are further needed to be determined.In this paper "image-object-image" model was used to determine the endpoints of the corresponding lines.As shown in figure 6, first of all, the corresponding lines were projected to their corresponding object elevation planes on the reference and search images respectively,  In the experiments, we conducted two sets of experiments with UCX aerial images to evaluate the effectiveness of the proposed method.Show in figures 7 to 8 respectively.Image resolution is 0.045m.Figures 9 to 10 shows the line extraction results by LSD for two pairs images (Rafael et al., 2013).The number of initial corresponding points obtained using improving plane-sweep matching approach with two pairs images are 4501 and 5717 respectively.

matching results
As the results of the " one-to-more " and " more-to-one ", the number of the initial corresponding line groups are different on the reference and search images.For example, in the first experiment, 670 corresponding line groups are obtained in the matching results, including 769 lines on the reference image and 870 lines on the search image respectively, "one-to-one" corresponding line is obtained by mergying.Two groups of experimental results are shown in the figure 11 and figure 12.
It can find out that the method adopted in this paper can get good results.But there is still a little error, which mainly happens on the edge of the tall building, vertical face, and the texture fracture.On one hand the reason is that using the corresponding points with the line to determine the plane of the elevation values on the edge of the building, on the other hand the reliability of the similarity of gray-window constraint is weak for the linear neighbourhood with the small overlap, which will be further optimized subsequently.

CONCLUSIONS
Line matching algorithm for aerial image combining image and object space similarity constraints is put forward in this paper.This algorithm has the following characteristics: using edge point matching results instead of traditional feature point matching to constraint further line matching, which is more conducive to guarantee straight neighborhood containing more points, with effective constraint candidate line range; utilizing the combination of image and object matching mode, it can enhance the reliability line matching results; the fracture of straight line extraction is effectively considered in the process of matching on reference and search images, and the matching results of "one-to-many", "many-to-one", "many-to-many" are merged, eventually to get the corresponding line of "one-toone", at the same time the continuity of the edge is also improved; object space plane as an intermediary is used to determine the endpoint of the corresponding line, effectively avoid the endpoint is difficult to determine when the line is parallel the epipolar.
Given the line constraint is strict in this paper, with respect to the images themselves extraction lines, the number of the corresponding lines is few.While vertical line and inclined line are not specific researched in this algorithm, next these two aspects will be further in-depth study.
Line matching: Straight line matching mainly includes four steps: (1) Determine the candidate straight lines; (2) Identify the corresponding line; (3) Line complicated; (3) Determine the endpoint of the corresponding line.Each step will be introduced in detail in section3.

Fig. 2
Fig.2 Neighborhoods of lines on the reference and search images is the gray similarity measure of correlation windows which obtained from the reference image and the search image by correction window warping procedure, respectively.As shown in figure4.

Fig. 3
Fig.3 Vertical distance and overlap distance between lines . Next the points with the farthest distance were selected, ③ and ②, and were projected to image respectively.Taking endpoint ③ for example, this point was came from the projected point of line 1 2 n l on the search image, and then projected to image in contrast, got point 3. On the reference image, it was perpendicular straight lines was the final endpoint.

Fig
Fig.6 Determination of endpoints for corresponding lines 4. EXPERIMENT 4.1 Data Description

Fig. 11 .
Fig. 11.Patch #1 -Red lines are correct matches, blue lines are wrong matches, and green lines are line endpoints are extended to occlusion areas in one of the image, but the result is correct.. Matches: 670(796/870) line sets.Incorrect: 7. Occlusion: 1