Triangulated Irregular Network based Seamline Determination for Fast Image Stitching of Multiple UAV Images

To maintain the efficiency of UAV (unmanned aerial vehicle) remote sensing, rapid image stitching is essential to make multiple UAV images into a seamless mosaic image. Relief displacement introduces variations in object appearance for each image, causing mismatch errors at mosaic seamlines. Traditional approaches involve orthorectifying images using DSMs (digital surface models). While these approaches allow for accurate image stitching, they do not cope with the advantages of UAVs due to their time consumption. In contrast, fast image stitching techniques that do not use orthorectification are well suited for UAV image processing. Related researches have attempted to optimize seamlines to eliminate the errors caused by relief displacement without the use of DSMs. We propose to utilize a TIN (triangular irregular network) of tiepoints to effectively eliminate errors caused by relief displacement while maintaining the fast speed of image stitching. In this study, a TIN is constructed based image tiepoints whose ground coordinates have been obtained through bundle adjustment. The edges of the TIN are used to generate seamlines for image stitching, and the facets of the TIN are used to select minimal images for image stitching and to optimize seamlines. Image stitching results of our proposed method had small error of 1-2 pixels and the processing time of less than 10 minutes for 97 UAV images. This study showed that the proposed method could stitch multiple images while maintaining stable quality using only geometric clues of a TIN of tiepoints.


Introduction
UAVs (unmanned aerial vehicles) offer ease of operation and cost-effectiveness, making them a popular choice for various applications.Flying at low altitudes, UAVs swiftly capture highresolution images.This leads to their widespread use in fields like agricultural monitoring and construction site management (Tsouros et al., 2019;Chan et al., 2015).Despite their advantages, the limited field of view of UAVs necessitates the acquisition of numerous images to cover an entire area of interest.To enhance the efficiency of UAV remote sensing, rapid stitching of multiple UAV images into a seamless mosaic image is essential (Kim et al., 2020).However, relief displacement introduces variations in object appearance across images, causing mismatch errors at the junction regions on the mosaic image (Yuan et al., 2020).Traditional approaches involve orthorectifying images using DSMs (digital surface models).However, creating a super-resolution DSM at the level of ground sampling distance of UAV images is very time-consuming and expensive (Zhang et al., 2023).As a result, orthorectification and image stitching, which entails resampling a large number of pixels in a super-resolution image, is an extremely time-intensive process (Wu et al., 2022).In the pursuit of fast mosaic image generation, our research focuses on eliminating the orthorectification process through optimizing seamline determination by a TIN of tiepoints.Existing studies employ seamline determination methods based on image pattern analysis, such as graph cut or optical flow (Li et al., 2018).Recently, there has been a surge in the exploration of deep learning-based methods for this purpose (Li et al., 2017;Dai et al., 2021).However, it was difficult to remove all the error caused by relief displacement using non-geometric clues such as brightness.Building upon our previous work, which introduced a rapid image stitching technique using a TIN (triangulated irregular network) of tiepoints (Yoon and Kim, 2023), this study proposes a TIN-based seamline determination and optimization method for fast and robust image stitching.

Proposed Method
Figure 1 is a flowchart of our proposed method.In the step of TIN construction with tiepoints, image tiepoints are generated to construct a TIN.Tiepoints are extracted from pairwise images and their outliers are removed by triplet verification.Then, bundle adjustment corrects EOPs (exterior orientation parameters) of the images and determines ground coordinates of tiepoints.Next, a TIN is generated with the ground coordinates of inlier tiepoints as its node.Utilizing the three nodes of TIN facets, facets are assigned to each image and the slopes of facets are calculated.In the step of seamline generation, initial seamlines for image stitching are determined according to TIN nodes and edges.The image area corresponding to an assigned TIN facet becomes the coverage of the mosaic image for each image.In addition, the minimal images required for image stitching are selected according to overlap ratios of TIN facets between images.The outlines of TIN facets on the selected images are defined as the initial seamlines.In this study, we assume that the objects with relief displacement appear in the regions of TIN facets with high slope.In the step of seamline optimization, slope angle thresholding is applied to find high-slope TIN facet regions.Then, seamlines that pass through the high-slope region are removed from the initial seamlines.This process is applied to get rid of mismatch errors in image stitching.Furthermore, images are selected that can cover each entire area of high-slope region.Finally, in the step of TIN-based image stitching, images are mosaicked by TIN facets.The details are described in the following sub sections.

TIN Construction with Tiepoints
In this study, tiepoints are first extracted for bundle adjustment and TIN construction.The tiepoint extraction algorithm consists of detector, descriptor, and matcher steps.In the detector step, keypoints are extracted from the images, and their features such as gradients are calculated in the descriptor step.Then, the keypoints are paired by comparing the features between images and form tiepoints.Tiepoint extraction algorithm varies depending on the algorithm combination of its three steps.Following our previous work, we use the SURF (speeded up robust features) algorithm in this study.Next, EOPs of UAV images are corrected and the ground coordinates of tiepoints are determined through bundle adjustment.To improve the accuracy of EOP correction and ground coordinates of tiepoints, triplet tiepoints are used for bundle adjustment, as shown in Figure 2. First, the ground coordinates of a tiepoint are calculated using two images based the collinearity condition.Then, the ground coordinates are projected to the remaining image.The difference between the projected and original image points is defined as the reprojection error.Triplet tiepoints are determined by the inliers of the reprojection error verification.Triplet tiepoints through bundle adjustment have image coordinates and ground coordinates.They are used as TIN nodes in this study as shown in Figure 3.To build a TIN efficiently, triplet tiepoints are sampled at regular intervals.The image information that all three nodes have in common is assigned to a facet.Their ground information also determines the slope of the facet.The equation for the slope calculation using the three nodes of a TIN facet is given by Eq. ( 1) and ( 2).

Seamline Optimization based Slope of TIN Facet
In this study, we assume that the objects with relief displacement appear in the regions of TIN facets with high slope.High-slope TIN facets are detected by slope angle thresholding.The seamlines on the high slope region are excluded from the initial seamlines.Furthermore, images for mosaicking are selected as images that cover the entire high slope region, as shown in Figure 5.In the top image of Figure 5, the number represents image ID, and the red area is an example of high-slope TIN facets.The image 2 in red is the image that can cover the red area.The bottom image represents the optimized seamline of the proposed method.
Figure 5. Seamline optimization based slope of TIN facet.

TIN-based Image Stitching
The image stitching in the proposed method is based on the affine transformation model as shown in Eq. ( 5).Affine transformation is a model to reproduce parallel translation, rotation, scaling and shearing (Zheng et al., 2019), and its coefficients can be estimated from three or more tiepoints.As shown in Fig. 6, UAV images are mosaicked in the form of triangles of facets in the nonhigh-slope region and polygons in the high-slope region.
[ To verify the accuracy of the proposed method, the triplet tiepoints that are not sampled as TIN nodes are used as checkpoints.In this study, the ground points of the triplet tiepoints are assumed to be the truth.In Figure 7, the green xmark indicates the point where the ground truth of the checkpoint is shifted to the mosaic coordinate system.The red x-mark indicates the point where the image point of the checkpoint is projected to the mosaic coordinate system using the affine transformation coefficients.The distance between these two points is defined as the error of image stitching.

Results of TIN Construction
Table 1 shows the results of TIN construction.In Dataset 1, 175,615 tiepoints were extracted, and 53,062 triplet tiepoints were determined through reprojection error verification.Finally, 1,009 triplet tiepoints were used to build TINs, and 18,936 TIN were constructed.The TIN construction for Dataset 1 took 9 minutes 29 seconds.In Dataset 2, 112,451 tiepoints were extracted and 65,555 triplet tiepoints were determined.4,614 triplet tiepoints were used for TIN construction, fewer than the nodes in Dataset 1.There were 8,465 TIN facets built, and the TIN construction for Dataset 2 took 5 minutes 5 seconds.Figures 9 and 10 show the slope maps of the TIN facets of Dataset 1 and Dataset 2. The slope angle ranged from 0 to 90 degrees, and in the figures, the larger the slope, the brighter it is.The slope of the TIN facet was higher around objects such as buildings and trees.The TIN slopes were mostly high around buildings and very small on flat ground.We concluded that tiepoint-based TINs were a good representation of the terrain.

Results of Seamline Generation
Table 2 shows the processing time using all UAV images for image stitching, and Table 3 shows the processing time using the minimal images.In Dataset 1, the processing time using all 97 images for image stitching was 58.55 seconds.By image selection based on image overlap, 41 images were selected for image stitching and the processing time was reduced to 24.12 seconds.In Dataset 2, the processing time using all images for image stitching was 42.51 seconds, while the processing time using the selected images was reduced to 16.28 seconds.
As shown in the seamline generation results in Figure 11 and 12, we were able to utilize the TIN facets to extract the minimal images needed for image stitching.Processing time for image stitching (seconds) 24.12 16.28 Table 3. Results of image stitching using minimal images.
Figure 11.Result of seamline generation using minimal images for Dataset 1.
Figure 12.Result of seamline generation using minimal images for Dataset 2.

Results of Seamline Optimization
Figures 13 and 14 show the results of extracting high-slope TIN facets.The red areas in the figures represent the high slope regions detected by slope angle thresholding.In both Dataset 1 and 2, the high-slope TIN facets were mostly distributed around buildings.In this study, we targeted the yellow areas in Figure 13 and 14 to confirm improvement results.Figure 15 is zoomed-in images of the targeted areas in both datasets.These showed that errors such as mismatches and distortions occurred in the targeted areas.
Figure 16 and 17 show optimized seamlines from the initial seamlines of Figure 11 and 12.In both figures, the red area indicates the target area of this study.The optimized seamlines were formed by avoiding the high-slope areas.As shown in Figure 18, the initial seamlines crossing the target region had been removed in the optimized seamlines.These results were also observed in Dataset 2, as shown in Figure 19.

Image Stitching Results of the Proposed Method
Figure 20 and 21 show the final image stitching results of the proposed method.The yellow regions in the figures indicate the target regions of this study.The mismatches and distortions between the mosaicked images along the optimized seamlines were mostly eliminated.Figure 22 shows that there were no errors in the target regions of both datasets.This result represented a significant improvement from the initial image stitching results in Figure 15.Table 4 shows the accuracy assessment results using triplet tiepoints.The initial image stitching resulted in an error of 31.7093pixels in Dataset 1 and 11.6237 pixels in Dataset 2. After seamline optimization, the final image stitching resulted in an error of 2.1861 pixels on Dataset 1 and an error of 0.9848 pixels on Dataset 2. Through TIN-based seamline optimization, our proposed method could effectively achieve an accuracy of 1-2 pixels.

Conclusions
In this study, we used a TIN of tiepoints for seamline generation and optimization of image stitching.We constructed the nodes of a TIN by tiepoint extraction and bundle adjustment without the use of additional geospatial data.We also used a simple terrain model using the TIN to quickly stitch images.We proposed an intelligent scheme to construct stitching seamlines to eliminate errors caused by relief displacement.The slope maps of TIN facets showed that it described the terrain and the area of relief displacement well.The slope angle thresholding could effectively detect the relief displacement areas.Finally, the mosaicked image along the seamline optimized by the TIN had a very small error of 1-2 pixels.Furthermore, the processing time from TIN construction to image stitching was 9 minutes 53 seconds for Dataset 1 and 5 minutes 22 seconds for Dataset 2. This supports that our proposed method maintained a fastprocessing speed.This study showed that the proposed method could stitch multiple images while maintaining stable quality using only geometric clues of TINs.We expect that our method could contribute to fast and effective UAV image stitching.

Figure 1 .
Figure 1.Flowchart of the proposed method.

Figure 3 .
Figure 3. TIN construction based ground point of tiepoints.

Figure 7 .
Figure 7. Accuracy assessment for TIN-based image stitching

Figure 13 .
Figure 13.Results of high-slope TIN facet extraction for Dataset 1.

Figure 14 .
Figure 14.Results of high-slope TIN facet extraction for Dataset 2.

Table 1 .
Results of TIN construction.

Table 2 .
Results of image stitching using all images.

Table 4 .
Accuracy of image stitching of the proposed method.