Precise Relative Geometric Correction for Multi-Sensor Satellite Images

Rapid progress in satellite technology has led to a noticeable surge in availability of Earth observation satellite images, which are being collected daily from satellites deployed worldwide. However, even with advanced satellite positioning equipment, there are still diverse level of remaining positional errors. This is a hindrance to satellite image utilization. Therefore, positional errors between satellite images must be corrected before utilization. Relative geometric correction of satellite images is a technique that adjusts geometric displacements based on their relative positional relationships in image or object space. In this study, we propose homography-based bundle adjustment for relative geometric correction of multi-sensor satellite images. Our method aims to estimate optimal ground plane on which images are projected and quickly generate result image. For experiments, orthorectified satellite images with various resolutions and georeferencing information were employed as input data. The experiment results showed that the average error, which was initially 4.96 pixels before relative geometric correction, was decreased to 1.73 pixels after applying the proposed method.


INTRODUCTION
As a result of increasing number of Earth observation satellite being launched, every part of the Earth is now daily covered by satellite imagery.These satellite images have been transformed into vast volume of big data that hold immense potential for utilization.The conversion of these satellite images into extensive datasets offers numerous opportunities for various applications, ranging from urban planning to environmental monitoring.Moreover, advanced processing techniques, such as time series analysis, enable the extraction of valuable insights for continuous Earth monitoring and change detection.To ensure accurate analysis of satellite imagery, precision in transforming between the image and ground coordinates is essential.To enhance the accuracy of the transformation, satellite image is commonly subjected to geometric correction and orthorectification using ground control points (GCPs) and digital elevation model (DEM) before being provided to users (Kim and Im, 2003;Xiong and Zhang, 2009;Son et al., 2021).However, even after applying geometric and orthorectification processes, relative positional errors between satellite images remain, which can be challenging for further applications.
To mitigate relative positional errors, image processing techniques such as image registration or image stitching are utilized with applying transformation models to the unaligned image (Zitova and Flusser, 2003;Reji and Vidya, 2012;Misra et al, 2022.).However, image processing techniques are difficult to apply when processing multi-sensor satellite images due to differences of ground sample distances (GSDs) and image size.In addition, in case of processing multiple satellite images, there are difficulties such as selecting a reference image with consider of various image combinations and error accumulations in sequential processing.Therefore, to address these difficulties, a bundle adjustment technique is used to correct relative positional errors.Bundle adjustment is a technique used to simultaneously adjust all parameters to obtain accurate transformation between image coordinates and ground coordinates from multiple satellites images (Grodecki and Dial, 2003).Bundle adjustment can estimate precise rational function model (RFM), which is mathematical model composed of rational polynomials coefficients (RPCs), with GCPs and tie points.However, securing high-precision GCPs presents practical challenges, ranging from time constrains and cost to policy limitations.Therefore, essential strategy in this pursuit involves embracing relative geometric correction methods that operate independent of GCPs.By focusing on estimating relative positions between satellite images, researchers and practitioners can reduce the burden associated with GCP acquisition and real-time geometric correction.
In this study, our objective is to construct a bundle adjustment technique for multi-sensor satellite images, addressing relative geometric errors with only tie points.Our approach involves reconstructing the adjustment problem using homography transformation rather than RFM.During the experiment, satellite images with various resolution were employed as input data.These data were designed to encompass diverse regions and conditions to validate the proposed method.To evaluate the effectiveness of our approach, we calculated modelling errors and reprojection errors using adjusted tie points and independently measured check points.Additionally, we produced result images by applying the estimated homography model and confirmed the relative geometric correction.

METHODOLOGY
To apply relative geometric correction, multi-sensor satellite images and georeferencing information are required as input data.Our method consists of four main steps.Figure 1 provides the processing flow of our proposed method.The first step is tie point extraction, which utilizes the georeferencing information of image to improve accuracy and reduce the time required for feature points matching.The second step involves constructing matrices from the extracted tie points to form observation equations.In this step, image coordinates of the tie points are applied as coefficients in the equation to form matrices for bundle adjustment.Next, a homography model is estimated for image transformation, and recursive estimation method is applied to update observation weights, ensuring a more rigorous approach.The final step is result image generation by applying the estimated homography model.This step is essential to generate images that have been corrected for practical applications such as change detection.

Tie Point Extraction
In the process of relative geometric correction, tie points are considered important basic units.In our method, scale-invariant feature transform (SIFT) algorithm, known as scale and rotation invariance, is used to automatically extract distinctive feature points from satellite images.The SIFT algorithm is noteworthy in that it can obtain sub pixel level image coordinates, providing finer granularity in representing feature points.However, since the descriptor of SIFT-based feature points is 128-dimensional vector structure, it takes a long time to match a large number of feature points.To improve processing time of tie point extraction, we utilize georeferencing information held by satellite images.We calculate overlapping area between satellite images to minimize the area performed for tie point extraction.The overlapping area can be easily calculated using the conversion formula between image and ground shown in Equation (1).We calculate image coordinates for overlapping area, then use these image coordinates to clip the respective region.Additionally, to enhance the accuracy of initial tie point matching, we divide the computed overlapping area into nine regions and index the extracted feature point according to the respective region they belong to.As shown in Figure 2, the reference image is divided into exactly nine regions, and the target image is divided by apply a small margin.The reference region and corresponding target region with margins are matched.We assume that features between the corresponding indexes should be matched since the satellite images used have already been geometrically calibrated to some level of accuracy.
After initial tie point matching, we apply the random sample consensus (RANSAC) algorithm to filter tie points (Fishchler and Boles, 1981).The RANSAC algorithm is an iterative robust estimation method used to remove outliers in the tie point matching process.Combining the tie point matching process with the RANSAC algorithm improves the accuracy and robustness of extracted tie points.This combination allows us to extract high-quality tie points, which serve as a reliable reference to the next step of relative geometric correction.

Bundle Adjustment Matrix Construction
To perform rigorous bundle adjustment, observation equations representing transformation between satellite image coordinates and ground coordinates must be defined.Since satellite images contain georeferencing information, the transformation between the image and ground in the form of an affine transformation can be expressed as a general perspective form of homography transformation (Equation ( 2)).
where Col and Row are measured image coordinates Col' and Row' are virtual projected ground coordinates For estimating the homography transformation parameters, it is crucial to select appropriate a reference image for initializing homography matrix.reference image is the basis for initializing the homography parameters.We select a reference image as the image with the highest count of extracted tie points.We focus that parameter adjustment occurs within a virtual image space.Therefore, the homography parameters for the reference image are initialized with an identity matrix, while those for the remaining images are initialized by estimating the homography transformation model using image coordinates of the extracted tie points as shown in Figure 3.
By combining the concept of adjustment within virtual image space with established observation equations, our bundle adjustment can be defined by the following Equation ( 3) and ( 4).The extracted tie points are utilized as virtual image coordinates to generate virtual reference frames.For the image coordinates of each tie point i, two observation equations can be established using k virtual image coordinates in j image for image coordinates. (3) As shown in Equation ( 3) and ( 4), there are two observation equations for each tie point extracted from satellite images.In our bundle adjustment method, the adjustment parameters are image-specific homography model and virtual ground coordinates of tie points.Since the model is a nonlinear system, linearization is required.By applying first-order Taylor series expansion in Equation ( 3) and ( 4), it can be expressed as a linearized model using initial value and their increments () as shown in Equation ( 5).
0 +   +  = 0 (5) where To estimate the increment value, we reconstructed Equation ( 5) using a least square approach with Gauss-Markov model as shown in Equation ( 6).

Rigorous Bundle Adjustment
Bundle adjustment is the process of estimating the increments of the parameters to be adjusted, which typically involves multiple iterations and updates to compute final parameter estimates.The results of these estimates are influenced by the weight of the observations.When using absolute ground coordinates such as GCPs, the weight matrix can be given an accuracy implied by the control points.However, in a relative correction process that utilizes only tie points, the weights of tie point observation are ambiguous, and using same weights at each iteration may lead to unstable estimation.In our method, we recalculate the weight matrix at each iteration and apply it to perform a more rigorous bundle adjustment.To recalculate the weight matrix, the adjustment result is evaluated at each iteration of the bundle adjustment.The adjustment result can be estimated by covariance matrix of residual.The covariance matrix of residual can be calculated using Equation ( 7) (Mikhail andAckermann, 1976, McGlone et al, 2013).  is the covariance matrix before adjustment, and the estimated new weight matrix ( ̂ −1 ) for next iteration can be calculated as shown in Equation ( 8).
These calculations provide valuable information for evaluating the accuracy and reliability of the estimated parameters and enable iterative improvements in subsequent rigorous bundle adjustment iterations.

Result Image Generation
To produce the result image demonstrating the relative geometric correction of satellite images, image transformation is accomplished by utilizing an image resampling method that relies on the relationship between image coordinates and ground coordinates.While the primary aim of relative geometric correction is to rectify existing pixel errors between satellite images, it is equally vital to minimize distortion of the original georeferencing information.Our bundle adjustment is rooted in a homography transformation assuming a 2D plane transformation.This means that the adjusted homography parameters operate within a virtual image space, projecting onto a virtual reference frame.Consequently, the adjusted homography parameters yield results at the image level, and immediate estimation of georeferencing information to the ground for each image can not occur.
The generation of the resulting images involves resampling the input images using the inverse of the homography models.This technique pre-calculates the size and position of the resulting images and extracts pixel values from the original image for each corresponding pixel.The pixel position can be calculated using the inverse of the homography model.Finally, the resulting image retains its original georeferencing information.This information describes the transformation relationship between the image and the ground in the form of an affine transformation.
When applied to the resulting image, we consider the coordinates of the top-left corner, the ground sample distance (GSD) in the row and column directions, as well as any rotation.
Other images are also projected onto the reference frame using homography parameters to create the resulting image.However, during the final step of coordinate assignment, the georeferencing information from the reference image is used to prevent pixel boundary mismatches with the reference image.

TEST RESULT
To apply our proposed method, we selected different satellite images with overlapping area.Relative geometric correction experiments were conducted using a total of four datasets as shown in Table 1.All datasets were designed to include overlapping areas to facilitate tie point extraction for relative geometric correction.Each dataset consists of two or more images, and the images were selected based on the region.The images were carefully selected to cover various geographical regions and environments.All satellite images used in the experiments were processed through geometric correction and orthorectification.They contained georeferencing information.For datasets consisting of three images, Dataset C and D, the initial relative errors were calculated based on the re-projection errors between image excluding the reference images.It is notable that relative positional errors existed even between the satellite images processed through geometric and orthorectification corrections.
Figure 5 visually represents the satellite images.On the right side of Figure 5, an enlarged satellite image is displayed.As depicted in Figure 1, All datasets had overlapping regions for tie point extraction.In the enlarged images, the misalignments of same objects were considerably significant.These are interpreted as relative positioning errors.We conducted rigorous bundle adjustment using only tie points for the four datasets.To validate the performance of our proposed method, we utilized check points which were manually extracted.The extracted check points were utilized to calculate reprojection errors using estimated homography parameters.To compute these errors, we projected image coordinates into a virtual image space using the homography model and then reprojected the projected ground coordinates into other images.In Dataset A and B, the results of our proposed method exhibited relatively lower model errors compared to the 2D homography transformation method.The check point errors of our proposed method were slightly higher compared to the 2D homography transformation; however, this difference could be considered negligible.
The effects of our method were more pronounced in the experiments of Dataset C and D. In this case, the relative check point accuracy with a fixed reference image was accurately estimated.However, the reprojection errors between other images excluding the reference image showed significant differences.In contrast, the check point errors in our proposed method were evenly distributed across all images.This advantage was attributed to our proposed method, which corrected relative positional errors by considering all images comprehensively.
Figure 6 displays the original satellite images before performing relative geometric correction and the result images generated using our proposed method.The original images in Figure 6 exhibit relative positional errors, which are also depicted in Table 1 as relative errors.To validate the effect of positional error correction, we visually compared the identified objects in the result images by enlarged images in each dataset.The proposed method successfully mitigated relative positional errors.These improvements are visually evident in Figure 6, where significant enhancements in object alignment and notable reduction in positional discrepancies between images can be observed.Figure 6.Enlarged result images with our proposed method.

CONCLUSIONS
In this paper, we proposed a new method designed to solve the problem of relative geometric correction for multi-sensor satellite images.Our proposed relative geometric correction method did not require GCP and utilized tie points only.We applied a bundle adjustment approach based on homography transformation model.We also applied a region-based matching method that utilized the georeferencing information inherent in satellite images to improve performance and reduce processing time.Our homography based bundle adjustment overcomes the limitations of image-to-image registration method and improves performance.By recalculating a reasonable weight matrix for each bundle adjustment iteration, we were able to apply a more rigorous bundle adjustment method.
When considering a RapidEye image as the reference for the dataset, an average relative positional error of 1.73 pixels was observed.With visual demonstration, improved alignment and enhanced reliability and accuracy were confirmed.In conclusion, bundle adjustment using a recursive approach could be a robust and efficient solution for relative geometric correction across multi-sensor satellite images.By effectively correcting positional error using tie points and homography models, we improved the overall accuracy and alignment between images.We hope that this research can contribute to the advancement of satellite image processing, offering more accurate and reliable opportunities for Earth observation data.

Figure 1 .
Figure 1.Flow chart of proposed relative geometric correction.

Figure
Figure 4. Result image generation method

Table 1 .
Properties of used datasets.

Table 2
shows the experiment results.All errors shown in the experiment results were reprojection errors.The errors were categorized into two groups: model errors and check point errors, as indicated in Table 2. Model errors measure how well the extracted tie points follows estimated homography model based on those tie points.Check point errors represents the reprojection errors.Since check point errors do not influence the model estimation, the accuracy of check points serves as the most reliable indicator.Figure 5. Location and relative positional errors of used datasets.

Table 2 .
Result of relative geometric correction.