Triplet Epipolar Images for Spaceborne Three Linear Cameras

: Epipolar images can reduce conjugate points searching from a two-dimensional to a one-dimensional space, which significantly improves the robustness and efficiency of dense matching. However, two images matched the lack of redundancy and suffered from occlusions. The third image could introduce two more stereos and improve the accuracy and completeness of the three-dimension reconstruction, which were adopted by spaceborne three linear cameras (TLCs), the ALOS PRSIM and ZY-3. In this study, we propose a new triplet epipolar images generation method for spaceborne TLCs. Triplet epipolar images were defined as any two of the three epipolar image pairs. The triplet epipolar geometry requires the imaging rays of any conjugate point to be coplanar. For high-resolution satellite images, parallel projection can be used to approximate the imaging geometry of satellite images. Therefore, a coplanar principal optical axis is a fundamental requirement for spaceborne TLCs. We propose a general workflow to generate triplet epipolar images (TEIs), which includes free-net bundle adjustment of TLCs, building a triplet epipolar geometry, correcting the y-parallaxes, and generating a rational function model (RFM) for TEIs. The ZY3-02 satellite images were used to validate the proposed method. The root means square error (RMSEs) of the free-net bundle adjustment in the image space was 0.185 pixels, which proved the fine intrinsic accuracy. After compensation, the RMSEs of the y parallaxes of the three epipolar image pairs were 0.295, 0.310, and 0.370 pixels. Owing to the simple geometry of the TEIs, the RMSEs of the RFM replacements were within 0.001 pixels.


Introduction
Epipolar images create all conjugate points of two images in the same row, which are generated after relative orientation or bundle adjustment.Hence, epipolar images can reduce the twodimensional correlation to one dimension along the epipolar line for three-dimensional (3D) reconstruction with two images, which was adopted in photogrammetry 50 decades ago (Helava and Chapelle, 1972).Correlation methods have evolved from normalised cross-correlation (NCC) to semiglobal matching (SGM) (Hirschmuller, 2008) and machine learning (Poggi et al., 2022).However, epipolar images play a vital role in 3D reconstruction.Moreover, the reliability of the dense matching between the two images is weak because of the lack of redundant observations.Triplet stereo images from three linear cameras (TLCs) and agile satellites (Loghin et al., 2020) were used for 3D reconstruction (Raggam, 2006).
Epipolar linear push-broom cameras have been studied since SPOT-1 (Otto, 1988;Zhang and Zhou, 1989).Epipolar geometry depends on the sensor models of high-resolution satellite images (HRSIs).Gupta and Hartley (1997) and Habib et al. (2005) studied the epipolar geometry of linear push-broom cameras under a line-central projection with constant velocity and attitude.In such cases, the epipolar lines are hyperbolic.Ono (1999) proved that the epipolar line is straight when the affine model is used as a sensor model for HRSIs.Kim (2000) derived hyperbola-like curves using rigorous sensor models.The absence of physical meaning makes epipolar geometry of rational function model (RFM) complex.Ye et al. (2009) derived a hyperbolic epipolar geometry with zero elevation and calculated the coefficients using RPCs.In general, the epipolar curve of a push-broom camera is hyperbolic, whereas a straight line can be used in some cases.It is difficult to derive the epipolar curves for HRSIs using geometric sensor models.
Several quasi-epipolar image-resampling methods have been proposed for HRSIs.These methods can be categorised into two types: object space and image space.In the case of an object, the epipolar direction in the object space is determined using a geometric sensor model, and both images are rectified with a reference elevation along the epipolar direction (Wang et al., 2011).This method is based on the affine model, in which the epipolar lines are straight and parallel.However, HRSIs are line-centred projections.Hence, a vertical parallax exists for the entire image.In the image space, the epipolar curves are determined by fitting the conjugate or projected points.Zhang and Zhou (1989) used the left scanning line as the epipolar line, as suggested by Dowman (Dowman, 1984).Moreover, they estimated the polynomials of the right epipolar line using conjugate points.Kim (2000) derived a local epipolar line using the projected points.Pan et al. (2011) extended the endpoints to obtain global epipolar lines and reconstructed RFMs for epipolar images.A similar approach has been developed independently (Oh et al., 2010).
The geometric constraints of the three images were recognised by Mikhail (Mikhail, 1962).Yang et al. (2023) proposed a minimum parameter solution for the trifocal tensors of HRSIs in an affine geometry.These studies demonstrate the general constraints for the three images.Some photogrammetrists have noticed that the epipolar image pair can be extended to the third image of TLCs (Bhalerao et al., 2013;Yue and Tang, 2022).This is because the epipolar planes of TLC images are coplanar (Pan, 2017).However, the triplet epipolar geometry remains unknown.
In this study, we present a general case of triplet epipolar imaging.Subsequently, workflows for the generation of triplet epipolar images (TEIs) are proposed, including the relative orientation, building triplet epipolar geometry, parallax compensation for perspective approximation, and RFM generation.The TLC images were used to validate the proposed method.

Triplet Epipolar Images
Point P was observed by three cameras, C1, C2, and C3, and there were three image points, p1, p2, and p3.According to the epipolar geometry, two of the three cameras can generate an epipolar plane that intersects two image planes with epipolar line pairs.If the three epipolar planes are coplanar, there are triplet epipolar lines, any two of which are epipolar line pairs.In general, there is a single set of triplet epipolar lines when object P is in the trifocal plane, which is determined by the three perspective centres.The same rows of the triplet epipolar images are triplet epipolar lines.In this case, the two conjugate epipolar lines induced by any image point p are the conjugate epipolar lines.Given all epipolar lines passing epipole, any points in one image should be collinear with the two epipoles corresponding to other two images.Therefore, the collinearity of the three projection centres is a necessary and sufficient condition.
As a special case of weak-perspective geometry, the triplet epipolar geometry of a parallel projection requires all rays of the same object to be coplanar.Because of its very narrow field of view, an affine model has been developed for high-resolution satellite images (Fraser and Yamakawa, 2004;Okamoto et al., 1999).As proven in previous studies, the affine model is insufficient for the sensor orientation of HRSIs (Jacobsen, 2007), and a central perspective-to-affine conversion is required (Zhang and Zhang, 2002).A parallel projection can be used for the subimages of the HRSIs.However, the TLCs from the same orbit cannot guarantee the coplanarity of the three views, which would be three lines on an elliptical cone if the roll angles are not zero.

Triplet Epipolar Images Generation
The fundamental requirement of the triplet epipolar geometry is that all conjugate rays are coplanar.Therefore, the relative orientation should be the first step in making all corresponding points intersect with each other.The triplet epipolar geometry was examined.With parallel projections, the epipolar curves were straight lines.Subsequently, the conjugate points were used to correct the y-parallax, which was introduced using the parallel projection approximation.The RFMs were then regenerated for the TEIs.
(1) Relative Orientation of TLCs Currently, the rational function model is used as a geometric sensor model for high-resolution satellite images and is defined as  Relative orientation generally estimates the minimised degrees of freedom, which depend on the epipolar geometry of the HRSIs.For TLC images, the relative orientation adds constraints to the sample directions, as illustrated by (Pan, 2017).In general, free-net bundle adjustment can also cause all the corresponding rays to intersect with each other.This is an over-parameterisation of the relative orientation.The affine compensation model in image space is widely used for HRSIs.(2) Building the Triplet Epipolar Geometry.The triplet epipolar geometry was estimated through a parallel projection in the Cartesian coordinate system.First, the overlapping areas between the three images were calculated.
The central point P was then determined by minimising height hmin.Using this point, the corresponding image points, p1, p2, and p3, were calculated.When the maximum height hmax is given, the three object points P1, P 2, and P3 can be calculated using the RFM, as illustrated in Figure 2.
The basic requirement for the triplet epipolar geometry is that the three vectors PP1, PP2, and PP3 are coplanar.Therefore, the triplet epipolar plane was estimated using two rays, PP1 and PP3.The projection errors for PP2 were calculated.A triplet epipolar plane exists if the projection error is within subpixels.The height range can be obtained using open-source DEM or RFM.
Using the triplet epipolar plane PP1P3, a local coordinate system P-XEYEZE can be built.The XE axis was defined as P1P3, and the ZE axis was perpendicular to the X-axis in the triplet epipolar plane.YE is defined using the right-hand rule.In this coordinate system, the y-parallaxes are in the YE direction.With this definition, a similar transformation between the Cartesian coordinates and triplet epipolar coordinates can be performed as: where R is the rotation matrix from triplet epipolar coordinates to Cartesian coordinates and (XC0, YC0, ZC0) is the origin of P-XEYEZE.
Therefore, a grid PE in the XEYE plane was established to rectify the original images.The grid size was calculated using the finest spatial resolution.To correct the tilt of the P-XEYEZE coordinates, the image point PE was projected onto the reference height.The projected point was then transformed into geodetic coordinates and reprojected onto the original image space (r,c).
The algorithm is as follows.
• Transform the triplet epipolar image point (re, ce) to P-XEYEZE coordinates (XE, YE, 0) with the grid size and origin.

•
Transform the grid point (XE, YE, 0) to Cartesian coordinates (XC, YC, ZC) with the similarity transform (3) as the origin of a ray.

•
Extraction of the Z-axis of P-XEYEZE in the ray direction • Making the ray intersect with the eclipse earth with reference height, the intersection corresponds to the ground point (X, Y, Z).

•
Calculate the image coordinates (r,c) using the RFM after bundle adjustment.
(3) Correction of the y-parallax The algorithm above operates when the rays are parallel to the affine model.Owing to the central perspective in the sample direction, a y-parallax was introduced.
The conjugate points of the TCLs were inversely transformed into TEIs using the above algorithm.Subsequently, the corresponding points in the TEIs were used to estimate the affine compensation model in the y-direction as follows: where (re1,ce1) and (re3,ce3) are the coordinates in the epipolar images; 21 r  and 23 r  are the y-parallaxes, which were calculated using re1, re2, and re3; and f1, …, f6 are the affine parameters.These parameters can be calculated using a leastsquares estimation.Combining the above y-parallax correction with algorithm, the triplet epipolar images can be generated.
(4) Rebuilding the RFM for TEIs The RFM of triplet epipolar images can be reconstructed in a terrain-independent manner (Tao and Hu, 2001).To build the grids of the control points, several elevation layers were used to calculate the ground points (X,Y and Z), as mentioned in the algorithm.Checking grids were used to evaluate replacement accuracy.

Datasets and Experiments
The ZY3-02 TLCs are forward (FWD), nadir (NAD), and backward (BWD) cameras.Furthermore, the ground sample distance (GSD) of NAD was 2.1 m, while the GSD of FWD and BWD was 2.3 m in sample and 2.5 m in line.The dataset covering Changsha obtained on 13 February 2017 was used for the experiments.The roll angle of platform was approximately −4.05 , and further details are presented in The zero y-parallaxes are the gold standard for epipolar imaging.However, the y-parallaxes were not superior in terms of the relative orientation accuracy.Therefore, a free-net bundle adjustment with highly accurate tie points was first conducted.Subsequently, TEIs were generated with and without the yparallax correction.The replacement accuracy of the RFM EPIs was verified.Finally, the y-parallaxes were validated using highly accurate conjugate points.

Triplet Epipolar Images of ZY3-02
Our previous work showed that sparse points were insufficient to illustrate intrinsic geometric properties such as distortions and attitude jitter (Pan et al., 2021).Therefore, very dense tie points (TPs) were extracted for the TLCs, feature tracking was performed, and only the tie points in three images were retained.Finally, Least Square Matching (LSM) was used to refine the TPs, and there were 424, 182 tie points.Owing to their similar texture, few TPs existed in the mountainous and downtown areas.The affine compensation model was used for the free-net bundle adjustment.The overall root mean square errors (RMSEs) in the image space were 0.164, 0.086, and 0.185 pixels in the sample, line, and image planes, respectively, proving the excellent accuracy of the ZY3-02 TLCs.The NAD residuals are shown in Figure 3.In addition, high-frequency attitude errors in the roll angle were observed, which may be introduced by the linear interpolation of attitude records and attitude noises (Pan et al., 2016).
The three image rays were almost coplanar.After building the triplet epipolar plane with ray PP1 and PP3, the reprojection errors of PP2 was 0.05 m, approximately 0.02 pixels, when the hmax was 681.7 m and hmin was −-2.4 m.The small projection errors might be introduced due to the roll angle.
The TEIs were generated with and without y-parallax compensation.Image matching was used to check the y parallaxes of the TEIs.Compared with the original image, the epipolar image rotated 90in the counterclockwise direction because of the major relief displacement in the track direction for the TLCs.In this case, the row coordinates of the conjugate points must be identical.Therefore, the y-parallaxes were calculated using the differences in row coordinates for the three image pairs BWD-NAD, NAD-FWD, and FWD-BWD.
Significant y-parallaxes were observed before compensation, as shown in Figure 4. To calculate the parallel axes, 330, 347 conjugate points were extracted.The RMSEs of BWD-NAD, NAD-FWD, and FWD-BWD were 0.481, 0.422, and 0.735, respectively.The maximum parallaxes were greater than two pixels in FWD-BWD.The parallaxes in each row were close and varied with the row number.After compensation, the y-parallaxes are significantly reduced, as illustrated in Figure 5.The RMSEs of 327 and 646 conjugate points were 0.295, 0.310, and 0.370 pixels, respectively.Compared to Figure 3, high-frequency attitude errors cause fringes in the column.However, the parallel axes were larger than the RMSEs of the orientations.As illustrated in Figure 5 Because of the simple geometry of the TEIs, the replacement residuals of the RFM were small.As shown in Table 2, the RMSEs of RFM replacement were within 3.0E-4 pixels in the sample, line, and image planes for all three images.Therefore, TEIs with RFMs can be directly used for 3D reconstruction.

Conclusions
Epipolar geometry is a fundamental constraint in stereo images.
In this study, we demonstrated that TEIs exist if any of the conjugate rays are coplanar.Using parallel projection, the triplet epipolar geometry of TLCs was studied and a workflow was proposed to generate TEIs for TCLs.The workflows consisted of free-net bundle adjustment of TLCs, building the triplet epipolar geometry of TLCs, correcting the y-parallax, and generating RFM for TEIs.The proposed method was validated using ZY3-02 TLCs.The RMSEs of the free net bundle adjustment was 0.185 pixels, indicating an excellent intrinsic accuracy.In addition, y-parallax compensation, which can reduce the RMSEs of the y-parallaxes from 0.735 to 0.370 pixels, is required.The RMSEs of RFM replacement were within 0.001 pixels.Dense matching using TEIs will be studied in the future.

Figure 1 .
Figure 1.Triplet epipolar geometry of three perspective images.
, C) are the normalised image coordinates, (P, L, H) are the normalised object coordinates, and ,the coefficients of the cubic polynomials.
ec0 and er0 are the shift compensation in the sample and line directions, respectively.Moreover, er1 and ec1 are the drift compensation parameters because the line coordinate r is timerelated.

Figure 2 .
Figure 2. Geometry of the triplet epipolar images.

Figure 3 .
Figure 3. NAD residuals in the image space after bundle adjustment.

Table 1 .
The elevation ranged from 0 to 681 m.

Table 1 .
The dataset of the experiments.