CALIBRATION EVALUATION AND CALIBRATION STABILITY MONITORING OF FRINGE PROJECTION BASED 3D SCANNERS

In this work a simple new method for calibration evaluation and calibration stability monitoring of fringe projection based 3D scanners is introduced. This method is based on high precision point correspondence finding of fringe projection sensors using phase values in two perpendicular directions and epipolar geometry concerning calibration data of stereo sensors. The calibration evaluation method can be applied in the measurement process and does not require any additional effort or equipment. It allows the evaluation of the current set of calibration parameters and consideration of the stability of the current calibration over certain temporal progression. Additionally, the quality of distortion correction can be scored. The behavior of three fringe projection based 3D stereo scanner types was analyzed by experimental measurements. Results of the different types of scanners show that calibration may be stable over a long time period. On the other hand, suddenly occurring disturbances may be detected well. Additionally, the calibration error usually shows a significant drift in the warm-up phase until the operating temperature is achieved.


INTRODUCTION
Contactless scanning of the surface of different measuring objects is increasingly required in industry and technique, scientific research, cultural heritage preservation and documentation, and in medicine.Fringe projection is the basic principle of a family of such 3D scanners.Flexibility, measuring accuracy, measurement data volume, and fields of application always increase.However, it should be guaranteed that the promised accuracy which is established theoretically can be also achieved under real measurement conditions.
High precision measuring systems based on image data require high precision optical components.The measuring accuracy, however, depends additionally on the quality of the geometric description of the components.The procedure of determination of the geometry of the optical components of a 3D scanning system is performed in the process of camera calibration.The correctness of calibration is crucial for the quality of a photogrammetric system and essential for its measuring accuracy.
The set of calibration data will be produced in the process of initial calibration in the production process of a certain 3D sensor device.However, it is usually not known how stable the initial calibration is over a longer time period.As Luhmann describes (Luhmann et al. 2006), modern photogrammetric measurement systems based on active structured light projection can achieve a measurement accuracy of up to 1:100000 compared to the length extension of the measuring field.However, such high accuracies can only be achieved, if the geometry of the measuring system is stable over the time between calibration and measurement.This is, unfortunately, only the case, if certain measurement conditions strictly hold.However, in the practical use this cannot always be ensured as e.g. reported by Hastedt (Hastedt et al. 2002) or Rieke-Zapp (Rieke-Zapp et al. 2009).
The stability of the calibration data of certain cameras which are used for measurement tasks has been recently analysed by several authors.Mitishita (Mitishita et al. 2003) analyses the interior orientation parameters from small format digital cameras using so called on-the-job-calibration. Shortis (Shortis et al. 2001) examines the calibration stability for a certain digital still camera.Läbe (Läbe and Förstner 2004) determines the geometric stability of low-cost digital consumer cameras.Habib (Habib et al. 2005) analyses the stability of SLR cameras over a long period (half a year).An interesting approach presents Gonzales (Gonzales et al. 2005) in his work.He analyses the stability of camera calibration depending on the proposed calibration technique.
An extensive review of the uncertainty of the epipolar geometry is given by Zhang (Zhang 1998).In his work several techniques for estimating the fundamental matrix and its uncertainty are presented.Dang (Dang et al. 2009) introduces a method for continuous stereo self-calibration by camera parameter tracking based on three different geometric constraints.His work also includes a detailed description of the sensitivity of the 3D reconstruction depending on erroneous calibration parameters.
As it could be observed by consideration of 3D sensor measurements, sometimes systematic errors occur.These errors are characterized by incorrect scaling or deformation of the shape.However, such errors are very difficult to detect, because they are first typically small, and second, the true exact size and the detailed shape of a measuring object is usually unknown.In this work, a novel methodology for calibration parameter evaluation and calibration stability supervision especially for fringe projection based stereo scanners is introduced.Application examples are given for the time dependent behaviour of the calibration quality of different fringe projection based 3D sensors as well as simulation examples.

Phasogrammetry
Phasogrammetry is the mathematical connection of photogrammetry and fringe projection.The classical approach of fringe projection is described e.g. by Schreiber (Schreiber and Notni, 2000).It can be briefly outlined as follows.A fringe projection unit projects well defined fringe sequences for phase calculation onto the object, which is observed by a camera.Measurement values are the phase values obtained by the analysis of the observed fringe pattern sequence at the image coordinates [x, y] of the camera.The 3D coordinates X, Y, Z of the measurement point M are calculated by triangulation, see e.g.(Luhmann et al., 2006).The calculated 3D coordinate depends linearly on the phase value.

Epipolar Geometry
The epipolar geometry is a well-known principle which is often used in photogrammetry when stereo systems are present.See for example (Luhmann et al., 2006).It is characterized by an arrangement of two cameras observing almost the same object scene.A measuring object point M defines together with the projection centres O 1 and O 2 of the cameras a plane E in the 3D space (see also Figure 1).The images of E are corresponding epipolar lines concerning M. When the image point p of M is selected in camera image I 1 the corresponding point q in camera image I 2 must lie on the corresponding epipolar line.This restricts the search area in the task of finding corresponding points.In the following we consider a system consisting of two cameras C 1 and C 2 and one projector in a fix arrangement.

Camera Calibration
Camera calibration is the determination of the intrinsic and extrinsic parameters (including lens distortion parameters) of an optical system.It has been extensively described in the literature, e.g. by Luhmann (Luhmann et al. 2006), Chen (Chen and Brown 2000), Brown (Brown 1971), Tsai (Tsai 1986), or Weng (Weng et al. 1992).Different principles are applied to perform camera calibration.The selection depends on the kind of the optical system, the exterior conditions, the effort to be pushed, and the desired measurement quality.In case of the calibration of photogrammetric stereo camera pairs, the intrinsic parameters of both cameras should be determined as well as the relative orientation between the cameras.Intrinsic parameters include principal length c, principal point pp = (u, v), and distortion description.
The position of the camera in the 3D coordinate system is described by the position of the projection centre O = (X, Y, Z) and a rotation matrix R obtained from the three orientation angles  (pitch angle),  (yaw angle), and  (roll angle).
Considering stereo camera systems, only the relative orientation between the two cameras (Luhmann et al. 2006) has to be considered, because the absolute position of the stereo sensor is usually out of interest.Lens distortion may be considerable and should be corrected by a distortion correction operator D. Distortion may be described by distortion functions as e.g. proposed by Tsai (Tsai 1986), or Weng (Weng et al. 1992), or by a field of selected distortion vectors (distortion matrix), as suggested by Hastedt (Hastedt et al. 2002) or Bräuer-Burchardt (Bräuer-Burchardt et al. 2004).

Basic Assumptions
Let us consider a fringe projection based stereo scanner with two cameras and one projector in a certain arrangement (see also Figure 1).3D surface data are aquired by a measurement from one sensor position using either epipolar constraint (mode m 1 ) or fringe pattern sequences in two orthogonal projection directions (mode m 2 -see Schreiber and Notni, 2000).Point correspondences with sub-pixel accuracy are found using equal phase values in both camera images.Whereas using mode m 2 always guarantees to find correct point correspondences, searching on epipolar lines (mode m 1 ) leads to point correspondence errors.However, mode m 1 needs half the length of the image sequence and hence image sequence recording needs half the time only.
The calibration of the sensor is performed once in the laboratory and is assumed to be fix over a longer time period.Calibration data include intrinsic and extrinsic parameters and a set of distortion parameters or a distortion matrix for each camera.Calibration may be performed again, of course, if it seems to be necessary.However, this means usually high effort.Alternatively, self-calibration (Schreiber and Notni 2000) can be performed at every measurement.However, conditions may be too poor to obtain robust parameters from self-calibration.

Goal
Especially moved sensors are susceptible to mechanic shocks and vibrations.These influences may disturb the current calibration, and calibration parameters become erroneous.Additionally, it is well known that calibration data are temperature dependent.Thermic changes in the environment or increase of working temperature of the sensor may also influence the calibration parameters.
The goal of this work is to describe the current state of the set of calibration parameters of fringe projection based stereo sensors.Small errors can be compensated by parameter correction whereas considerable errors should imply the decision to perform a new calibration.

Epipolar Line Error
Obviously the correct position of the epipolar lines strongly depends on the accurate values of the calibration data set.Therefore calibration errors directly influence the correctness of point correspondences, if searched on the epipolar lines.Hence, if corresponding points are located exactly on the corresponding epipolar lines this implies that the calibration data are "good".However, this is not sufficient for the that the calibration data are correct.Certain calibration parameters may be disturbed without having a considerable influence on the position of the epipolar lines but leading to a disparity error which leads, subsequently, to a depth error of the reconstructed 3D points.On the other hand, if the epipolar line position is erroneous, calibration data must be disturbed.
Let us consider two corresponding points p i and q i .The epipolar line position error err pos (p,q) i is defined as the perpendicular distance of the correct corresponding point q i to the epipolar line g i , defined by p i and the set of calibration parameters.Further the rms epipolar line error E rms of the image pair is defined by where n is the number of considered corresponding point pairs.These points should be well distributed over the images.Additionally, the maximal epipolar line error E max can be considered: Alternatively, a value of the 95% percentile may be chosen for E rms in order to prevent using outliers.Actually, epipolar line position error err pos (p,q) i is signed in contrast to rms epipolar line error E rms .This is meaningful, because the sum of all err pos (p,q) i may be near zero whereas the amount of rms epipolar line error E rms may be considerable.

Analysis of Calibration Parameter Error Influence
Let us consider the parameters of interior orientation c i , , 2 (altogether 18 parameters) and the distortion D i , describing the geometry of the stereo sensor.Because of symmetry reasons (only relative orientation between the cameras is of interest) only twelve parameters (omitting X 1 , Y 1 , Z 1 ,  1 ,  1 ,  1 ) are considered.Note that the meaning of the parameters may be different regarding either terrestrial or aerial orientation, respectively.
Assume an aerial orientation of the stereo camera pair with small pitch (or tilt) angles  i , a yaw (or gear) angle difference between the cameras near triangulation angle, and small roll (or rotation) angles  i for both cameras.Assume X-axis alignment approximately parallel to the baseline.Estimation of the influence of erroneous calibration parameters and image coordinates to the epipolar line error err pos was performed.It was derived from intercept theorems and collinearity equations (see e.g.Luhmann et al. 2006).Results of error influence estimation are given in table 1, where x and y are the centered image coordinates, and d is the distance to the measuring object.A more detailed analysis of the error influence was performed by Dang (Dang et al. 2009).

Calibration Evaluation
The idea for calibration evaluation is to simply describe the calibration quality by the amount of the rms epipolar line error according to equation (1).In order to get a significant value, representative points should be extracted being well distributed over the images.Hence, a virtual grid of image coordinates is defined.The number n of grid points should be at least n = 100.For all points p i in the image of camera C 1 the corresponding points q i in the image of camera C 2 are found by use of the rotated phase algorithm.Table 1: Influence of calibration parameter errors for a certain aerial arrangement

Approach for Calibration Correction
The knowledge of the amount of the mean epipolar line error implies the idea to manipulate the calibration parameters such that the mean epipolar line error becomes minimal.This was performed with the help of a newly developed algorithm, which is described by the authors (Bräuer-Burchardt et al. 2011).
It can be briefly explained as follows.In a first step an analysis of the influence of the single calibration parameters is performed.A reduced set of parameters with considerable and different sensitivity (see next section) is selected.The remaining parameters (in our experiments between three and seven) are systematically changed thus that a minimization of the mean epipolar line error is achieved.This will be achieved by an iterative algorithm.The manipulated parameters are now used as the current calibration data.Usually, using this method the rms epipolar line error may be reduced to a value of near or below 0.1 (pixel).This is very accurate and allows finding point correspondences on the epipolar lines with high precision.
This algorithm may be extended performing the minimization of the scaling error, too.Scaling error realizes the consideration of those parameters having a poor sensitivity to the epipolar line error (Bräuer-Burchardt et al. 2011).However, for the determination of the scaling error additional information is necessary which cannot be extracted from the phase data of an arbitrary measuring object.Here an object with well-defined lengths should be used, e.g. a grid pattern.

Simulation of Calibration Parameter Error Influence
First, simulations showing the possible difference of the influence of the single parameters on the epipolar line error were performed.It must be noted that the influence of the parameters strongly depends on the actual geometric arrangement of the sensor.Hence, simulations are performed in such a way, that the geometric situation of a real sensor is approximated, i.e. a sensor in aerial arrangement with a measuring field of about 20 mm x 15 mm and a triangulation angle of about 15°.A meaningful error of each of the parameters of the exterior and interior orientation of the sensor was assumed.The same parameters of the two cameras show analogous influence.Some parameters show similar behaviour as others (Y ~  ~ v, X ~  ~u).The results confirm the theoretic analysis (see table 1).

Simulation Example of a 3D Measurement
In order to illustrate the effect an erroneous calibration parameter set simulations was performed showing a scene with four spheres with two different diameters (1 mm and 2 mm) and a frustum of a cone, respectively.3D reconstruction was performed using original and disturbed data.The (meaningful) amount of disturbance (E rms = 1, 2, and 5, respectively) was chosen according to experimental results.See Figures 2 and 3 which illustrate the effect of disturbed calibration parameters.
Considering the spheres, it can be seen that the effect of parameter errors is hardly detectable from the subjective evaluation of the sphere shape, even if the error is big (five pixel).However, decreasing completeness can be observed.Contrary, the results of the frustum of a cone imply that errors can be detected earlier by user observation.
Table 2 documents the error D of the sphere diameter measurement and the maximal length measurement error len mx obtained by the distance measurement between two sphere centre points.Additionally, the mean standard deviation SD mn of the 3D points on the sphere surfaces is given.The size of the simulated scene was about 20 mm x 15 mm, the sphere diameters D were D 1 = 1.98 mm and D 2 = 3.96 mm and the distance between the spheres 11.9 and 16.8 mm, respectively.It can be seen that epipolar line error has a considerable and nonlinear influence on the diameter error.Moreover, the smaller the diameter the bigger is the percentage error.Length error, however, is weak and increases proportionally.

Experiments on Real Data
In the following a number of experiments using real sensor data are documented.The temporal course of the epipolar line error is considered using three different sensor types (see Figure 4).Sensors DS and CS are hand held, light weight and mobile scanners with low spatial resolution whereas sensor HS is a high resolution sensor (5 Mpixel) designed for flat measuring objects.More details are given by the authors (Kühmstedt et al. 2007, Munkelt et al. 2007, Bräuer-Burchardt et al. 2011).
The measurements were performed as follows.A white nearly plane object was chosen as measurement object.Measurements of the coordinates of corresponding points of rectangular grid points well distributed over the image plane were performed using a 16 phase algorithm of orthogonally rotated phase directions.It can be assumed that the point correspondence error is negligible.For each corresponding pair of points the epipolar line position error was determined by the perpendicular distance of the measured point q i to the calculated position of the epipolar line g i .Mean epipolar line error was calculated using equation ( 1) and used as feature size of the calibration quality.
By the measurements two aspects of calibration stability should be checked.First, the stability over a longer time period (several weeks) was analysed.For this experiment evaluation measurements were selected with warmed up sensors.The result of epipolar line error of the sensor DS 1 is illustrated in Figure 5.It can be seen that the calibration remains stable over the quite long period of almost four weeks.At the days of "higher" errors, operating temperature may be outside the target range.However, the operating temperature should be in a certain welldefined range.The next experiment was the consideration of the calibration stability from switching on the device until the reaching of the operating temperature.These experiments were performed for all three scanners and repeated several times.
By Figures 6 and 7 the behaviour epipolar line error over 120 minutes of the rms epipolar line error of scanner DS is documented for two days.Decreasing error in the "warm up" phase is due to the heating up of the scanner to operating temperature of about 40°.The warming up time takes between 30 and 45 minutes.Last but not least sensors of type CS were checked.Figure 10 shows the rms epipolar line error over 30 minutes from switch on of sensor CS 1 .Scanner CS 2 showed similar behaviour.

DISCUSSION AND OUTLOOK
The three analyzed scanner types show similar behavior of the calibration quality represented by the rms epipolar line error.
After being switched on the sensors need a certain time to reach their operating temperature.The duration of this warm-up phase is different depending on the properties of the sensors and can be exactly determined by the proposed method.
Sudden changes of the calibration data which may be caused by mechanic influences as shocks or vibrations are reliably detected by the method.Depending on the shape of the measuring object these changes are not necessarily noticed by the user as it showed the sphere example.However, subsequent correction may be performed by improvement of the current set of calibration parameters.We proposed a systematic and iterative change of between three and seven calibration parameters and minimization of the epipolar line error (Bräuer-Burchardt et al. 2011).However, it should be analyzed, whether the proposed correction is sufficient in the current case.Because only the position of lines is optimized, the corrected set of calibration parameters is only better than the erroneous one, but not really true.New calibration has possibly to be performed in the case of strongly disturbed calibration.
Future work should be addressed on the improvement of the current calibration correction procedure, the monitoring of several 3D scanners over longer periods (several months), and the fully automation of the developed algorithms so far.When scaling error can be determined, a correlation analysis of the rms epipolar line error and scaling error should be performed.This will possibly allow a blind correction of more calibration parameters than three.
Furthermore, we plan to develop a new algorithm which realizes the correction of the calibration parameter set using an estimation of the fundamental matrix (see Zhang 1998).First attempts showed a low numeric robustness but the method should be improved.

SUMMARY AND CONCLUSION
In this work a simple new method for calibration stability monitoring of fringe projection based 3D scanners was introduced which allows considering the stability of the current calibration over certain temporal progression.The behavior of three types of fringe projection based 3D stereo scanners was analyzed by experimental measurements.
Only in the case of occurrence of calibration parameter errors without showing the epipolar line error effect the proposed method is not sensitive and will not detect a disturbed calibration.However, this case is very unlikely.
Calibration evaluation by the proposed method should be applied if epipolar constraint is used in order to realize point correspondences and the stability of the set of calibration parameters may be disturbed by thermic (warm-up) or mechanic (shocks or vibrations) influences.Correction is necessary according to the amount of the detected error and the requested measuring accuracy.The proposed method is ideal for determination of the end of the warm-up phase and for correction of the point correspondence in the warm-up phase.

Figure 1 :
Figure 1: Stereo camera arrangement with fringe projector

Figure 2 :Figure 3 :
Figure 2: Sphere measurement (simulation) with E = 0, E = 1, E = 2, andE =5 (from left to right) The first sensor DS (seeKühmstedt et al. 2007) has a measurement field of about 20 mm x 14 mm.The image size is 516 x 778 pixel leading to a local resolution of about 35 µm.Two sensors DS 1 and DS 2 were analysed.The second sensor HS (see Bräuer-Burchardt et al. 2011) has a measurement field of about 50 mm x 40 mm.The image size is 2448 x 2048 pixel leading to a local resolution of 20 µm.The third sensor CS (see Munkelt et al. 2007) has a measurement field of about 240 mm x 175 mm.The image size is 640 x 474 pixel leading to a local resolution of 350 µm.Two devices CS 1 and CS 2 of sensor type CS were analysed.

Figure 5 :
Figure 5: Rms epipolar line error and corrected rms epipolar line error of scanner DS 1 over a period of 26 days

Figure 8 :
Figure 8: Rms epipolar line error and corrected rms epipolar line error of scanner HS over a period of 22 days Next, the behaviour in the warm-up phase was checked at five days.See the results of one measurement illustrated by Figure 9.The progress of the rms epipolar line error at the other days was almost identic.

Figure 9 :
Figure 9: Rms epipolar line error of scanner HS over two hours from switch on, 2011/04/06, measuring object: plane

Figure 11 :
Figure 11: Effect of insufficient distortion correction of CS 2 : the vectors indicate epipolar line error scaled by factor ten, epipolar lines run vertically, image section is cutted above and below

Table 2 :
Influence of calibration parameter error E on measurement errors of two spheres