A NOVEL APPROACH TO CAMERA CALIBRATION METHOD FOR SMART PHONES UNDER ROAD ENVIRONMENT

: Monocular vision-based lane departure warning system has been increasingly used in advanced driver assistance systems (ADAS). By the use of the lane mark detection and identification, we proposed an automatic and efficient camera calibration method for smart phones. At first, we can detect the lane marker feature in a perspective space and calculate edges of lane markers in image sequences. Second, because of the width of lane marker and road lane is fixed under the standard structural road environment, we can automatically build a transformation matrix between perspective space and 3D space and get a local map in vehicle coordinate system. In order to verify the validity of this method, we installed a smart phone in the ‘Tuzhi’ self-driving car of Wuhan University and recorded more than 100km image data on the road in Wuhan. According to the result, we can calculate the positions of lane markers which are accurate enough for the self-driving car to run smoothly on the road.


INTRODUCTION
Camera calibration is the crucial step in photogrammetry, which is the determination of parameters necessary to establish the projection equations between the world coordinate system and image geometry [1]- [2] .In self-driving car, obtaining the camera parameters called Pos information is the necessary part for the 3D reconstruction which is a great implement in the advanced driver assistance systems [3]- [4] .
There are many approaches for camera calibration.The commonest way is to make full use of the control points in the world coordinate system and their corresponding image points in the image plane to solve the parameters by the collinearity equation or direct linear transformation, namely spatial resection or DLT [5]- [6] .Another typical one based on monocular vision, is to use the geometrical relationship of the three vanishing points to calculate the parameters called camera self-calibration [7][8] [9] .However, both methods mentioned above are not practical and applicable in self-driving car due to their complexity in computation and the requirement in control points.The first method needs considerable time to match the corresponding points.The second is ineffective when there are no three vanishing points.Therefore, in this paper, we propose a new camera calibration method that uses single monocular image by estimating the single vanishing point and width of the lane marker.It is not only greatly simplifies the computation, but also performs well in precision.This method was applied in the TuZhi car and the stable results suggest its promising future in the application of real time self-driving.

Lane Detection
At first, we can detect the lane marker feature points in a Transform maps the image plane into the parameter space and count the line parameters.After the Hough Transformation, the lines are shown clearly in the image plane.

Extraction of Vanishing Points
A set of parallel lines in the real scenes which do not parallel to the image plane will be converted to a set of intersecting lines when projected to the image plane.And the vanishing points which contain the direction information of the lines are defined by the intersection points of the lines [10] .Finding the vanishing point greatly helps determine the estimation of attitude parameters of the camera since the relation between vanishing points and attitude parameters.In addition, in the road system, this process of analyzing will be greatly simplified by the lane marker detection.Once the number of line segments that we detect is large, this method is slow and lack of accuracy.Therefore, we put forward another method which is efficient in computation and of great accuracy, since such an inefficient method is not applicable in the real time detection.
As the solution, we separate the line segments according to their locations in the image into two set of lines, one is belonging to the left lane markers, another is belonging to right lane markers.
During the computation of the intersection point, we only calculate the intersection points from different sets.We choose separately the line segment with a length threshold, mi mj from sets and calculate the intersection points weighted by the length of the line segment li lj involved in the calculation.
The location of the corresponding vanishing point hypothesis can be computed as [11] ij = ij V m m  (2) and the corresponding weight set to .And because of the high effectiveness of lane detection algorithm, the inaccuracy of the vanishing point is less than 2 pixels.Once the model is built, a 2D point can be projected into a 3D point by using the attitude parameters and the equalities.In other words，with the connection between the two coordinate system created, the 3D reconstruction of the road will be easily to complete.
According to the parameters mentioned above, the formula revealing the relationship between the two coordinate systems can be derived [12][13] Fig 7 .the camera and road model where u, v=image coordinate; x, y, z=space coordinate; θ=pitch angle; γ=course angle; alpha=transverse angel of view; Longitudinal angle of view; h=camera height m, n=size of the image;x0,y0,z0=camera's space coordinate.
From the formula above，another formula which transform the world coordinate into image coordinate can be derived Thus we are creating the relationship between the two coordinate systems.According to the definition of vanishing point, we can easily get the expression of it.In this part, we let x tend to infinity.The result is: So far,it is easily to find that the attitude parameters are highly related linear with the vanishing points.Therefore, it is very convenient to look up for the location of the vanishing point to compute the attitude parameters.

Spatial Constraint to Solve Camera Height
Once we get the attitude parameters, we can use the formula above or create the Rotation matrix to recover the connection between the real space and images.We get a point in the image and then transform it into the point in the real world or the scene.
For the camera height h, we put forward a new and highly effective approach to calculate it.Because of the width of lane marker and road lane is fixed under the standard structural road environment, we can automatically build a transformation matrix between perspective space and 3D space and solve the h parameter.
For the edge points on the lane markers, we find the corresponding points by the regulations of parallel.By the least square fitting, not only can the parameter h can be solved, but also the accuracy of attitude parameters can be greatly improved.
Strictly speaking, because of the existence of other vanishing points, the distance in the real world of the selected points are not equal to the lane width.But we can prove that this error is less than 2 pixels which is a very small error that can be negligible under the condition of γ<5°.

Experiment
We test our algorithm in the self-driving car 'TuZhi' (figure 8)with smart phone.The focal length is 1090mm,each frame's size is 1280×720.With the result of lane detection, the vanishing point with high accuracy can be calculated.According to the formula above, the attitude parameter and camera height h can be estimated.Thus, we apply inverse perspective transformation on the image to reconstruct the road.With a set of continuous images, the road can be stitched.In addition, the width of the road or the distance between cars can be estimated accurately.
From figure 9,the results of the 3D reconstruction are clearly shown.All the roads and lane markers become a set of parallels again.

Conclusion
We have proposed a simple and real-time approach for the camera calibration, based on the vanishing points and the geometrical constraint of the lane marker width.By using the images from smart phone installed on the 'TuZhi' car, the result is accurate enough for many applications.As its simplification in computation, it can be widely used in self-driving car.
However, this method's precision is closely related to the accuracy of detection of vanishing point.It still has some improvement in the work.

Fig 1 .
Fig 1.The process of lane detection

Fig 5 .
Fig 5.The diagram of vanishing point

Fig 6 .
Fig 6.The center of the red circle is vanishing points

Fig 8 .
Fig 8. Experiment result, figure a is the original image, figure b is the result of 3D reconstruction The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B5, 2016 XXIII ISPRS Congress, 12-19 July 2016, Prague, Czech Republic This contribution has been peer-reviewed.doi:10.5194/isprsarchives-XLI-B5-49-2016