CAMERA CALIBRATION FOR UAV APPLICATION USING SENSOR OF MOBILE CAMERA

Recently, 3D measurements using small unmanned aerial vehicles (UAVs) have increased in Japan, because small type UAVs is easily available at low cost and the analysis software can be created the easily 3D models. However, small type UAVs have a problem: they have very short flight times and a small payload. In particular, as the payload of a small type UAV increases, its flight time decreases. Therefore, it is advantageous to use lightweight sensors in small type UAVs. A mobile camera is lightweight and has many sensors such as an accelerometer, a magnetic field, and a gyroscope. Moreover, these sensors can be used simultaneously. Therefore, the authors think that the problems of small UAVs can be solved using the mobile camera. The authors executed camera calibration using a test target for evaluating sensor values measured using a mobile camera. Consequently, the authors confirmed the same accuracy with normal camera calibration.


INTRODUCTION
Recently, unmanned aerial vehicles (UAVs) have been noticed for periodic inspection to maintain and preserve Japanese civil engineering buildings.Previously, inspection work in structures such as bridges and under the elevated road, was performed by humans in dangerous environments.In such situations, it is believed that using UAVs will lead to safe inspections.Furthermore, multi-view images captured by UAVs during regular inspections and high-density point cloud data created using Structure from Motion (SfM) and Multi View Stereo (MVS) enables the creation of 3D models and quantitative evaluations of the displacement of a structure.However, lowcost UAVs that are currently sold have some issues: they are small payloads and have very short flight times.A low-cost UAV has an approximate flight time of 15 min.Therefore, there is a limit on the weight of a sensor that can be mounted on the UAV; it is necessary to devise such replace a sensor mounted on the UAV for each measurement.On the other hand, the use of high-performance and lightweight mobile devices (e.g., Android and iOS devices) has progressively and rapidly spread in Japan (Figure 1).These mobile devices have a digital camera as well as GPS and several sensors such as an accelerometer, a gyroscope, and a digital compass.By integrating some of the sensors built into a mobile device, it is possible to estimate the position and orientation of the device.In this case, noise contained in sensor values can be denoised using a Kalman filter (Fuse, 2014).Moreover, mobile devices are lightweight at approximately 0.15 kg, and it is presumed to be capable of solving the problems of UAVs.In this study, the authors described and evaluated camera calibration techniques for UAVs using images and orientation parameters of sensor values from mobile devices.

Android
Android devices have digital cameras as well as various sensors.However, Android is included in various devices from different manufacturers, and some models do not have many mounted sensors.The left pane of figure 2 shows the various sensors of the Android device used in this paper, and the right pane shows the sensors in another device.Therefore, the Android device must be carefully selected.In this case, X and Y axes must be exchanged by equation ( 1) when taken android device rotating 90 degrees.Therefore, the axes of coordinate system must be carefully oriented (Google Inc., 2015).
where x, y, z = Android sensor values.

EXPERIMENTAL PROCEDURE
Figure 4 shows the test target (H: 640 mm, W: 480 mm, D: 20 mm; 3 rows in the center) used in this paper.The white-circled points are control points for camera calibration and the other 86 black-squared points are check points.The digital cameras shown in Tables 1 and 2 took five triplet images of base-height ratio 0.34 with altitude varying between 0.6-0.8m to keep a uniform photo scale.Furthermore, in the experiment using the device of Table 1, each optical axis was adjusted to be consistent with the image taken by a consumer grade digital camera.Moreover, image and sensor values were synchronize using Internet time because in this paper, the device is connected to a network of all devices.○：Control points □：Check points  2 is denoised using the Kalman filter, and rotation by a certain angle is performed.Furthermore, in order to compare this result performed to normal camera calibration.After the camera calibration, the accuracy was estimated using average root-mean-square-error (RMSE) for check points.The XY and Z accuracies of the center coordinates of each blackcircled point were ±0.05 mm, and these image coordinates were obtained as area centroid by image-processing procedures.

Device Name
CyberShot DSC-HX60V

Consumer Grade Digital Camera and Android Device
Experimental results using the consumer grade digital camera and sensor value measured by the Android device are shown in figure 5, in which "Normal" is the result of normal camera calibration and "Sensor Angle" is the result of camera calibration using the sensor value as the initial value measured by the Android device.Furthermore, "Normalization" is divided by the standard error in equation ( 3) from each result.Consequently, a greater "Normalization" implies that a result has error lower than the standard error.
where σX0, σY0, σZ0 = standard error H = altitude f = focal length B = base line σP = pointing accuracy (here is estimated 0.1 pixels) Figure 5 shows almost the same result in both cases.Furthermore, Table 3 shows the relationship between the normal camera calibration result and camera calibration using the Android device measuring the orientation result of the triplet image.Table 3 shows that the measurement value of the Android device can be the same as that of the normal camera calibration.Therefore, the orientation obtained from the Android device is considered to be available as the initial value.

Location Calibration Result Sensor Values
Left

Use of Only the Android Device
The equipment that can be mounted on a UAV is restricted as the flight time decreases with increasing payload.The Android device is lightweight and features any built-in sensors and a digital camera of high resolution.The Android device can measure simultaneously using the sensor and digital camera.The experimental results using only the Android device is shown in figure 6, which shows a large RMSE compared with the standard error.However, the result of normal camera calibration and camera calibration using the sensor value is showed the same tendency.Therefore, the large RMSE is assumed to be caused by the calibration techniques used in this study.Accordingly, the authors think that the sensor values of Android devices can be used for photogrammetry.

CONCLUSION
This paper evaluated the camera calibration techniques using images and the orientation parameter of sensor values from mobiles device for UAVs.Normal camera calibration with a consumer grade digital camera yielded the same result as with calibration using the sensor value with an Android device.
In the experiment using the image and sensor values of the Android device, an RMSE greater than the standard error was obtained.This result has a different trend from the result obtained using a consumer grade digital camera.However, the authors believe that the large RMSE was caused by the camera calibration technique used because the same error was obtained in normal camera calibration.Therefore, the sensor value of an Android device can be used in a UAV, as a large difference is not observed in the measurement accuracy.Future tasks include camera calibration using the sensor value and GPS of an Android device and mounting an Android device on a UAV.

Figure 1 .
Figure 1.User and spread ratio

Figure 2 .
Figure 2. Difference of mounted sensor