SEAMLESS CO-REGISTRATION OF IMAGES FROM MULTI-SENSOR MULTISPECTRAL CAMERAS

Abstract. Small-format, consumer-grade multi-camera multispectral systems have gained popularity in recent years. This is specifically due to the simplicity of their integration onboard platforms with limited payload capacity, such as Unmanned Aerial Vehicles (UAVs). Commercially available photogrammetric software can process the image data collected by these cameras to create multispectral ortho-rectified mosaics. However, misalignments of several pixels between spectral bands have been observed to be a common issue when employing these solutions, which can undermine the spectral and geometric integrity of the data. Besides, in advanced processing workflows such as object detection and classification with deep learning algorithms, band-to-band co-registered images are needed rather than one mosaic. We propose a two-fold solution for seamless band-to-band registration of images captured by five cameras integrated into a miniature multispectral camera system. This approach consists of 1) a robust self-calibration of the multispectral camera system to accurately estimate the intrinsic calibration parameters and relative orientation parameters of all cameras; 2) a single capture, band-to-band co-registration method based on trifocal constraints. This approach differs from existing literature since it is fully automatic, does not make any assumptions about the scene, does not use any best-fit projective or similarity transformations, and does not attempt cross-spectral feature-point matching. Our experiments confirm that the proposed co-registration method can accurately fuse multispectral images from a miniature multi-camera system and is invariant to large depth-variations in the captured scene.