On the Perspectives of Image-to-Lidar Constraints in Dynamic Network Optimisation

Abstract. The evolution of airborne mapping witnesses the introduction of hybrid lidar-camera systems to enhance data collection, i.e. to obtain simultaneously high-density point-cloud and texture. Yet, the common adjustment of both optical data streams is a non-trivial problem due to challenges associated with the different influences of errors affecting their mapping accuracy including those coming from navigation sensors. Stemming from a special form of graph-based optimization, the dynamic networks allow rigorous modeling of spatio-temporal constraints and thus provide the common framework for optimizing original observations from inertial systems with those coming from optical sensors. In this work, we propose a cross-domain observation model that leverages pixel-to-point correspondences as links between imagery and lidar returns. First, we describe how the existence of such correspondences can be introduced into optimizations. Second, we employ a reference dataset to emulate a set of precise pixel-to-point correspondences to assess its prospective impact on the common (rather than cascade) optimization. We report the improvement in the estimated trajectory attitude error with lower quality IMU and thus the point-cloud registration. Finally, we study whether such correspondences could be contained from freely available deep learning networks with the desired accuracy and quality. We conclude that although the introduction of such camera-to-lidar constraints has significant potential, none of the studied machine learning networks can fulfill the requirement on correspondence detection in terms of quality.