Advancements on Semantic Real-Time UAV Mapping

Abstract. This paper presents key improvements in real-time ortho image generation and scene understanding for disaster management and first responders. Through the introduction of an Inertial Measurement Unit, a depth estimation network and a trained network for scene segmentation, it is possible to produce end-to-end real-time ortho and semantic maps. Since datasets containing inertial data are sparse, the results of the pipeline were verified on a flight, which was recorded and post-processed as a ground truth with ground control points using the standard photogrammetric workflow. The reported errors are in the same range as a post-processed ortho map on raw Global Navigation Satellite System measurements, however, produced in real time. Semantic segmentation results demonstrate surprising levels of accuracy and robustness, but reveal a need for more comprehensive data acquisitions and benchmarks.