TRANSFORMER-BASED METHOD FOR SEMANTIC SEGMENTATION AND RECONSTRUCTION OF THE MARTIAN SURFACE

Abstract. The last decade has witnessed a great advance in deep space exploration, such as the rover missions to Mars. Semantic information on the Martian surface is garnering more attention, for its ability to distinguish the surface landforms for rover traverse planning and facilitating 3D reconstruction. The state-of-the-art studies on semantic segmentation exclusively leveraged transformer-based methods, and the results have been verified to outperform the traditional convolutional neural networks. However, few datasets concerning the Martian surface have been generated, and the publicly available network models were all trained on the common Earth dataset. Constructing a pixel-wise semantic segmentation dataset requires lots of human labor, especially for training a large transformer network. Furthermore, the results of semantic segmentation were typically used for intuitive visualization but seldom exploited in the 3D reconstruction pipeline. To address these problems, this paper presents the following three contributions: (1) introducing an approach to generate a large dataset for Mars in a semi-automatic way; (2) development of a novel variant of transformer designed for multi-view semantic segmentation to improve the accuracy; (3) development of a semantic-aware dense image matching method for improved matching performances assisted with the semantic information. Experimental results using the dataset collected at the Zhurong landing site on Mars have shown superior performances of the proposed methods as compared with traditional methods.