INSTANCE SEGMENTATION OF 3D MESH MODEL BY INTEGRATING 2D AND 3D DATA

Abstract. Buildings are an important part of the urban scene. In this paper, a novel instance segmentation framework for 3D mesh models in urban scenes is proposed. Unlike existing works focusing on semantic segmentation of urban scenes, this work focuses on detecting and segmenting 3D building instances even if they are attached and occluded in a large and imprecise 3D surface model. Multi-view images are first enhanced to RGBH images by adding a height map and are segmented to obtain all roof instances using Mask R-CNN. The 2D roof instances are then back-projected onto the 3D scene, the accurate 3D roof instances are obtained using a novel 3D clustering method and two post-processing steps which preserve the largest connected region and remove the model ambiguity. Finally, the 2D convex hull of each 3D roof instance is calculated and the model is divided within the range into building instances. The performance of the proposed methods is evaluated using real UAV images and the corresponding 3D mesh models qualitatively and quantitatively. Results revealed that the proposed method could effectively segment the model of the urban scenes and building instance is obtained, the over-segmentation masks can be clustered correctly into roof instances and the under-segmentation masks caused by image segmentation errors are eliminated.