3D Façade Element Extraction from Image-based Instance Segmentation and Scale-Invariant Object Contour Points

Abstract. Real-world 3D reconstructions of building façades in LoD3 and beyond are not yet widely available on the mass market due to financial and technological barriers, as well as the challenges of automated modeling. We propose a novel method for extracting 3D façade elements using image-based instance segmentation and scale-invariant object contour points (SIOCP). Our methodology uses RGB images, camera parameters, absolute 6DoF pose and position, as well as LoD2 building information. The images are processed using instance segmentation with YOLOv8 and SAM, complemented by classical and enhanced algorithms for line and edge detection. The SIOCP method refines object contour lines from instance segmentation by incorporating LoD2 building data and 6DoF information. Subsequently, the keypoints are estimated and the single-camera image 6DoF pose is reconstructed using a PnP solver. From these 6DoF poses a photogrammetrically point cloud is generated, and semantically- and instance-segmented with SuperCluster. The segmentation results are intended for future comparisons with other point clouds and LoD3 reconstructions. The presented approach is still under development, so the current results are limited. In summary, this paper introduces a key component of our vision for LoD3 reconstruction by using handheld devices.