Effective integration of geometric and semantic data within Building Information Models (BIM) is essential for the efficient life cycle management of modern facilities. However, maintaining accurate as-is BIM models for existing buildings remains a significant challenge, as manual updates are labour-intensive and full 3D reconstruction is often impractical for incremental changes. In such cases, image-based approaches offer a fast and flexible alternative, but require reliable alignment of 2D imagery with existing BIM geometry. To address this challenge, this study introduces a streamlined pipeline for semantic enrichment that uses a single-image visual localisation approach to directly align 2D imagery with existing BIM geometry. The proposed method integrates transformer-based panoptic segmentation (Mask2Former) with a closed-form Perspective-n-Line solver to estimate 6-degrees-of-freedom (6-DoF) camera poses. The novelty of the proposed approach lies in the explicit use of semantic information as a geometric constraint to guide the selection of 2D–3D correspondences for pose estimation. Semantic labels are utilised to filter line correspondences, ensuring that only stable architectural boundaries (e.g., walls, floors, and ceilings) are used in the registration process. Such semantic filtering stabilises correspondence selection, effectively mitigating pose ambiguity in repetitive indoor layouts or scenes where structural elements are partially obscured by furniture and clutter. Experimental results confirm high accuracy, achieving a median position error of 9.84 cm and an orientation error of 1.05° in complex indoor environments. This robust registration framework provides a reliable foundation for the downstream semantic enrichment and digital twin updates.