OpenStreetMap (OSM) is a representative open geospatial platform that provides free access to major spatial objects, including buildings worldwide, constructed through crowdsourcing-based manual digitization. However, subjective differences among contributors and the absence of unified quality control standards have led to the accumulation of positional offsets and boundary shape errors in building polygons. To address this issue, studies using deep learning-based semantic segmentation for OSM quality improvement have been conducted. Nevertheless, Transformer-based segmentation models exhibit an under-segmentation tendency that merges adjacent buildings into a single object, along with limitations in precise boundary delineation. To overcome these challenges, this study proposes a two-stage framework that integrates SegFormer, which excels in global context recognition, with SAM 2, which is capable of precise boundary segmentation. In the first stage, SegFormer semantically segments building regions from a true orthoimage, and in the second stage, SAM 2 infers object-level precise boundaries using the bounding boxes of OSM polygons as box prompts. The two results are combined into a prior probability map, enabling uncertain boundary regions to be re-evaluated in an unsupervised manner. In experiments conducted over the Suseo-dong area in Gangnam-gu, Seoul, the proposed method achieved a BIoU of 70.40%, an improvement of 23.85 percentage points over OSM building data, with consistent performance gains across all evaluation metrics. This framework offers scalability applicable to any region worldwide without additional label construction, provided that high-resolution true orthoimagery and OSM data are available.