Accurate damage localization is essential for infrastructure inspection, but conventional segmentation methods rely on dense pixel-level annotations that are costly to obtain and difficult to scale. This paper presents a foundation model-based pipeline for data-efficient damage localization in built infrastructure. The proposed workflow combines DINOv3 features for image-level classification, Grad-CAM for weak localization, and the Segment Anything Model (SAM) for prompt-guided pixel-level segmentation. The resulting masks are further transferred into 3D space for spatially contextualized visualization. <br />The pipeline is evaluated on two case studies. On a subset of Sewer-ML, three representative sewer defect classes are used to compare pretrained backbones and to qualitatively assess downstream localization. The DINOv3-based classifier achieves a higher average F2-score than a Google ViT baseline, reaching about 0.72 versus 0.64. On a custom historic masonry dataset, the method is quantitatively evaluated for material-loss segmentation using manually annotated test masks. The proposed heatmap-guided prompting strategy achieves a mean Dice score of 0.69 and a mean IoU of 0.53, while the classification stage reaches an F2-score of 0.99. A proof-of-concept experiment further demonstrates that segmented damage regions can be visualized within a larger local 3D scene. Overall, the results show that the proposed foundation-model based pipeline can support data-efficient and spatially meaningful damage localization across different infrastructure domains.