Three-dimensional (3D) building models with accurate geometry and realistic textures remain essential for city information modeling and digital twin applications. However, photogrammetric reconstructions consistently suffer from severe texture defects caused by occlusions, shadows, distortions, and projection errors. Existing approaches either rely on rigorous photometric optimization that demands topological correctness and multi-view imagery, or employ flexible AI-driven generation that leverages semantics but often lacks geometric constraints. This paper presents a novel hybrid framework that exploits architectural regularities—specifically massing decomposition and partial symmetries—to guide high-fidelity texture refinement. We first decompose building meshes into mass-aligned convex volumes using MorphCut. Textures are then reprojected onto these volumes, followed by Building Section Skeletons to pair symmetric facades and establish precise geometric correspondences. Finally, generative AI is applied using symmetry-aware constraints to achieve contextually accurate inpainting and correction. Pilot studies on three Hong Kong buildings demonstrate robust decomposition, faithful texture transfer, and effective defect mitigation, while revealing current limitations of zero-shot vision language models (VLMs) in preserving floor counts and structural regularity. The proposed symmetry-guided pipeline notably advances the reliable and semantically coherent reconstruction of textures for complex urban buildings.