Accurate and timely building damage detection (BDD) is crucial for disaster emergency response. Although deep learning-based change detection methods have made significant progress in remote sensing, their practical application in disasters still faces two major challenges: (1) Existing high‑accuracy models are typically computationally complex and difficult to deploy for real‑time inference on edge devices. (2) Model performance heavily relies on large amounts of annotated data, but disaster data are extremely scarce. To address these challenges, this paper proposes a novel lightweight Local-Global Interaction Network (LGINet) for efficient BDD. The core of LGINet is the proposed Local-Global Interaction Unit (LGIU), which achieves efficient fusion of detailed and contextual features through a dual‑path architecture and channel‑wise cross‑attention mechanism. Furthermore, a Frequency Difference Enhancement Unit (FDEU) is proposed to generate more accurate damage features, and contrastive learning is employed to reduce the model’s sensitivity to weather conditions and its reliance on annotated data. Experimental results on the xBD and WBD datasets show that LGINet achieves F1-scores of 81.76% and 80.91%, respectively, with an inference speed of 47.83 FPS. It achieves the best balance between accuracy and efficiency, outperforming existing methods.