Grotto temples are valuable World Cultural Heritage sites and provide important physical evidence for the study of ancient civilizations. However, prolonged exposure to natural and human factors has caused surface deterioration in these grottoes. Conventional deterioration identification methods have shown clear limitations in quantification, automation, and large-scale application. To address this issue, this study focused on the Yungang Grottoes, a World Cultural Heritage site. We collected and processed multisource historical and contemporary images and built a high-quality annotated dataset covering three typical deterioration features: peeling, crack, and human factor. We then developed an improved YOLO11 model and introduced a spatial attention mechanism to dynamically direct the model toward critical deteriorated regions, thereby improving detection accuracy. The experiments showed that the model achieved recognition confidence levels of 88.53%, 86.42%, and 84.78% for the three typical deterioration features, respectively. These results provide a new technical approach for automated deterioration identification of stone cultural relics in grotto temples.