Towards Roof Material Identification by Fusing Aerial and Street View Imagery

Abstract. Accurate roof material detection is essential for 3D building modeling, urban planning, climate change mitigation, and infrastructure maintenance. Roof materials directly influence key building properties, including thermal behavior and solar reflectance, and enable urban planners to prioritize climate-resilient infrastructure in areas exposed to extreme weather conditions, thereby helping to mitigate the urban heat island effect. Remote sensing and aerial imagery offer great potential for identifying roof materials. Recent advances in deep learning, particularly convolutional neural networks (CNNs), have significantly improved the performance of roof material detection using images.
Despite their success, approaches relying solely on aerial imagery remain inherently limited. From a top-down perspective, many roofing materials, such as asphalt and metal, exhibit similar spectral characteristics, making them difficult to distinguish. Furthermore, aerial imagery is often affected by shadows, occlusions, and limited texture visibility, particularly in dense urban environments or regions with complex illumination conditions.
In this study, we propose a novel multimodal framework for roof material classification that leverages both aerial orthoimages and ground-level imagery derived from GoPro video recordings. The proposed method utilizes spectral and textural features extracted from both nadir and off-nadir images to improve roof material detection. The framework employs a dual-branch architecture in which aerial and ground-level image features are extracted independently using CNN-based feature extractors. A decision-level late fusion strategy with learnable weights is designed to combine the predictions from each branch. The proposed method classifies four roof material classes, including asphalt, metal, membrane, and gravel, and achieves an overall accuracy of 91%.