Haul roads are essential to open-pit mines, acting like the mine’s circulatory system. Keeping accurate, up-to-date maps of these roads is critical for maintenance, safety, and efficient material handling, yet automating this task is challenging. Traditional deep learning models that rely only on RGB images often fail in mining environments, where road surfaces resemble bare earth, dusty terrain, or shadowed areas. To address this, we propose a dual-encoder transformer that combines UAV-captured RGB images with DSM data using stage-wise cross-attention, leveraging both visual and topographic information. Two SegFormer encoders process each data type separately, creating detailed feature representations that are fused at each stage. This allows the model to learn specialized information while sharing knowledge between modalities. A lightweight All-MLP decoder produces the final segmentation map. We tested our method on a high-resolution dataset of 12,000 tiles from the Mildred Lake open-pit mine in Fort McMurray, Canada. Our model achieves 80.8% mIoU, 88.7% F1-score, and 73.7% road accuracy, outperforming an RGB-only baseline by 3.3%, 2.4%, and 7.8 points, respectively. Ablation studies demonstrate that including DSM data consistently improves recall and road detection, especially in areas where RGB information alone is ambiguous or terrain is complex.