Improving Tree Species Detection for Operational Forestry: The Role of Dataset Design

Abstract. Accurate tree species mapping from high-resolution aerial imagery is essential for forest monitoring, yet deep learning approaches remain highly sensitive to training data design and often fail to generalize from curated datasets to operational conditions. This study aims to evaluate how dataset composition and model configuration influence species-level tree detection. Using 10 cm RGB imagery and 18,379 partially labeled annotations across 10 classes, where only a subset of tree crowns is annotated per image, three dataset configurations are compared: unbalanced masked, balanced masked, and mixed masked–unmasked imagery. A YOLO-based object detector is trained with optimized hyperparameters, and an extended architecture incorporating a channel attention mechanism is assessed. Results show that class balancing the dataset yields the highest validation performance (F1 = 0.83, mAP₅₀ = 0.87), while mixed datasets reduce performance (F1 = 0.56, mAP₅₀ = 0.56). The attention mechanism partially improves results (F1 = 0.60, mAP₅₀ = 0.62), particularly for conifer species and dead trees. However, evaluation on full aerial mosaics reveals contrasting behavior: masked models tend to over-detect with overlapping predictions, whereas models trained on unmasked or mixed data exhibit strong omission errors. These findings highlight a trade-off between validation performance and operational behavior, and demonstrate that partially labeled training data limit generalization. For practical applications, fully labeled datasets and evaluation on complete aerial images are critical for reliable and transferable tree species detection.