This study explores the efficacy of vehicle-assisted monitoring for bridge damage assessment, emphasizing the integration of diverse sensor data sources. A novel method utilizing a deep neural network is proposed, enabling the fusion of fixed sensors on bridges and onboard vehicle sensors for damage assessment. The network offers scalability, robustness, and implementability, accommodating various measurement types while handling noise and dynamic loading conditions. The main novel aspect of our work is its ability to extract damage-sensitive features without signal preprocessing for future bridge health monitoring systems. Through numerical evaluations, considering realistic operational conditions, the proposed method demonstrates the capability to detect subtle damage under varying traffic conditions. Findings underscore the importance of integrating vehicle and bridge sensor data for reliable damage assessment, recommending strategies for optimal monitoring implementation by road authorities and bridge owners.