Construction plans integrate visual and textual information that is essential for construction projects. However, the huge diversity of formats of these plans poses challenges for automated analysis. This paper presents a novel correspondence model that links objects and texts in construction plans, providing a unified approach to interpreting various formats, such as scanned blueprints, CAD drawings, and digital construction documents. Leveraging deep-learning-based object detection and text recognition techniques, our model establishes semantic correspondences between visual and textual elements. We integrate CLIP-based models with ViT-based encoders as part of our approach to enhance feature extraction and correspondence learning. By employing a threshold-based determination, our model effectively resolves cases where a single text passage may describe multiple objects or where a single object is referenced by multiple pieces of text. This capability enables the model to establish robust correspondences between objects and texts, laying a strong foundation for subsequent semantic understanding and information extraction. We evaluate its effectiveness on labeled datasets and demonstrate that our model achieves high precision, recall, F1-score, and accuracy. Hence, we provide a feasible approach to establishing object-text correspondences in construction plan analysis. The results suggest its potential to serve as a foundation for further exploration in the automated analysis of technical drawings, particularly in the context of quality assurance and construction project planning.