<p>The paper presents an effort to develop learning models based on the massive amounts of data that has been accumulated over the past decades during the process of digital documentation of heritage structures around the globe especially those in disaster zones.</p><p>The development of an ontology is proposed that describes heritage buildings, their sites, and major hazard events that may cause damage to them. This ontology can serve as a repository for documenting heritage structures and provide highly structured data for developing machine learning systems that can identify patterns of damage from recorded image data. For heritage structures in seismic zones, the first step in ontology development is analyzing available earthquake information about the event and the damage information. The resulting model will create links between information items, for example relating the extent of the damage of an element to the earthquake magnitude and its distance from the epicenter. The ontology may also include collected images from previous earthquake events, with links to the objects in each image. Special tools will focus on selecting sub-models to be included in a machine learning model. For example, if the learning objective is to identify the damage and its extent from an image, then the rules will select the features in the model that relate to structural damage and identify each type of damage. It is hoped that this work will help develop learning systems that speed up processing of large volumes of image damage data collected from heritage sites.</p>