Heat insulation performances of the existing building stock are known rarely since the attributes of the utilized insulation material was not documented during the construction stage. Moreover, strong winds and earthquakes can dislodge building roofs and cause openings at the window frame-wall junctions, increasing heat loss from the building. For these reasons, it is necessary to regularly measure the thermal insulation performances of existing buildings. Considering the number of existing buildings, automating the thermal insulation performance assessment will provide significant savings in terms of time and money. In this study, a framework system has been developed that measures the thermal insulation performance of buildings with minimal human intervention. In the first stage, building facade is recorded in video format with a thermal camera. Image frames with the least blur is obtained from the using the Variance of Laplacian method. To ensure accurate classification, entropy-weighted contrast enhancement is applied. To make the system's training data available, areas of heat loss were manually marked on the image. The resulting image dataset was divided into training and test data, and the YOLOv8 Instance Segmentation model was trained using these data. Then, heat loss was determined on other thermal images using the trained model. Case studies have shown that the developed model can successfully identify heat loss regions.