ENHANCED ALGORITHMS TO EXTRACT DECAY FORMS OF CONCRETE INFRASTRUCTURES FROM UAV PHOTOGRAMMETRIC DATA

: Nowadays, inspection and maintenance of infrastructures is one of the most crucial issues to be faced. The structural performances should be constantly checked, considering that especially ageing stock and extreme weather events deteriorate the network infrastructure over time. However, economic reasons and the practical difficulty of carrying out a targeted inspection makes the detection a major challenge. In most of the western countries, a high percentage of infrastructures has reached and exceeded the nominal life, so the detection plays a fundamental role for their proper functioning. In this paper, experimental algorithms were tested to extract the decay forms on a portion of concrete pillar. These algorithms were applied on a metric digital image, obtained by photogrammetric output. The finally accuracy was calculated considering as ground truth the visual inspection performed by a concrete expert and the confusion matrices, the overall accuracy and the kappa-coefficient were evaluated. Then, a comparison with existing algorithms was made using the same methodology. The stacking of the original RGB image with two of the experimental algorithms could increase the accuracy up to 5%, while the common ones did not produced significant improvements. The final purpose of this work was to propose a semi-automatic/automatic procedure to detect concrete decay forms and, therefore, help the infrastructure management. The awareness is essential to provide the best practices to ensure safety and adequate performance over time.


INTRODUCTION
Inspection and maintenance of infrastructures is, nowadays, a hot topic.Ageing stock and extreme weather events, mainly, deteriorate the network infrastructure.Their structural performance should be checked periodically, but this is not always possible, both because of the difficulty to practically carry it out and because, sometimes, insufficient funds are allocated to infrastructure management.In most of the western countries, a high percentage of bridges, roads, viaducts were built between the 1950s and the 1970s, so considering that the infrastructure's average age is over fifty years, the detection plays a fundamental role for their proper functioning.The awareness related to the numerous problems of infrastructures ageing is essential to provide the best practices to ensure safety and adequate performance over time.Many countries are studying technical ways to survey and monitor transportation corridors like bridges, viaducts, highways (ARTBA, 2022, MIT, 2020, Maurey et al., 2019).This is possible through a constant cognitive process and the application of methods and techniques that periodically monitor and control the structures and infrastructures.Traditionally, the methodology involves concrete expertise with visual inspection but it is usually time-consuming, and it may be subjected to the operator experience.Recently, the new concept of Structural Health Monitoring (SHM), which is an interdisciplinary subject that involves synergistic knowledge and experience technologies in civil, mechanical, control and computer engineering, has been developed for investigation and monitoring of structures and infrastructures.SHM includes 4 levels of analysis: detection, localization and quantification of damage and prevision of remaining life (Zinno and Artese, 2021).In this context, geomatic techniques play a fundamental role for continuous detection and damage quantific-ation over infrastructures by using the traditional survey techniques (Zschiesche, 2022), or LiDAR, photogrammetry, remote sensing (optical and SAR), and image analysis methods.Although the classic survey techniques can provide highly reliable real-time data of structural conditions (Alicandro et al., 2021, Zschiesche, 2022), they cannot be always performed both because of economic reasons and of the difficult-to-access areas.Therefore, the aforementioned techniques can be exploited to develop new and alternative methodologies to fulfil this need.Among these, UAV photogrammetry can provide useful data for inspections and control of infrastructures.It can acquire and process large quantities of metric data in relatively short times and low costs.Moreover, it is useful to investigate difficult-toaccess areas remotely, guaranteeing the safety of the operator and performing non-destructive and non-invasive survey.UAV photogrammetry outputs can be exploited with Digital Image Processing (DIP) to extract the main features from digital images.Several DIP, including enhancement, noise removal, edge detection, line detection, morphological functions, colour analysis, texture detection, wavelet transform, segmentation, clustering and pattern recognition, alone or combined, can help to highlight and extract cracks and decay forms (Jahanshahi et al., 2009).In literature, different approaches are tested for this aims but they are principally applied to extract and analyse cracks (Ioli et al., 2022, Rau et al., 2017, Prasanna et al., 2012).Other studies, which use Object-Based Image Analysis (OBIA), can automatically extract and classify not only cracks but also other decay forms (Zollini et al., 2020, Belcore et al., 2022).The aim of this work was to test innovative algorithms to enhance the extraction of features such as cracks, spalling areas, exposed rebars, and washout of reinforced and pre-stressed concrete viaduct.The experimentation compared algorithms already used in literature and new algorithms within the world of artificial intelligence.Specifically, the ACM Systems (Active Connection Matrix) (Buscema, 2005) were tested and compared with the common ones.This algorithms were developed for medical image analysis but they were also recently tested in territorial field and for cultural heritage applications (Zollini et al., 2023, Alicandro et al., 2018).Important information could be extracted by analysing the reciprocal positions occupied by pixels.Both the common and experimental algorithms were compared to enhance the features in the UAV photogrammetry output images.They were processed as input to perform the Object-Based Image Analysis (OBIA) (Blaschke, 2010).Various algorithms both in the segmentation and in the supervised classification steps were evaluated.A thematic map consisting of different objects, such as crack, spalling area and so on, was directly extracted and confusion matrices with overall accuracy and kappa coefficient were calculated considering as ground truth the image classified by a concrete expert.The final objective was to propose a semi-automatic/automatic procedure to detect the decay forms and, therefore, help the management of the infrastructures.Identifying and monitoring the concrete deterioration with fast methodologies is essential for conservation, maintenance and safety guarantee.

MATERIALS AND METHODS
An overpass located in Via Campo di Pile, L'Aquila (Italy), was considered as case study.It was built between the late 80s and early 90s.Overall length of the straight structure is 225 m, consisting in 9 spans, with supported beams.Pillars are monolithic elements of reinforced concrete, which have rectangular shape section 2 X 3 m and variable height according to the orography.The overpass is surmounted by 5 beams for each span, of rectangular section with variable height from 1 m to 2 m and base of 2 m; the length from the interlocking section is 3.8 m.One pillar was chosen for the photogrammetric survey and for the subsequent processing tests.The photogrammetric survey was conducted by a Flynovex Hexacopter UAV equipped with a Sony alpha 6000 camera.The acquired images were elaborated with Agisoft Photoscan Pro to reconstruct the photogrammetric model of the pillar.9 Ground Control Points (GCPs) and 4 Check Points (CPs), previously acquired with TS30 Total Station, were used to solve, respectively, the Bundle Adjustment and to evaluate the quality of the model, obtaining a final RMS (Root Mean Square) error of about 1.5 cm (Zollini et al., 2020).A high definition orthomosaic of a representative portion of the pillar (Region Of Interest, ROI) was generated starting from the 3D model.The ROI chosen for the analysis is shown in Figure 1.Then, many experimental algorithms, called Active Connec- tion Matrix (ACM) Systems, were tested to enhance the spectral and geometric characteristics of the image.
The ACM have been implemented from Semeion Research Centre.This patent considers each image as an active matrix (network) of connected elements (pixels) that develops over time.Each digital image stores the maximum amount of information within the pixel values and their relationships.So both the pixels' values and the weights with their neighbors is taken into account.Furthermore, it is possible to obtain important information by analyzing the reciprocal positions occupied by pixels.The systems are classified into three orders of complexity, according to the type of evolution over time.In the first order of complexity, the values of connections are initialized once, at the beginning, and then remain fixed while pixel values evolve over time until convergence.The situation is specular in the case of second order, where the pixels values are fixed while the connections values are updated at each iteration, after initializing them.Finally, the third order of complexity includes models in which both the pixels and the connections change over time.For a complete presentation of ACM algorithms, see (Buscema, 2005, Buscema andGrossi, 2010).In order to extract the edges, i.e. the geometry of the image, Automata Rule (AR), New Interactive Activation and Competition (New IAC), New Constraints Satisfaction Network (New CS) and J-Net Dynamic were tested.The best edge extraction was performed by the New CS because it better emphasized the geometry.AR and New IAC were noisier, so some borders were not correctly extracted.The same happened with the J-Net Dynamic, which belongs to the third order of complexity, where both the connections and units are dynamic.It can easily extract the edges if there is an important contrasts between elements, but in this case, the spectral response between the background and the decay forms was similar, so the extraction was not representative of the correct elements.Considering the contrast enhancement, the Contractive Maps (CM) and the CM Squashed were tested.
The first one gave the best results because, compared to the other, it enhanced the spectral pixel values, i.e. the contrast between different elements, without smoothing the edges.To sum up, the New CS (Figure 2) and the CM (Figure 3) were chosen.Segmentation is the process of dividing an image into segments that have similar spectral, spatial, and/or texture characteristics (Jin, 2012).This is the most delicate step of the process because it influences the entire workflow.This step relies on the watershed transformation, which sorts pixels by increasing greyscale value, then begins with the minimum pixels and "floods" the image, partitioning the image into basins (regions with similar pixel intensities) based on the computed watersheds (Roerdink and Meijster, 2003).The result is a segmentation image, where each region is assigned the mean spectral values of all the pixels that belong to that region.The Edge segmentation method was chosen within the available algorithms because it draws lines along the strongest intensity gradients (working as an edge detector).It computes a gradient image using a Sobel edge detection method, where the highest pixel values represent areas with the highest pixel contrast.The watershed algorithm is applied to the gradient image rather than the original image.The watershed algorithm floods the image starting with the lowest gradient values (the uniform part of the objects) to the highest gradient values (the edges).Many scale levels were tested and the best value was 30, as it was a good compromise between the number and the kind of segments.Then, the Full Lambda Schedule merging method combines adjacent segments, by increasing the merging value.In this case, a value of 0 was set, in order to preserve the smallest decay forms like cracks and to avoid their merging with different elements.
Then, the training classes were defined and selected (Figure 4).The selection was performed by choosing the representative samples of different features (decay forms) among the segments created in the previous step and by assign them to classes before performing the supervised classification.Within the classification, the Support Vector Machine (SVM) was visually compared with other two algorithms designed also for this step, K Nearest Neighbor (KNN) and Principal Component Analysis (PCA), but both of them generated more noise in the image, especially confusing the crack class with other elements.SVM is a supervised classification method derived from statistical learning theory and it separates the classes with a decision surface that maximizes the margin between the classes.The surface is often called the optimal hyperplane, and the data points closest to the hyperplane are called support vectors (Chang and Lin, 2011).A threshold of 17 was set.This indicates the level of confidence that the closest segments represent the same class as that segment.Finally, the main output was exported.The same procedure with the same classes were applied on the three input images and the classification raster was exported for each input data.

RESULTS AND ASSESSMENT
OBIA results are shown in Figures 5, 6, 7, which correspond to RGB, RGB + CM and RGB + CM + New CS classified images, respectively.Visually, it can be observed that the OBIA results were improved by considering the RGB image stacked with the ACM Systems.In the original image, indeed, there is added noise and misclassification of many elements, particularly the spalling and the formwork lines.In order to quantitatively evaluate the results, the confusion matrices with the overall accuracy and the kappa coefficient were obtained considering as ground truth the where i is the class number; N is the total number of classified values compared to truth values; mi,i is the number of values belonging to the truth class i that have also been classified as class i (i.e., values found along the diagonal of the confusion matrix); Ci is the total number of predicted values belonging to class i; Gi is the total number of truth values belonging to class i.
The accuracy and metrics assessments are illustrated in Tables 1, 2, 3 for each input data.The overall accuracy resulted to be 64%, 67% and 69% while the kappa coefficient was 0.32, 0.36 and 0.37 considering, respectively, the RGB, the RGB + CM and the RGB + CM + NewCS images.

DISCUSSION
The aim of the paper was to test the applicability of experimental algorithms on a metric orthophoto to extract the main decay forms.From the results, what transpires is that the background was correctly classified in 73% of cases in the RGB image.This value increases to 75% and 77% considering the RGB + CM and the RGB + CM + NewCS, respectively.The major source of error was related to the washout.In some areas, this decay form has not a strong spectral response, making it similar to the one of the background.The cracks have been correctly classified over the 24%, 32% and 25% considering the 3 input images, but little percentages were classified as exposed rebars and spalling.This is understandable because the exposed rebars have similar spectral response to the cracks, and geometrically both elements generally have elongated shapes.As far as the spalling is concerned, it corresponds to the areas of concrete which have cracked and delaminated from the substrate for many reasons, like freeze thaw cycling, expansive effects of Alkali Silica Reaction, exposure to fire, or, as it is in this case study, because of the corrosion of embedded steel reinforcement bars.So, it is normal that spalling areas contain cracks or microcracks.The fact that, with the application of the NewCS, the percentage decreases compared to the RGB + CM image could be explained because this algorithm could not well recognize the edges of the thinnest cracks.The exposed rebars were correctly classified almost entirely, around 85% for all the input.Spalling and washout, instead were generally confused with each other, because both the decay forms coexist in the same portions.The best performance was obtained by the RGB + CM + NewCS image.However, the CM enhances of about 3% the overall accuracy when combined with the RGB image.By adding, also, the NewCS, the accuracy increases of about 5%.So, although these systems were born for different purposes, they can enhance both the geometry and the spectral contrast in a photogrammetric output, leading to higher accuracy also in the infrastructure detection field.For further comparison, the same methodology was applied using two processes commonly used in literature using the same concept: one algorithm to enhance the spectral contrast and another one the geometry of the original image.In this case, for contrast enhancement, the histogram equalization algorithm (Pizer et al., 1987) was applied.
The main idea of the algorithm is to use the whole available dynamic.A histogram over the image is computed and the whole dynamic is exploited, e.g. it tends to flat the histogram.This leads to obtain a gain for each bin that transform the original histogram into the flat one, and the gain is applied on the original image.To emphasize the geometry, the Sobel algorithm was applied.This filter uses the Sobel operator (Vincent et al., 2009) to calculate the image gradient and then finds the magnitude of this gradient vector.Therefore, the feature extraction was performed on two input images, one composed by stacking RGB + Histogram equalization algorithm bands and the other the RGB + Histogram equalization + Sobel algorithms.The confusion matrices, the overall accuracy and the kappa coefficient were calculated.To simplify, only the values of the overall accuracy and the kappa coefficient are here reported: 64% and 62%, 0.28 and 0.25, respectively.Not only are the values lower than the ones obtained with the experimental algorithms, but they are also equal to or less than the original RGB image.Therefore, there was not any particular enhancement using the literature algorithms.For better clarify, the obtained overall accuracy and kappa coefficient are diplayed in Table 4.

CONCLUSIONS
In conclusion, the aim of this work was to test new experimental algorithms on the orthophoto obtained by the photogrammetric process applied on a concrete pillar.The algorithms that the authors have used are called ACM Systems, and they have the ability to define the edges and distinguishing noise and salient forms within the image.It was found that they can be used on metric orthophotos to analyse concrete deterioration.The main purpose of this work was, indeed, to extract the decay forms in order to find a fast and automatic methodology helpful for the continuous monitoring of structures and infrastructures.
Although the analysis needs to be further investigated and improved, the first approach provided promising results.It was demonstrated that the final accuracy is enhanced more by using the two experimental systems, namely the CM and the New CS than the two common algorithms, i.e. the histogram equalization and the Sobel.It was proved that the ACM Systems are useful algorithms for territorial and cultural heritage analysis but also for concrete decay forms inspection.Future studies could be not only improve the obtained results by testing other machine learning algorithms that follow the statistic behaviour of the ACM, but also test the methodology on different concrete samples and decay forms.

Figure 1 .
Figure 1.Orthophoto chosen to perform OBIA analysis, composed by Red, Green and Blue (RGB) bands.

Figure 2 .
Figure 2. Application of New CS algorithm on the original RGB image.

Figure 5 .
Figure 5. OBIA applied on the original RGB image.

Figure 7 .
Figure 7. OBIA applied on the + CM + New CS image.

Table 2 .
1. Confusion matrix obtained by validating the OBIA on the RGB image with the ground truth.Overall accuracy 64%, kappa coefficient 0.32.Confusion matrix obtained by validating the OBIA on the RGB + CM image with the ground truth.Overall accuracy 67%, kappa coefficient 0.36.The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-1/W1-2023 12th International Symposium on Mobile Mapping Technology (MMT 2023), 24-26 May 2023, Padua, Italy

Table 3 .
Confusion matrix obtained by the OBIA on the RGB + CM + NewCS image with the ground truth.Overall accuracy 69%, kappa coefficient 0.37.

Table 4 .
Summary of the overall accuracy and kappa coefficient obtained in this work The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-1/W1-2023 12th International Symposium on Mobile Mapping Technology (MMT 2023), 24-26 May 2023, Padua, Italy