Study on Fault Monitoring Technology of Photovoltaic Panel Based on Thermal Infrared and Optical Remote Sensing

: Rapid access to the operating status of Photovoltaic (PV) panels and troubleshooting can save management and maintenance costs for the development of PV power plants, which is important for PV power plant management and power generation capacity assurance. The use of remote sensing technology to identify the faults of photovoltaic panels has developed rapidly, however, current research usually relies only on a single optical data source to identify and count the area of PV panels in a PV electric field, although there are literature on PV panel fault detection, only the surface fault identification of PV panels is tested, while the internal faults (such as panel bad points or bad lines) cannot be identified because of the limitations of optical remote sensing. In this paper, a photovoltaic panel fault monitoring technology based on multi-source remote sensing is proposed. The optical and thermal infrared hybrid data combined with deep learning technology are used to achieve rapid and accurate fault identification and localization of PV panel arrays. It can not only automatically identify PV panels that are obscured by dust and foreign objects, but also locate PV panels that have bad dots or bad lines, which greatly improves the ability and effectiveness of remote sensing PV panel fault monitoring. The high-resolution unmanned air vehicle (UAV) optical image and thermal infrared image are applied in this experiment. The Mask R-CNN algorithm is used to accurately locate and number the photovoltaic panel of the optical image. Then, the fault scene classification model is established for the multi-type fault characteristics of the optical image and thermal infrared image within the panel range, so as to identify five types of faults, such as dust cover, branch cover, bird droppings cover, internal bad points and bad lines of PV panel, which effectively solves the problem that the single optical remote sensing image cannot identify the internal component faults of the photovoltaic panel under normal conditions.


Introduction
Photovoltaic solar energy as a green and clean energy is one of the main sources of energy for the world's low-carbon development (OREGAN and GRATZEL, 1991).As the basic unit of PV power plant, PV panel and its operational status directly determines the power generation efficiency and market benefits.For photovoltaic power systems, it is of great significance to quickly obtain the operating status and troubleshooting of PV array facilities (Andò et al., 2015).It can not only save management and maintenance costs for the efficient development of PV power stations, but also provide an important guarantee for China 's low-carbon development and the realization of the "double carbon" strategic goals of "carbon peaking and carbon neutrality goals".With the rapid development of remote sensing wide-area monitoring and multisource collaborative recognition technology based on artificial intelligence, related technologies (Bradbury et al., 2016) have been gradually applied to PV power station array detection and system maintenance, and have become an important research hotspot for remote sensing green production capacity applications.
The advantages of remote sensing technology, such as low cost, non-contact, large area and multi-source observation, have been gradually applied to the inspection and management of photovoltaic power plants, and a series of achievements have been achieved (Song et al., 2018;Tan et al., 2023).The remote sensing monitoring of PV power station is usually based on a single high-resolution optical remote sensing data source.The semantic segmentation or target detection algorithm based on deep learning is used to accurately locate the peripheral contour of the PV array panel in the photovoltaic power station, and then calculate the use area and occupation area of the photovoltaic array in the whole area (Da Costa et al., 2021).Compared with the traditional remote sensing image classification algorithm, this remote sensing monitoring algorithm has achieved higher recognition accuracy, and is also widely used in the area statistics, change monitoring and photovoltaic capacity supervision and maintenance of photovoltaic array panel in photovoltaic power stations.However, optical remote sensing monitoring can only be applied to the statistics and land cover planning of the normal operation of photovoltaic arrays, as well as the identification of faults or hidden dangers caused by external objects such as excessive surface area ash, bird manure attachment or branch cover in photovoltaic array blocks.However, due to the fact that the photovoltaic array with internal faults (short circuit, partial aging, crack) is not obvious in the optical remote sensing image (Murillo-Soto and Meza, 2021), it is impossible to detect and troubleshoot this part of the fault.
With the rapid development of thermal infrared remote sensing image technology, the abnormal infrared band characteristics presented by the internal damage parts of the photovoltaic array board can be used to detect the internal faults of the photovoltaic array board.At the same time, it has achieved good monitoring results in a small range, which makes up for the lack of optical remote sensing monitoring capabilities (Ge et al., 2022).Therefore, only relying on a single remote sensing data source to detect the fault of photovoltaic array can not achieve the goal of fault detection and functional maintenance of remote sensing photovoltaic power station.It is necessary to use the support of multi-source remote sensing joint data such as optical and thermal infrared.In this paper, optical remote sensing data and thermal infrared remote sensing data in the same area are used as mixed data sources, combined with deep learning target detection algorithm and scene classification algorithm to realize intelligent photovoltaic panel fault identification, which opens up new applications and ideas for the application of multisource remote sensing.

The Algorithm Flow
In order to accurately identify the state of each PV panel, the minimum unit of the PV panel array is first located and identified, and then the multi-source remote sensing image features in each panel are extracted.According to the fault scene classification model, the state of each panel is judged, so as to complete the identification of PV panel faults in all regions.It is divided into the following three steps (as shown Figure 1) : Firstly, in the experimental training area, optical remote sensing images and deep learning target detection technology (Mask R-CNN (He et al., 2018)) are first used to identify and accurately position PV panels in the PV array.Secondly, by using the PV panel positioning information and multi-source remote sensing images, the PV panel target optical remote sensing and thermal infrared remote sensing fault image sample sets are made respectively.The optical remote sensing fault identification samples include surface dust accumulation and foreign object covering faults, and the thermal infrared remote sensing fault identification samples include thermal infrared abnormal spot (bad point, bad line) faults.Based on the above two sample sets, the PV panel fault identification models of optical remote sensing and thermal infrared remote sensing are trained respectively.Finally, in the experimental verification area, the PV panel positioning model and fault identification model are combined with multi-source remote sensing images for accuracy verification.

Photovoltaic Panel Positioning based on Optical Remote Sensing
Since the photovoltaic panel units are tilted (as shown Figure 4 ) and the subsequent fault analysis needs to rely on the range information of each panel based on this step, the target detection using only four-corner box positioning cannot meet the demand.The Mask R-CNN algorithm with mask segmentation branch is used to realize the instance identification of photovoltaic panel units.Mask R-CNN is applied to the instance segmentation of photovoltaic panel units, which is mainly optimized in three aspects: Firstly, according to the presentation characteristics of photovoltaic panel targets in remote sensing images, in the data enhancement part, the hue and brightness enhancement are deleted, and the rotation enhancement algorithm is added to strengthen the samples according to 15 degrees of rotation (as shown Figure 2).In the feature extraction part, swin-transformer is used as the feature backbone.Based on its powerful feature extraction ability, it can be better adapted to the high-resolution UAV photovoltaic panel discrimination.The use of a hybrid loss combining DiceLoss and CrossEntropyLoss (as shown in Formula ( 1)) in the semantic segmentation network header can better meet the boundary extraction accuracy and recall requirements of the photovoltaic array mask.
, y is the true value of the label, ' y is the model inference value, j is the number of input sample categories,  is a very negative value, to prevent the denominator from being 0 in the calculation, and can also play a role in smoothing LossDice; is the probability of the predicted value;  is the error harmonic coefficient, and 0.5 is used in this paper to indicate that the two losses contribute the same amount of semantic information to the detection of photovoltaic panel targets.

Fault Scene Classification based on Multi-source Remote Sensing
Based on the imaging mechanism, the optical remote sensing image can identify the occlusion faults such as dust, branches and bird droppings on the surface of the photovoltaic panel.The thermal infrared remote sensing image can identify the bad points and bad lines inside the photovoltaic panel.In view of the fact that a photovoltaic panel may have the above two types of faults at the same time, the optical-based fault scene and the thermal infrared-based fault scene classification model are established respectively.The photovoltaic panel fault classification system is shown in Table 1.
Table 1.photovoltaic panel fault classification system

PV fault identification model based on optical remote sensing
Aiming at the resolution of optical remote sensing images, the fault sample size obtained by the previous instance segmentation is not unique and unevenly distributed.The number of samples stored in the fault scene usually accounts for a relatively small proportion (less than 1/100).Therefore, the sample must be normalized first.The length and width of the sample are uniformly adjusted to 330 * 330, and the sample enhancement algorithm (does not include hue, chroma and noise enhancement algorithm) is used to equalize the number of four types of fault samples in the optical image.In the feature extraction stage, ResNet101 is used as the backbone network, and the final fully connected layer output is adjusted to 4 to establish an optical remote sensing photovoltaic fault detection classifier.The loss function uses CrossEntropyLoss.

PV fault identification model based on thermal infrared remote sensing
Due to the low spatial resolution of thermal infrared remote sensing, the fault samples of thermal infrared remote sensing photovoltaic panels obtained based on the instance segmentation range are small, and the length and width are usually only 23 * 25 pixel size.In order to enhance the recognition ability of the network, the bicubic interpolation algorithm is used to upsample it and unify it to 330 * 330 size.
For the class imbalance problem, the enhancement algorithms such as rotation, color, and brightness are also used to achieve the balance of the number of samples of "good_inf, bad point, and bad line'.Because the thermal infrared features are not obvious, the feature backbone of the scene classification uses the DenseNet201 network.The output of the last fully connected layer is modified to 3, and the loss function also uses CrossEntropyLoss.

Experimental Data
The optical and thermal infrared remote sensing data used in this paper are UAV images with spatial resolutions of 0.5 centimeters and 4 centimeters respectively.The preprocessing of optical remote sensing images is through ground control point/tie point (GCP/TP) collection, bundled adjustment, terrain ortho-rectification and clipping.True color images with RGB bands are obtained.The thermal infrared data is based on the optical image as the reference data, through the control point collection and regional bundled adjustment, the correction result is corresponding to the optical remote sensing position.Thermal infrared remote sensing images are rendered by standardized unified layered color rendering to form false color images in RGB format.As shown in Figure 3, the abnormality of photovoltaic panel power generation components will lead to thermally sensitive abnormal reactions, which are usually manifested as bad points or bad lines, but the optical images at the same position cannot detect this fault.
In Figure 3, the upper row is an optical image, and the lower row is a thermal infrared image at the same position.The red frame in the left column is marked as the fault of the bad line in the thermal infrared, and the blue frame in the right column is marked as the fault of the bad point in the thermal infrared detection.However, the PV panels with internal faults were all seen to be intact from the optical images.
Figure 3. Optical and thermal infrared data of UAV

Experimental area
In this paper, two centralized photovoltaic power plants located in Jianhu County, Yancheng City, Jiangsu Province (covering an area of 16336.8m 2 and 63583.7m 2 , respectively) are selected as the main research objects.A total of two optical and corresponding thermal infrared images (as shown in Figure 4a and 4b) are obtained for the training of instance segmentation model and fault scene recognition model.At the same time, a photovoltaic power plant in Zhangjiagang City, Suzhou City is selected as a validation area, covering an area of 7858.7m 2 (as shown in Figure 4c).

(a)
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-1-2024 ISPRS TC I Mid-term Symposium "Intelligent Sensing and Remote Sensing Application", 13-17 May 2024, Changsha, China   According to the method strategy in the second section, the fault detection in this section is divided into two stages.The first is to use the optical remote sensing image to identify the fault scene.Secondly, the thermal infrared remote sensing is used to identify the fault scene, and the two are implemented in series.Finally, the multi-dimensional attribute table of vector data result is used to obtain the final fault information.

Photovoltaic panel fault training based on optical remote sensing
On the two optical data in the training area, the results of the automatic positioning and numbering of the photovoltaic panel are used to manually interpret the optical remote sensing fault markers for each panel (good_opt, dust, branches, bird droppings, as shown in Figure 7), and then the optical remote sensing photovoltaic panel fault scene sample set Sopt_fault is made.Because the good PV array units account for a large proportion, the category samples are extremely unbalanced.For example, the good category is more than 8000 times that of the dust accumulation fault.If it is directly used for training, it will lead to serious inter-class imbalance.The sample enhancement algorithm is used to balance the categories, among which the fault categories are few.A variety of enhancement methods are used to expand.If there are many fault categories, a few methods are used.Finally, the sample orders of magnitude of the four categories are basically the same ( as shown in Table 2 ).

Photovoltaic panel fault training based on thermal infrared remote sensing
Using the same method, based on the two scenes of thermal infrared data in the training area, the thermal infrared image blocks are extracted by using the results of automatic positioning and numbering of Mask R-CNN, and the thermal infrared fault markers (good_inf, bad points, bad lines ) are carried out according to the feature judgment.Because the sample range of photovoltaic panel fault scene based on thermal infrared remote sensing is small, it is first upsampled to 330 * 330 size, and then through the sample enhancement algorithm, the problem of intra-class imbalance of photovoltaic panel thermal infrared fault scene is solved.The enhancement effect of some samples is shown in Figure 8, and the final sample SInf_fault statistics are shown in Table 3.

Inferencing photovoltaic panel faults based on multi-source remote sensing verification area
The experimental process of automatic fault identification of photovoltaic panels in the verification area is as follows : Firstly, the positioning range and number of photovoltaic panels are obtained by using the method of section 3.

Conclusion
In this paper, an intelligent detection algorithm of photovoltaic array fault based on multi-source remote sensing data is proposed.In the method, the fast and accurate positioning and unique code number of photovoltaic array are realized by using the wide-area non-contact advantage of remote sensing combined with deep learning technology, which improves the management efficiency of array board management of photovoltaic power station.Secondly, combined with optical remote sensing, the surface fault identification of intelligent photovoltaic array panel is realized, which provides an efficient strategy for regular cleaning and accurate surface maintenance.
Combined with the automatic high-precision internal fault detection of photovoltaic panel realized by thermal infrared remote sensing, the hidden fault of photovoltaic array is explored, which improves the quality of fault detection of photovoltaic array.This paper proposes a new model for collaborative observation and application of multi-source remote sensing images, which opens up and innovates the field of multi-source remote sensing and intelligent algorithm study.

Figure
Figure 1.The algorithm flow

Figure 4 .
Figure 4. Image data of training area and verification area 3.3 Photovoltaic panel instance segmentation Two optical images in the training area are taken as the main samples.The labeled data are in ArcGIS Shapefile format.The photovoltaic panel target is marked as a real semantic frame (not a rectangular frame).Due to the large size of the whole image (19,653 * 30,798), this paper uses a sliding frame of 1024 * 1024 and a training sample.The moving step size is 512, and the generated samples are shown in Figure 5, and the entire training area generates a total of about 10,915 photovoltaic panel instance segmentation sample label pairs.

Figure 6 .
Figure 6.Verification area photovoltaic panel inferencing and numbering

Figure 7 .
Figure 7. Photovoltaic panel fault scene classification sample set based on optical remote sensing

Figure 8 .
Figure 8. Photovoltaic panel fault scene classification sample set based on thermal infrared remote sensing Photovoltaic panel with bird droppings cover and bad point

Figure 9 .
Figure 9. Photovoltaic panel fault identification results based on multi-source remote sensing joint inferencing in the verification area

Table 2 .
Statistical table of the number of optical fault samples before and after enhancementBased on the training sample Sopt_fault, ResNet101 is used as the feature extraction backbone, and the cross entropy is used as the loss function to update the weight gradient by back propagation.