ARTIFICIAL INTELLIGENCE FOR REAL-TIME MONITORING OF LOGS ON THE MADEIRA RIVER: A CASE STUDY ON JIRAU HYDROELECTRIC PLANT

: The Jirau and Santo Antônio hydroelectric plants in Rondônia implemented a methodology using high-range cameras and artificial intelligence technology to address the challenge of managing logs transported by the river during floods. By applying machine learning techniques and neural networks, the system automatically monitors log transport and accumulation. Python 3, along with libraries like OpenCV, PIL, Numpy, and Pytorch, was utilized for efficient implementation. The methodology includes frame selection, log and debris segmentation, perspective correction, and log counting. Training was conducted using annotated images, and the detection process involved color segmentation, noise removal, and morphological operations. The calculated log and debris occupancy results were stored in a SQL database and presented on Power BI dashboards. The system aims to improve log management, ensuring power generation and ecological order are safeguarded.


INTRODUCTION
In general, the Amazonian rivers are recognized for the large number of logs and floating and submerged debris they transport, especially during the flood season. The ecological role that this material plays in the aquatic environment is associated with the colonization and reproduction dynamics of various aquatic organisms, as well as carbon export. One of them is the Madeira River, which is one of the main tributaries of the Amazon River and stands out due to its ecological, hydrological, and socioeconomic importance in the Amazon rainforest region of South America. With a length of approximately 3,250 kilometers (2,020 miles) and a vast drainage basin spanning parts of Brazil, Bolivia, and Peru, it is recognized as one of the largest and most consequential rivers in the Amazon Basin. In this context, the Jirau and Santo Antônio hydroelectric plants, located on the Madeira River, face many challenges in the operation and management of the large quantity of logs transported by the river during flood periods. The difficulties faced are generally associated with the efficiency of Log Boom systems available on the market, which are used to direct the logs along the reservoir to the spillway structures designed for their downstream discharge from the dam. These challenges require the search for more efficient and appropriate solutions for quantification of logs in these hydroelectric plants, to ensure safe and effective operation of the structures, minimize environmental impacts, and optimize the utilization of water resources in the region. The initial attempt to quantify fluvial logs using sensors took place in France in 2009, in the Ain River (Macvicar et al., 2009). However, due to the limitations of the technological resources available at that time (low-mobility cameras and drones), the monitoring data was obtained at insufficient temporal scales to support a predictive model of log transport and a monitoring program. Since 2017, technological advancements have allowed for an increased temporal resolution in monitoring the dynamics of fluvial logs, as evidenced by studies conducted in the Rhône River, France (Benacchio et al., 2017). However, the establishment of continuous monitoring plans for fluvial logs still faced challenges, including factors such as climate, lighting conditions, image coverage area, camera positioning, and high variability in woody material density within reservoirs. More recently, other techniques have been employed to improve the data obtained from cameras. (Spreitzer, Tunicliffe, and Friedrich, 2019) utilized SFM photogrammetry method to assess the volume of surface log accumulations in the Whakatiwai River, New Zealand, including the volumetrics of other materials retained among the log components. In Brazil, the use of photogrammetry techniques for the physical characterization of woody materials has been limited to wood storage yards. (Figueiredo et al,2016) employed RGB cameras mounted on Remotely Piloted Aircraft (RPAs) to obtain highdensity point clouds and estimate the volume of wood stored in storage yards. Currently, the use of advanced Artificial Intelligence (AI) techniques, such as deep learning, to monitor rivers through cameras continues to be a promising area of research. AI, combined with image processing and data analysis, enables the development of automated systems capable of detecting, identifying, and tracking specific objects and phenomena in the captured images. In the context of the Madeira River and the Jirau Dam, these techniques have been applied to monitor and identify floating logs, debris, water flow patterns, and other relevant features. Deep learning algorithms have been trained on large datasets, allowing the system to learn to recognize specific patterns and perform real-time automated analysis. Additionally, this system has assisted in the early identification of unwanted events, such as excessive accumulation of logs in the structures, enabling faster and more efficient cleaning measures to be taken. However, it is important to emphasize that the successful implementation of these techniques was made possible due to adequate infrastructure, including the strategic positioning of cameras, efficient processing of large volumes of data, and the ability to integrate monitoring results into decision-making dashboards. Therefore, this project aims to introduce a methodology that utilizes cameras and Remotely Piloted Aircraft (RPA) for monitoring and quantifying logs at the Jirau Hydroelectric Power Plant.

METHODOLOGY
The development of methodologies utilizing RPA e cameras sensors, artificial intelligence techniques, process automation algorithms, relational databases, and dashboards has been undertaken to monitor features in Jirau Hydroelectric Power Plant structures, including logs and debris. These methodologies encompass three primary activities: sensor-based monitoring, artificial intelligence analysis, and result presentation. Using RPA equipped with image sensors, flights are conducted to capture images of the power plant structures. These images are processed using artificial intelligence algorithms to identify relevant features such as logs and debris. In addition to RPA imaging, video cameras were installed at strategic points for continuous monitoring of the power plant structures. These cameras capture real-time footage, enabling the detection of events and identification of potential issues or anomalies. The data collected through RPA imaging and video cameras are stored in a relational database. Through process automation algorithms, this data is processed and transformed into relevant information. The information is then presented through dashboards and intuitive interfaces, allowing users to visualize and analyze the results clearly and efficiently.

Data Collection:
The monitoring campaigns consisted of on-site surveys using photogrammetric sensors onboard RPA to monitor the accumulation of logs on the structures of the Jirau Hydroelectric Power Plant as presented in Figure 01. The four areas mentioned above represent the main structures in relation to log retention, which is the focus of the R&D study. The table 1 summarizes the surveyed areas and the adopted nomenclature pattern. This table provides an overview of the specific areas that were surveyed and monitored during the project. Each area is identified by a unique name or code according to the adopted nomenclature pattern. These pieces of information facilitate the organization and reference of the monitored areas throughout the study. Regarding the aerial surveys, two types of flights were conducted: manual and autonomous. Manual flights were performed without any support software, solely relying on the pilot's expertise. The purpose was to conduct a panoramic inspection of the structures through visual verification of debris retention and occurrences in log occupancy structures, as presented in Figure 2.  On the other hand, according to the example presented in Figure  3, autonomous flights were conducted with the support of software responsible for communication with the RPA, standardizing parameters such as image overlap, flight altitude, speed, and route to be followed. Daily, two campaigns were conducted, one in the morning and another in the afternoon. The first campaign started around 7:30 AM. This timing was chosen to anticipate the interference caused by the operation of ferries in clearing accumulated logs during the night. Autonomous imaging of the structures was conducted to obtain orthophotos and calculate the concentration area of the logs.
The same procedure is repeated in the afternoon campaign, around 4 PM. It is worth noting that, at this time, priority was given to areas where no ferry was operating to avoid interference during the flyover and discrepancies in logs and debris occupation between what was recorded and what was retained. The flights were conducted at an altitude of 120 meters with a lateral and longitudinal overlap of 80%.
2.1.2 Orthophoto generation: In the context of water surface monitoring, there may be situations where images capture only the water without objects that allow for correlation between the images. This can make it challenging to generate products such as orthophotos. However, the presence of logs or other elements, such as riverbeds, plays a crucial role in the generation of these products. Logs and other elements present in the monitored area serve as visual references that enable the establishment of homologous points between the images, thereby facilitating the necessary correlation for rectification and orthophoto generation. These elements provide reliable reference points that help create a consistent foundation for the photogrammetric intersection process. Therefore, the presence of logs and other elements in the monitoring area is crucial to ensuring the quality and accuracy of orthophotos. These elements allow for the establishment of image correlation, enabling rectification and the creation of a faithful representation of the imaged region. Figure 4 illustrates the produced orthophotos, which served as the starting point for the vectorization process to extract the area.

Orthophoto vectorization:
To calculate the area of the logs in the orthophotos, a process of vectorization of log features was performed. This involved digitizing the areas corresponding to the logs in the orthophotos, transforming them into vector structures. Initially, geoprocessing was conducted manually using QGIS, an open-source, cross-platform geographic information system (GIS) software that allows for the visualization, editing, and analysis of georeferenced data. In order to enhance the orthophoto vectorization process, specific artificial intelligence techniques using computer vision were employed. Computer vision encompasses a wide range of image processing applications, including facial recognition, object detection, image classification, image segmentation, motion tracking, among others. To accomplish this, machine learning and deep learning models were tested. The primary models employed were Random Forest, which generates random decision trees to establish decision rules, and Convolutional Neural Networks (CNN), a model specifically designed for pattern recognition in images. The entire process was implemented using the Python programming language, utilizing various modeling and geoprocessing libraries, including Shapely, GDAL, Scikit Learn: and PyTorch. Initially, for training prediction models within both frameworks, appropriate data sampling was required to ensure that the data representation accurately reflected the target population. This was crucial to avoid biases and distortions in subsequent analysis and predictions. The sampling process involved highresolution orthophotos with a substantial number of logs, the objects of recognition, following a manual feature identification process through manual vectorization in the QGIS software. The subsequent stages encompassed the creation of training, validation, and test datasets, model training, prediction algorithms, and georeferencing of the results. The Figure 5 shows the result of the automatic vectorization method.

Figure 5. Result of the automatic vectorization method
These monitoring campaigns were conducted from December 2020 until the completion of the project in April 2023. During this period, over 2749 orthophotos were obtained for area counting, along with more than 500 panoramic photos and several video shots. These records enabled the monitoring and assistance in managing the floods of the Madeira River throughout this period, providing valuable information to the responsible team. When comparing the three campaigns, a significant difference in log accumulation is evident for CF1 over the periods. This difference can be attributed to the rupture of the log boom on the right bank, which occurred at the end of January 2021. This event resulted in significant changes to the remaining log boom structure, directly affecting the flow of logs and debris. As a result, a higher log accumulation was observed during the second campaign. Furthermore, there are periods with a significant reduction in log accumulation, which is a result of the cleaning operations performed on the power plant structure. During the second campaign, there were few spillway openings, contributing to the decrease in log accumulation.
In the third campaign, it was observed that the installation of the new log boom, as Figure 6, had significant benefits in terms of log containment and spillway operation. During spillway openings, there was no need to use barges to assist in the removal of materials, as was the case with the previous log boom. This demonstrates the efficiency of the newly installed structure.

Figure 6
New log boom right bank

Camera monitoring
To identify the length, area, and count of logs and debris, computer vision techniques were employed using frames from real-time monitoring camera videos. Initially, the structures of the hydroelectric power plant were categorized into two groups: • Structures with log accumulation (Log boom right bank LBMD, Spillway (VT), Powerhouse 1 (CF1), Log boom left bank LBME) • Structures with log passage -continuous flow of logs (Abunã Bridge (PA), Jirau Waterfall (CJ)) The flowchart in Figure 7 outlines the methodology's development process, including the installation of cameras, realtime video acquisition, generation and testing of prediction models, and continuous processing to obtain values for area, length, and log count. The programming language used was Python, version 3, which provides compatibility with the latest versions of libraries and frameworks for developing advanced deep learning models using neural networks, as well as routines for task automation. The main open-source libraries used were as follows, with the complete list available in the project's development repository on GitHub: OpenCV: This library was employed in computer vision, facilitating the techniques applied in image and video processing, detection, tracking, and analysis. PIL (Python Imaging Library): This library provides support for various image operations and was used for image manipulation. Numpy: This library offers a wide range of routines for arithmetic and logical operations on arrays, among other functionalities. Pytorch: This is one of the primary tools for developing deep learning models. CUDA and cuDNN: These Nvidia libraries are notable for accelerating processing on Graphics Processing Units (GPUs) and optimizing the performance of deep neural networks.

Camera installation:
An overview of the application of video cameras for monitoring is presented in Figure 8, which shows the image capture element (camera on the left) and the boundary ranges, in space, for the classes of Identification, Recognition, Observation, Detection, and Monitoring.

Figure 8 Image processing ranges for object tracking
Six strategic points were defined for the installation of cameras, with a total of 11 cameras, aiming to observe regions of the Madeira River and the structures of the UHE. Figure 9 presents the defined points for 4 locations, while the other 2 are located at the Abuna Bridge and Jirau Waterfall..

Figure 9
The installation points for the camera towers The installed cameras are connected to a Network Video Recorder (NVR), which is responsible for storing the video footage from the monitoring system. The cameras are connected using network cables, fiber optic cables, and Wi-Fi transmitters. This setup ensures efficient data transfer and reliable connectivity between the cameras and the NVR for seamless video recording and storage. 2.1.5 Training dataset development: To enable the training of the tested segmentation models, two datasets were created: one focused on object detection and the other on segmentation. The object detection dataset was developed using the LabelImg tool. It consists of 93 annotated images. All annotations were manually performed, classifying objects into two categories: "log" and "debris". Rectangular bounding boxes were used to delineate the logs and debris in the images. An example of an annotated image using this tool is shown in Figure 10.

Figure 10
Example of an annotated image using LabelImg While there are several strategies for object detection, they have a limitation when it comes to accurately calculating the area and volume of logs and debris. Object detection allows for identifying the location of logs and piles of debris in the images; however, it is based on the use of rectangular bounding boxes. This approach includes the surrounding water area along with the objects of interest, which reduces the precision of the area calculation. The segmentation approach offers an alternative that overcomes the limitations of object detection strategies. Segmentation methods enable the identification of areas corresponding to selected classes, regardless of their shape. This characteristic allows for more accurate calculation of log and debris areas. To train the segmentation models, it was necessary to annotate all objects of interest with polygons. The annotation process was carried out using the labelme tool. An example of an annotated image using this tool is shown in Figure 11.

Figure 10
Example of an annotated image using Labelme The following categories were used for annotation: debris, dense debris, log, partial log, log boom, boat, and bank. The division of the debris category into two subcategories was done to enable a more precise volume calculation, allowing for separate weighting in situations with scattered debris and large debris piles. The log category was also divided for the same purpose. In various situations, logs may be occluded, and this category allows for compensation of this occlusion, reducing the error in the final volume calculation. One of the limitations encountered during the annotation process is the time required to annotate an image accurately. To optimize this process through semi-automated annotation, after training the neural network with an initial version of the dataset, new images were passed to the algorithm to be segmented, as shown in Figure 12.

Figure 12
Segmented image After segmentation, the generated mask was transformed into a JSON file that contains the boundaries of the segmentation polygons. These files were opened in Labelme and adjusted as needed, reducing the time required for annotation.
2.1.6 Continuous processing of monitoring videos: To process the real-time monitoring videos obtained from camera infrastructures, a continuous computational processing flow was created, as shown in Figure 12. For frame selection, the Lucas Kanade algorithm was chosen, as it provided the best performance in terms of frames per processing time, achieving a processing rate of 120 FPS. The Lucas Kanade algorithm is focused on the task of optical flow, enabling the identification of motion in various points of the images. Tracking is performed on pre-defined points, and Figure 13 shows the application of this method within the Region of Interest (RoI) of a test image obtained on-site. In an oblique image, the pixel size on the ground, known as Ground Size Dimension (GSD), varies as objects move away from the camera, while in an orthogonal image, the GSD remains constant. To minimize computational cost, tests were conducted using perspective transformation algorithms from different libraries: OpenCV, Pillow, and Scikit-image. The OpenCV algorithm was considered the most suitable for this application. For the perspective transformation tests, arbitrary control points were used since the goal was to compare the performance of different algorithms. After the cameras were installed, specific points of interest were identified for each camera to determine the precise parameters for perspective transformation.
2.1.8 Area calculation and logs quantity: After the perspective correction step, an image is obtained where all pixels correspond to the actual size on the ground and represent the previously defined classes obtained through the application of the segmentation model. The output of the FCN-8s model is a mask where each color represents a class in the problem. As a result, no additional corrections are needed for calculating the area occupied by logs and debris. The area calculation can be performed by counting the pixels corresponding to each class. To convert the size in pixels to the size in meters, it is necessary to define the ground size represented by each pixel and perform a multiplication. From the segmented images, a contour detection method is applied, which allows counting the existing logs in each of the new images. The visible size of the log is identified by calculating the Euclidean distance between the endpoints of the log, obtained through its contour. In addition to the log size, we capture information related to its diameter. 2.1.9 Database: The data generated by the system was stored in a relational database using the SQL Server management system. By connecting the codes to the database, it became possible to save and access the results in real-time. Subsequently, Power BI was chosen as the tool to present the counting and accumulated area results of the logs. The dashboards were created with the objective of ensuring an efficient and personalized presentation of the data, as shown in Figure 14.

SUMMARY
The monitoring methodologies adopted at Jirau HPP, through drone flights and camera installations, have proven to be effective in data collection and generating relevant information for management and decision-making. With consistent results, these approaches have contributed to the monitoring of Jirau HPP activities, providing accurate data, enabling detailed analysis, and facilitating preventive and corrective actions. The use of advanced technologies such as neural networks and automated processing systems has optimized processes, increased productivity, and provided continuous and real-time monitoring. Therefore, monitoring and data analysis become essential for the safe and efficient operation, ensuring environmental preservation and the supply of quality energy