AUTOMATING THE MEASUREMENT OF RED CORAL IN SITU USING UNDERWATER PHOTOGRAMMETRY AND CODED TARGETS

A photogrammetry tool dedicated to the monitoring of red coral populations in situ has been developed by LSIS in Marseille (France). This tool is used to collect in an efficient and precise manner key data for the study of the population dynamics of red coral. In selected red coral populations, scuba-divers obtain a series of photographs from the permanent plots (about 2 m2) on an annual basis. To facilitate the photographic sampling and measurements, the scuba-divers use a 20x20 cm quadrat to cover the permanent plots. The analysis of the photographs provides reliable measurements on colony sizes (basal diameter and maximum height), occurrence of breakage of colonies and the occurrence of necrosis. To minimize the divers’ tasks during the acquisition phase, we opted for stereoscopic acquisition using a single device to easily adapt the measurement procedure to the scene configuration. The material is quite light, one camera and two electronic strobes and a simple procedure with two photographs taken for each site. To facilitate the measurement phase of colony sizes; the exploitation of photographs consists of four key steps: orientation, scaling, measurement of the characteristic points of coral colonies and result validation (checking measurement consistency to detect possible errors in measurement or interpretation). Since the context of the shooting can vary widely, dominant colors, contrast, etc. may often change. In order to have a stable and common reference in all photographs independently of the site, we decided to always include a quadrat in the scene which then will be used for the orientation and scaling phases. The automation of orientation and the lack of constraints to adapt the analytical technique to the features of each site offer the possibility to multiply field surveys and to measure hundreds of quadrats from several different populations in a very efficient manner. The measurement results are exported into a spreadsheet application and integrated into the biologists’ workflow. The results obtained from different red coral populations displaying contrasting characteristics (small versus large colony sizes) are presented and discussed at the end of this article.


INTRODUCTION 1.1 Measuring coral in situ
The Mediterranean red coral Corallium rubrum (Linnaeus, 1758) is a slow-growing coral which can reach 50 cm in height and live for up to 100 years (Garrabou & Harmelin 2002, Marschal et al. 2004).The red coral is mainly distributed in the Mediterranean Sea and forms dense populations in dim-light habitats between 10 and 600m.Its red calcareous skeleton has been appreciated for its use in jewelry since ancient times.Current populations are dominated by small sized colonies except in populations within marine protected areas (MPAs) or in deep water where harvesting activities are limited or never occurred.Moreover, red coral is also suffering from the effects of climate warming which can severely increase mortality rates.Finally, scuba divers often unintentionally damage red coral colonies with their gear, making it harder for the coral to recover from other disturbances.Monitoring the current state of red coral populations is a basic step in designing sound management and conservation plans for this threatened species.In situ surveys in red coral populations are difficult since they usually develop in deep water (>20 m depth) limiting the diving time and in semi-dark habitats (overhangs, cave entrances, vertical walls).In addition, red coral are fragile, rending difficult the application of direct measurements as is currently done for other benthic species.Underwater photogrammetry has been successfully applied to the monitoring of red coral populations (Ledoux et alii, 2010;Linares et alii, 2010).However, the measurement of colonies at different times is time-consuming and makes it difficult for large-scale applications.

Related work
In recent years various teams of biologists have experimented with photogrammetry or computer vision (the difference is more subtle) to measure, count, describe living organisms or the seabed itself.From a light photogrammetric approach (Sánchez et alii, 2009)in deep water to the use of a specifically-designed, sophisticated underwater vehicle (Singh et alii, 1999)or, since a long time, using specific stereoscopic photography equipment (Hollister, 1984), biologists worked hard to measure benthic comunities using photogrammetry (Abdo et alii, 2006;Madjidi & Nagahdaripour, 2003;Sanchez et alii, 2009;Sánchez et alii, 2009;Shortis et alii, 2007;Singh et alii, 1999).Shortis (Shortis et alii, 2009) provides a clear and synthetic overview.We can note a similar evolution in marine biology and underwater archaeology: geographical data must now be considered and results have to be linked with geographic information systems (GIS) ((Canese et alii, 2009;Giusti et alii, 2009)).In this same direction, photogrammetry and GIS approaches can be used for the spatial analysis of red coral colonies in relation to genetic distribution.We can look at the work of Jean-Baptiste Ledoux on this topic (Ledoux, 2010) page 10.The use of photogrammetry allows a better approach to study spatial distribution compared to the traditional approach used in this community.(Bythell et alii, 2001)

Introduction
In this paper, we present new developments on how to avoid manual time-consuming pre-processing procedures through the automation of orientation and scaling of photographic records.In this way, marine biologists (the experts) only focus on the measurements of red coral colonies.We present a semi-automated method composed of two main steps.The first step involves the orientation and scaling of a set of photographs of the entire colony and is fully automatic.It includes 2D point detection, photo orientation, and the scaling of a local reference system definition.The second step starts after the first one and uses its results, i.e. a set of oriented photographs that offer the biologist an interface developed for surveying relevant points on red coral in order to compute the needed data.
Figure 1.Four photographs taken on site showing the same small 20x20 cm zone.

On site
To carry out the surveys, we adopted two sampling schemes: (i) permanent plots and (ii) random quadrats.
(i) For the permanent plots were set up using PVC screws fastened to holes in the rocky substratum.Each plot was variable in length, depending on the complexity of the substratum, and 40 cm wide.The total area covered in each site was about 2 m 2 which allowed monitoring hundreds of colonies.Transects were photographed using different reflex cameras (Nikon D70, Nikon D300) with a housing and two electronic strobes.In each sampling a cord was deployed between the screws to help positioning the 20x20 cm quadrats sequentially along the cord (above and below the cord length).At each position, two photographs from each quadrat (using two slightly different angles, approximately 30°) were obtained.
(ii) In the random quadrat sampling scheme, we used the same type of quadrat (20 x 20 cm) and photographic equipment as above but it was positioned randomly within the populations.Regardless of the sampling scheme, from the photograph pairs we were able to estimate the basal diameter, the maximum height and the number of branches of each colony found within the permanent plot.Moreover, we were able to calculate the density of colonies, the necrosis rate, the breakage of branches and the presence of signs of recent harvesting.However, permanent plots were photographed on an annual basis and we could monitor the fates of colonies and evaluate changes in their sizes.
Surveys of red coral populations using underwater photogrammetry started in 2002 in one site near Marseille and they have been progressively extended to different areas of the NW Mediterranean Sea.

Pattern recognition
In the context of coral measurement, the implemented system is based on the use of the coded targets.This approach has certain advantages; mainly it is easy to use, gives a semantic link between what is seen and the knowledge base of the system through a coded pattern and the calculation of the camera pose is performed in real-time and with high accuracy.
The principle of the coded target method is based on the deployment of a number of special graphics whose geometry is known.Instead of trying to directly recognize the objects in the real world, it is only necessary to recognize the targets that are linked to them.As coded targets' geometry is known, it is possible to optimize the image processing used to recognize them.There are many visual tagging systems.We can cite the CyberCode system which was formally proposed by Rekimoto (Rekimoto, 1998)and called Matrix before being renamed CyberCode in (Rekimoto & Ayatsuka, 2000).This approach has been implemented in many applications including mobile telephones (Rohs & Gfeller, 2004).
The ARToolkit system (Kato et alii, 2000) is a library that can be used to create augmented reality applications.In particular, it contains a system of square targets with a black border and an internal pattern.The process to extract and identify the code is similar to the process of the previous system.The code is recognized by calculating the correlation coefficient between the standardized image and the standardized reference database.
In order to increase the recognition rate of targets, some variations have been brought.These variations are based on the calculation of the correlation rate between the extracted pattern and the reference patterns that are learned by the system (Owen et alii, 2002).However, the performances of the system can be greatly improved.This is why most of the research laboratories have been developing their own systems that closely look like the previous system.We can cite as an example Fiala's ARTag system (Fiala, 2010).The company InterSense (Naimark & Foxlin, 2002) has also developed its own system of coded targets.It relies on circular targets.Although it is not the first of its kind, it is more successful because the processing is embedded in the electronics of a hybrid system that is made of several sensors.Like the other systems of circular targets, it is necessary to have several landmarks in the field of view so as to calculate the camera pose.Indeed, each target gives a particular point.The position of its center is the primitive that is used for geometrical calculations.CanTag (Rice et alii, 2006) is a 3D Open Source solution that provides tools for the generation and tracking of targets in a 3D space.Unlike the previous systems, it can generate both circular and square targets.The final user can select the best set of targets for any given scenario.It also offers a set of tools for image processing that allow the user to adapt to different situations.Although the concept is similar to the ARTag system, APRILTag (Olson, 2011) is completely Open Source and its algorithms are well documented.This system is better than its predecessor; it implements a robust and fast detection method of lines.Its digital coding system is strong and has a better robustness to occlusion, distortion and lighting changes.
The study of the state of the art allowed us to look more closely at the architecture of the most popular systems of coded targets.
We noticed fundamental differences between the square target and circular targets.At similar size, square targets encode more symbols and are more robust to different noises, whereas circular targets provide better performance in terms of location and accuracy of the pose.This is mainly due to the fact that the center of the circle is easier to locate than the corners of a square.
The goal of our application is to provide biologists with a photogrammetric survey tool that is easy and fast to implement.It involves placing 10x10mm coded targets on quadrants which will be automatically identified and accurately located by our system.
Our targets have an identical geometry to those of APRILTag.
They are square-shaped markers with a large black border surrounding a black-and-white coded pattern composed of 16 modules.The identification is fast since the pattern inside does not need to be normalized and correlated with the reference patterns stored in memory.
Targets are located by first binarizing the image, and then contours are extracted and transformed into snakes.A polygonal approximation of each snake is computed.For each quadrangles, the system will determine if it stores an identified coded pattern of not.
In order to improve its robustness to occlusions, we implemented a new line detector.For this, we modified the method of chaining and we added geometric constraints.This method allows us to detect occlusions without any identification errors.
a: Before the improvement b: After the improvement Figure 5. Robustness test.
In Figure 5.a, we simulated additive noise, impulse noises and some occlusions and applied APRILTag algorithms.Figure 5.b shows the behavior of our algorithm on this same image.We note that thanks to the processing previously cited, all corners have been detected (green dots appear at each corner of all targets).
To improve target localization, we modified the circular target algorithm.We then added a black dot on the white center of the target.To easily detect this point, an area of interest is computed using the target corners.After thresholding, the coordinates of the central sub pixel point are computed using a weighted average (by grayscale neighbors).
Figure 6.Target with central dot.
Our interest in the detection of central dot is the fact that it is a simple and robust solution.Indeed the central points are more reliable than the corners because they are more stable to the orientation of the camera, and they do not require postprocessing (chaining, filtering ...).We see clearly the top left corner is not well located in Figure 6.a.Knowing the geometry of the target, our method detects the central dot, despite the occultation of the top right corner.
The distribution of targets on the quadrat enables an unambiguous identification of each quadrat; a unique signature is generated by calculating the distances between targets to identify them automatically.
In Figure 6, thirty-two 10x10mm targets are placed together with a space of 3 mm.Our system can decode and locate them with precision.
In our application (Figure 7), the quadrat has 20 coded targets, which are automatically detected and used for photograph orientation, quadrat identification and scaling.

Automatic photogrammetric pipeline
A fully automatic system for the orientation of a set of unordered photos is proposed using an algorithm based on the fact that the set of photos is always small (maximum of 10 photos).Under this condition, the following algorithm can be used: 1. Extract features based on combining: a. Tag ruler target (from 1 to 20, fixed on the quadrat) b.SIFT and SURF algorithms for acquiring other points (Kalia et alii, 2011;Lowe, 2004;Mahiddine et alii, 2012) were used in the beginning but now we switch to FAST descriptor (Rosten et alii, 2010).).The result is a set of blocks containing a set of oriented photographs.A post-processing step can check if there is some way to merge these models by common points of photographs.

Manual measurement
This application provides a huge set of oriented photographs to the marine biologists in order to measure relevant elements on coral colonies.Biologists are the only ones able to make qualitative observations associated with measurements and other annotations made when measuring the photographs.They can choose two oriented photographs that they want to measure and the system will produce result files in several formats: XML, Excel, and PLY for 3D visualization.
Figure 8. Stereo interface for manual plotting.The user can benefit from auto correlation with epipolar constraint, epipolar line, management of already measured object, and the belonging observation, consistency check.

Control and consistency check
Each quadrat has 20 targets in a random position which provides a unique signature for each quadrat which is detected by matching a set of random distances between targets.As there may be some occlusion, we verify all the possible distances and compute a score.Currently we have several teams using several quadrats and the system is able to manage 12 different quadrats in the automatic EO and scale based on the set of photographs.

An interdisciplinary work
The work presented in this paper is based on ten years of interdisciplinary work where computer science researchers had to understand the real need of the biologists and the hard constraint of diving, and on the other hand, biologists had to change their methods in the field and in the lab.The method implemented in terms of photographs approach, scene illumination, practical feasibility on site, photogrammetric method, minimizing time on site and giving an estimator of the accuracy provided by the measurement method and a consistency check regarding the data collected by experts.Two kinds of scenarios were implemented: measurement of colony characteristics, such as base diameter, height, etc.This is done by making a set of photographs of a colony surrounded by a known quadrat.These measurements are stored in a local reference system, dependent of the quadrat placed on the scene.The second one is the measurement of zones delimited in the rock substrate in order to monitor changes over time on the zone; i.e. the measurements are stored in a local reference system expressed by relevant, fixed targets present on the zone.The result is a set of direct or indirect measurements made on the red coral colonies or by using a dense cloud of points postprocessed after photograph orientation.Finally in order to help the experts read the final documents, we are able to provide 3D models as well as generate 2D orthophotos that are accurate and reliable.

Future work
The use of underwater photogrammetry already enables researchers to obtain novel information on the size-structure population dynamics of red coral which provided new quantitative evidence of the effect of harvesting and the recovery of red coral populations dwelling in marine protected areas.Likewise, for the very first time we obtained reliable measures of growth rates on height in this species.The photographic records were also useful to quantify the necrosis rates on the populations affected by mass mortality events related with anomalous warming.In this study we focused on only one species, the red coral, but photographs contain information about many other species which could be analyzed in the future to provide more global information on the functioning of benthic communities.The use of underwater photogrammetry was proven very valuable, but a new avenue of applications for the study of such complex communities is already available with the development of affordable geo-referenced mosaic techniques.These mosaics will offer marine biologists new data sets of spatial related information which was just unachievable few years ago.
Based on almost ten years of collaboration we will use these data, collected during these years, to study the evolution of coral during this time even if the methodology, implemented algorithms and photograph technology has changed.

Figure 2 .
Figure 2. Permanent plots, with fixed and covered targets in Scandola, Corse.
2. Match points (target by identification) 3. Make all possible relative orientations, i.e. creating all possible stereo models, oriented using the 5-point algorithm proposed by Stevenius (Stewenius et alii, 2006) 4. Compute a score for each stereo model based on 3D point residuals, number of points, point repartition on the image, ratio median distance from the point to the first camera center and the distance between the two cameras (depth / basis), and median intersection angle for 3D points.5. Choose the best relative model to start the global orientation.6. Select a list of candidate models to increase this seed model.7. Choose the next added model according to its score and its connectivity with the existing block.8. Add oriented photo and new observations to already existing 3D point and creating, if needed, new 3D points.9. Run a Bundle Adjustment based on SBA (Lourakis & Argyros, 2009) with additional constraints including points belonging to a plane and distances between measured points.As the targets on the quadrat are well known, we have a correct scale at each iteration.10.If residuals are correct, choose the next stereo candidate (Step 7).11.Otherwise, close the current set of oriented photographs and start a new model-block selecting the best starting stereo pair (see Step 5

Figure 9 .
Figure9.One of the five distance constraint after final bundle adjustment.The bundle adjustment is able to process additional constraints as distances between points; this is done for final scale and accuracy control, all the residuals on the constraint distance are less than 0.5mm.After verifying the accuracy, a check for gross error is performed by comparing the measured point coordinate with the bounding box computed after densification and computation of a dense cloud of points.Finally a consistency check module is performed on the measured points by comparing the default values of the studied colonies.A consistency check module, based on the work made by Julien Seinturier during his PhD(Seinturier, 2007; Seinturier  et alii, 2003)  is continuously updated with the current plotting.

Figure 10 .
Figure 10.Dense cloud of point measured automatically to check the manual measure.

Figure 11 .
Figure 11.Dense cloud of point measured automatically to check the manual measure.3. RESULT ANALYSIS Underwater photography has been used to monitor 10 populations in the NW Mediterranean Sea.Permanent plots have been set up in 5 marine protected areas (Parc Natural del Montgrí, les Illes Medes i el Baix Ter, Réserve Naturelle Marine de Cerbère-Banyuls, Parc Marin de la Côte Bleue, Parc National de Calanques et Réserve Naturelle de Scandola).Except for the populations within the Parc National de Calanques near Marseille, which was created in 2012, populations benefit from protective measures for the last decades.The analysis of photographic surveys allowed to obtain the following main results: (i) data on size-structure of