EVALUATING DENSE 3D RECONSTRUCTION SOFTWARE PACKAGES FOR OBLIQUE MONITORING OF CROP CANOPY SURFACE

Crop Surface Models (CSMs) are 2.5D raster surfaces representing absolute plant canopy height. Using multiple CMSs generated from data acquired at multiple time steps, a crop surface monitoring is enabled. This makes it possible to monitor crop growth over time and can be used for monitoring in-field crop growth variability which is useful in the context of high-throughput phenotyping. This study aims to evaluate several software packages for dense 3D reconstruction from multiple overlapping RGB images on field and plot-scale. A summer barley field experiment located at the Campus Klein-Altendorf of University of Bonn was observed by acquiring stereo images from an oblique angle using consumer-grade smart cameras. Two such cameras were mounted at an elevation of 10 m and acquired images for a period of two months during the growing period of 2014. The field experiment consisted of nine barley cultivars that were cultivated in multiple repetitions and nitrogen treatments. Manual plant height measurements were carried out at four dates during the observation period. The software packages Agisoft PhotoScan, VisualSfM with CMVS/PMVS2 and SURE are investigated. The point clouds are georeferenced through a set of ground control points. Where adequate results are reached, a statistical analysis is performed.


INTRODUCTION
Crop Surface Models (CSMs) as introduced by Hoffmeister et al. (2010) are 2.5D raster surfaces representing absolute plant canopy height.Using multiple CMSs generated from data acquired at multiple time steps, a crop surface monitoring is enabled.This makes it possible to monitor crop growth over time and can be used for monitoring in-field crop growth variability which is useful in the context of high-throughput phenotyping.CSMs can be generated using different sensor types such as laser scanners (Tilly et al., 2013(Tilly et al., , 2014) ) or from overlapping red, green, blue (RGB) imagery both from aerially acquired nadir imagery (Bendig et al., 2013) and oblique imagery (Brocks and Bareth, 2014) using Structure-from-Motion (SfM) and dense 3D reconstruction techniques.Comparisons of different software packages and algorithms for dense 3d reconstruction have been performed (Eltner and Schneider, 2015;Remondino et al., 2014;Grenzdörffer, 2014;Dall'Asta and Roncella, 2014;Brutto and Meli, 2012), but none for the specific scenario tested in this study.Here, we compare the suitability of several software packages for the generation of CSMs from oblique stereo imagery acquired from a terrestrial platform with converging image axes.

STUDY SITE
The study site was a summer barley field experiment at the Campus Klein-Altendorf (N 50°37 27 , E 6°59 16 ) of the University of Bonn's Faculty of Agriculture The experiment was setup by the CROP.SENSe.net(http://www.cropsense.uni-bonn.de)interdisciplinary research network that is working towards nondestructively analyzing and screening plant phenotype and crop status such as nutrients and stress.For this field experiment, nine barley cultivars were cultivated with two nitrogen treatments (40 kg N/ha and 80 kg N/ha) in three repetitions.Figure 1 shows an overview of the study site.

Structure-from-Motion and Multi-View-Stereo
Structure-from-Motion (SfM) (Ullman, 1979) and Multi-View-Stereo (MVS) (Seitz et al., 2006) are techniques to derive 3D information from 2D red, green, blue (RGB) imagery that depict the same object from different points of view and are thus overlapping.Specifically, SfM is used to derive internal and external camera orientation from a set of images, therefore reconstructing the 3D geometry of a scene.This is realized by using algorithms such as the well-known Scale Invariant Feature Transform (SIFT) (Lowe, 1999(Lowe, , 2004)).Using this approach, a sparse point cloud containing the detected feature points is generated.This, along with the scene geometry, i.e., the camera coordinates for each RGB image used as input, can then be used to generate a dense point cloud representing the objects pictured in the images.For this, MVS algorithms are used.

Agisoft Photoscan
Agisoft Photoscan is a commercial dense 3D reconstruction software packages that allows the user to generate spatial 3D data from 2D imagery.It uses SFM and dense 3D reconstruction algorithms and provides a graphical user interface that is easy to use.Due to its commercial black-box nature, the exact algorithms used are not known.According to the developers, it uses an approach similar to SIFT for detecting feature points and then estimates internal and external camera orientation using a greedy algorithm, refining the result using a bundle adjustment (Semyonov, 2011;Triggs et al., 2000).Dense 3D reconstruction is performed using different algorithms such as MVS or pair-wise depth map computation, depending on user settings.Photoscan georeferences the generated point clouds through ground control points.The GCPs' coordinates have to be entered and they need to be marked on the input images.
3.1.2SURE SURE (Rothermel and Wenzel, 2012) is a commercial software package for dense point cloud generation from an overlapping set of images and its orientation.A free version is available for research purposes.It generates up to one 3D point per pixel of input images, allowing a highly-detailed result.Because it requires the images orientations as input, previous processing using other SfM software packages is required.

Data Acquisition
Oblique stereo imagery of the field experiment was acquired daily at three times throughout the growing period by using two Samsung Galaxy Camera EK-GC100 smart cameras.The cameras feature a 4.1˜86.1 mm focal length (23˜483 mm 35 mm equivalent) lens, a 1/2.3 " BSI CMOS sensor with a maximum resolution of 16 megapixels (4608 x 3456), ISO settings from 100 to 3200, a shutter speed from 16 to 1/2000 s and an aperture from f/2.8 to 5.9 along with Wifi and mobile network connectivity.They were mounted at a height of 10 m and a horizontal distance of 3.6 m between each other on a hydraulic hoisting platform.Image taking was automated by using a custom-developed Android application that acquired images three times daily at different exposure times.Details of the data acquisition application have been shown by Brocks and Bareth (2014).Images were acquired from May 02 until June 30, 2014, with some breaks due to technical issues with the power supply.

3D reconstruction
To be able to compare the different software packages, all image pairs acquired on the dates of the manual plant height measurements where processed using PhotoScan, SURE and VisualSfM with PMVS/CMVS2.The dates of the image acquisition were May 08, May 23, June 05 and June 17.
For PhotoScan, the following workflow was observed: After adding the image pair to a processing workspace, the ground control points were marked on the images and their coordinates entered.
Then, the photos were aligned and the alignment was optimized using the default settings.Finally the dense point cloud was generated with the "high" quality setting and the generated georeferenced dense point cloud was exported for analysis.
Using SURE, a different workflow was needed: Because SURE does not contain an SfM component, internal as well as external camera orientation parameters for all used images need to be provided by a different program.For this purpose, camera positions and undistorted images were exported from Photoscan using the InPhoto format .prj.Then, for each image pair, with the parameter for the minimum intersection angle between the cameras "minangle" set to "1" to ensure that most of the field would be covered by the resulting point cloud.With the default intersection angle of 4 degree, large parts of the test field further from the cameras would not be covered by the generated dense point cloud.By using the camera orientation generated by PhotoScan, the resulting point clouds were correctly georeferenced.
For the VisualSfM and CMVS/PMVS2 workflow, first the image pairs were added in the VisualSfM GUI and then the "match images" function was used to detect feature points.Then, the "sparse reconstruction" function was used to create a sparse point cloud.In this step, VisualSfM runs a multicore bundle adjustment and the external camera parameters are reconstructed.. Finally, for the dense point cloud reconstruction, CMVS/PMVS2 was run.As with SURE, the minimum intersection angle was set to 1 degree to ensure coverage of the whole field.To georeference the generated point cloud, several approaches were used: One integrated directly into VisualSfM and two external solutions.
The integrated solution applies a transformation to the generated dense point cloud based on the X/Y coordinates of the GCPs in the input images.Because this this method did not produce satisfactory results, two other approaches external to the VisualSfM software were also used: The SfM-Georef software (James and Robson, 2012) works similar to the internal method: GCPs are defined, their coordinates entered and then marked on the input images used to create the point clouds.Then, using camera models and orientations defined by the VisualSfM output, the GCPs' 3D coordinates in the SfM coordinate system can be calculated and used to transform the point cloud to the coordinate system of the GCPs.The third approach uses the v.ply.rectifyGRASS GIS add-on (Metz, 2012).This necessitated the manual detection of the GCPs in the dense point clouds generated by CMVS/PMVS2.Using their coordinates, a 3D orthogonal rectification of the point cloud is performed.
For a statistical analysis, we interpolated raster surfaces from the dense point clouds using an inverse distance weighted (IDW) algorithm.A cell size of 1 cm was chosen for the generated raster, and the 12 closest points to each raster cell within a distance of 0.5 m were used in the calculation.This results in areas with a very low point density to not be shown in the raster surface.The generated crop surface raster represents the crop canopy height above the sea level.To generate CSMs containing just the absolute plant height, we subtracted a base elevation DEM from the generated raster datasets.Zonal statistics including mean, minimum and maximum elevation were then calculated for the observed plots.To eliminate border effects in this step, the actual plot borders were buffered by 0.3 m.

RESULTS
According to a visual inspection of the generated dense point clouds, Photoscan achieved the best results.Usable point clouds were generated for all sampling dates.For some of the dates, (May 23 morning, June 5 noon and evening), parts of the field were not covered.The generated point clouds contained no noise or other artifacts.Because SURE generated usable dense point clouds for all dates only with the evening acquisition and we want to compare fairly with PhotoScan, we performed the statistical analysis for the CSMs generated for that acquisition time.Figure 6 shows the linear regressions of mean CSM-derived plant height and manually measured plant height for the evening acquisition time for both SURE and Photoscan.For this regression, all plots with a mean point density higher than 100 per m 2 were considered.It can be clearly seen that Photoscan performs better in deriving the plant heights with an r 2 value of 0.83 compared to 0.56 for SURE.The root mean square error (RMSE) for Photoscan equals 0.10 m, while the RMSE for SURE equals 0.21 m, again showing that Photo-Scan performed better.This study aimed to evaluate several dense 3D reconstruction software packages for their suitability for crop canopy surface monitoring from an oblique angle using stereo imagery.the three tested software packages PhotoScan, SURE and VisualSfM with PMVS/CMVS2, PhotoScan provided the best results and is most suited for further investigation, despite its status as a commercial software package without access to the source code and the lack of knowledge regarding which algorithms are exactly used to process the data.The fact that PhotoScan clearly performed better than SURE is in contrast to the evaluation performed by Remondino et al. (2014) and most likely explained by the fact that in this study, only one stereo pair acquired at an oblique angle was used reconstruct the 3D scene geometry.This is a corner case not reflected by the test datasets used in that study.
The difference in noise between SURE and PhotoScan appears possibly due to some kind of filtering in PhotoScan's processing.Due to the black box nature of the software, the cause for the lack of noise can not be exactly determined.Another possible explanation is that PhotoScan deals better with smaller minimum intersection angles between observing cameras.The noise in the SURE results is most pronounced in the region furthest from the camera position, where the intersection angles are smallest.
While PhotoScan and SURE provided usable results, VisualSfM in combination with PMVS/CMVS2 did not.The distortion of the true geometry of the field in the results provided by Visu-alSfM with PMVS/ CMVS2 could be caused by the image alignment algorithm used in VisualSfM not being able to deal correctly with this corner case.Another possibility is that the camera selfcalibration implemented in PhotoScan performs better than the one from VisualSfM.Regarding noise and point density, the results of PMVS/CMVS2 look promising, but further analysis in the context of this study was impossible due to the incorrect geometry of the generated point cloud.
In summary, this study shows that from the tested software packages, PhotoScan is best suited for crop height monitoring from oblique angles when using a single stereo image pair.
Figure 1.Study site

Figure 2 .Figure 4 .
Figure 2. Top-down view of point clouds for the morning of May 08 SURE did not perform as good as Photoscan.For one acquisition date, no point cloud was generated at all, and for two other dates, there were large parts of the field where the point cloud obviously does not represent the crop surface correctly.Compared to the point clouds generated by Photoscan, the point clouds contained noise not seen in the Photoscan point clouds, especially along the borders of the point cloud and at distances of more than 40 m

Figure 6 .
Figure 6.Regressions of mean CSM-derived plant height and manually measured plant height, evening acquisition time 5. DISCUSSION AND CONCLUSION

Table 1 .
Descriptive statistics of plot-wise mean heights of manual measurements and generated CSMs for plots with point density greater than 100/m 2 Table1shows descriptive statistics for the CSMs generated by SURE and PhotoScanfor the evening image acquisition date: The range and standard deviation of the plotwise heights is much larger for the CSMs generated by SURE, while the values for the PhotoScan CSMs are much closer to those of the manual plant height measurements.This again shows that the results generated by PhotoScan are superior to those of SURE.