Evaluation of Automatically Generated 2D Footprints from Urban LiDAR Data

: Over the last decade, several automatic approaches have been proposed to extract and reconstruct 2D building footprints and 2D road profiles from ALS data, satellite images, and/or aerial imagery. Since these methods have to date been applied to various data sets and assessed through a variety of different quality indicators and ground truths, comparing the relative effectiveness of the techniques and identifying their strengths and short-comings has not been possible in a systematic way. This open contest was designed to overcoming this shortcoming. Specifically, participants were asked to submit 2D footprints (building outlines and road profiles) derived from ALS data from a highly dense data (approximately 225 points/m 2 ) across a 1km 2 of central Dublin, Ireland. The proposed evaluation strategies were designed to measure not only the capacity of each method to detect and reconstruct 2D buildings and roads but also the quality of the reconstructed building and road models in terms of shape similarity and positional accuracy. The evaluated methods will represent those submitted as part of IQPC15.


INTRODUCTION
The availability of three-dimensional (3D) point clouds offers a potential resource for wide ranges of applications (e.g.environmental planning and monitoring, computational simulation, disaster management, security, telecommunications, location-based services).Urban, two-dimensional (2D) footprints, which primarily include 2D footprints of buildings and the road network, play important roles in these applications and can be a major resource for generating final 3D models.For example, Laycock and Day (Laycock and Day, 2003) generated 3D building models by extruding 2D building footprints with the building height derived from aerial laser scanning (ALS) data.Furthermore, a part of the digital road map can be subsequently generated based on a 2D road profile.A number of researchers have addressed the problem of extraction and reconstruction of 2D building footprints and 2D road profiles from ALS data, satellite images, and/or aerial imagery (Boyko and Funkhouser, 2011;Clode et al., 2005;Kwak and Habib, 2014;Lafarge and Mallet, 2011;Zhang et al., 2006).Proposed methods have been tested on different data sets, and the authors have also used various evaluation criteria and ground truth resources.For example, Boyko and Funkhouser (Boyko and Funkhouser, 2011) manually generated ground truth of a road network and proposed five comparative quantities (completeness, correctness, quality, average spill size and prevailing spill direction) to evaluate extracted roads.This causes difficulty in generating a consistent comparative assessment of the methods.Thus, this contest is called participants to submit resulted 2D footprints (building outliers and road profiles) from ALS data provided by the contest organizers for evaluation.The contest also opens a challenge in detecting and reconstructing road profiles from ALS data only because several current methods required fusion data.The success of this contest can possibly provide useful information for establishing strategies for automatic urban 2D footprints from ALS data.
The contest uses a highly dense point cloud (225 million points covering approximately 1km 2 area) of Dublin, Ireland's city centre.The data has Cartesian system coordinates and intensity values and was merged from 44 flight strips.The flight plan was design to maximize data acquisition on the building facades.
The participants are asked to submit the results of automatically generated 2D building footprints (Task A) and/or 2D road profiles (Task B) from three pre-designated sub-areas of the study area.The contest organizers will evaluate submitted results based on the ground truth provided the Ordinate Survey Ireland (OSI) and OpenStreetMap (OSM).The task description, ground truth, and evaluation for each task are presented in Section 3-5.Finally, a winner and two runners up for each task will be selected based on the overall evaluation scores.

DATA DESCRIPTION
The test area is approximately 1km 2 and consists of 205 blocks, each of which may contain in excess of a dozen buildings per block, as shown in Figure 1.The typical building is 11-15m in height, less than 5m in width, and 6-10m in length (Clarke and Laefer, 2012).The buildings are mostly closely spaced or abutting each other, with some sharing an adjoining wall, commonly referred to as a "party wall".
The dataset was acquired by ALS using the FLI-MAP 2 system, which generated 1000 pulses for each scan line.The system operated at a scan angle of 60 degrees.The quoted accuracy of the FLI-MAP 2 system is 8 cm in the horizontal plane and 5cm in the vertical direction, including both laser range and navigational errors (Hinks, 2011).Acquired points were provided in a global coordinate system with reference to the National Irish Grid (Irish Grid), relating to the use of a Global Navigation Satellite System (GNSS) to determine the aircraft position during scanning.The FLI-MAP 2 system is capable of recording up to four echoes for each emitted pulse and spectral data with intensity values.
The dominant directions of the flight tracks were chosen as north-east, north-west, south-east and south-west.The flight attitude varied between ~380-480 m (as low as possible with respect to approval by the Irish Aviation Authority), with an average elevation of ~400m.A total of 2,823 flight path points were collected during data acquisition.As a result, a point cloud was merged from 370,154 scan lines resulting in a typical density of 225 points/m 2 .The echo distribution is shown in Table 1.The vast majority of points were first echoes.Secondary echoes constituted only a small portion of the points, as the overwhelming majority of surfaces in the study area was formed of solid objects (i.e.streets and buildings).For further information about this ALS data, participants are referred to Hinks (2011).The data set was organized into 9 tiles, each covering 500m x 500m (Figure 1), which is 5.8 Gb in size and stored with a LAS format.Three subsets of the data are used for this competition (Figure 2).Area 1 contains sparse buildings, a large green area, and trees.Area 2 has both building blocks and buildings sharing walls, as well as some trees.Area 3 contains mostly low brick buildings and no trees.
The data of each area (Area 1, 2 and 3) were extracted from the original data set and is 1.1 Gb in size in an ASCII file (Zip file), where each row represents x-, y-, z-coordinates and intensity of each data point, or in a LAS format (3.8 Gb).Either the data sets of the study area or these of the the contest are can be downloaded through the webpage of IQmulus project.

TASK AND SUBMISSION
Task A is to extract a point cloud affiliated with buildings and to reconstruct 2D building footprint from these points.In urban areas, buildings are often abutting or sharing an adjoined wall.These buildings also have similar morphology (i.e.height, width or similar roof configuration), which poses a major challenge for automatic algorithms in building detection and building boundary reconstruction.Participants were asked to submit the results from their algorithms in two sets of file results: (1) ASCII files containing ALS data sets of each building, where each row represents the x-, y-, and zcoordinates of the data points and (2) ASCII files containing the building footprint describing as a polygon, where each row contains both the x-and y-coordinates of the polygon vertices.The file naming convention should be "Building_X1_Y1_X2_Y2", where the pairs X1and Y1 and X2 and Y2 are two opposite corners of the bounding box of the dataset on a horizontal plane.
Task B is to extract a point cloud of the roads and to reconstruct the 2D road profiles including the pavement edges.Similar to Task A, the participants were asked to submit two ASCII files containing: (1) ALS data points of the road network and (2) the polygons describing pavement edges of the road network.Furthermore, the submitted files were to be named akin to Task A using the coordinates of two opposite corners of the bounding box of the road segment.
In the case that only ASCII files containing data points of either buildings or road network were submitted, the building footprints and the pavement edges of the road were to be generated by the organizer's algorithm for further evaluation.

GROUND TRUTH
The "ground truth" consisted of 2D footprints provided by the OSI.The 2D footprints primarily contain 2D building boundaries and road profiles (centre and edges).However, buildings and road network can be changed over period time, which may not update in OSI 2D footprints.The building boundaries and road centres derived from OSM were to be a supplement resource.The 2D footprints from OSI and OSM are shown in Figure 3.It can be seen clearly a majority of footpaths in Area 1 are not available in OSI 2D footprint and they can derived from OSM data.

Task A
The evaluation process identifies the level of locational deviation, the level of shape similarity, and the positional accuracy of the extracted building footprint (ExB), with respect to the ground truth building (GtB).For the location deviation, quality indicators involving True Positive (TP), False Positive (FP) and False Negative (FN) measure the overall extraction and reconstruction of the building footprints.These indicators can be determined after mapping extracted results onto the GtB.If the object from the ExB matches one from the GtB, it is TP.If the object from ExB does not match to one from the GtB, it is FP; otherwise, if the object from GtB does not match one from the ExB, it is FN.These quality indicators are illustrated in Figure 4. Completeness, correctness, and quality will be the measured indicators of the submitted results, which are expressed in Eq. 1, 2 and 3. (3) For the level of shape similarity and positional accuracy, the building footprint from ExB will be considered, if this building overlaps any building from the GtB.To measure a shape similarity, the differences in area and perimeter of each building are computed, which are given in Eq. 4 and 5.
where A GtBi = the areas of the building footprint from GtB A ExBi = the areas of the building footprint from ExB L GtBi = the perimeter of the building footprint from GtB L ExBi = the perimeter of the building footprint from ExB n = the number of the buildings Finally, summing the absolute, mean, and standard deviation of these differences (area and perimeter) are used to express the shape similarity.
Furthermore, a positional accuracy can be described in terms of the accuracy and conciseness of the building footprint, which is performed through establishing orientation and corner errors.
The orientation error () is the angle between L ExBj (a side of the extracted building footprint i, ExBi) and L GtBj (a side of the ground truth building footprint i, GtBi), where L GtBi is the closest side to the L ExBj .For details of determining a pair of L GtBj and L ExBj , readers can refer to Truong-Hong and Laefer (Truong-Hong and Laefer, 2015).In addition, the vertices'/corners' error (d) is defined as the Euclidean distance between the corners of the ExBi to its nearest corner derived from the GtBi.The evaluated indicators are expressed as Eq.s 6 and 7: where L ExBj = the side length of ExBi  j = the angle between the L PREDi and the L GTi L GTi = the side length of GtBi m = the number of boundary lines in the building footprint of interest n = the number of the corners in ExBi In these error measurements, L ExBj was introduced to avoid a heavy penalization for short, extracted, boundary lines (Okorn et al., 2010).Subsequently, average and standard deviations are used to measure distributions of these quantities.

Task B
Similar to Task A, the evaluation process of Task B identifies the level of locational deviation and the positional accuracy of the extracted road profile (ExR), with respect to the ground truth road (GtR).Based on the minimum bounding box of GtR, a 2D grid with the cell size of 1m x 1m is generated.When the GtR is mapped onto the 2D grid, the cell, C GtRi (x,y), has a value of 1, if any pavement edge or road surface of the GtR overlaps the cell (green cells in Figure 5); otherwise the value of C GtRi (x,y) is 0 (white cells in Figure 5).Furthermore, if the pavement edge overlaps to C GtRi (x,y) = 1, the cell is divided into two parts called inside road (C GtRInti (x,y)) and outside road (C GtROuti (x,y), where C GtRInti (x,y) is a part of the cell having the centre drops between two pavement edges of the road; otherwise it is C GtROuti (x,y).Notably, a total area of C GtRInti (x,y) and C GtROuti (x,y) equals the cell area = 1m 2 .This rule is also applied for ExR when projecting ExR onto the 2D grid.and ExR, as expressed in Eq.s 8-10 and Figure 6.The positional accuracy can be determined through differences in location and orientation of the road edges between the ground truth and extracted roads.A distance and angle between the road edges from the ground truth and the extracted results are proposed to measure those differences.A pair of road edges (L GtRi from GtR and L ExRi from ExR) is initially extracted.For that, if C GtRi (x,y) = 1, a pavement edge segment from GtR overlapping C GtRi (x , y) is computed, which is called L GtRi .Then, if the angle between the L GtRi and a horizontal direction (n x = [1,0]) is less than or equal to 45 degrees, a pavement edge segment of the ExR on the same vertical grid to the C GtRi (x , y) closest to the L GtRi is the L ExRi .Otherwise, the pavement edge segment of the ExR on the horizontal grid to the C GtRi (x , y) closest to the L GtRi is the L ExRi (Figure 7).

 
where n = the number of pairs of the road edge segments   j = the angle between the L GtRi and the L ExRi Finally, the winners of each task are selected based on the overall evaluation of the output quality, where all evaluated quantities are weighted equally.

CONCLUSION
Automatic approaches have been proposed to detect and reconstruct building and road profiles from ALS data.Previously, these methods were evaluated by using different data sets associated with various different criteria and ground truths.That precludes a rigorous comparison of the advantages and disadvantages of each method.This paper presents the objectives of the track in the IQPC 2015 contest related to automatic building and road detection and reconstruction.The contest was run on dataset consisting of ALS data captured over 1km 2 of the Dublin's city centre with a typical data density of 225 points/m 2 .The success of this contest can possibly provide useful information for establishing strategies for automatic urban 2D footprints from ALS data.
An evaluation strategy was proposed to benchmark the results in terms of the capacity of the submitted results in detecting and reconstructing building and road outlines.The evaluation process identifies the level of locational deviation, the level of shape similarity, and the positional accuracy of the extracted building footprints and road profiles, with respect to the ground truth building and road.The contest was launched in March 2015.The test datasets remain available on the webpage of the track.Participants are welcome to submitted for future evaluation.

Figure 1 .
Figure 1.Acquired ALS area in Dublin central and ALS tiles (contest area outlined in red)

Figure
Figure 2. Contest Areas

Figure 3 .
Figure 3. Ground truth from OSI and OSM

Figure
Figure 4. Illustration of determining TP, FP, and FN

Figure 5 .
Figure 5. Classification of cells The locational deviation, completeness, correctness, and quality indicators mentioned in Task A are measured, where these parameters can be determined from Eq.1-3.True Positive (TP), False Positive (FP) and False Negative (FN) are computed by comparing cell values from two 2D grids represented by GtR and ExR, as expressed in Eq.s 8-10 and Figure 6.
the cell value from GtR C ExRi = the cell value from ExR C GtRInti = the areas of a part of C GtRi inside GtR C ExRInti = the areas of a part of C ExRi inside ExR

Figure 6 .
Figure 6.Illustration of computing TP, FP and FN areas

Table 1 .
The data set is now publicly available.Echo distribution of acquired ALS points