PRIMITIVE-BASED BUILDING RECONSTRUCTION BY INTEGRATION OF LIDAR DATA AND OPTICAL IMAGERY

Abstract. Airborne LiDAR data and optical imagery are two datasets used for 3D building reconstruction. In this paper, the complementarities of these two datasets are utilized to perform a primitive-based 3D building reconstruction. The proposed method comprises following steps: (1) recognize primitives from LiDAR point cloud and roughly measure primitives' parameters as initial values, and (2) select primitives' features on the imagery, and (3) optimize primitives' parameters by the constraints of LiDAR point cloud and imagery, and (4) represent 3D building model by these optimized primitives. Compared with other modelbased or CSG-based methods, the proposed method is simpler. It only uses the most straightforward features, i.e. planes of LiDAR point cloud and points of optical imagery. The experimental result shows this primitive-based method can accurately reconstruct 3D building model. And it can tightly integrate LiDAR point cloud and optical imagery, that is to say, all primitives’ parameters are optimized with all constraints in one step.


INTRODUCTION
3D reconstruction of buildings is an important approach to obtain the 3D structure information of buildings, and has been widely used in the applications of telecommunication, urban planning, environmental simulation, cartography, tourism, and mobile navigation systems.It has been the major topic of photogrammetry, remote sensing, computer vision, pattern recognition, surveying and mapping.Traditionally, photogrammetry is the primary approach for deriving geospatial information through the use of multiple optical images.Optical imagery has sharp and clear edges, so the 3D information derived from photogrammetric measurements consists of accurate metric and rich descriptive object information (Mikhail et al., 2001).But it is hard to obtain dense 3D points on the building's surface because of the matching problem at the homogeneous or occluded places.Also because of matching problem, it is hard to generate 3D building model automatically by photogrammetry (Schenk and Csatho, 2002).
Since it was introduced in the 1980s, as a promising method, Light Detection And Ranging (LiDAR) technology is used in the applications of acquiring digital elevation data.Because LiDAR technology is fully automated for generating digital elevation data, many researchers have paid attention to the technology and its applications (Ackermann, 1999;Arefi, 2009;Habib et al., 2004;Mayer et al., 2008;Rottensteiner and Briese, 2002;Zhang and Li, 2006).Although LiDAR point cloud has dense 3D points, these points are irregularly spaced, and don't have accurate information regarding breaklines such as building boundaries.Thus, the reconstructed 3D building's model is not very accurate (the accuracy depends on the points density), not only the shape but also the position of the building.Obviously, to generate a more accurate 3D building model using LiDAR point cloud, the help of other datasets with accurate boundaries is necessary.
Both ground plan and optical imagery satisfy this requirement.Compared with ground plan, optical imagery has the advantages of easy availability and up-to-date state.A variety of research has been conducted using LiDAR point cloud and optical imagery, whatever data-driven or model-driven approaches (Habib, 2009;Kim, 2008;Tarsha-Kurdi et al., 2007b;Wang, 2008).The existing methods have some drawbacks.Firstly, most of these methods use edges as the features to connect LiDAR point cloud and optical imagery, the data processing procedure is complex due to the edge detection, filtering, combination and other operations.Secondly, the LiDAR point cloud and optical imagery are often processed respectively, and then the results are combined simply.
In this paper, using airborne LiDAR data and optical imagery, a primitive-based 3D building reconstruction method is proposed to overcome the problems mentioned above.Two datasets are tightly integrated, and the accurate 3D building model can be acquired by the straightforward and simple features.The organization of this paper is as follows.In section 2, the proposed primitive-based method is described in detail, including motivation, workflow, and explanation of some crucial steps.Section 3 is the result of the preliminary experiment and some discussions.Finally, we draw the conclusion and identify the work of near future.

Selection of Reconstruction Method and Features
In this section, two crucial points will be explained, i.e., the selection of method and features for building 3D reconstruction.
There are two reasons for the selection of primitive-based method to reconstruct 3D building model.
Firstly, LiDAR point cloud has dense 3D points, but these points are irregularly spaced, and don't have accurate information regarding breaklines such as building boundaries.On the contrary, optical imagery has sharp and clear edges, but it is hard to obtain dense 3D points on the building's surface.In order to reconstruct 3D building model by integration of LiDAR point cloud and optical imagery, the selected object must have clear edges and dense surface points at the same time.Obviously, primitives, for example, box, gable-roof and hip-roof can satisfy this requirement.Suitable primitives will "glue" LiDAR point cloud and optical imagery.
Secondly, from the point view of computation, primitive-based representation of 3D building model has less parameters.For example, to represent a box, 3 parameters (width, length and height) are used to represent the shape; together with 3 parameters for position and 3 parameters for orientation, totally 9 parameters are enough to determine the shape and locate the box in 3D space.So the solution can be calculated easily and robustly.
For the selection of features, it is crucial because it affects the complexity of the process and the accuracy of the reconstructed 3D building model.As we have seen, LiDAR point cloud and optical imagery have different characteristics, so different features will be selected for these two datasets.The features should be as straightforward and simple as possible, so that they can be easily located and accurately measured.Plane is the feature that we selected for LiDAR point cloud, and corner is the feature that we selected for optical imagery.Using these straightforward and simple features, the computational procedure is simplified, and the result can be obtained precisely and robustly.
Because of above reasons, we select primitive-based method to reconstruct 3D building model, and plane feature for LiDAR point cloud and corner feature for optical imagery.In this paper, only two kinds of primitives are studied, i.e., box and hip-roof.

Workflow
Fig. 1 shows the workflow of this primitive-based 3D building reconstruction method.The numbers denote the order of processing.Finally, 3D building can be represented by these primitives with the optimized parameters.
In the following paragraphs, the detail of each step will be depicted.A building consisting of several hip-roof primitives is used as an example to facilitate the understanding.Fig. 2 is the LiDAR point cloud of this building, and Fig. 3 is the corresponding optical image.They are a part of an experimental data of Strasbourg, France.This experiment was carried out in September, 2004.In this experiment, LiDAR sensor is Optech ALTM 1225, flight height is 1440m, and point density is 1.3 points/m 2 ; camera sensor is Vexel UltraCamX, flight height is 2330m, and GSD (Ground Sample Distance) is 15cm.

Recognize Primitives:
In order to carry out the primitive-based 3D building reconstruction, a building must be decomposed into several primitives.It is hard to perform this task all by computers.At current stage, the recognition of primitives is done in interactive mode.The decomposition is not unique, it depends on the operator.After analyzing both LiDAR point cloud (see Fig. 2) and optical image (see Fig. 3), this building can be decomposed into 5 hip-roof and 1 box primitives.

Measure Initial Parameters:
W h e n t h e w a y o f decomposition is determined, the parameters of each primitive should be measured.Height parameters can only be derived from LiDAR point cloud.Plane parameters, such as length, width and primary orientation of the building can be obtained from LiDAR point cloud or optical imagery.The parameters related with relative displacement and rotation to the adjacent primitive should also be determined.
As mentioned earlier, the measured parameters will be used as initial values and will be optimized in the following bundle adjustment procedure.It doesn't need to measure these parameters accurately.So the process can be simplified and the efficiency will be high.

Extract Features
Corners are detected/selected on the optical imagery.All visible vertexes are measured; the pixel coordinates of x and y directions are recorded.Every vertex has a unique ID; the bigger numbers in Fig. 3 are these IDs.The pixel coordinates of these corners will be used as the observed values of Collinearity Equation and contribute to the cost function.
Planes are detected/selected in the LiDAR point cloud.In Fig. 3, these planes are labelled with smaller numbers.Every plane has a unique ID, and the plane equation is calculated using all 3D points that form this plane.These 3D plane equations will be used as constraints and contribute to the cost function.

Compute Features
Based on the type and parameters of primitives, the 3D coordinates of the primitives' features, such as corners, can be calculated.For example, in Fig. 4, the bottom four corners' coordinates are (0,0,-height), (length,0,-height), (length,width,height) and (0,width,-height).They will be used as model/computed values in the following bundle adjustment procedure.Certainly, before these model values are substituted into equations, the coordinates must be transformed from the primitive's local coordinates system to the global coordinates system.This can be done by 3D rigid body transformation using a serial of translation and rotation parameters of primitives.

Optimize Parameters (Bundle Adjustment)
On the basis of previous preparation works, all primitives' parameters can be optimized by a bundle adjustment procedure.To achieve this goal, an important thing is to construct a simple and effective cost function.
Our idea is straightforward and simple.When a 3D building model has correct shape and is located in the correct place in 3D space, two conditions will be satisfied.Firstly, the backprojections of primitive's vertexes (computed features) on the optical image should perfectly superpose on the measured corners (extracted features).Secondly, the primitive's vertexes should be exactly on the planes which are formed by LiDAR point cloud.
Two mathematical models, i.e., Collinearity Equation and 3D Plane Equation are used to represent these two conditions respectively.And the cost function is established based on these two mathematical models.The inputs of this cost function are observed values, model values, and initial values above.The cost function can be optimized by bundle adjustment technique.When the bundle adjustment procedure is finished, the optimized/refined primitives' parameters will be outputted.
So far, 3D building can be represented by these primitives with the optimized parameters.The experimental result will be shown and some discussions will be made in the next section.

Experimental Result
Fig. 5 shows the reconstructed 3D building model of the example data.The wireframes of the model are overlaid on the original LiDAR point cloud.It can be seen that they are fitted very well.
Figure 5. Reconstructed 3D building model Because there are no surveyed data of this building, we use another way to verify the effectiveness of the proposed primitive-based 3D building reconstruction method.As shown in Fig. 6, the wireframes of both initial model and optimized model are back projected to the optical image, which is used to reconstruct this building.They are represented by yellow lines and white lines.The region circled in Fig. 6 is zoomed and shown in Fig. 7. Obviously, the initial polygons diverge from the real building's edges.While, the optimized polygons fit the real building's edges very well.
As shown in Fig. 3, there are totally 27 vertexes.The distances of the vertexes in the initial model and the optimized model compared to the imagery are computed.The result is shown in Tab. 1.It can be seen, the mean value and standard deviation are improved in both x and y directions after optimization procedure.In this experiment, only one image (Fig. 6) is used.In order to reinforce the effectiveness of proposed method, the optimized model is also back projected to another image (Fig. 8), which covers this building but is not used during the reconstruction procedure.The optimized polygons fit the real building's edges very well too.
Figure 8. Wireframes of optimized model on the image which doesn't involve in optimization procedure

Discussion
From this preliminary experiment, it can be seen that the proposed method can utilize the complementarity of imagery and LiDAR point cloud, and a model is established as soon as the cost function is solved.This is only the first step, perhaps the reconstructed model is not very accurate, because: 1.The 3D parameters and 2D features were measured roughly by manual work.2. The actual building doesn't exactly coincide with the used simple model.3.At present, only one image is used during optimization procedure.
Besides, there exist some deficient places.
1.The accuracy of the reconstructed model is only verified semi-quantificationally. 2. Only two simple primitives are used, i.e., box and hip-roof.3.There are many manual works.For example, extraction of 2D corner features and 3D plane features, selection of primitives and measurement of the initial parameters of these primitives.4.Not all images covering the studied building are used during the data processing procedure.5.Only the data of one separated building was studied and processed.
To make the proposed method more automated and to apply this method to a wide urban area, there are some places needing improvement in subsequent research.1.To represent more complex buildings, more primitives such as cylinder and sphere will be considered, and boolean operations of primitives will be used.2. To improve the accuracy and the efficiency, instead of current manual work, either specific humaninteractive software (Brenner, 1999) or some automatic algorithms will be developed, for example, extracting features from LiDAR point cloud and optical imagery (Tarsha-Kurdi et al., 2007a), recognizing types of primitives and obtaining the initial parameters of primitives.3. To apply to a wide urban area where a lot of buildings in it, some in-depth topics will be studied, i.e., building detection, extraction and segmentation from LiDAR point cloud and optical imagery (Hu, 2003;Tarsha-Kurdi et al., 2006).

CONCLUSIONS AND FUTURE WORK
Airborne LiDAR data and optical imagery are two datasets used for 3D building reconstruction.In this paper, the complementarities of these two datasets are utilized to perform a primitive-based 3D building reconstruction.The preliminary experimental result demonstrates the feasibility of the proposed 3D building reconstruction method.
The proposed primitive-based method is a kind of model-based method, so it has the following merits as other model-based methods.
1.For buildings with regular shapes, it can increase efficiency than point-by-point measurement.2. The operator only needs to select primitives and approximately measure parameters as initial values, while the labor-consuming precise measurement is carried out by computer.3. The inner constraints guarantees the geometric nature unchanged after reconstructing.4. It is possible to reconstruct the whole building even if a part of it is occluded.
Beyond that, the proposed primitive-based method in this paper is simpler, because it only uses the most straightforward features, i.e. planes of LiDAR point cloud and points of optical imagery.Further more, it can tightly integrate LiDAR point cloud and optical imagery, that is to say, all primitives' parameters are optimized with all constraints in one step.
The following researches will be carried out in the near future.1.All images covering the reconstructed building will be used to improve the accuracy and integrality of the reconstructed 3D model.2. The corner features will be extracted more precisely from the optical imagery, so that they can be used to improve and validate the accuracy of reconstructed 3D model.3. The result will be validated quantificationally or by in-situ data.

Figure 1 .
Figure 1.Flowchart of the proposed method 1. Recognize primitives and measure initial parameters.With the help of optical imagery and LiDAR point cloud, the building is decomposed into several primitives.Then the primitive's parameters are measured roughly on LiDAR point cloud and optical imagery, such as length, width, height, orientation and translation of the primitive.These measurements can be used as fixed values (constraints) or initial values in the following bundle adjustment procedure.2. Extract features.Corners are detected/selected on the optical imagery, and planes are detected/selected in the LiDAR point cloud.These features will be used as observed values/observations in the following bundle adjustment procedure.3. Compute features.Based on the type and parameters of primitives, the 3D coordinates of the primitives' features, such as corners, can be calculated.They will be used as model/computed values in the following bundle adjustment procedure.4. Optimize parameters (bundle adjustment).Using Collinearity Equation and 3D Plane Equation as mathematical models, a cost function can be established.The inputs of this cost function are observed values, model values, and initial values above.When the bundle adjustment procedure is finished, the optimized/refined primitives' parameters will be outputted.

Figure
Figure 2. LiDAR point cloud

Figure 6 .
Figure 6.Wireframes of initial model and optimized model

Table 1 .
Statistics of distances of vertexes