USING VERTICAL PANORAMIC IMAGES TO RECORD A HISTORIC CEMETERY

In 1919, during colonization of the West Region of São Paulo State, Brazil, the Ogassawara family built a cemetery and a school with donations received from the newspaper Osaka Mainichi Shimbum, in Osaka, Japan. The cemetery was closed by President Getúlio Vargas in 1942, during the Second World War. The architecture of the Japanese cemetery is a unique feature in Latin America. Even considering its historical and cultural relevance, there is a lack of geometric documentation about the location and features of the tombs and other buildings within the cemetery. As an alternative to provide detailed and fast georeferenced information about the area, it is proposed to use near vertical panoramic images taken with a digital camera with fisheye lens as the primary data followed by bundle adjustment and photogrammetric restitution. The aim of this paper is to present a feasibility study on the proposed technique with the assessment of the results with a strip of five panoramic images, taken over some graves in the Japanese cemetery. The results showed that a plant in a scale of 1:200 can be produced with photogrammetric restitution at a very low cost, when compared to topographic surveying or laser scanning. The paper will address the main advantages of this technique as well as its drawbacks, with quantitative analysis of the results achieved in this experiment.


INTRODUCTION
The cities of the west region of the State of São Paulo in Brazil were founded due to the expansion of coffee cultivation and the building of the São Paulo Railway (Fig. 1).Urban centres were created with distinctive landscape and with strong local identities.
In this context, the Japanese immigration played an important role in land use.In the 1910s, this region was entirely occupied by native forests.In 1919 during colonization, an epidemic of yellow fever killed many people in the Brejão Settlement, a district of Álvares Machado.The nearest cemetery was in Presidente Prudente, at a distance of 15 km, that should be traversed on foot, and to avoid this inconvenient and dangerous travel, the Ogassawara family decided to bury the victims there, purchasing 5 acres for the construction of the cemetery and the school with donations received from the newspaper Osaka Mainichi Shimbum, in Osaka, Japan (Takenaka, 2003).Since then, 784 people were buried including Japanese immigrants and one Brazilian.The cemetery was closed by President Getulio Vargas in 1942 during the Second World War.
Since 1980 the cemetery is officially protected by the CONDEPHAAT, a state body dedicated to the protection of historic, archaeological, artistic and touristic heritage (CONDEPHAAT, 2013) in the State of São Paulo.Even with this official and legal protection, a full and detailed inventory was not developed yet, which restricts proper maintenance.As a result, some parts of this historical cemetery were modified and original features were changed.
The 1988 Brazilian Constitution states in the Article 216 that "cultural heritage consists of Brazilian assets of material and immaterial nature, taken individually or together, making reference to identity, action, memory of the various groups that form the Brazilian society, in which are included: I -forms of expression; II -ways of creating, making and living; III -the scientific, artistic and technological creations; IV -works, objects, documents, buildings and other spaces intended for artistic and cultural expressions; V -urban complexes and historical, natural, artistic, archaeological, paleontological, ecological and scientific sites.
Brazil is signatory of the Convention on the protection of world heritage and the Convention on the intangible cultural heritage.
The protection of cultural property in Brazilian territory is guaranteed by a Federal Law which sets the rules for inventory of property belonging to the "National Historical and Artistic Heritage" as well as the protection that these goods shall be liable towards its preservation and conservation.
There is a recognized need for low cost, fast and reliable techniques for cultural heritage recording (Patias, 2007) and in developing countries, like Brazil, this is more evidenced.Several new technical developments have made available devices which can be used for photographic recording with advantages with respect to the conventional cameras and processes.Digital cameras have now high geometric resolution at a very affordable price (Grussenmeyer and Guillemin, 2011).Besides the conventional objectives, fish-eye lenses are now available with good cost benefit ratio, with acceptable aberrations and distortions.Several recent works have dealt with the problem of fish-eye geometric modeling (Abraham and Forstner, 2005;Kedzierski et al., 2009;Schneider et al., 2009) and it can be considered that they can be used for photogrammetric projects, provided that rigorous calibration has been performed.Direct georeferencing techniques can also be employed using GNSS receivers and integrated Inertial Measurement Units (IMU) to provide the position and attitude of the camera in each station.
The aim of this paper is to present a combined technique for acquisition of high resolution georeferenced panoramic images using near vertical panoramic images taken with a digital camera with fisheye lens.An integrated INS (Inertial Navigation System) combining a GNSS receiver and IMU can be used to provide position and attitude.The technique was used to record a unique historical Japanese cemetery, in Álvares Machado, São Paulo State, Brazil.The technical details and results achieved will be presented and discussed.

METHODOLOGY Some Features of the Japanese Cemetery
As previously stated, since 1919 until 1942, when the cemetery was closed, 784 people were buried including Japanese immigrants and one Brazilian.
The architecture of the Japanese cemetery is a unique feature in Latin America.It covers an area cut by a deep valley, having on one side the cemetery and a small chapel (Fig. 2.a) and on the other an old school house (Figure 2.c) a wooden stage and toilets (Fig. 2.b).A simple portal marks the entrance to the cemetery.The graves are mostly simply furnished with just a wooden upright with Japanese ideograms with the name of the person buried.Some of them are more refined with the use of granite or brick (Fig. 2.d), but retaining its character of simplicity and clean look.In the course of time, some tombs were altered with new elements added to the original, mainly due to religious changes in some families.
Early in 1919, the burials took place one after one more or less randomly; in 1921 the tombstones were ordered in rows organizing the layout of the cemetery.There are 26 rows with a total of 729 graves.
Nowadays, religious celebrations take place in the cemetery and chapel and cultural activities to honour the dead in the open courtyard near the old school.
As it was previously mentioned, even considering its historical and cultural relevance, there is a lack of geometric documentation about the location and features of the tombs and other buildings within the cemetery.
There is a need for detailed inventory with both geometric measurements and pictorial records.There are several trees inside and around the cemetery (Fig. 1.a) and the internal sidewalks are narrow.The requirements to produce a plant in a scale of 1:200 and the needs for photographic record, naturally favoured the adoption of photogrammetric techniques, but to achieve this scale very low flight would be necessary.Even with a low flight with an UAV (Unmanned Aerial Vehicle) some occlusions would occur.Considering these requirements and the lower costs of terrestrial imaging, it was proposed to use vertical terrestrial images taken with fish-eye cameras.

The proposed technique
The proposed technique is an alternative to provide detailed and fast georeferenced information about the area.It is proposed to use near vertical panoramic images taken with a digital camera with fisheye lens as the primary data followed by bundle adjustment and photogrammetric restitution.Additionally, an INS can be used, attached to the camera, to provide camera position and attitude.The system has to be firstly calibrated to estimate the camera Interior Orientation Parameters (IOPs), the lever arm and the boresight angles.

Camera calibration
Due to the internal geometry of the fisheye lenses, there is a severe distortion effect with strong scale variations in the images requiring rigorous calibration, which result in different parameters when compared with conventional perspective lenses.
The camera was firstly calibrated in a terrestrial test field using a conventional bundle adjustment with the Collinearity and Equidistant models with Conrady Brown lens distortion model.Eq. 1 presents the Collinearity equations with the additional parameters.
) are the point 3D coordinates in the photogrammetric reference system (Eq.( 2)); (x',y') are the image point coordinates in the photogrammetric reference system; (x 0 ,y 0 ) are the coordinates of the principal point (PP) and; Δx and Δy are the effects of the systematic errors (Eq.( 3)).
In which: r ij (i and j from 1 to 3) are rotation matrix elements that relate the image and the object reference systems; (X, Y, Z) are the point coordinates in the object reference system; and (X CP , Y CP , Z CP ) are the coordinates of the perspective centre (PC) in the object reference system.
In which K 1 , K 2 , K 3 are symmetric radial distortion coefficients; P 1 e P 2 are the decetering distortion coefficients; A and B are the affinity parameters; The Collinearity equations are generally used in the calibration process, however the image acquisition with fisheye lens camera does not follow the collinearity condition.The rays are refracted toward the optical axis.Most of the fisheye lenses are constructed following the equidistant or equisolid-angle projection (Schneider et al. 2009).In this paper, however, the Collinearity and the equidistant models (Eq. 1 and 4) were assessed (Abraham and Förstner, 2005) because of the small differences when considering other models as those presented in Abraham and Förstner (2005).In both models, the additional set of equations encompassing the lens distortions (Conrady-Brown model) were also used (Brown, 1971).
Equations ( 1) or (4) can be used to set a system of non-linear redundant equations, in which the image coordinates are observations and the IOPs, the Exterior Orientation Parameters (EOPs) and object point coordinates are the parameters.This system can be solved by the Least Squares Methods provided that some constraints are imposed such as the ground coordinates of object points, object distances or observations on the EOPs (Brown, 1971;Clarke and Fryer, 1998).

Direct georeferencing
With a combination of relative GNSS positioning and IMU, data acquired by DG can be used to estimate the EOPs of a sensor.This approach is known as Direct Sensor Orientation (DSO) or, when combined with Ground Control Points (GCPs), Integrated Sensor Orientation (ISO) (Blázquez and Colomina, 2012).
The physical shift between GNSS antenna and the external nodal point of the camera is called lever arm and must be estimated, normally by direct measurements.The misalignment angles between IMU and the camera photogrammetric reference system, also called boresight misalignment, must be included into the system calibration, since it is unfeasible to determine it with direct measurements.Boresight misalignment and the inner orientation have to be determined over a controlled reference area (Yastikli and Jacobsen, 2005).

Image acquisition procedure
To acquire strips of vertical terrestrial images with data provided by DG, a mobile platform mounted over and ordinary pallet carrier was adapted to carry all the equipment, as can be seen in Figure 3.a.A telescopic pole attached to a tripod was used to raise up the integrated system up to approximately 4.8m above the ground.
A portable computer connected to the system is used to record the raw data measurements.stamp in the GPS data that will make feasible the direct determination of the coordinates of the exposure station.The results with both models are presented in the Experiments and Results section (Section 3).

Bundle adjustment
The block of images can be oriented simultaneously with bundle block adjustment, either with the original images or with the rectified ones.The EOPs determined by the integrated INS can be used as constraints in the bundle adjustment.
To achieve that, the image coordinates of both control and check points have to be measured and corrected for the systematic errors.Bundle adjustment uses the Least Squares Method to solve a system of nonlinear equations (collinearity equations -Eq. 1) in which the image coordinates are the observations and the EOPs and the ground coordinates of the tie points are the unknowns.After determining the EOPs, point clouds, mosaics and photogrammetric plotting can be produced.

EXPERIMENTS AND RESULTS
The aim of this paper is to present a feasibility study on the proposed technique with the assessment of the results with a strip of five panoramic images, taken over some graves in the Japanese cemetery.Nikon D3100 digital camera with Bower fisheye lens was used in this study.
The camera was firstly calibrated in a terrestrial test field using a conventional bundle adjustment with both the Collinearity and Equidistant models with Conrady Brown lens distortion model.
Figure 5 shows one of the 12 images used in the camera calibration process.The terrestrial calibration field is composed of 139 coded targets, using the ARUCO style (Muñoz-Salinas, 2012).These targets have two main parts: an external crown, which is a rectangle and 5x5 internal squares that can code 10 bits of information.With this scheme, 1024 values can be encoded.A Public software developed by Muñoz-Salinas ( 2012) was adapted to automatically perform the location, identification and accurate measurement of the four corners of the external crown of targets of the calibration field (Silva et al., 2012).With this software, most of the existing coded targets are automatically extracted.Some missing corners were interactively measured to provide enough points with suitable geometry for camera calibration.
After automatic measurement (complemented by interactive measurement) of corners targets in 12 images, suitably acquired for camera calibration, 3216 observations were generated, with 411 ground points.The object coordinates of the corners were measured with topographic methods with an accuracy of 3 mm.
The CMC software (Calibration with Multiple Cameras - Bazan et al., 2009), developed in-house, which uses the well know collinearity model with Conrady Brown distortion model, was used for the first calibration trials.This software was modified to use also the equidistant models including the Conrady-Brown distortion equations.
The results of camera calibration with the collinearity equations are presented in Table 3. Results of the camera calibration process with the equidistant model.
Comparing the parameters values and their estimated standard deviations achieved with both models (Tables 2 and 3) it can be seen that the σ Naught and the standard deviations of the parameters are lower with the equidistant model, and, for this reason, these parameters were used to generate the resampled images.The estimated parameters are also different, mainly the lens distortion coefficients.The value of K 1 , estimated with the conventional collinearity model, is more than ten times the value estimated with the equidistant model.This happens because the collinearity equations do not model the deflection of rays caused by the fisheye lenses and this effect has to be absorbed by the lens distortion coefficients.
After camera calibration the images were resampled to correct for the distorted geometry caused by the fisheye lenses.The resampling uses the IOPs computed in the calibration step with the equidistant model, generating an image with a tie shape (see Figure 4.c).To check the results of the camera calibration, one image of the test field (Figure 4) was resampled and the targets corners were measured automatically.A space resection procedure was performed, using the estimated IOPs, the measured image points and the corresponding ground coordinates.The values of the camera positions estimated by both the calibration process and the space resection were compared and the differences were around 1 mm.Also, the image residuals after the space resection were around 1 pixel, showing that the bundle generated from the resampled image fits the ground points.
The images acquired over the cemetery were then resampled with the equidistant model parameters.The data collected with the INS system was processed and the coordinates of the exposure stations were generated with a standard deviation of 15 cm.The attitude angles were also computed, but they are used only as initial approximation due to the lack of boresight angles, which were not estimated in this first feasibility study.
A set of 7 ground control points and 2 check points were also surveyed in the cemetery using a combined process with some stations determined by GPS tracking in static mode and other by combination of intersection and radiation.The estimated standard deviations for the ground control points are within 1cm.
Figure 6.Configuration of the photogrammetric project with the images, GCP and check points.
The resampled images were imported in the Leica Photogrammetric Suite (LPS) along with the coordinates of the exposure stations and the GCP coordinates.Firstly, the image coordinates of the GCP were interactively measured with the classic point measurement tool.Then, tie points were also interactively measured and, finally some tie points were automatically extracted.Owing to the perspective displacements caused by the height of objects and image blurring, the number of tie points correctly extracted by the software was lower than in conventional aerial images.This happens because the tie points transfer process uses area based matching which is sensitive to geometric changes such as existing in the images acquired by fisheye lenses.Nevertheless, 389 object points were finally estimated in the bundle adjustment.Most of the tie points interactively measured are grave corners that will be used to produce the final plot.
Figure 6 shows the configuration of the project with the images footprints, GCP and tie points.The bundle adjustment was computed with the LPS software.Some results of the bundle adjustment are summarized in Table 4.It can be seen that the σ Naught is 2.3 pixels, which is likely owed to the errors in the image measurements mainly in the areas out of the image centre, which are more affected by the distortion and image aberrations.Nevertheless, the achieved results in the object space coordinates can be considered compatible with the application.
In Table 4 the RMSE of the differences among intersected and measured control points in the object space are less than 3 cm.These values (m X , m Y and m Z ) were computed from the differences among values estimated by aerial triangulation (rays intersection) and values surveyed with GPS techniques in field.Also, it can be seen in Table 4 that the discrepancies in the 2 available check points are less than 3 cm in X, Y and Z.The average estimated precisions of object points, from the covariance matrix of the bundle adjustment, are less than 3 cm in X and Y and 6 cm in Z.These figures have to be further analyzed, because some points can be affected by blunders and these values can be improved.In order to check the accuracy achieved in a photogrammetric restitution, the grave corners were measured as tie points, and their coordinates were estimated in the bundle adjustment.The resulting cloud points were then imported in a CAD system and the corner points were connected to generate a topographic plant.The accuracy of this plant was assessed by comparing the distances between some corners with values directly measured with a measuring tape and the discrepancies are around 3 cm, as it was expected.

CONCLUSIONS
The paper proposed a combined technique for acquisition of high resolution georeferenced panoramic images, addressed the main advantages of this technique as well as its drawbacks, with quantitative analysis of the results achieved in an experiment with real data.The studied object was a historical Japanese cemetery in west region of São Paulo, Brazil.The main problems with the images are the occlusions, the strong scale change, aberrations and blurring.Even with these problems, the results showed that a plant in a scale of 1:200 can be produced with photogrammetric restitution at a very low cost, when compared to topographic surveying or laser scanning.This process will be used to produce the whole plant of the cemetery; some further developments are necessary to embody the DG techniques reliably.
Figure 1.(a) São Paulo State and Brazil; (b) The West Region of São Paulo.
Figure 3. (a) The acquisition unit; (b) details of camera and INS (c) the Nikon D3100 digital camera and; (d) Bower fisheye lens.
The integrated INS (SPAN CPT -Novatel), composed by a GPS antenna and IMU, was linked to a digital camera with fisheye lenses in nadir viewing to acquire DG data and to enable the DSO ((a) A mosaic with the resampled images (b) Original vertical image acquired by the camera with fisheye lenses; (c) resampled image.Strips and block of images were formed by moving the camera and the whole mount along lines (Figure 4.a).After the calibration step, in which the inner orientation parameters are estimated, it was possible to generate a resampled image (Figures 4.b and 4.c), in order to facilitate point location and transfer in the images with a commercial software.The camera is triggered by a remote control and the flash hot shoe sends a pulse that creates a time

Figure 5 .
Figure 5. Example of one image of the calibration field with ARUCO coded targets.

Table 1 .
Technical data of the Nikon D3100 digital camera and Bower fisheye lens.
The acquisition unit is composed of a Nikon D3100 digital camera with Bower fisheye lens (see Table1for details); a SPAN-CPT Inertial Navigation System from Novatel; a dual frequency GPS antenna; a portable computer to record the INS data; a tripod and a telescopic pole; cables; power supply unit and; camera remote control.

Table 2 .
Table 2 while the results with equidistant model are summarized in Table 3. Results of the camera calibration process with the Collinearity model.

Table 4 .
Results of the bundle adjustment with the resampled images.