AN IMAGE-BASED APPROACH FOR THE ARCHITECTURAL MODELING OF PAST STATES

During the modelling process, 3D models are not eas ily conceived to support changes over time. On the contrary, for the description of cultural heritage, it is often necessary to disp lay not only the actual state of buildings but also their previous states so to understand their modifications. The construction and structuring of spatio-temporal models of cultural heritage demand a double conceptual effort: on one side, 3D models must be r econstructed and structured in space according to a rchitectural concepts; on the other one, such elements should follow the criteria of temporal decomposition. For this reason, links between structured elements should be established to keep track of changes over tim . Moreover, only the current state can be reco nstructed in a rigorous way using a combination of different 3D measurement tec hniques (such as laser scanning and photogrammetry) ; on the contrary, knowledge about past states is conditioned by missi ng elements, whose morphology and dating can reveal uncertain. For this reason, this paper describes a methodological approach to m ake use of the existing iconographic corpus for the analysis and the 3D management of building transformations. The aim is to establish a relation between the iconography use d for the hypothetical reconstruction and the 3D representation that depen ds o it. As a result, 3D representations can be us ed like visualization systems capable of reflecting the amount of knowledge produ ce studying historic buildings. * Corresponding author. This is useful to know for c mmunication with the appropriate person in cases with more than one author.


INTRODUCTION
The study of historic buildings can be based on the collection and analysis of the surveyed elements (if existing) and on iconographic sources such as photographs, drawings, engravings, paintings or sketches.The whole of surveyed elements and iconographic sources describing past states allow historians and archaeologists to verify and validate their hypothesis on the construction techniques and on the evolution of a building.On the basis of these kinds of data, three aspects should be taken into account.Firstly, due to missing elements and uncertain sources, formulating assumptions on the temporal distribution of building states is a difficult task.Secondly, combining 3D reconstructions and traditional 2D documentation is today an efficient way to verify and display analysis and interpretation of historical sites.Finally, this widespread approach has many limits if we consider the possibility of using computer graphics techniques in order to analyze, manage and display scientific content related to the building history.For this reason, iconography can be used for different goals.Firstly, it offers geometric information on anterior states; secondly, it can suggest the visual appearance of a building at a particular period; thirdly it can confirm the temporal distribution of shape transformations, and finally, it becomes a visual support for the study of building transformations.This paper describes a methodological approach that can be applied to a wide range of complex buildings having undergone significant transformations over time.Laying on iconography for the analysis and the reconstruction of building transformations, this approach joins three main aspects in a complete workflow: the spatial and temporal referencing of 2D iconographic sources, a temporal state distribution, and the use of spatial and temporal relations established between 2D iconography and 3D representation for the visual browsing of information.

OVERVIEW OF MODELING APPROACHES IN CULTURAL HERITAGE
3D modelling can concern various temporal states: present and past states.For this reason, in this section we distinguish modelling approaches according to the following goals of the reconstruction.

Representing present states
In the case of existing architectures, a large volume of published studies describes the techniques adopted for heritage site reconstruction.Some approaches are based on the survey and the automatic reconstruction of architectural elements by means of laser scanning (Sideris & Roussou 2002).Other approaches use to extract from points clouds significant profiles used for the following stage concerning the interactive reconstruction of shapes.However, as affirmed by (Remondino et al. 2009), both these methods can be constrained by the building position and morphology that can limit the point acquisition.
Image-based modeling and rendering techniques are used more and more for 3D building reconstruction (P.E. Debevec et al. 1996).Multi-images solutions are based on establishing projective relations between a set of photographs of a single scene in order to extract characteristic points for the 3D reconstruction workflow (Werner & Zisserman 2002;V.H. et al. 2009).Single-image solutions make use of knowledge related to the geometric invariants of some primitives in order to estimate the composition of elements in the photographed scene (Taylor et al. 1996).As various researches have demonstrated, these techniques can be combined in order to get the best result according to the level of detail needed for the study.All these techniques can be integrated to achieve the most complete realistic result (De Luca 2006;Remondino et al. 2009).

Representing past states
If the goal is to represent past architectures, other techniques are adopted for their reconstruction.In this case, as the level of detail reduces, interpretation level increases.Some flexible solutions are adopted for the 3D reconstruction of past states.These approaches are based on the photogrammetry principles applied to old photographs, paintings with correct perspective and postcards (S.F.El-Hakim 2001).This method uses different constraints such as points, coordinates, surfaces and topological constraints.In the case of paintings with pseudo-perspectives (S.F. El-Hakim et al. 2008), this method requires that part of the site still exist, so as to create an image in correct perspective and to use it to reconstruct the entire site.
Other approaches basing on Constraint Satisfaction Problem allow old photographs to be order in time and after the calibration stage, textures are automatically applied on 3D models (Schindler et al. 2007).Also procedural methods allow past states to be reconstructed by means of generative mesh modelling that produces complex manifold surfaces from simpler ones (Haegler et al. 2009;Maïm et al. 2007).Finally, some approaches use CAAD methods to reconstruct past states by deduction and tend to distinguish hypothetical parts and historical states displaying them by color coding (Theron et al. 2007), by transparency (Kensek et al. 2004), wireframe or broken lines (Strothotte et al. 1999).

FUNDAMENTAL PROBLEM
All the studies reviewed so far, however, suffer from the fact that geometry reconstruction of present and past states are currently treated as disjointed.Difficulties arise when an attempt is made to manage evolutions.Recent studies fail to consider two fundamental issues: firstly, buildings can undergo transformations over time, and secondly, the reconstruction of disappeared states lays on a more or less wide inconsistency level.

Building transformations and lifespan
Potential changes of buildings trough their existence are abundant and various.In fact a building can undergo both physical transformations and qualitative ones.Some transformations affect the entire building lifecycle (buildings are built, destroyed, rebuilt, they can also be extended, attached to other ones, or divided into several buildings), others transformations imply only certain parts of its morphology (so they undergo minor geometrical modifications or their components can be displaced elsewhere).In addition, factors involving transformations can cause gradual and slow changes, or sudden ones.For instance, a demolition can take few minutes or hours, a construction can take some months or several years, furthermore historical building deteriorations may take centuries or millennia.So, temporal granularities can vary consistently.

Iconography and inconsistency
In cultural heritage, information about artifacts is transmitted in accidental way, by means of known sources and discoveries.Sources can be very heterogeneous in the type and the quality of the provided information.The more we go back in time, the more sources are not sufficient to provide comprehensive information on morphology and lifecycle of buildings.Three factors characterize information provided by the iconographic corpus.Firstly, the source quality depends on its state of preservation, on the representation tools and on the purpose of the description.Sometimes sources provide just rough information on building shapes and proportions; sometimes they offer more metric information.Secondly, the consistency level depends on the reliability of information communicated by sources, and it bases on the comparison with other sources.
Finally, the objectivity level is linked to the source author, because he often transfers graphically his viewpoint according to his life context, and sometimes he represents architectures according to projects not yet realized or completed.As graphic sources are very heterogeneous, interpretation problems can concern spatial information (shape and position) and temporal one (construction and demolition periods, or still lifecycle).Assumptions are a particular case of uncertainty interpretation applied to architecture.Particularly, historians and archaeologists base on different ways of reasoning to formulate their hypotheses: the deduction and the analogy process.

THE ICONOGRAPHY-BASED APPROACH
The goal of this study is to establish a relation between the iconography used for the reconstruction of historical states and the 3D representation depending on it.As a consequence, this research describes the iconography-based modeling process of historical states and the 3D model structuring according to the temporal point of view.The complete methodological approach is based on the analysis of the existing iconographic corpus and the management of building transformations.As this approach is wide, only some steps will be detailed in the next paragraphs.To simplify the comprehension, we explain this methodology describing an historical site whose lifecycle is particularly reach of transformations and for which we dispose of several sources: the Trophée des Alpes at la Turbie.Changes undergone by this site (can be known by means of historic photographs of the twentieth century and older engravings concerning the transformation of the roman monument into a fortress.Furthermore, Formigé's drawing supposes the original shape of the Augustan monument at the first century BC (see Figure 2).

3D modelling of the current state
Our method starts with the elaboration of an accurate 3D model of the building current state (see Figure 3).This step, not detailed in this paper, is based on a hybrid acquisition (3D laser scanning and photogrammetry), and is followed by a reconstruction approach already described in (De Luca et al. 2006).This 3D model makes the general support for the following operations of space-time iconography referencing like for the semantic structuring step.In order to manage the temporal structuring, this 3D model was produced by taking into account requirements of structure disassembling (block per block).The 3D model is structured by means of associations of concepts expressed through graph description.The spatial reference of morphological features is based on the method described in detail in (De Luca 2006).Basing on the chosen point of view, procedures allow creating groups, subgroups and entities.
Figure 3. 3D model of the building current state elaborated mixing laser scanning and photogrammetry techniques.

Spatial referencing of sources
This method establishes a projective relation between iconography and the 3D model of the building current state by means of a spatial resection procedure.A detailed description of this approach has already been presented in (De Luca et al. 2010).The spatial resection of old photographs poses various problems.Firstly, the lack of information relating the camera used at the shooting time can be important.Secondly, the image format can have a lower precision degree in the case of images that have been modified or trimmed.In this case, intrinsic parameters estimation becomes the most difficult task.For the estimation of the spatial resection we use a versatile camera calibration algorithm (Tsai 1986) needing at least 11 2D/3D correspondences with the intent of providing intrinsic and extrinsic parameters of the camera.In order to obtain a better parameter estimation, characteristic points are accurately selected so to distribute homogeneously the 2D/3D correspondences: they should be chosen on the image maximal extension and depth range of the 3D scene.For this experience, six old photographs with global errors (average of re-projection errors) included between 6 and 25 pixels have been superposed (see Figure 4).

4.2.2
Other projections.Sources can follow other projection rules.Actually a work is in progress in order to integrate other kinds of sources into the system.In the case of iconography with cylindrical projection, 3 correspondences 2D/3D are sufficient to determine the scale of the iconographic source in the 3D work space and to determine the projection reference plane.Once the plane is defined, it can be shift along the projection straight.As for iconography in perspective projection, in order to reduce the correspondence error, it is better to choose 2D/3D coordinates on the image maximal extension.In the case of pseudo-perspectives/pseudoaxonometries, a manual method is adopted.It consists in referencing manually the image basing on the existing elements, than to adapt the image plane position, orientation and scale according to the elements to be modelled.
For all these cases, sometimes there can be morphological incoherencies between building past states (2D iconography) and the surveyed state (3D model).That introduces various problems in the 2D/3D correspondences selection step.The solution consists in leading this step using the unmodified parts of the building.

Temporal referencing of sources
If the information level is consistent, iconography should be referenced in time, too.A procedure allows the user to attach temporal attributes, including the snapshot time and the state described by the snapshot.Each state represents a period which artifact does not undergo changes in, it is therefore characterized by starting and ending dates.Attributes are attached through Mel scripts of type addAttr and setAttr.

3D modelling and structuring of past states
The construction of a geometric model of past states is based on the comparison between two historical states represented by the iconography (2D) of the past state and the 3D representation (of the current or past state already modeled).In order to structure the model, several tools detailed below have been customized.Three modeling operations permit to restitute the geometry of architectural entities according to the temporal distribution of transformation events: adding, modifying and deleting.Surface creation allows automatically temporal and transformation attributes to be generated.

Modeling principles:
for the position and orientation of created, modified or deleted entities the image-basedmodeling capabilities are used: entities are controlled by means of the perspective projection rules on the image.Specifically, the spatial resection of the photographs generates a working environment which permits to retrieve geometrical information from the current state 3D model, laying on coordinates, straights, reference planes, etc.In order to take into account the information lack due to the absence of historical contemporary images or to the geometric inconsistencies between two states, different interactive modeling solutions were explored using the intersection principle: the intersection of two projection lines corresponding to homologous points selected on two images (both of them related to the same temporal state or two different ones); the intersection between the projection line of a selected point on the image and an axis of the 3D scene; finally the intersection between the projection line of a selected point on the image and a reference plane previously defined in the 3D scene.
The characteristic points extracted from the shapes are used in order to position, handle and deform geometric entities.Different interactive modeling techniques are used: primitive adjusting, face extrusion, meshing, etc.When geometric information about shapes is missing or uncertain, standard geometric properties of architectural shapes -i.e.parallelism, orthogonally, symmetry, etc. -are taken into account.

Structuring principle:
Once the source is space-time referenced, it becomes the guide for time structuring.Temporal referencing is based on the comparison between the 3D model and the analyzed source.Referencing can be manual or automatic.In this last case, the referencing bases on time slots data already assigned to the source.If the procedure is performed automatically, such attributes are attached to entities starting from the operations of creation, division, union and destruction.They identify each state and refer to the beginning and end of the construction and demolition periods.Through them, creation, existence and demolition intervals can be easily deducted.A procedure allows temporal attributes to be updated, too.Even if reconstruction should not follow a specific temporal order, temporal referencing is easier if two consecutive states (in chronological or anti-chronological order) are analyzed.To save the history of each morphological element, the kind of relations composing building lifecycle should be declared.These attributes describe the history of transformations (simple morphological variation of artifacts or more complex ones such as division, union or reconstruction).
An automatic procedure stores the kind of change, the time period concerning a specific change, entities and groups involved before and after a transformation and any new created group.Attributes describe the basic cases of creation, division, union, destruction, reconstruction and degradation, but other kinds of transformation can be inserted by the user.

Reliability level
If sources are heterogeneous and imprecise, then space-time 3D model needs to be qualified according to its reliability.In particular, some tools are customized to take into account several factors.The first concerns the reconstruction classification and describes the hypothetical or certain level of the 3D model.The second one is the type of uncertainty: it concerns the spatial level (entity presence/absence in the iconography, source quality, geometric resolution level, and certainty of reasoning) and the temporal level (temporal information provided by iconography, granularity of time intervals, certainty of reasoning).The last one defines the adopted interpretation process (analogy, deduction or mixed).

Verification tool
Structuring the model according to time stratifications can result more or less complex according to the number of states and transformations.Thus, a constant control of the structuring should be taken into account in order to optimize the time of work and revision.In order to do that, various manipulation, verification and modification tools have been introduced in the interface.These tools will not be detailed in this paper: for further readings, see (Stefani 2006).

Visual enrichment
At this step the reconstructed geometry can be visually enriched by texture extraction and projection starting from iconography.For instance, in the case of perspective projections, the camera image plane is projected on the 3D model and the part of the image corresponding to the reconstructed geometry is isolated.Geometry can be enriched by textures extracted by one or more images.As iconography concerning past states is not abundant, usually textures are extracted starting by a single image.

Attributes and representation storage
Once the 3D model is structured according to time, automatic procedures ensure the export of the geometric entities and the affected attributes.3D representations are exported in .NMO format through conversion methods.Spatial, temporal, certainty, and transformation attributes are stored in a .txtformat file by means of fopen/fprint procedures.By means of this file, attributes can be imported into a MySQL relational database for web-based applications (De Luca et al. 2007).

VISUAL BROWSING
In the domain of architectural heritage, geometry is not sufficient to understand historical site transformations.The 3D scene remains the main visualization and comprehension tool, but other visualization supports (Tufte 2001) are needed to understand relations among artifacts and to manipulate geometries.Our approach aims at analyzing the actual relevance of processed spatial and temporal data.With the aim to build a visual interface for understanding spatio-temporal building transformations, we are working on the development of a visual browsing system that connects iconography sources to the 3Dmodel by means of a temporal graphic notation and a spatial search engine.
Our web system (actually in development) is based on a threepart architecture: 1-real-time geometry manipulation: a 3D scene developed in Virtools DEV allows the download, display and manipulation of different restitutions; 2-building transformation visualization: a SVG interface allows the reading of historical transformations by means of server-side dynamic graphs generated with PHP on MySQL; 3-online consultation: a PHP page allows the user to select projects, temporal states, assumptions, and to display of data.
Data is organized in a MySQL developed relational database, containing information regarding geometric entities (spatial and temporal information and certainty degree).
In order to understand cultural heritage, the representation system needs to go beyond simple geometric restitution: it is necessary to take into account different visual solutions to understand and compare data.The qualitative attributes linked to geometrical entities during the modeling stage are used as entries for a dynamic representation system based on Information Visualization techniques (Stefani et al. 2010).The system focuses mainly on the following kinds of relations.

Spatial relations
Our search engine for iconography retrieval is based on the concept that 3D model allows for the access to information about a building using its morphology as interface (Dudek & Blaise 2003).Spatial referencing is used like a way to establish relations between 3D and 2D information (Busayarat et al. 2008).Two main queries are possible: the entity-based and viewpoint-based iconography searching.

Entity-based query:
this search allows finding the whole of iconographic sources related to a 3D entity by visualizing (with relative precision) the place where it was, or where it should have been, or where it could have been (see Figure 7).It uses a ray intersection method to determine which images are oriented to which object.The system tests the intersection between the camera frustum of stored images and the geometry of a selected entity (taking occlusions into account).

View-point based iconography query:
it determines iconography close to the current position and orientation of a navigation camera in the 3D scene by intersecting the spatialized iconography frustums and the observation camera ones.Three parameters are used in order to filter the results of the intersection process: firstly, the distance between the navigation camera and the position of each optical center concerning iconographic sources; secondly, the angle between the orientation of the navigation camera and the orientation of each iconographic source; finally, the distance between the current temporal state (displayed in the 3D scene) and the temporal state concerning the founded iconography.

Temporal relations
The visual browser displays states and relations between them.The following tasks are possible.

Transformation display:
transformations are represented by means of historic graphs allowing the display of evolutions and mutual relations among buildings throughout time (see Figure 8 and Figure 9).For all the lifespan of a building, each graph retains the transformation history, including complex transformations such as union, division and reconstruction.Each graph represents the link between the 3D scene and transformations, and interacts with geometry to display transformations both on the scene and in the graphic notation.Moreover, the representation scale of graphs can be adjusted: transformations are displayed at different scale levels according to the dimension of the selected time period.In this way, some small transformations which are not visible on a great scale become visible on a smaller one.These graphs are generated dynamically.For each building element, an algorithm firstly orders dates, and secondly, according to the transformation affected to each date, it assigns some graphical symbols (circles, smoothed rectangles, arrows, etc.) so to compose the temporal notation.

5.2.2
Ancientness display: colour coding can be useful to better distinguish building components having different temporalities.Colour tones can be adopted according to the site context: for instance, if dates are too close to one another, visualization by colours not proportional to dates will offer a more useful result than a proportional visualization (Figure 9).

Uncertainty
Hypothesis is distinguished by certain artefacts by means of variation of the shading property of the entities (figure 10).The diffuse colour parameter is set according to 4 percentage levels corresponding to: existing elements modelled by survey techniques; entities belonging to past states modelled by modifying operators; entities created in past states by means of adding operators; missing entities modelled inferring shapes by analogy and deduction techniques.The current work has underlined deeper reflections that constitute a limit of this approach and pose the basis for future researches.Firstly, when experts analyze historical sites, sometimes additional kind of queries can be needed.This can imply the necessity to change the spatial granularity.However, in the current investigation, the spatial structuring is difficult to modify because of the large amount of spatial data stored in the database.Secondly, results have shown that the 3D model can be the common factor linking heterogeneous sources.Actually we are working on image semantics in order to associate on one hand a set of semantic layers to iconographic sources and, on the other hand, to qualify the enriched model with further visual information concerning its textures.

Figure 1 .
Figure 1.Pipeline for the iconography-based approach The presented functions are included in the NUBES platform, developed at the MAP Laboratory.In particular, the modelling process is achieve with NUBES Forma, a 3D reconstruction tool based on point clouds and photogrammetry, developed as a Maya Plug-in in MEL (Maya Embedded Language) and C++.It is displayed in NUBES Visum, a 3D Web Information System at the architectural scale, developed in PHP/MySQL and Virtools DEV.

Figure 2 .
Figure 2. Ancient photographs describing evolution of Trophée des Alpes in the early 20 th century.
projection.In the case of photographs, a set of correspondences established between the photograph (2D) and the current state model (3D) are necessary to determine the camera geometrical model associated to the photograph(Hartley & Zisserman 2004).It uses the focal length and distortion (intrinsic parameters) and the translation and orientation (extrinsic parameters) related to the geometrical model of the image.

Figure 4 .
Figure 4. Spatial referencing of historic photos on the 3D model of the building current state.

Figure 5 .
Figure 5. Modifying tool: split operation of the actual state of the colonnade according to entities visible in the previous state.
diagrams: as entities are described by temporal attributes, a Mel procedure enables to display state diagrams describing the lifespan of each geometric entity of the 3D scene.They represent the existence states and display the dates of creation, of demolition, and the duration.These diagrams can be used as a complement to the 3D environment: in fact the user can modify temporal attributes interactively and easily in the interface by reducing or expanding diagrams, so modifying the time values stored in the geometry table.

4. 6 . 2
State visualization: if temporal data are linked to the 3D model, the objects of the scene are automatically referenced in time (setting key frames).By means of a timeline, it is possible to display states in chronological or anti chronological order, to freeze temporarily the animation, to slow or accelerate it, or still to move forward or backward the timeline cursor step by step.4.6.3State manipulation:in order to verify the structure and the model, the interface allows on one side the formulation of temporal queries (by dates or periods) to select elements of the 3D scene, and on the other side, multiple visualizations of 3D entities (according to affected attributes) to understand and verify the structuring before the validation of the model.

Figure 6 .
Figure 6.User interface displaying state diagrams, timeline and window attributes.

Figure 7 .
Figure 7. Spatial queries: entity-based searching of iconographic sources

Figure 8 .
Figure 8. Temporal notation for the description of building transformations.

Figure 9 .
Figure 9. Ancientness display by means of color tones.

Figure 10 .
Figure 10.Uncertainty visual coding for the hypothetical representation of a temporal state 6.CONCLUDING REMARKS This research has proposed a complete workflow integrating different tasks in order to model and visualize architectural sites.The spatio-temporal analysis of buildings is possible by means of an extensive use of iconography.Despite of the heterogeneity of sources and the difficulty of creating a digital mock-up covering the lifecycle of buildings in cultural heritage, a privileged link between spatial and temporal information has been established, by choosing a semantic structuring according to the point of view of the study.Moreover, present and past states of the site have been connected in order to follow the history of transformations.The current work has underlined deeper reflections that constitute a limit of this approach and pose the basis for future researches.Firstly, when experts analyze historical sites, sometimes additional kind of queries can be needed.This can imply the necessity to change the spatial granularity.However, in the current investigation, the spatial structuring is difficult to modify because of the large amount of spatial data stored in the