3 D PAINTING DOCUMENTATION : EVALUATION OF CONSERVATION CONDITIONS WITH 3 D IMAGING AND RANGING TECHNIQUES

The monitoring of paintings, both on canvas and wooden support, is a crucial issue for the preservation and conservation of this kind of artworks. Many environmental factors (e.g. humidity, temperature, illumination, etc.), as well as bad conservation practices (e.g. wrong restorations, inappropriate locations, etc.), can compromise the material conditions over time and deteriorate an artwork. The article presents an on-going project realized by a multidisciplinary team composed by the ENEA UTICT 3D GraphLab, the 3D Optical Metrology Unit of the Bruno Kessler Foundation and the Soprintendenza per i Beni Storico Artistici ed Etnoantropologici of Bologna (Italy). The goal of the project is the multi-temporal 3D documentation and monitoring of paintings at the moment in bad conservation’s situation and the provision of some metrics to quantify the deformations and damages.


INTRODUCTION
Heritage conservation and preservation should be always coupled with adequate and accurate documentation and surveying actions.The concept and application of "preventive conservation and monitoring" became a widespread concept in Europe after the Vantaa ICOM-CC conference in Finland (2000) where everyone agreed on the importance of a careful monitoring of the environmental conditions surrounding cultural heritage.Several actions took place after the Vantaa conference and a number of rules and solutions have been sought in the following years.The main factors impacting on the conservation state of an artwork were recognized to be, as far as microclimate is concerned, relative humidity, temperature and light.Biological aspects such as insect attacks were also considered relevant and therefore to be studied and monitored.Another important identified factor was the air's quality and air dispersed pollutants which are deposited on the artwork's surface and may cause a noticeable decay even in a short time.Recently, several scientific projects -often EU-funded -started whose aim was to define and meet the most up-to-date guidelines in preventive conservation (ERA -Environmental Research for Art Conservation, MIMIC -Microclimate Indoor Monitoring in Cultural Heritage Preservation, LiDo (Light Dosimeter for Monitoring Cultural Heritage, Development, Testing and Transfer to the Market, SMOOHS -Smart Monitoring of Historic Structures, LaserACT -Laser multitask non-destructive technology in conservation diagnostic procedures, etc.).One of the most successful and recent idea was to consider some of the above mentioned issues not separately but the resulting global impact factor as all causes act concurrently.Nowadays the installation of climate control units to provide human comfort as well as good conservation conditions inside museums is facilitating conservation and preservation actions.On the other hand museums' depots and storage rooms which host large part of collections are sometimes not maintained in a stable environmental conditions due to lack of funding and bad practice.This inappropriate practice is crucial for all artworks, particularly for paintings conservation (especially for those realized on wooden supports).These kinds of artworks indeed are very sensitive to the variation of some parameters like light, temperature and humidity.10829:199910829: (Mibac, 1999)).According to these guidelines the optimal thermo-hygrometric parameters for the conservation of paintings on wooden support are shown in shown in Table 1.When thermo-hygrometric parameters are altered, wooden paintings can suffer of support's deformation, resulting in curving and warping, which can affect the depicted surface causing cracks, pigments detaching until the worst and unrepairable break of the support (Fig. 1, a).
Starting from this assumption and following the concept of preventive conservation and monitoring, it was decided to carry on a monitoring activity on some selected paintings.A traditional monitoring of thermo-hygrometric parameters was combined with 3D imaging and ranging techniques.Realitybased 3D techniques (Remondino & Rizzi, 2010) can indeed offer a useful support to identify and measure the conservation's status of paintings on wooden tables assessing, at submillimetres level, the deformation of the support assuming it was originally flat.For this project, three paintings were selected among a collection in the deposits of the Pepoli -Campogrande Palace in Bologna (Italy):  Saint John the Baptist (198 x 62.5 cm) by Giacomo Francia, ca XVI century;  Cretan icon (85 x 55 cm), ca XVII century (Fig. 1, right);  Oil on canvas (30 x 20 cm) by Ludovico Cavalieri, ca XIX century.The three paintings (the first two are on a wooden support while the third one is an oil on canvas) were digitized using photogrammetry and structured light scanning techniques with the following aims:  analyse the geometric features of the paintings (e.g.brush strokes);  visualize additional information about the style;  influence the overall visual impression of the painting;  measure the shape of the wooden support;  assess the deformation's status assuming an original planarity of the support;  monitor the shape of paintings on a medium/long term period;  produce high-resolution orthoimages for digitization, analyses and restoration purposes.Beside these aims, the project is seeking an efficient costeffective procedure for a systematic 3D recording of paintings in order to study the deformations occurred over time and plan the optimal conservation conditions and policies.The reasons behind the use of both imaging and ranging techniques are to evaluate and compare the benefits and drawbacks of each technique in terms of:  time necessary to digitize a complete painting (front and back side);  operational difficulty and limitations;  dependence on location constraints.At the end of the project, the comparison between the two techniques will allow to draft guidelines for systematic 3D surveys in application scenarios similar to those shown in this article.

RELATED WORKS
In the field of painting's conservation there is a number of noncontact techniques which allow detailed analyses of the artwork's surface and support in the visible and non-visible spectrum.Multispectral imaging and 3D surveying techniques can be used for different purposes to provide reliable data to restorers for planning the restoration process or adjust the environmental conditions.In recent years the study and characterisation of artworks' surface by means of different non-invasive digital techniques is quickly evolving (Fontana et al., 2005;Blais et al., 2007;Lahanier et al., 2008;Remondino et al., 2011;Granero-Montagud et al., 2013).Typical outputs are usually represented by pigment identification, colour measurements, extraction of geometric features (brush strokes details), shape' measurements, etc. Range-based 3D modelling techniques (Guidi et al., 2004;Akca, 2007;Blais et al., 2008;Breuckmann, 2011), such as laser scanners and structured light sensors, and image-based 3D modelling technique, such as photogrammetry (Robson et al., 2004;D'Amelio & Lo Brutto, 2009;Barazzetti el al., 2010) can provide for precise and reliable 3D geometrical and radiometric information useful for detailed analyses and inspections.Both approaches allow the retrieval of very high geometric details with reported spatial resolutions ranging from 60 µm and 400 µm.With respect to related works, the study presented in this paper reports the analysis of the whole paintings (both front and back sides), 3D reconstructions with dense image matching methods for measuring fine geometrical details on the painting surface and the use of spherical targets for accurately connect the two sides in a unique reference frame.Comparisons between active scanning and photogrammetry are also reported.

SURVEY CAMPAIGNS
The structured light scanning and photogrammetric acquisitions were realized within one week, to guarantee neither significant shape variations nor changes in environmental conditions (temperature, humidity, etc.) between the surveys.Both surveys were planned to deliver similar geometrical results with comparable sub-millimetre spatial resolution.Different strategies for each technique were exploited to solve the main problem of surveying a thin flat object on both the sides.Besides the difficulties arisen from the geometric shape of paintings and their brilliant surfaces, many constraints were present, generally dictated by the location and preservation of the paintings themselves.The vulnerability of this class of objects requires a careful project planning that takes into account their sizes, surface materials as well as limited mobility.Indeed, the space in the deposits is usually very narrow and, furthermore, rarely the paintings can be moved from the room in which they are stored.For this reason the locations where the paintings are situated represent one of the main constraints that do not allow to find a standard procedure applicable to any case study.From this point of view the deposits of Pepoli -Campogrande Palace in Bologna can be considered a very complicate scenario for photogrammetric and laser scanning acquisitions (being the space available in the deposit's room approximately 3x2 square meters) and so a representative case study.

RANGE-BASED SURVEY
The paintings digitization was performed exploiting a structured light Breuckmann SmartScan 3D HE instrument.This active sensor allows the simultaneous recording of 3D geometries and colour information.The instrument features two cameras with a resolution of 5Mpx (2.452x 2.056 pixel), a field of view of 240 x 200 mm and a measuring depth of ca 150 mm.This ensures a X-Y spatial resolution of 100 µm for single range map.According to the manufacturer specifications, the active system can achieve a X-Y spatial resolution of 26µ according with the VDI 2634 specification (Luhmann, 2000;VDI/VDE, 2002, 2008).In order to avoid likely deformations during the registration of the range maps due to low overlapping, a sheet of polystyrene used as additional plane constrain was placed behind the artwork.Some nails were skewered on the polystyrene sheet all around the painting and some magnetic targets (4 mm diameter) placed on top of them (Fig. 2a).During the scanning process (4 days for all the 3 artworks), the targets were acquired together with the borders of the wooden artwork in order to create the frame of the painting, filled later with the scans of the internal parts.This method allowed to avoid deformations which can occur when digitizing flat surfaces.
In order to acquire on both sides of the artworks, a 2 mm adhesive targets were placed on the lateral edges.
The traditional range data processing pipeline was applied using the Optocat proprietary software package.The surrounding range maps were pre-aligned using the automatically recognized targets while the internal range maps were manually registered.
Finally an ICP minimization procedure refined the alignment of all the acquired maps delivering a unique and dense point cloud for every artwork (Fig. 2b).Table 2 summarizes the overall RMSE values after the ICP processes.It is worth notice how the global RMSE of the alignment is significantly higher when front and back sides of the painting are merged together in a unique reference system (Cretan icon).
After the alignment process, all the range maps of each painting were merged together in a single polygonal model (Table 3) with a spatial resolution of 0.4 mm.

PHOTOGRAMMETRIC SURVEY
A photogrammetric survey of a painting can be considered a special case of surveying a flat surface i.e. a relatively simple shape to model.Today digital photogrammetry can be a feasible and flexible solution and a quasi-standard procedure may be found to suggest general best practices for systematic 3D surveys.The main difficulties arise when both the front and back of the painting must be surveyed in a unique reference frame as it is the case of this project where a sub-millimeter global accuracy was required.The painting's shape is such that a standard camera network consisting of a sequence of overlapping photos all around would not be feasible or at least would be unreliable.Indeed, the front and back parts of the paintings would be connected by the sides of the painting that are normally very narrow leading to a geometrically unstable connection of back-to-front parts.These difficulties can be overcome using external object points (e.g.targets) positioned strategically around the painting.
For the presented artworks, the main purpose of the photogrammetric survey was to record the paintings' 3D shape and a faithful photographic color reproduction with a spatial resolution of ca 0.2 mm, demanded to monitor the surface and the presence of possible fractures.
In order to survey the front and back of the paintings, two separate surveys for each painting were planned.These surveys were intended to be processed separately and then merged using a similarity transformation in a unique frame by means of common spherical targets.
The paintings were placed vertically on an easel and 10 spherical targets attached to two photographic light-stands were placed on the lateral sides in order to be visible, at the same time, from the front and the back side of the painting (Fig. 3).The camera used for the survey was a Nikon D3X featuring a 24 megapixel full-frame CMOS sensor (6 m pixel size).Given the requirements of sub-millimeter accuracy for 3D geometry and 0.2 mm spatial resolution for the texture, a photographic scale of 1:30 was selected.With such scale, the ground sample distance (GSD) of the images was about 0.18 mm.A 50 mm prime lens was mounted on the Nikon camera yielding to a camera-to-object distance of about 1.5m.The camera's autofocus was disabled and the lens focusing ring fixed with a piece of insulating tape to avoid accidental changes of interior orientation parameters during the photogrammetric survey.
In order to maximize the rapidity of the photogrammetric survey and achieve high accurate results, an automatic photogrammetric approach including self-calibration was planned for each painting.To avoid projecting coupling between interior and exterior orientation parameter due to the flat shape of the object a proper camera network was planned including convergent and rotated images.This step is mandatory if reliable results are expected and to avoid a degradation of object points precision (Fraser, 1982;Kraus, 1997;Nocerino, 2013Nocerino, , 2014)).In all the photogrammetric projects, two scale bars in horizontal and vertical positions were placed at the bottom and aside the painting.The paintings were properly lighted up using two calibrated photographic lamps.
To reduce the specular reflections, images were acquired using polarized lights.Images were mandatorily taken on a stable tripod and each image was checked for possible blur presence caused both from very long exposure time and from limited depth of field.Indeed, in this project using an aperture value of f/16 the expected depth of field resulted about 340 mm (considering a circle of confusion of 12 m that is also the diffraction limit for the selected aperture value).The extension of depth of field is an important additional parameter that must be considered when surveying a heritage asset in quasi macro photogrammetry (Menna et al., 2012).
The images were automatically processed using Agisoft Photoscan, the extracted tie points were filtered and reduced in number using the procedure proposed by Nocerino et al. (2013Nocerino et al. ( , 2014) ) which takes into account multiplicity of rays and their spatial distribution.The filtering procedure was necessary to keep only well distributed reliable points and to reduce the computational time when using standard bundle adjustment software.Tie points were then imported in Photomodeler and a self-calibrating bundle adjustment was run to obtain main statistical values and inter-correlations between camera parameters (all values missing in Photoscan).
For the Saint John the Baptist painting, at the end of the bundle adjustment, the automatic tie point extraction on the painting surface achieved a RMSE of image observations of 0.45 pixels.
The average spatial precision the derived 3D coordinates was 0.035 mm on the painting plane and 0.075 mm along the depth axis, corresponding to a relative global precision of 1:52,000 (Fig. 4-middle).High redundancy with up to 25 rays per each tie point was achieved (Fig. 5-right).For the other paintings similar values were obtained.
The centres of the spheres were marked using the semiautomatic tool implemented in Photomodeler.It is not clear from software documentation if the semi-automatic algorithm is able to remove the eccentricity of imaged spheres.Given the high RMSE residuals of about 0.9 pixels on the spheres, it is likely that the eccentricity remains uncorrected.The average 3D precision of the spheres was respectively 0.1 mm on the painting plane and 0.2 mm along the depth axis, corresponding to a relative global precision of 1:9,100.Further analyses are being carried out to improve the accuracy of spheres marking using an in-house developed algorithm.
The front and the back of the paintings were merged in an unique reference frame after computing a similarity transformation by means of the spheres.For the Saint John the Baptist painting, the average RMS of residuals of the transformation was about 0.4 mm in the paintings plane and 0.2 mm orthogonally to it.The differences between expected precision and residuals from the similarity transformation indicates that some systematic effects are present in 3D computation of spheres' centers.A dense image matching procedure was afterwards run in Photoscan with a sampling interval of 0.7 mm producing a point cloud of about 3 million points per side.An orthophoto with a spatial resolution of 0.2 mm/pixel was also produced (Fig. 5 -a) and available online through HDview free plugin (http://www.afs.enea.it/project/graphlab/hd/francia/battista_francia.htm).

FEATURES AND DEFORMATION ANALYSES
The produced 3D models were then inspected and analyzed to identify any features not visible at first sight.During the initial visual inspection the Cretan icon showed significant bending toward the borders, whereas the Saint John the Baptist painting didn't show any apparent deformation.
To enhance hidden characteristics, some shading algorithms were applied.Inlays were identified in the lateral and upper part of the painting (noticeable also during visual inspection) and at the bottom, which were not evident.Assuming the original planarity of the wooden support of the Saint John the Baptist painting, a best fitting procedure was run.The result shows a slight deformation which reaches its negative and positive peaks on the lower left side (19.9 mm) and on left up corner (16.9 mm) for a maximum absolute range of about 37 mm (the entire painting spans ca 198 x 62.5 cm).By modelling in threedimensions both the front and back of the paintings in a unique reference system it is possible to correlate the observed deformation pattern for further restoration interventions.In Figure 5b and 5c the front and back parts of the painting are shown side by side.The back has been flipped horizontally to be compared directly with the front.The deformation pattern seems to be quite dependent on the wooden frame on the back, especially in the upper part.
The evident deformation of the Cretan icon (Fig. 1b) translates into a characteristic colour pattern of deformation (Fig. 6) and was assessed in the order of ~17 mm.The oil on canvas by Ludovico Cavalieri was visualized with different shading techniques (Fig. 7).Surface details (i.e.brush strokes) were highlighted revealing elements of the author's artistic style as well as providing elements for the restoration process (detachment, loss of pigments, etc.).

CONCLUSIONS
The article reported a study on 3D range-and image-based techniques used to provide for an efficient cost-effective noninvasive procedure for a systematic 3D recording and multitemporal monitoring of paintings.High resolution orthoimages were obtained from the photogrammetric method to identify and document fine fractures of painting surfaces.A method for inspecting the colour degradation over time is being also implemented using the procedure exposed in (Remondino, 2011).Systematic orthophoto, generated at different epochs, will be compared by supervised inspections and with automatic methods.Shading techniques were also used for exploiting the potentialities of 3D techniques in characterizing authors' style.
As already stated, the choice of the right digitization technique is fundamental but it's primarily dictated by the final goal of the project and considering time, local constraints, required accuracy and budget.
The structured light technique allows to acquire highly accurate and reliable geometric data at a very fast speed (acquisition time of ca 1 second per range map in fast mode).Drawbacks are represented by the cost of the active sensor, its transportability, the need of a powerful computer for data processing (a workstation 8 double cores, 24 GB RAM was used) and, normally, a time consuming post-processing phase (for example the Cretan icon required 2 hours of data acquisition and ca two days of post-processingthe same artwork needed ca 30 min for the image acquisition and 2 hours of post-processing).A low-quality texture (with respect to SLR cameras), often influenced by the environmental conditions, is generally achieved however a separate photographic setup can be realized where possible.

Figure 1 .
Figure 1.An example of a damaged painting (a).A Cretan icon showing a clear bending effect (b).Both paintings are located in the depots of the Soprintendenza per i Beni Storico Artistici ed Etnoantropologici of Bologna, Italy.
a) 3D structured light scanning survey of the Saint John the Baptist painting.On the upper part of the image a closer view of the used targets is shown.b) The 195 range maps of the Saint John the Baptist painting after the final ICP alignment which closed with 40 m RMSE.

Figure 3 .
Figure 3. Photographic setup for the Saint John the Baptist painting with two lights symmetrically on the sides of the camera.

Figure 4 .Figure 5 .
Figure 4. Photogrammetric processing of the Saint John the Baptist images.Left: distribution of the automatically extracted tie points over an image and some of the (ten) spheres used for connecting the back and front of the painting.Middle: camera network geometry and tie points in 3D -colors represent the estimated spatial precision (sigma xyz) of the derived 3D coordinates (units in mm).Right: color-coded visualization of the multiplicity of the derived sparse point cloud, i.e. number of images in which each 3D point is visible.

Figure 6 .
Figure 6.Cretan icon: an image of the artwork with the spheres and bars used during the photogrammetric processing (above).The result of the best fitting plane of 3D coordinates (in mm) obtained with the structured light system (below).

Figure 7 .
Figure 7. Oil on canvas by Ludovico Cavalieri: an image of the artwork (above) and the structured light system's 3D model visualized in shaded mode (below).The brush strokes are clearly visible thanks to the submillimetre geometric resolution of the digitization.

Figure 8 .
Figure 8. Deformation analysis on the Saint John the Baptist painting assuming a planar shape of the artwork.Photogrammetric 3D model (left) and structured light scanner 3D model (right).

Table 1 .
Thermo-Hygrometric parameters for the conservation of optimal chemical and physical conditions of artworks.The most used specifications for indoor climate control in Italian museums are contained in the Italian standard UNI

Table 2 .
The final merged models consisted of millions of polygons as shown in the table below.RMSE values of the alignment procedures.

Table 3 .
Range-based surveying data and modelling results.