Automated full-3D digitization system for documentation of paintings Maciej Karaszewski*a, Marcin Adamczyka, Robert Sitnika, Jakub Michońskia, Wojciech Załuskia, Eryk Bunschb, Paweł Bolewickia a Warsaw University of Technology, Dept. of Mechatronics, Sw. A. Boboli 8,02-525 Warsaw, Poland b Wilanow Palace Museum, St. Kostki Potockiego 10/16, 02-958 Warsaw, Poland ABSTRACT In this paper, a fully automated 3D digitization system for documentation of paintings is presented. It consists of a specially designed frame system for secure fixing of painting, a custom designed, structured light-based, high-resolution measurement head with no IR and UV emission. This device is automatically positioned in two axes (parallel to the surface of digitized painting) with additional manual positioning in third, perpendicular axis. Manual change of observation angle is also possible around two axes to re-measure even partially shadowed areas. The whole system is built in a way which provides full protection of digitized object (moving elements cannot reach its vicinity) and is driven by computer-controlled, highly precise servomechanisms. It can be used for automatic (without any user attention) and fast measurement of the paintings with some limitation to their properties: maximum size of the picture is 2000mm x 2000mm (with deviation of flatness smaller than 20mm) Measurement head is automatically calibrated by the system and its possible working volume starts from 50mm x 50mm x 20mm (10000 points per square mm) and ends at 120mm x 80mm x 60mm (2500 points per square mm). The directional measurements obtained with this system are automatically initially aligned due to the measurement head’s position coordinates known from servomechanisms. After the whole painting is digitized, the measurements are fine-aligned with color-based ICP algorithm to remove any influence of possible inaccuracy of positioning devices. We present exemplary digitization results along with the discussion about the opportunities of analysis which appear for such high-resolution, 3D computer models of paintings. Keywords: automated 3D shape measurement, structured light, documentation of paintings, cultural heritage 3D digitization
1. INTRODUCTION During last ten years, the quality and accuracy of 3D scanning devices increased greatly. Currently, a system capable of achieving spatial resolution of 10000 points per square millimeter is not unusual. While accuracy of such systems is exceptional, their main serious drawback is their small working volume which can be inscribed in a prism, side of which is at most few centimeters long. The advantages which can be gained while digitizing objects with high resolutions cannot be underestimated - having full 3D and color digital information allows to inspect whole surface details remotely, without physical interaction with the object (which often requires its removal from exhibition). Quality of 3D representation of object's surface achievable with current scanning devices gained the interest of conservators and art historians. The 3D models are being perceived as a next step (from 2D photography) in documentation of cultural heritage objects. In comparison with photography, 3D digital objects allows for many more applications and research of unique works of art. Professional use require however objectivity and repeatability of digitization process. The only way to achieve this is the automation of the whole process. Some systems, used for automatic 3D models obtaining, were created [1], [2], [3]. They were developed to be universal, allowing to digitize objects of various shape. For this purpose such systems usually use robot arm and usually a turntable. This solution is however not ideal for digitization of objects like paintings, which are usually of simple shape (in most cases it is a rectangular prism with very small depth). Therefore to measure them, rotating table is not necessary, and the use of robot arm makes the system expensive and complicated, especially when the painting to be digitized is large. As the collections of precious paintings are vast and numerous, the development of the system aimed solely at their digitization is well-grounded. *
[email protected], phone 48 22 2348394, fax 48 22 2348601, http://ogx.mchtr.pw.edu.pl
Optics for Arts, Architecture, and Archaeology IV, edited by Luca Pezzati, Piotr Targowski, Proc. of SPIE Vol. 8790, 87900X · © 2013 SPIE CCC code: 0277-786X/13/$18 · doi: 10.1117/12.2020447 Proc. of SPIE Vol. 8790 87900X-1 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 09/12/2014 Terms of Use: http://spiedl.org/terms
2. SYSTEM MODULES We present the complete, automated system for digitization of paintings. It is composed of the measurement head (structured light scanner [4]), fixed to manually adjustable head (providing pan, tilt, yaw and depth adjustment), which in turn is fixed to servo controlled custom built XY positioner. The painting is fixed to the specially designed frame which allows for strain-less but firm securing of the object. This frame is placed in a distance from all moving parts, ensuring that even in the event of any breakdown or software errors, the painting cannot be damaged by any part of measurement system. The important part of the system is the controlling and processing software, along with visualization module. The software is developed to cope with datasets as large as few Terabytes (one measurement uses at average 500MB [5], the number of such measurements can reach 2500 for large paintings). After data processing is complete, the visualization module is able to present this data in real time, optimizing the resolution of displayed data to the virtual camera parameters and position. In this section, all vital system components are presented. 2.1 Measurement head As it was stated before, the system consists of structured light measurement head. This device was custom built to meet conservators' requirements, especially no ultraviolet nor infrared radiation emission in the direction of measured object. Only low intensity visible light (400-700 nm) is allowed and the emission occurs only during measurements, it is blocked (light sources are turned off) during positioning and pauses. Such measurement head is built on the basis of commercially available multimedia projector (Casio XJ-A250 [6]), with custom lens, allowing for close range focusing (250mm from the device) and small image size (50mm x 50mm). The new optics block and no heat or UV radiation towards image require completely changed cooling system, which was carefully designed to ensure also a low level of vibrations. After the modifications, series of tests have been performed to ensure that the projection system meets conservator’s demands. Among those tests, the spectrum of projector emission towards the object has been measured with professional spectrophotometer at WUT (Table 1). The results of the measurement show little differences from spectrum plot available in projector’s documentation [7] (Figure 1), especially regarding blue line (about 20nm shorter wavelength), but nevertheless neither ultraviolet nor infrared radiation is emitted. . Table 1. Spectrum of different parts of projector’s lighting module.
Light source
Minimum observed wavelength
Maximum observed wavelength
Blue lasers block
440 nm
455 nm
Green phosphorus
498 nm
605 nm
Red LED
613
654
Figure 1.Spectrum of projector’s radiation towards the object [7]. Blue and red stripes mark the visible light range.
The visible light projected onto the digitized object cannot transfer too much energy. The exposition values(measured in lx*h) for typical conditions in the museum are presented in Table 2. Exposition measured before and after projector
Proc. of SPIE Vol. 8790 87900X-2 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 09/12/2014 Terms of Use: http://spiedl.org/terms
modification is presented in Table 3. Amount of energy emitted during measurements (after modification) is low enough to be accepted by art conservators. Table 2. Exposition values for different brightness in museum conditions.
Brightness [lx]
Time [hours]
Exposition [lxh]
50
8 (one working day)
400
100
8 (one working day)
800
500 (conditions not allowed by conservators, typical brightness in workplace)
8 (one working day)
4000
Table 3. Measured exposition for projector during typical measurement.
Brightness [lx] 710520 projector)
Time [hours]
(off-the-shelf
24850 (modified projector)
Exposition [lxh]
Equivalent of normal exposition [hours]
0.0041 (15 seconds)
2913
59 (7 working days!)
0.0041 (15 seconds)
102
2.04 (1 hour environment)
for
100
lx
The 3D model and real view of the projector are presented at Figure 2. The detector used in this measurement head is a standard high-quality DSLR, namely Canon EOS 60D [8]. The use of such camera is beneficial because it is excellent at the reproduction of colors. To improve quality of color information, special lighting ring is used, mounted on the same optical axis as camera's lens, ensuring no parallax error. The white balance parameter is calibrated with certified calibration unit. Figure 2 presents different stages of measurement head development – computer model, modification of projector (prototypes) and the computer model of the final design.
a)
b)
c)
d) Figure 2. Measurement head: a) prototype 3D model with modified elements, b) projector cooling system, c) modified projector, d) production model.
Proc. of SPIE Vol. 8790 87900X-3 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 09/12/2014 Terms of Use: http://spiedl.org/terms
The prototype measurement heads (Figure 2 b and c) have been used since August 2012 for digitization of paintings, usually with automated digitization system based on the industrial robot (Figure 3). This setups have been used for estimation of resolutions, accuracy, color representation quality and other parameters of measurements required by art conservators.
Figure 3. Prototype measurement head mounted on the automatic digitization system. On the right, the easel with painting is visible.
2.2 Positioning devices The measurement head is fixed to commercial photographic head, which allows for adjusting its roll, pitch and yaw angles (Figure 5). This feature is used to avoid direct reflections of light from projector into detector's lens (Figure 4).
Q,l
Figure 4. Direct reflections of light from projector observed by detector (marker with red rings).
Proc. of SPIE Vol. 8790 87900X-4 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 09/12/2014 Terms of Use: http://spiedl.org/terms
Additionally, the head can be shifted manually toward and backward the scanned surface, to alleviate different thickness of various paintings (for example painted on wood or canvas).
Figure 5. Manual adjustment of measurement head. The pitch-roll-yaw head and linear stage are highlighted in blue.
The whole module is fixed to the carriage of servo controlled XY positioner (Figure 6). This device is used to move measurement head in the plane parallel to painting's surface. The positioning axes are independent, allowing to position the carriage in any XY position within it's working range. Servomechanism are fully controlled by measurement software and the (highly accurate) coordinates of the carriage are used for initial integration of obtained directional measurements.
.
Figure 6. Model of the measurement system. Positioning rails are marked red and green.
Proc. of SPIE Vol. 8790 87900X-5 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 09/12/2014 Terms of Use: http://spiedl.org/terms
2.3 Measurement control software The servomechanisms driving XY positioner are controlled with the application running on the computer connected to measurement head. This application allows for starting and suspending automatic digitization process (which can be resumed later). The positions which have to be reached by the measurement head are calculated on the basis of expected measurements overlap percentage (Figure 7) and coordinates of measurement head, placed manually by the operator in the three corners of the painting (Figure 8). In this case, “manually” means that operator uses the application’s button to steer the carriage to face painting’s corners. Coordinates of the head, obtained this way, along with measurement volume size and required overlaps allows for calculation of head’s positions required for digitization of the whole painting.
Figure 7. Measurements overlapping. Three subsequent measurements (red, yellow and green) are presented, the overlapping is 50%.
Figure 8. Estimation of painting’s dimensions with three corners method.
2.4 Processing software During digitization process, the clouds of points obtained from subsequent measurements are saved on the hard disk of controlling computer, along with head’s coordinates. Depending of the processing power of the computer used, their processing can be started immediately or after measurements are finished. At first, each cloud is processed separately. The operations performed with this data include removal of noise points by Hausdorff grouping [9], analysis of distance between point and plane fitted to its neighborhood. Also, the lowest quality points (quality is defined as the modulation of structured-lighting patterns) are removed due to the fact that such points are often burdened by high amount measurement noise. After all directional measurements are processed, all of them are loaded into the processing application. This application is especially designed to allow for processing of large size data (even terabytes). Not going deep into the details which are out of this paper’s scope, using advanced memory usage algorithms the application can load as little part of the measurements’ data as possible. With the whole model of painting loaded, a special version of ICP [10] algorithm (optimized to use mainly color information) is run for fine data integration (initial one is not needed due to the fact that head’s coordinates are known during measurement). After this operations, the whole model is processed to allow for real-time visualization. The details of this algorithm are also beyond the scope of this paper, but after it is completed, the data is divided into different levels of detail, calculated for specified virtual camera (used in visualization module) parameters (horizontal and vertical resolution and field of view). 2.5 Visualization module The very important part of presented system is a visualization module. It allows for real-time presentation of measured data to users. Of course, the whole model cannot be visualized at once without some data simplifications. The application calculates required density of the data visible in the lens of the virtual camera (this density should be as low as possible, but high enough to assure continuity of observed surface – Figure 9). The algorithm predicts the possible movements of the camera and loads required data in advance. This procedures, along with good data preprocessing
Proc. of SPIE Vol. 8790 87900X-6 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 09/12/2014 Terms of Use: http://spiedl.org/terms
(mentioned in the previous section), assure very good application responsiveness, especially when fast hard disk is used for data storage (the PCIExpress SSD drive is recommended).
a)
b)
c)
Figure 9. Visualization module: a) Data from various levels of detail loaded depending on local camera configuration (marked with red rectangle); b, c) fragment of van Gogh painting’s copy for different zoom levels with more dense data (but for smaller visible area) loaded for close-up.
3. AUTOMATED MEASUREMENT The course of automated digitization process is presented at Figure 10. It begins with placement of the object to be digitized on the fixing frame. This part is done by conservators, to ensure that the painting is handled with proper care. In the next step, the measurement head is moved by the operator to three corners of the painting (left-top, right-top, rightbottom) to establish the dimensions of the scanning space. After this procedure is completed, the measurement head is returned to left-top corner of the painting and required overlapping of directional measurements is defined by the operator. This action ends the preparation stage of the process and the measurements may begin. Start of digitization process
ixing the painting to holding frame
Processing individual clouds of points Directional measurement
e
\IL Right- bottom comer reach
anual positioning of measurement head to face left -top comer of painting
anual positioning of measurement head to face right -top comer of painting
anual positioning of measurement head to face right -bottom
Global ICP correction
comer o painting Moving scanning head to the next position
( Preprocessing for visualization
Setting required overlapping of measurements
Preparation stage
Measurement stage
Processing stage En of digitization process
Figure 10. Course of automated digitization process.
Operator starts the measurement process from the controlling application. The scanning head begins the operation of collecting directional cloud of points. After that, the head is moved horizontally (along X axis) to the next position and another directional measurement is done. When the carriage reaches the right edge of the painting, it is moved vertically (down, along Y axis) and horizontally (to the left edge) in a typewriter – style (Figure 11). The return to left edge at the beginning of each line is performed to avoid any hysteresis of horizontal positioning mechanism. The measurement process can be suspended and resumed at any moment. The procedure is ended when scanning head reaches right-bottom corner of the painting.
Proc. of SPIE Vol. 8790 87900X-7 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 09/12/2014 Terms of Use: http://spiedl.org/terms
Figure 11. Directional measurements (yellow rectangles) sequence.
After all measurements are complete, the processing of data begins. At the end, the complete 3D model of painting, usually optimized for visualization is saved to disk, thus finishing the whole digitization process.
4. EXEMPLARY RESULTS OF DIGITIZATION OF PAINTINGS The prototype system for digitization of paintings was used to create 3D models of three paintings from the collection of Palace Museum at Wilanow. The properties of those paintings and used measurement heads are described in Table 4. It is clear that manual digitization of such objects with such high resolution (implicating high number of directional measurements) is at best very hard, while automatic process takes up to few days. The photos of digitized objects, along with their 3D models are presented in Tables 5-7. Table 4. Characteristics of digitized objects.
Object
Characteristics
Size
Number of directional measurements
Digitization time (with data processing)
Portrait of Jan III (Wil. 1197)
Unknown painter, oil on canvas, dated: 1675 – 1700
340 mm x 440 mm
472
71.5 hours
Portrait of Jan III (Wil. 1348)
Unknown painter, oil on canvas, painted after year 1683
730 mm x 600 mm
876
119 hours
Fragment of copy of “Vincent’s Chair with His Pipe”
Original artist – Vincent van Gogh, painted in year 1888, Arles.
100 mm x 100 mm
20 (not whole surface has been digitized)
4 hours
Table 5. Painting 1, small portrait of Jan III Sobieski.
Photo (by A.
3D model
Surface details – cracks
Surface details (fragment 4mm x 4mm)
Proc. of SPIE Vol. 8790 87900X-8 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 09/12/2014 Terms of Use: http://spiedl.org/terms
Indyk)
, Height map
Map of surface roughness
Local height map
Table 6. Painting 2, large portrait of Jan III Sobieski.
4,.,3%
400
" $4.1Z,
'
ir'',"
Surface details (50mm x 50mm) Local height map 3D model
I Photo (by Z. Reszka)
Close-up (50mm x 50mm)
Close-up (0.7mm x 0.5 mm) – eye
Local curvature visualization
Detail
j
Local height map for the eye
Isometric view on the detail
The two paintings, presented in Tables 5 and 6 were fully digitized with measurement head of 1600 points / mm2 resolution. As no lighting ring nor external light source was used, the texture was taken with illumination from projector, thus its quality is a matter for improvement (due to differences between brightness of subsequent measurements). Many surface details, like cracks in the protective varnish, are clearly visible in the measurements (Table 5).
Proc. of SPIE Vol. 8790 87900X-9 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 09/12/2014 Terms of Use: http://spiedl.org/terms
Table 7. Painting 3, Vincent van Gogh’s “Vincent’s Chair and His Pipe” copy.
Detail – ripple of paint
Surface structure
Height map
Local curvature map
Height map of surface structure
Histogram of height map
Histogram of curvature map
Histogram of local height map
3D model (fragment)
,r a
Photo
i
On the measurements of copy of van Gogh’s painting, the details of paint can be find easily. Ripples of paint, local cracks – all of those elements can be located and measured (for example little bulges of paint, shown in fourth column of Table 7 have average height of 0.4 mm). More advanced measurements, along with cross-sections etc. can be of course performed. Moreover, even the shape of paintbrush and the parameters of its movements can be analyzed and used for example to asses authenticity of the painting.
5. SUMMARY In this paper, the automated measurement system for digitization of paintings has been presented. It consists of three main modules – the structured-light based measurement head with custom projector, optimized to meet rigorous conservators demands and requirements, XY positioning system controlled by PC-computer and software for processing and visualizing large scale data (up to few terabytes). The prototype measurement system was built for pilot studies of digitization of paintings. It was used to obtain 3D models of two XVII century paintings of Polish king, Jan III Sobieski and a small fragment of copy of Vincent van Gogh’s painting “Vincent’s Chair with His Pipe”. The results of those studies served as the basis of the presented system’s development. Feature works will contain the optimization of measurement process and improvement of quality of color information. It is planned that multispectral measurement head will be designed to obtain color information in LAB standard.
Proc. of SPIE Vol. 8790 87900X-10 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 09/12/2014 Terms of Use: http://spiedl.org/terms
ACKNOWLEDGMENTS This work has been partially supported by the Ministry of Culture and National Heritage (Poland) by KULTURA+ framework (2011-2015), Grant No. N R17 0004 06/2009 “Realization of the idea of preventive conservation by the means of precise 3D documentation” financed by the Polish Ministry of Since and Higher Education and Statutory Work of Warsaw University of Technology.
REFERENCES [1] Callieri, M., Fasano, A., Impoco, G., Cignoni, P., Scopigno, R., Parrini, G. and Biagini, G., "RoboScan: An automatic system for accurate and unattended 3D scanning," Proceedings - 2nd International Symposium on 3D Data Processing, Visualization, and Transmission. 3DPVT 2004, 805-812 (2004). [2] Karaszewski, M., Sitnik, R. and Bunsch, E. , "On-line, collision-free positioning of a scanner during fully automated three-dimensional measurement of cultural heritage objects," Robotics and Autonomous Systems 60(9), 1205-1219 (2012). [3] Scott, W. R., Roth, G. and Rivest, J. -. , "View planning for automated three-dimensional object reconstruction and inspection," ACM Computing Surveys 35(1), 64-96 (2003). [4] Sitnik, R., Kujawinska, M. and Załuski, W., "3DMADMAC system: Optical 3D shape acquisition and processing path for VR applications," Proceedings of SPIE - The International Society for Optical Engineering, 1-12 (2005). [5] Sitnik, R. and Karaszewski, M. , "Automated processing of data from 3D scanning of cultural heritage objects," Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 6436 LNCS, 28-41 (2010). [6] Casio USA , "Casio USA - Slim projectors," (2013). [7] Schmitt, E. , "Casio XJ-A 130 bis 245 LED & Laser statt UHP ! ," (2010). [8] Canon USA , "Canon USA - DSLRs," (2013). [9] Guerra, C. and Pascucci, V., "3D segment matching using the Hausdorff distance," IEE Conference Publication, 1822 (1999). [10] Besl, P. J. and McKay, N. D. , "A method for registration of 3-D shapes," IEEE Trans.Pattern Anal.Mach.Intell. 14(2), 239-256 (1992).
Proc. of SPIE Vol. 8790 87900X-11 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 09/12/2014 Terms of Use: http://spiedl.org/terms
