Perceptual approach for unsupervised digital color restoration of cinematographic archives Majed Chambah~, Alessandro Rizzi#, Carlo Gatta#, Bernard Besserer~, Daniele Marini* ~
L3I, Université de La Rochelle, France E-mail:
[email protected],
[email protected] #
Dept. of Information Technology - University of Milano/Italy E-mail:
[email protected],
[email protected] *Dept. of Information Science - University of Milano/Italy E-mail:
[email protected]
ABSTRACT The cinematographic archives represent an important part of our collective memory. We present in this paper some advances in automating the color fading restoration process, especially with regard to the automatic color correction technique. The proposed color correction method is based on the ACE model, an unsupervised color equalization algorithm based on a perceptual approach and inspired by some adaptation mechanisms of the human visual system, in particular lightness constancy and color constancy. There are some advantages in a perceptual approach: mainly its robustness and its local filtering properties, that lead to more effective results. The resulting technique, is not just an application of ACE on movie images, but an enhancement of ACE principles to meet the requirements in the digital film restoration field. The presented preliminary results are satisfying and promising. Keywords: Digital film restoration, color fading, automatic color correction, color constancy, ACE.
1. INTRODUCTION Since the 1950s, monopack color film became the standard on which millions of cinematographic works were recorded. A couple of decades later, it turned out that this process was chemically unstable, causing the fading of whole film stocks with time. Usually, a bleached color release print is the only available record of a film. Since the bleaching phenomenon is irreversible, photochemical restoration of faded prints is not possible, hence the incontestability of digital color restoration. The technique proposed here for the color digital restoration of faded movies is based on a perceptual approach, inspired by some adaptation mechanisms of the human visual system (HVS), in particular lightness constancy and color constancy. The lightness constancy adaptation makes us stably perceive the scene regardless changes in mean luminance intensity and the color constancy adaptation makes us stably perceive the scene regardless changes in color of the illuminant. The work that preceded this article focused, on one hand, on the capabilities of color constancy-like methods to restore faded films [3][5] and the steps of digital color film restoration [1][2][4]. On the other hand, an algorithm for digital images unsupervised enhancement, called ACE for Automatic Color Equalization was also presented [6][7][23]. It provided experimental evidence for correcting automatically the color balance of an image using a perceptual approach.
138
Color Imaging VIII: Processing, Hardcopy, and Applications, Reiner Eschbach, Gabriel G. Marcu, Editors, Proceedings of SPIE–IS&T Electronic Imaging, SPIE Vol. 5008 (2003) © 2003 SPIE–IS&T · 0277-786X/03/$15.00
We present in this paper the first results of a new technique for digital color film restoration based on ACE. This work is the first result of the collaboration between the L3I lab of Université de La Rochelle and the Department of Information Technology of University of Milano. After a detailed presentation of ACE algorithm, we present the new version of ACE for color digital movie restoration. We report and discuss the results of our experiments and we give some future prospects of this work.
2. ACE: AUTOMATIC COLOR EQUALIZATION ACE, for Automatic Color Equalization, is an algorithm for digital images unsupervised enhancement. It is based on a new computational approach that merges the "Gray World" and "White Patch" equalization mechanisms, while taking into account the spatial distribution of color information. Inspired by some adaptation mechanisms of the human visual system, ACE is able to adapt to widely varying lighting conditions, and to extract visual information from the environment efficaciously. ACE implementation follows the scheme shown in figure 1: a first stage accounts for a chromatic spatial adaptation (responsible for color constancy) and a second stage, dynamic tone reproduction scaling, configures the output range to implement an accurate tone mapping. The first stage merges the gray world and white patch approaches and performs a sort of contrast enhancement, weighted by pixel distance. The result is a local-global filtering. The second stage maximizes the image dynamic, normalizing the global lightness. No user supervision, no statistics and no data preparation are required to run the algorithm.
Fig 1: ACE basic schema In figure 1, I is the input image, R is an intermediate result and O is the output image; subscript c denotes the chromatic channel. 2.1 Chromatic / Spatial adaptation The first stage, the Chromatic/Spatial adaptation, produces an output image R in which every pixel is recomputed according to the image content, approximating the visual appearance of the image. Each pixel p of the output image R is computed separately for each chromatic channel c as shown in equation (1).
r ( I ( p ) − I ( j )) d ( p, j) j∈ Im, j ≠ p Rc ( p) = rmax ∑ j∈ Im, j ≠ p d ( p , j )
∑
(1)
I( p) − I ( j ) accounts for the basic pixel contrast interaction mechanism, d(p, j) is a distance function which weights the amount of local or global contribution, r(⋅) is the function, discussed below, that accounts for the relative lightness appearance of the pixel.
Proc. of SPIE Vol. 5008
139
The lower part of the fraction has been introduced to balance the filtering effect of pixels near the border, avoiding a vignetting effect [23]. A sort of lateral inhibition mechanism is realized by computing the difference between each pixel value and all other pixels of the selected image subset. This “difference mechanism” enhances the contrast, tuned by r(⋅) , the chromatic adaptation function. The distance d(⋅) weights the global and local filtering effect. It is well known that both global and local adaptations are present in the HVS. Global models, in fact, are not able to simulate several local chromatic adaptation effects, e.g. simultaneous contrast or Mach bands [23]. Different d(⋅) functions have been tested. Some functions seem to achieve better results than others, but a best function has not yet been found. For the tests in this paper we have chosen the Euclidean distance r. For each pixel of the image, r(⋅) , together with d(⋅) , control the contrast interaction, accounting for the spatial channel lightness adaptation. It computes all the single contributions of the image content (weighted by d(⋅) ) to each final pixel value in the output image. To perform a gray world mechanism, r(⋅) has to be an odd function, while the white patch mechanism is obtained by a non-linear enhancement of small differences between neighbor pixels. We have tested different r(⋅) functions, trying to implement an effective white patch mechanism. Figure 2 displays the tested functions, the codes mean: LI=Linear, SG=Signum, SA=Saturation.
Fig. 2: r(⋅) functions Linear and Signum functions can be seen as limit cases of Saturation function with unitary or infinite slope respectively. For the test of this paper we have chosen Saturation with a slope value 20. 2.2 Dynamic tone reproduction scaling This second stage maps the intermediate pixels array R into the final output image O. In this stage not only a simple dynamic maximization can be made (linear scaling), also different reference values can be chosen in the output range to map into gray levels the relative lightness appearance values of each channel. According to the chosen reference point an additional global balance between gray world and white patch is added. The following two scaling methods can be used to obtain a standard 24 bit output image from the signed floating point array R. The first simple method scales linearly the range of values in Rc independently on each channel to the range [0,255] by the formula: Oc ( p ) = round [ sc ( Rc ( p ) − mc )] where sc is the slope of the segment [(mc,0),(Mc,255)], with M c = max[ Rc ( p)] and mc = min[ Rc ( p )] . p
p
In this case the linear mapping fills exactly the available dynamic range without further adaptation.
140
Proc. of SPIE Vol. 5008
The second alternative method, yields better results, by scaling linearly the values in Rc with the following formula
Oc ( p ) = round [127.5 + s c Rc ( p )] using Mc as white reference and the zero value in Rc as an estimate for the medium gray reference point to compute the slope sc. For this reason, the available dynamic could not be used entirely, or tones around the very dark values could be lost. Alternatively, some values in Oc can result negative. In this case the values lower than zero are set to zero. The second method adds a global gray world adaptation in the final scaling, thus the dynamic of the final image is always centered around the medium gray.
3. ACE FOR COLOR DIGITAL MOVIE RESTORATION The principal characteristic of ACE [6][7][23] is its local data driven color correction, inspired by some adaptation mechanisms of the human visual system. ACE is able to adapt to unknown chromatic dominants, to solve the color constancy problem and to perform an image dynamic data driven stretching. Moreover, ACE algorithm is unsupervised and needs little involvement from the user. These properties make it suitable for film restoration, a problem in which there is no reference color to compare the results of the filtering. The only reliable criterion is the pleasantness and naturalness of the final image. Faded movie images are drab and have a poor saturation and an overall color cast. This is due to the bleaching of one or two chromatic layers of the film. Since we deal with lost chromatic information, restoring the color of faded movies is more intricate than balancing the colors of an image with a color cast due to an illuminant shift. Figure 5 shows the original faded image of figure 4 processed by ACE. Even though the cast was removed, the colors of the image are still drab. This is due to the fact that bleached movie images are more damaged than images with a cast due to an illuminant shift. Vividness (saturation) of the image colors is a major issue in digital color movie restoration. The naturalness of the image and its histogram shape is also important. The new unsupervised technique, that we present, is not just an application of ACE on movie images, but an enhancement of ACE principles to meet the requirements in the digital film restoration field. To restore the vividness of the image colors, we enhance the saturation of the “real” colors of the image before removing the cast and balancing the colors with ACE (see figure 3). To avoid increasing the color cast, we use a non-uniform saturation enhancement technique presented in [3]. It consists in stretching the bounding ellipsoid of the points according to the principal axes in CIELAB color space. Unlike uniform saturation increase methods, this color enhancement technique avoids increasing the color cast all over the image and enhances the “real” colors of the image. Figure 6 shows the image of figure 4 after non-uniform saturation enhancement.
Saturation enhancement
Color balance ACE
Original images
Restored images
Fig. 3: Steps of color restoration Once the colors of the image have been revived, the next step consists in removing the color cast, balancing the colors and correcting the contrast of the image using ACE. Inner parameters of the ACE algorithm have to be properly tuned and new functions have to be added to meet the requirements of image and histogram shape naturalness. Figure 7 shows
Proc. of SPIE Vol. 5008
141
the result of ACE on saturation enhanced image of figure 6. The recovered colors are vivid, but the image is noisy and has an unnatural look due to the equalized histogram shape as shown on figure 9.Trying to avoid these unwanted behaviors we followed a conservative approach choosing 0.2 for saturation parameter as a trade off between a good Color Constancy behavior and a solution that doesn’t amplify excessively the original low contrast background noise. This preliminary choice that needs further investigation derives from previous tests [6]. To preserve the natural histogram shape and to obtain natural images two functions have been added in the ACE algorithm. A new scaling function called WP+GW (black) and a new “keep original gray” (fade to black) feature. In particular the WP+GW (black) scaling is the same as WP+GW but with a modified mapping that fills exactly the available dynamic solving the problem of losing dark tones using WP+GW. The “keep original gray” feature has been devised to relax the GW mechanism in the second stage: instead of centering the chromatic channels around the medium gray, “keep original gray” preserve the original mean values; this results in histograms more similar in shape with original ones. Relaxing GW mechanism in the second stage does not affect the Color Constancy property of ACE since it derives from WP/GW mechanism in the first stage and from global WP approach in the second one.
4. EXPERIMENTAL RESULTS We tested different parameters to find the best configuration for faded images restoration. We run the experiments on several images coming from different films of different epochs and with different bleaching behaviors and color casts. The criteria used to evaluate the quality of the restoration are visual judgment [8] according to the naturalness of the aspect of the image (colors and contrast) and the naturalness of the histogram shape (smooth, regular and stretched shape not an equalized one). To judge the fidelity of the colors of the restored image we focus on memory colors [9] [10] like the blue of the sky, the green of the foliage, skin tones, achromatic zones, etc... We varied different parameters like the scaling function, the shape function, and some features like “keep original gray”. To get a restored image with balanced vivid natural colors without noise, and having a natural histogram shape, the best shape function is saturation. Figure 18 shows the result of ACE on the image of figure 6 with linear shape. The image has still a strong cast. Using a saturation shape guarantees the removal of the color cast. The most suitable parameter of saturation shape function is 0,2. Under this value the image becomes noisy, the contrast is not ideal (see figure 16) and the histogram shape is unnatural as shown on figure 17. The WP+GW scaling function (see Sect. 2.2) does not give an image with a good contrast especially in the shadows, as shown in figure 10, since the shadows are shifted as shown in figure 11. A linear scaling gives a more stretched histogram (figure 13), but the result is still not satisfying (figure 12). The best result is given by WP+GW black scaling function combined with the “keep original gray” feature, since this configuration gives a good contrast all over the image, whether in shadows, mid-tones or highlights. Figure 14 shows a well restored image using this configuration combined with a saturation shape with a parameter of 0,2. The colors of the image are natural and vivid. The contrast and the histogram shape are natural as well as illustrated by figure 15. Figure 23 shows another well restored image by the same configuration. The polar hue histograms of figures 21 and 25 show the chromatic diversity of the images restored by this method which confirms the suitability of this technique to digital film color restoration.
5. DISCUSSION The experimental results gave evidence for the suitability of the developed technique to restore color faded movies. This technique has several advantages: It uses a perceptual approach with global and local effects, is unsupervised, needs little involvement from the user and gives good results. The major drawback of this method is its slowness. The algorithm can run on a local LUT instead of a full image but the results have not the same reliability. To avoid running the algorithm on all the full images of the sequence, which takes a huge amount of time (see table 1), it is preferable to run the algorithm on a reduced image (at CIF resolution for instance), then do a linear color mapping [4] between the original faded image and the restored reduced image. The linear color mapping will provide us with the matrix that will restore the whole sequence at full definition. The linear processing gives a well restored image (figure 19) close to the target (the image processed by ACE), but it is not exactly the same since ACE is a non linear and local processing. Alternative accelerating methods are under test.
142
Proc. of SPIE Vol. 5008
In the meanwhile, to accelerate the ACE and meet the economical requirements in the digital restoration field, we rewrote the algorithm using Intel’s SSE instructions [22]. SSE instructions are SIMD (Single Instruction Multiple Data) instructions permitting to process a set of pixels with a single instruction. The first results are very promising as shown by table 1, the processing time was divided by nearly 5 (a gain factor of 80%) on a PC equipped with an AMD Athlon XP 1800+ processor and 512 of DDRAM. Table 1: Computing time of the original and the SSE optimized ACE algorithm
Image size 925 x 672 pixels 310 x 293 pixels
Algorithm ACE ACE w/ SSE ACE ACE w/ SSE
Processing time t 19 hours 3 hours 30 min 22 minutes 4 min 30 sec
tACE / tACE SSE 5,43
Gain factor 81%
4,88
80%
6. CONCLUSION We presented in this paper a new technique for digital restoration of color faded movies using a new unsupervised color equalization algorithm, based on a perceptual approach. This work results from the collaboration between the L3I lab of Université de La Rochelle and the Department of Information Technology of University of Milano. The developed technique consists in enhancing the saturation of the images, since after fading the images become too drab, then balancing the colors of the image using a new version of ACE algorithm (Automatic Color Equalization). To meet the requirements in the digital restoration field the inner parameters of the new version were properly tuned, some new functions were added and the whole algorithm optimized. The first obtained results are satisfying and promising. The future prospects of this collaboration consist in including the saturation enhancement processing inside the ACE algorithm, optimizing and accelerating the algorithm.
REFERENCES 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.
M. Chambah, B. Besserer, “Digital color restoration of faded motion pictures”. Computer graphics and image processing conference CGIP2000, Saint-Etienne, France, 2000, pp. 338-342. M. Chambah, B. Besserer, “Digital restoration of faded color movies: a four-step method”, 8th Color imaging conference CIC8, IS&T/SID, Scottsdale, AZ, USA, 2000, pp. 161-166. M. Chambah, B. Besserer, P. Courtellemont, “Recent Progress in Automatic Digital Restoration of Color Motion Pictures”, SPIE Electronic Imaging 2002, San Jose, CA, USA, janvier 2002, vol. 4663, pp. 98-109. M. Chambah, B. Besserer, P. Courtellemont, “Approach to Automate Digital Restoration of Faded Color Films”, IS&T Conference on Graphics, Imaging, and Vision CGIV 2002, Poitiers, France, avril 2002, pp. 613-618. M. Chambah, B. Besserer, P. Courtellemont, “Latest Results in Digital Color Film Restoration”, Machine Graphics and Vision (MG&V) Journal, 2002. (in press ) A. Rizzi, C. Gatta, D. Marini, “Color Correction between Gray World and White Patch”, Electronic Imaging 2002, 20-25/01/02, San Jose, California (USA). C. Gatta, A. Rizzi, D. Marini, “ACE: an Automatic Color Equalization algorithm” CGIV02 the First European Conference on Color in Graphics Image and Vision, 2-5/4/2002, Poitiers (France). M. Stokes, T. White, Color fidelity test methods, IS&T/SID’s color imaging conference proceedings, 1998, pp.258262. K. Töpfer, R. Cookingham, The quantitative aspects of color rendering for memory colors, IS&T’s PICS conference, 2000. C. Tuijn, W. Cliquet, Today’s image capturing needs: going beyond color management, IS&T/SID’s 5th color imaging conference proceedings, 1997, pp. 203-208. S.N. Pattanaik, J.A. Ferwerda, M.D. Fairchild, D.P. Greenberg, “A Multiscale Model of Adaptation and Spatial Vision for Realistic Image Display”, Proc. of SIGGRAPH98, pp.287-298, Orlando, Florida (USA), July 1998. A. C. Hurlbert, Formal connections between lightness algorithms, J. Opt. Soc. A, 1986, 3, 1684-1693 M. Ramasubramanian, S.N. Pattanaik, D.P. Greenberg, “A Perceptually Based Physical Error Metric for Realistic Image Synthesis” Proc. of SIGGRAPH99, pp.73-82, Los Angeles, California (USA), 1999.
Proc. of SPIE Vol. 5008
143
14. J. McCann, “Lessons Learned from Mondrians Applied to Real Images and Color Gamuts”, IS&T Reporter, Vol. 14, No. 6, November/December 1999. 15. G.W. Larson, H. Rushmeier, C. Piatko, “A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes”, IEEE Transactions on Visualization and Computer Graphics, Vol. 3, No. 4, October/december 1997. 16. J. Tumblin, H. Rushmeier, “Tone Reproduction for Realistic Images”, IEEE Computer Graphics and Application, November 1993, pp.42-48. 17. J. Tumblin, J.K. Hodgins, B.K. Guenter, “Two Methods for Display of High Contrast Images”, ACM Transactions on Graphics, Vol. 18, No. 1, January 1999, pp. 56-94. 18. J. McCann, “Lessons Learned from Mondrians Applied to Real Images and Color Gamuts”, IS&T Reporter, Vol. 14, No. 6, November/December 1999. 19. D. Marini, A. Rizzi, "A computational approach to color illusions", in: A. Del Bimbo Ed., Image Analysis and Processing, Springer Verlag, Berlin, pp.62-69, 1997 20. Marini D. Rizzi A, Rossi M., “Color Constancy Measurements for Synthetic Image Generation”, Journal of Electronic Imaging, 8, 9, (1999) pp. 394-403 21. Marini D., Rizzi A., "A Computationl Approach to Color Adaptation Effects", Image and Vision Computing, 18, 13 (2000) pp. 1005-1014 22. Intel Corporation, SIMD extensions in Pentium III, www.intel.com. 23. A. Rizzi, C. Gatta, D. Marini, “A New Algorithm for Unsupervised Global and Local Color Correction” ”, in press on Pattern Recognition Letters.
144
Proc. of SPIE Vol. 5008
Fig. 4: Original faded image to restore
Fig 5: Image of fig 4 processed by ACE. Even though the cast was removed, the colors are still drab. This is due to the fact that bleached movie images are more damaged than images with a cast due to an illuminant shift.
Fig 6: Image of fig 4 processed by our saturation enhancement algorithm. The real colors of the image are revived without increasing the cast.
Proc. of SPIE Vol. 5008
145
Fig 7: The result of ACE on saturation enhanced image of fig 6. The recovered colors are vivid, but the image is noisy, has unnatural look due to the equalized histogram shape as shown on fig 9.
146
Fig 8: Histogram of original faded image of fig4. The histogram is not stretched since the colors of the image are damaged.
Fig 9: Histogram of the image of fig 7 processed by ACE. The histogram is unnatural and equalized.
Fig 10: Result of ACE on image of fig 6 with saturation shape (parameter 0,2) and WP+GW scaling. The colors are vivid and natural but the contrast of the image is not good especially at shadows.
Fig 11: Histogram of fig 10. The highlights are equalized and the shadows are far from the edge, that is why the shadows are light.
Proc. of SPIE Vol. 5008
Fig 12: Result of ACE on image of fig 6 with saturation shape (parameter 0,2) and linear scaling. The contrast of the image is better than the contrast of the image of fig10.
Fig 13: Histogram of image of fig 12. The histogram is more stretched than the histogram on fig 11. That is why the contrast of image of fig 12 is better than of image of fig 10.
Fig 14: Result of ACE on image of fig 6 with saturation (0,2), WP+GW (black) scaling and with keeping original gray. This configuration gives one of the best results in terms of vividness of colors, contrast and naturalness of the histogram.
Fig 15: Histogram of fig 14. It is well stretched and has a natural shape which explains the naturalness of the image of fig 14.
Fig 16: Result of ACE on fig 6 with saturation (0,07), WP+GW (black) and with keeping original gray. When the saturation parameter is too small, the image is noisy, the contrast is not ideal and the histogram shape is unnatural (fig 17).
Fig 17: Histogram of fig 16. It has an unnatural shape which explains the noise on fig 16.
Proc. of SPIE Vol. 5008
147
Fig 18: Result of ACE on image of fig 6 with linear shape, WP+GW (black) scaling and with keeping original gray. The image has still a strong cast. This shows the usefulness of a saturation shape.
Fig 19: The result obtained by a linear color mapping between fig 4 and fig 14. The colors and the contrast of the image are very satisfying but not exactly the same as of fig 14, this is logical since we approach two non-linear treatments with a linear one. The linear color mapping permits to find the matrix that can restore the colors of the image. This matrix can be applied to the whole sequence, gaining hence a lot of processing time.
Fig 20: Hue polar histogram of the original faded image of fig4, showing its poor chromatic diversity.
148
Proc. of SPIE Vol. 5008
Fig 21: Hue polar histogram of the image restored of fig 14, showing its good chromatic diversity.
Fig 22: Original faded image to restore.
Fig 23: Image of fig 22 after saturation enhancement and ACE.
Fig 24: Hue polar histogram of the image of fig. 22, showing its poor chromatic diversity.
Fig 25: Hue polar histogram of the image of fig. 23, showing its good chromatic diversity.
Proc. of SPIE Vol. 5008
149
