Automatically choosing the source image in the coloring ... - CiteSeerX

Anais do IV Workshop em Tratamento de Imagens, NPDI/DCC/ICEx/UFMG, p. 77-89, Junho de 2003.

Automatically choosing the source image in the coloring process of grayscale images LUIZ FILIPE MENEZES VIEIRA1 RAFAEL DUARTE VILELA1 ERICKSON RANGEL DO NASCIMENTO1 FERNANDO ANTÔNIO FERNANDES JÚNIOR1 RODRIGO L. CARCERONI1 ARNALDO DE ALBUQUERQUE ARAUJO1 1

Computer Science Department - Federal University of Minas Gerais, Av. Antônio Carlos, 6627 ,31270-010, Belo Horizonte , MG , Brazil {lfvieira,vilela,erickson,junior,carceron,arnaldo}@dcc.ufmg.br Abstract. The process of automatically coloring a grayscale image by transferring color from a colored source image was developed recently. This process requires the human intervention to select the source image. We have developed techniques to automatically select the source image, turning the process completely automatic. Applications, such as movie coloring, could take advantage of this approach, reducing the time and the human intervention required on the process. To assess the techniques, we asked a relevant group of people if the resulting images appeared real.

1 Introduction Color can be added to grayscale images such as historic photos and movies to increase their visual appeal. Moreover, the visualization of certain grayscale scientic images by human beings can be facilitated by through luminance-preserving pseudo-coloring. In any of those cases, the problem of coloring grayscale images amounts to assigning to each pixel that is originally one-dimensional (i.e., has only a scalar value, luminance), another two dimensions (e.g., hue and saturation). Thus, this problem does not have an optimum solution and any reasonable solution to it must make use either of strong prior knowledge about the scene being colorized or of decisive human intervention. There is a great number of applications, which can take advantage of the grayscale image coloring pro cess. Improving scienti.c data is one area of those. For example, in the medical .eld we have magnetic resonance images, computed tomography images, X-ray images, etc. In the geography .eld, we have satellite images (Landsat), at laboratories, we have microscopic images (SEM). The main .eld is the movie industry. Classic movies and black and white photos can be colorized and have their quality improved. Coloring grayscale images could also be used as artistic e.ect. Some softwares allow grayscale images to be colorized by having a human meticulously hand-coloring each of the individual image regions [6]. Trying to decrease the human intervention on grayscale image coloring process, Welsh et. al. [9] developed a technique to transfer color from a source image to a grayscale image. With the same purpose, to decrease the human intervention in the process and to allow the process to be completely automatic, we developed a tool which permits the source image to be automatically chosen, extending the work of Welsh et al. [9]. Gonzales and Wintz [4] describe a simple approach for pseudo-coloring grayscale images of luggage acquired by Xray. Pratt [7] describes pseudo-coloring, a common technique for adding color to grayscale images, as an “image enhancement” technique because it can be used to “enhance the detectability of detail within the image”. The concept of transferring color from one image to another is based on work by Reinhard et al. [8]. They use a simple statistical analysis to impose one image’s colors characteristics on another. Hertzmann et. al. [5] developed a framework for processing images by examples. They use arti.cial intelligence techniques. A group of image is used to train the framework, and then it is used as an application to transfer characteristic from an image to another. Although all those works permit an image to be colorized, they all require human intervention. A variety of techniques for image matching are found in literaturee, including the use of RGB histogram, HSV histogram, texture mapping, etc. [4]. Each method works in a speci.c domain. We implemented four techniques to automatically choose an image from a database, and use this image as the source image in the process of transferring color to a grayscale image. The greatest advantage of this approach is to be able to

Anais do IV Workshop em Tratamento de Imagens, NPDI/DCC/ICEx/UFMG, p. 77-89, Junho de 2003.

select the image source without human intervention, turning the process completely automatic. Each technique had a di.erent algorithm to select the best image as the source image. Experiments were taken to assess them. This article will present the main concepts in transferring coloring to a grayscale image and the technique developed to select the best source image. In the next Section we give an overview of the system. In Sec- tion 3 we describe each of the techniques implemented to automatically choose the source image. The coloring process is describe in Section 5. And Section 6 presents the experiments and results. Finally, Section 7 describes the conclusion and future works. 2 System Overview We are given a grayscale image and we have a database of images from which we will select the source image. Our goal is to select a source image such that, when applying the transferring color process, the result image will look as real as possible. The .rst step is to extract relevant information from each image in our image database. This information is stored in the form of a vector and is denominated the signature of a image. All the signatures together are stored in a characteristic database. This is shown in Figure 1.

Figure 1 The process to select an image as a source image can be described as follows. The grayscale image is processed and its signature is constructed. The signature of the image is compared with all the signatures in the characteristic database. The image with the most likely signature is returned. This image will be used on the process of transferring color. This is shown in Figure 2.

Figure 2 The transferring color process takes two images, one with color and one grayscale and transfers color from the .rst to the second resulting in a new image, the result image. This is shown in Figure 3.

Anais do IV Workshop em Tratamento de Imagens, NPDI/DCC/ICEx/UFMG, p. 77-89, Junho de 2003.

Figure 3 3 Techniques 3.1 Techniques description to select the source image We developed four techniques to select the source image. They were based on the luminance, because this is the criterion adopted by the transferring color process [9], and also the grayscale histogram, which is used in image matching algorithms. They are called here: the Luminance Histogram, the Image Sub-sampling, the Luminance Histogram and Image Histogram after applying Sobel’s filter, and the Image sub-sampling and Image Histogram after applying Sobel’s filter. First we describe the characteristics that are common to all of them, and then we explain each of them. First we explain the fundamentals concepts about the L color space, the preprocessing step, and the extracting phase. The reason the L  color space is used is because it provides a decorrelated achromatic channel for color images [8]. This allows the transfer of the chromatic channels from the color image to the grayscale image without cross-channel artifacts. 3.2 Color space L Luminosity is the criteria adopted to transfer color between two Images [9]. It can be characterized in color space L . L, is a measurement of the luminance of an object.  is a measurement of red (  positive) or green (  negative);  is a measurement of yellow ( positive) or blue ( negative); Those coordinates ( and ) approximate to zero for neutral colors (white, gray and black) [1]. Figure 4 illustrates the L Color Space.

Figure 4 3.3 PreProcessing Initially all the color images from the database were converted to grayscale images, using the Craig formula:

Anais do IV Workshop em Tratamento de Imagens, NPDI/DCC/ICEx/UFMG, p. 77-89, Junho de 2003.

Y = 0.3 * R + 0.59 * G + 0.11* B Equation 1 Next, we realized the first step of the preprocessing phase designated to use the luminance. It consists of converting the images from the RGB color system to the /  color space by using the equations 2, 3, 4. Because the /  color space is a transform of the LMS cone space [10], we first convert to LMS space

 L   0.3811 0.5783 0.0402  r   M  = 0.1967 0.7244 0.0782  g        S   0.0241 0.1288 0.8444  b  Equation 2

L = log L M = log M S = log S Equation 3

  l   α  =      β    

1 3

0

0

1 6

0

0

 0  1  L   1 1   0 1 1 − 2  M   1  1 − 1 0   S   2

Equation 4 Once the images were in the same color space, we did a quantifying process on the images. Those operations have the objective of reducing the values we are working, since the human biological system realizes only a tiny slot of the luminance spectrum. We chose eighteen as the number of values, because this is the number found in the literature for the quantifying process for H ( hue ) in the HSV ( hsv-162 ) system. 3.4 Extracting After preprocessing all the images on the database, we extract the information relevant to selection of the source image. The color characteristics are obtained by using the image’s histogram. The histogram of a digital image I of length n and height m is a discrete function of the percentage of pixels with color k from image I de.ned as:

hI (k ) =

λkI n× m

Equation 5 where I k := {(x, y) . I : I[x, y] = k} is de.ned as the set of pixels from image I that has the same color k. Using the histogram function on matrixes with grayscale or luminance information obtained from the images, we will get, for each matrix, a vector with eighteen positions which corresponds to the possible values of tons. Each position of the

Anais do IV Workshop em Tratamento de Imagens, NPDI/DCC/ICEx/UFMG, p. 77-89, Junho de 2003.

vector will have the proportion of pixels that have a determined value in the image. This vector is formed by the characteristics extracted from the image. For each image, and for each technique, we construct a signature. The signature will be formed by the information the techniques use. For example, the techniques that uses grayscale information, the signature will be the originated vector from the gray levels histogram, and for the technique that uses luminance, the signature will be the originated vector from the luminance histogram. 3.5 Similarity In literature we .nd many classic measurements as Euclidean distance, Manhattan distance and others. In the project we chose the vectorial model similarity measurement, which is very used too. This measurement verifies the correlation between two vectors using the internal cross product. Those vector are: characteristics vector from the consults image and the characteristics vector from the database image. Given a consult image Q and the others images I from the database, the function MV( SQ, SI ) that calculates the similarity is defined as: c

MV ( SQ , S I ) =

∑S i =1

c

Q

(i ) * S I (i )

∑ SQ (i) * S I (i) * i =1

c

∑S i =1

Q

(i ) * S I (i )

Equation 6 where c is the size of the characteristics vector. 3.6 The Luminance Histogram technique In the Luminance Histogram technique, the histogram of the luminance values is the most important criteria to select the source image. As previously described, the luminance information is used on the selection of the source image because it is used on the transferring color process. The histogram is used because it is used in image matching algorithms. First, as described in the preprocessing Section, the images are converted to the l color space, and their histogram is constructed considering 128 intervals. Then, for each image on the database, a signature is created that consists of the 128 values of the respective luminance histogram of the image. The similarity among signature is used to compare any pair of images and to find the best matching image for a given image. Figure 5 illustrates the process.

Figure 5: The Luminance Histogram technique process. 3.7 The Sub-Sampling technique This technique uses sub-sampling, which is found in the literature in image matching algorithms. First, the images are converted to the L color space and divided equally in 128 regions of the same size. For each region, we calculate its luminance mean. All the means are grouped together and form the signature of each image. The similarity among signature is used to compare any pair of images and to find the best matching image for a given image. Figure 6 illustrates the process.

Anais do IV Workshop em Tratamento de Imagens, NPDI/DCC/ICEx/UFMG, p. 77-89, Junho de 2003.

Figure 6: The Sub-Sampling technique process. 3.8 The Luminance Histogram and Image Histogram after applying Sobel’s filter The third developed technique involves the first technique and the use of Sobel’s filter. This technique uses more information than just the luminance, it uses the border information too. As previously described, all the images I on the database are converted to the L color space. The luminance histogram of the image is constructed, as in The Luminance Histogram technique 3.6, but this time considering 64 values. The images I are converted to the grayscale, and then we apply the Sobel’s filter S to them ( R=S(I) ) to acquire the border information. The histogram of R will be the other 64 values that will form the signature of the image I for this method. Figure 7 illustrates the process.

Figure 7: The Luminance Histogram and Image Histogram after applying Sobel’s filter process. 3.9 The Image sub-sampling and Image Histogram after applying Sobel’s filter. Finally, we combined the sub-sampling technique with the Sobel’s filters. This technique is similar to the subsampling technique, only this time we consider the border information too. The images I are converted to the after applying Sobel’s

Anais do IV Workshop em Tratamento de Imagens, NPDI/DCC/ICEx/UFMG, p. 77-89, Junho de 2003.

filter process. L color space and divided equally in 64 regions of the same size. For each region, we calculate its luminance mean. All the means are grouped together and form the first 64 values of the signature of each image. The images I are converted to the grayscale, and then we apply the Sobel’s filter S to them ( R=S(I) ) to acquire the border information. The signature is completed with 64 values from the histogram of the image R obtained after applying the Sobel’s filter. Figure 8 illustrates the process.

Figure 8: The Image sub-sampling and Image Histogram after applying Sobel’s filter process. 4 Transferring Color to Grayscale Image Although the general problem of adding chromatic values to a grayscale image has no exact, objective solution, Welsh et. al. [9] propose a general technique for “colorizing” grayscale images by transferring color between a source, color image and a destination, grayscale image. It transfers only chromatic information based on matching luminance and texture information, and retain the original luminance values of the target image. The algorithm is explained below.

Anais do IV Workshop em Tratamento de Imagens, NPDI/DCC/ICEx/UFMG, p. 77-89, Junho de 2003.

Figure 9: Example of the color transfer procedure. First, each image is converted into the decorrelated L color space [8]. A small subset of pixels in the color image are selected as samples. Then, going through each pixel in the grayscale image in scan-line order the best matching sample in color image using neighborhood statistics is selected. The comparison to find the best matching sample is based on the luminance value and neighborhood statistics of the pixel. The luminance value is determined by the l channel in the lab color space. To create a better correspondence in the luminance range between the two images, they perform luminance remapping [5]. This linearly shift and scale the luminance histogram of the source image to fit the histogram of the target image. The neighborhood statistics are computed over the image and consist of the standard deviation of the luminance values of the pixel neighborhood of size 5x5. The best match is determined by using a weighted average of pixel luminance and the neighborhood statistics. The chromaticity values ( , channels) of the best matching pixel are then transferred to the grayscale image to form the final image. Further, swatches can be employed to match similar areas between the two images. In order to allow more user interaction in the color transfer procedure and to improve results, swatches are used between corresponding regions in the two images. The user selectively chooses the regions to place the swatches. Color transfer using swatches involves the same global image matching procedure but only between the source and target swatches. The colorized pixels in the target swatch regions are then used as the source pixels for the color transfer to the remaining non-colorized pixels using a texture synthesis approach. This consist of using the L2 distance described in [2][3]. Their approach works well on images when corresponding color regions between the two images also correspond in luminance values. Nonetheless, the process of selecting the source image is still dependent on human interaction. 5 Experiments and Results 5.1 Tool We developed a tool, in Java, that receives as input a grayscale image, allows to chose the selection criteria, and outputs the best selected image to work as source image in the color transfer process. The image to be select, will be one among all in the database.

Anais do IV Workshop em Tratamento de Imagens, NPDI/DCC/ICEx/UFMG, p. 77-89, Junho de 2003.

Figure 10: Tool Interface The tool interface, shown in Figure 10, allows the user to select the grayscale image, to select the criteria to be used to give the best images. Ant it also shows the grayscale image and the resulting images. The tool returns the best four images. In Figure 10 we show only the first two. To analysis our image matching criterions, we kept the original color image in the database. When querying for an image, we expected that the original image would be returned as the first image. To assess the image matching criterions, we supposed that we did not have the original color image, and always used the second returning image in the color transfer process. 5.2 Images Database The experiments used a database of natural images that was developed at the ENSEA University. It has fourteen classes semantically distinct: airplanes, cars, elephants, aerial views, bears, etc., in a total of 1200 images of color naturals scenes. An example of classes presented in the image database can been seen in Figure 11. Using this database, it was possible to test our developed techniques.

Figure 11: Natural Database. Here are shown three of the fourteen different classes of images. In the horizontal are the images of the same classes. From top to down, the classes are: faces, sunset and cars.

Anais do IV Workshop em Tratamento de Imagens, NPDI/DCC/ICEx/UFMG, p. 77-89, Junho de 2003.

5.3 Tests The tests were conducted in two phases. The first consisted of choosing randomly one image of each class. Since each class have a different number of images, it was randomly selected a number among one and six (the smallest number of images presented in a given class) and the nth image of each class was chosen to be in the set of test images. Using the result of that phase, we made a pre-analyze of the behavior of each method for the images in the test set. We could divide the image classes in two groups, the homogeneous group and the heterogeneous group. The homogeneous group was composed of images without a great luminance variation and with few objects (less than three) presented in the scene. The others images, with a visual perceptive luminance variation and the presence of more than three objects were classified as heterogeneous. Seven images of the homogeneous group and eight images of the heterogeneous group were selected to analyze the techniques developed and explained in Section 3. We applied the four developed techniques to all fifteen selected images to select the source image. For each technique, and for each one of the fifteen images, we obtained the resulting image by applying the transfer color method described in Section 4. We developed a web site to assess the resulting images. Along with the resulting images, it were eight images colorized by the transfer coloring process with randomly chosen image as the source image, and also, eight original images that were not colorized by process. Each image could be classified into four levels, as shown in Figure 12.

Figure 12: Example of images and its classifying options on the web site. 5.4 Results From the votes on each image over the Internet, we can measure the performance of each method for the set of images. As we can observe in Figure 13, the technique of Luminosity Histogram and the Sub-sampling are presented as the best alternatives for a general image. Randomly selecting a source image presents the worst result.

Anais do IV Workshop em Tratamento de Imagens, NPDI/DCC/ICEx/UFMG, p. 77-89, Junho de 2003.

Figure 13: Number of votes for each technique. We observe that the Luminance Histogram technique was the best technique for the homogeneous group. This is reasonable, since the homogenous images do not vary to much, the source image can be selected by using just the luminance information, as shown in Figure 14. At the heterogeneous, the Image sub-sampling and Image Histogram after applying Sobel’s filter presented the best results. As we can conclude, those image requires more information than just the luminance. The images resulted after coloring with a source image selected by this technique appeared to be, in its major part, reasonably colorized by more than 50% of the people who answered the questionnaire.

Figure 14: Percentage of votes for each technique among the homogeneous images.

Anais do IV Workshop em Tratamento de Imagens, NPDI/DCC/ICEx/UFMG, p. 77-89, Junho de 2003.

Figure 15: Percentage of votes for each technique among the heterogeneous images. 6 Conclusion Recently, the process of automatically coloring a grayscale image by transferring color from a colored source image was developed [9]. This process requires the human intervention to select the source image. We have implemented four techniques to automatically select the source image, turning the process completely automatic. They are all based on image matching techniques [5]. Dividing the test set images in two groups: homogeneous and heterogeneous and having a relevant group of people assessing the resulted images, we were able to identify the best techniques. We observe that the Luminance Histogram technique is the best technique for the homogeneous group. And at the heterogeneous the Image sub-sampling and Image Histogram after applying Sobel’s filter present the best results. Applications, such as movie coloring, could take advantage of this work, reducing the time and the human intervention required on the process. References [1] ”http://www.dafne.pt/produtos/colorimetria/fundamentos.html”. Web site. [2] A. Efros and T. Leung”. Texture synthesis by non-parametric sampling. In 7th IEEE International Conference on Computer Vision, pages 1033–1038, 1999. [3]

A. A. Efros and W. T. Freeman”. Image quilting for texture synthesis and transfer. In In Proceedings of ACM Siggraph, pages 341–346, 2001.

[4] R. C. Gonzales and R. E. Woods. Digital Image Processing. Addison-Wesley Publishing, 1987. [5] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. Image analogies. In Proceedings of the 28 annual conference on Computer graphics and interactive techniques, pages 327–340. ACM Press, 2001. [6] S. J. ”http://www.cinesite.com/core/press/articles/1998/10 00 98-team.html”. Cinesite Press Article, 1998.

th

[7] P. W. K. Digital Image Processing. ”John Wiley & Sons”, 1991. [8] A. M. G. B. S. P. Reinhard, E. Color transfer between images. In IEEE Computer Graphics and Applications, special issue on Applied Perception, pages 34–41, 2001.

Anais do IV Workshop em Tratamento de Imagens, NPDI/DCC/ICEx/UFMG, p. 77-89, Junho de 2003.

[9] T. Welsh, M. Ashikhmin, and K. Mueller. Transferring color to greyscale images. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques, pages 277–280. ACM Press, 2002. [10] G. Wyszecki and W. S. Stiles. Color Science Concepts and Methods, Quantitative Data and Formulae. Wiley, 1982.