Example-Based Color Stylization Based on Categorical Perception Youngha Chang∗ Dept. of Computer Science Grad. School of Information Science & Engineering Tokyo Institute of Technology Keiji Uchikawa‡ Dept. of Information Processing Interdisciplinary Grad. School of Science & Engineering Tokyo Institute of Technology
Suguru Saito† Dept. of Computer Science Grad. School of Information Science & Engineering Tokyo Institute of Technology Masayuki Nakajima§ Dept. of Computer Science Grad. School of Information Science & Engineering Tokyo Institute of Technology
Figure 1: Demonstration of our automatic color transformation result. The left image shows an input photograph and the middle image shows a reference image. Our algorithm automatically gets tendencies of color use from the middle image and transfers it to the input image. The resultant image is as shown in the right image.
Abstract
1
We describe a new computational approach to stylize the colors of an image by using a reference image. During processing, we take characteristics of human color perception into account to generate more appealing results. Our system starts by classifying each pixel value into one of a set of the basic color categories, derived from our psycho-physiological experiments. The basic color categories are perceptual categories that are universal to everyone, regardless of nationality or cultural background. These categories provide restrictions on the color transformations to avoid generating unnatural results. Our system then renders a new image by transferring colors from a reference image to the input image, based on this categorizations. To avoid artifacts due to the explicit clustering, our system defines fuzzy categorization when pseudo-contours appear in the resulting image. We present a variety of results and show that our color transformation performs a large, yet natural color transformation without any sense of incongruity, and that the resulting images automatically capture the characteristics of the color use of the reference image.
Color is one of the most influential and powerful elements in images and a variety of color processing methods have been proposed in the field of image processing. Results sometimes do not match human perception, however, because they capture the characteristics of color by treating them as signals. In this paper, we propose a method that takes human perceptual characteristics into account to produce attractive color transformations. The colors of images are often changed using image editing tools, which work very well for global color transformations. For more detailed color editing, such as transforming blue sky to pale blue, green leaves to a more yellowish green, and orange flowers to a more vivid and reddish hue, a user’s manual interaction is required to produce a satisfying result. In addition, an attractive result can usually be obtained only if the user has a good aesthetic sense. In our work, we introduce the concept of visual color perception, and with it we aim to implement a more intuitive, flexible, and automatic color transformation system that stylizes the colors of images based on a provided reference image.
CR Categories: I.3.3 [Computer Graphics]: Picture/Image Generation I.4.5 [Image Processing and Computer Vision]: Reconstruction—Transform methods I.4.6 [Image Processing and Computer Vision]: Segmentation—Pixel Classification I.4.10 [Image Processing and Computer Vision]: Image Representation— Statistical
There is a related work that enables an easy and effective color cor-
Copyright © 2004 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail
[email protected]. © 2004 ACM 1-58113-914-4/04/0008 $5.00
Keywords: example-based, basic color category, color transfer ∗ e-mail:
Introduction
[email protected]
† e-mail:
[email protected] ‡ e-mail:
[email protected]
§ e-mail:
[email protected]
91
rection by referencing another image [Reinhard et al. 2001]. No artistic talent is required for this color transformation because it only requires the user to set a reference image. This method uses a simple statistical analysis and it works very successfully; however, it tries to represent the overall characteristics of color use in an image using only two statistical values per color channel, and then transforms all pixel colors in an image according to the same statistical information. Therefore, when the source and target images are not compatible, a user has to manually set swatches for each color region and have to match between them. A similar approach is used to a challenging problem, colorizing greyscale images by example [Welsh et al. 2002]. This method matches luminance values between a greyscale image and a color image, and then transfers matched color values from the color image. This approach is also simple, and by using swatches, we can successfully transfer colors. Yet, there still remains the issue that it requires manual operation for some input images.
colors [Fairchild 1997]. The shift of color appearance is apparent when the color has very low saturation. These colors tend to be named as chromatic due to the color contrast with the achromatic surrounding colors. Therefore, we perform a new visual psychophysiological test where the test stimulus is surrounded by various color stimuli.
2.1
We used an sRGB video monitor, EIZO FlexScan T 766, for the experiment. Before the experiment, we carefully calibrated the monitor by measuring red, green, and blue gamma curves and obtained the maximum luminance levels of red, green, blue, and white for the monitor. We employed test colors from CIE 1931 xy chromaticity diagram at each luminance levels, Y , of 2, 5, 10, 20, 30, 40, 50, 60, 70, 80, 90, 95, and 98, where Y = 100 is referenced to the maximum luminance level of the monitor. The xy values were sampled on a regular grid, with intervals of 0.025, in the xy plane of every luminance level. Because these experimental results are used only for colors appearing in digital images, we chose color samples within the scope of sRGB. The total number of test colors was 1254.
Our color transformation aims to be more detailed and natural. It does not try to capture the overall characteristics of color use, but instead captures characteristics for each color unit. A color unit is a perceptually agreeable color category that provides restrictions on the color transformations to ensure that resulting images will not be perceived as odd, because the transformed colors will remain perceptually similar to the original colors. As humans can detect subtle changes in color, the resulting image can evoke different emotions after a color transformation. Therefore, color transformations are performed within the same perceptual color category to produce natural and attractive images. Our algorithm first categorizes pixels into perceptually agreeable color categories and measures the characteristics of the color spread in each category. Then, for each pixel in the input image, it finds a matching color values from the same category of the reference image, and replaces the pixel color value. A user interacts with our system only to provide an input image, on which to transfer color and a reference image, which provides an example of color use. Therefore, no artistic talent is required for the color transformation, because a user only sets one’s favorite image as a reference image. The most important aspect of this paper is to generate perceptually agreeable color categories, and for this we adopt the Basic Color Terms (BCTs) [Berlin and Kay 1969]. In the previous work of Berlin and Kay, 98 languages from several families were examined and it was found that developed languages have 11 universal BCTs, with the types and color ranges for these being similar among all of the studied languages. For example, in English, the BCTs are black, white, red, green, yellow, blue, brown, purple, pink, orange and gray. Many later researchers have confirmed this idea [Uchikawa and Boynton 1987; Rosch Heider 1972]. In our system, we divide the color space into basic color categories, and perform the color transformation within each category. In Section 2, we describe our experiment to determine the spread of each basic color category. In Section 3, we explain a method for determining the tendency of pixel spreads in each basic color category, and in Section 4, we discuss a method for performing the color transformations. We show a variety of results in Section 5, and in Section 6, we conclude this paper with suggestions for future work.
2
Apparatus and stimuli
2.2
Participants
We collected color-naming data from a group of 10 participants, aged 22 to 33. Five of them are male, and the rest are female. All these participants have normal color vision.
2.3
Procedure
(a) color stimulus arrangement
(b) experimental environment 1 deg.
marker
2 deg.
(c) naming range(within red rectangle), color stimulus size, marker placement
Figure 2: Experiment on color naming. In a dark room, we presented color stimuli on the sRGB video monitor as shown in Figure 2(b). Following a period of dark adaptation, the test stimuli were presented in random order. The arrangement of test stimuli were different at every experiment. As shown in Figure 2(a), the viewing angle of each color stimulus was set at 2 degrees and color stimuli were arranged on a uniform gray field. The space between stimuli were set at 1 degree of viewing angle. We did not set surrounding conditions as one single color nor Mondrian collage of random patches of color, because 1) the purpose of this experiment is not to measure the exact effects of color contrast for all possible color combinations; it is aimed to imitate the
A Psycho-Physiological Experiment for Estimating Basic Color Categories
There are several works that aim to capture the spread of basic color categories. In most of these works, one color stimulus is shown at a time with achromatic surroundings. In images, however, color appearance of a pixel is often affected by surrounding
92
# of stimuli # of participant # of experiment # of stimulus in one screen background
Our Previous Work 802 3 6(twice/person)
Our New Experiment 1254 10 10(once/person)
1 stimulus
various stimuli
gray
various colors
Next, the achromatic categories are expanded in our new method. We define “achromaticity” as the percentage of test color which is described as “black”, “white” or “gray” at least in one experiment. Achromaticity of new experimental results is 15.5% while that of previous results is 11.9%. In our previous work, because only achromatic colors are shown in the periphery, the saturation of the test stimulus is emphasized by saturation contrast. Therefore, most of the colors tend to be named as chromatic color terms, which results in small achromatic regions. On the other hand, because we put various color stimuli around the test stimulus in our new method, it better enables participants to perceive achromaticity for very low luminance colors.
Table 1: Differences between our previous work and new work. environment where various color stimuli are mixed together in one image; 2) we aim to avoid the color assimilation effect carefully; if we arrange color patches without space between stimuli, then many low saturation colors will be perceived as chromatic because of the color assimilation effect; 3) the achromatic background indirectly indicate that illumination color is achromatic, D65 color. The participant observed each stimulus for several seconds with 50cm distance from the monitor. The duration was not limited so the participant could observe for as long as needed to decide on the response of the color. The participant responded by naming the color by using one of 11 BCTs: red, green, yellow, blue, orange, purple, brown, pink, white, gray, and black. This is achieved by typing one of 11 keys on the keyboard. After typing, the participant receives voice feedback that confirms his choice. If he had a mistake in typing a key, then he can re-name that stimulus. The naming process was done for each stimulus in a raster order except for color stimuli which border the edge of the screen(see Figure 2(c)). This is because the stimuli outside the red square will have fewer interactions between surrounding color stimuli. Because so many stimuli are shown in one screen, there may be a possibility that the participant confuses which stimulus he should fixate next. For avoiding this problem, we made small markers at the corners of current fixated stimulus as shown in Figure 2(c). After answering all the test stimuli in one screen, the screen was filled with gray for 2 seconds in order to reject the influences of any afterimage. After finishing all the experiment, color stimuli were grouped according to participant’s categorization, and these sorting results were shown to him for re-confirmation. If he found mistakes in this step, he could rename that stimulus again. The responses of the participants were recorded in the computer.
2.4
(a) our previous work [Uchikawa et al. 1994]
(b) our new experiments
Figure 3: Comparison with results of our previous work and new work. In Figure 3, we visualize a part of our experimental results on the CIE L∗ a∗ b∗ color space and show the expanding of achromatic regions. We show resulting data whose a∗ b∗ values are located between -15 degrees and 15 degrees from the a∗ axis on the a∗ b∗ plane. We also show results of low saturation colors whose distance from L∗ axis is less than 20 on the a∗ b∗ plane. For visibility, we plot each naming result in grayscale according to its percentage of achromatic namings. If the test color is always named as black, gray, or white, then the test color is drawn in black; if the test color is never named as black, gray, nor white, then the test color is drawn in white. We can see that in our new experiment, achromatic regions are expanded. Moreover, because our new experiment are also done for lower and higher luminance levels than previous one, we can estimate the naming of lower and higher luminance colors more effectively. The effects of expansion of achromatic categories will be shown in Figure 6, in Section 5.
Experimental Result
In this section, we compare our new experimental result with our previous one. We clarify the main differences of these two approaches in Table 1. Because the sampling range of the test data is not the same, we only compare test colors that are placed within the intersection of these two.
3
3.1
There are no significant differences in the naming results near focal colors, and responses are in agreed with each other. For the colors which are placed near the boundary of basic color categories, responses are ambiguous in both methods. However, in our new method, responses are more ambiguous in some color stimuli. We define “variability” as the average number of BCTs used to describe each color sample. Variability in new experimental results is 1.81 while variability in previous results is 1.47. This is because in our previous work, we used one single achromatic surrounding color, which confines the color appearance of the test stimulus by one specific environment. On the other hand, in our new method, we randomly arranged a variety of colors near the test stimulus and these stimuli affect the color appearance of the test stimulus. Therefore, we can get a variety of naming probabilities in various situations, which are useful for practical use.
Calculating the Features of Color Spread for Each Basic Color Category Automatic Pixel Naming
In the previous section, we described our psycho-physiological experiment. In this section, we explain how automatic color naming is performed by using this. First, we categorize color space by using the experimental results of the color naming described above. We choose the CIE L∗ a∗ b∗ color space for our color transformation process because it is a perceptually uniform color space. The algorithm first transfers all the test colors of our color naming experiments from the Y xy color space to the CIE L∗ a∗ b∗ color space. It then divides the color space into 11 basic color categories, in a manner similar to Voronoi tessellation, by treating each data point as a site, and combining regions belonging to the same basic color category. The algorithm then performs automatic color naming for every pixel color of an image. For this, it transfers each pixel color
93
1 to the CIE L∗ a∗ b∗ color space, and determines the basic value pin i color category, Ci , to which it belongs.
3.2
3.3
In some cases, a user might select an input image having pixels pin i from category Ci , while the reference image has no pixels from Ci . In this case, the algorithm has no reference to transfer pin i . We propose the following two approaches to solve this problem.
Generation of Basic Convex Hulls
In this section, we explain how color features of an image are obtained in each basic color category. For capturing the location, shape, and center of the color distributions, we define a convex hull that encloses all of the pixel color values within a category. We refer to these convex hulls as basic convex hulls.
The first is simple and can reflect the user’s preference. Because our algorithm allows multiple reference images to be set, another reference image may be chosen for category Ci . Yet some argue that it may be difficult to prepare harmonious multiple reference images based on their color use. The second approach automatically estimates suitable color distributions of Ci , and is useful if all categories in the reference image have some tendencies in their color distributions. For this, we statistically analyze other catre f re f egories and estimate the size of chi , and its center of gravity ci . To do this, the algorithm separately estimates the hue(H), saturare f re f tion(S), and brightness(B) of ci . The values of S and B for ci are estimated by Equation 2.
However, in some cases, convex hulls cannot capture the features of the color distribution effectively. For example, if a basic color category, Ci , has a pixel color distribution as shown in Figure 4(a), then the process described above generates a convex hull chin i as shown in Figure 4(b). This contains large unused color regions because of the very small number of isolated pixel points. This results in the colors of many pixels in the input image being transformed into the unused color regions in the reference image. In this case, we remove such isolated and very sparse points so that the algorithm can capture the color features of each color category more efficiently. This process also helps to reduce any artifacts of the image noise.
(a) pixel spread
(b) convex hull for (a)
re f
re f
S(ci ) = S(cuniv ) + ∑j∈T w j (S(c j ) − S(cuniv i j )),
re f re f B(ci ) = B(cuniv ) + ∑j∈T w j (B(c j ) − B(cuniv i j )), re f the number o f pixels in ch j w j = the number o f pixels in re f erence image
(2)
where c is the center of gravity and T is the set of basic color categories in the reference image. H cannot be calculated in a similar way because the hue is represented as an angle of the color circle. Thus, instead of calculating a mean hue value, the algorithm calculates the values of yellowish(y), bluish(b), and reddish(r) separately for each category, as shown in Equation 3.
(c) after reconstruction
re f
x = w j (H(c j ) − H(cuniv j )), j ∈ T j = red : i f (x ≥ 0) y = y + x j = orange : i f (x ≥ 0) y = y + x j = yellow : i f (x ≥ 0) b = b + x j = green : i f (x ≥ 0) b = b + x j = blue : i f (x ≥ 0) r = r + x j = purple : i f (x ≥ 0) r = r + x j = brown : i f (x ≥ 0) y = y + x j = pink : i f (x ≥ 0) r = r + x
Figure 4: A problem of approximating by using convex hull. Ichimura took similar approach for extracting explicit clusters [Ichimura 1997]. For each pixel color p in Ci , the algorithm calculates the squared-Mahalanobis distance using Equation 1. d 2 = (p − ci )t Vi−1 (p − ci )
Estimating Convex Hulls with No Direct Reference
(1)
else else else else else else else else
b = b−x r = r−x r = r−x y = y−x y = y−x b = b−x r = r−x b = b−x
(3)
After calculating the values of yellowish, bluish, and reddish, we re f estimate H(ci ) by adding these values to H( f i ), as shown in Equation 4.
where Vi is the covariance matrix for pixel colors in Ci , and ci is the center of gravity of pixels in Ci . Next, the algorithm serializes the pixels in d 2 order and checks for any significant changes in d 2 values in the sequence. If there is such a point in the sequence, and its order in the sequence has a percentage larger than a threshold t, we delete the points having larger d 2 values. In our work, we used t = 97. Our algorithm then regenerates the basic convex hull re f chi by using the remaining pixel color values within the category, as shown in Figure 4(c). Finally, it remaps the deleted points onto re f re f points where the vector from pi to ci intersects with the boundre f ary of chi .
i = red : i = orange : i = yellow : ...
re f
H(ci ) re f H(ci ) re f H(ci )
)+y−b = H(cuniv i = H(cuniv ) +y−r i = H(cuniv ) + b−r i
(4)
re f
The algorithm then estimates the size of the convex hull chi , as shown in Equation 5. re f
1 In this paper, a superscript on the right side of a variable indicates where
pin
the variable belongs: means that the variable is from an input image, pre f means that the variable is from a reference image, and puniv means that the variable is from universal color space. Variables without a superscript are from either an input image or a reference image. A subscript on the right side of the variable, i, indicates the basic color category to which the variable belongs: i = {black | white | red | green | yellow | blue | brown | pink | orange | purple | gray}. Variables without a subscript are those that can be used for any one of the basic color categories. Variables shown in bold are vectors, while variables shown not in bold are points.
R(chi )
=
w
=
R(chuniv ) ∩ Scale(R(chin i ), w), i 1 n
∗ ∑k∈U
σkin σkre f
(5)
Here, R(ch) is the region of the convex hull ch, Scale(α , β ) scales α by β , σ j is the standard deviation of the pixel color distribution in category j, U is the set of basic color categories that appears in both images, and n is the number of categories in U. In this way, re f re f we can estimate the convex hull chi when Ci does not exist in the reference image.
94
4 4.1
Color Transformation
input image, then we can reduce the number. Therefore, the number of diffusion Nd (x, y) depends on color differences at regionboundary pixels in both input and resulting images, as described by Equation 8.
Matching Point Calculation and Replacement
Nd (x, y) = Pg (x, y) ∗ Linear(T (x, y),t), T (x, y), T (x, y) ≤ t , Linear(T (x, y),t) = t, T (x, y) > t out in out f (x, y) − f (x, y), f (x, y) − f in (x, y) > 0 T (x, y) = 0, f out (x, y) − f in (x, y) < 0
In this section, we describe a process for finding a corresponding color value in the reference image for every pixel color value in the input image. For each pixel color value pin i in the input image, its re f corresponding color value, pi , is given by Equation 6. re f
pi
=
in ||cin re f re f i − pi || re f (b − ci ) + ci in in ||ci − bi || i
(8) where f out (x, y) is the color difference at (x, y) in the output image.
(6)
Fourth, the algorithm diffuses pixel naming. It starts from each boundary pixel (x, y) and diffuses to neighboring pixels with Nd (x, y) iterations. A more precise statement of the algorithm is:
where ci is the center of gravity of the basic convex hull chi , and bi is the intersection point between chi and a line between ci and pi . Next, the algorithm substitutes each pixel color value in the input image with its corresponding color value calculated as described above. In this way, we can produce an image whose color features are similar to those of the reference image.
4.2
while there exist (x,y) such that Nd (x, y) > 0 do: for each pixel (x,y) in an image do: if Nd (x, y) > 0 then diffuse color naming at (x,y) Nd (x, y) = Nd (x, y)-1 for each its neighborhood pixel (x0 , y0 ) do: if Nd (x, y) > Nd (x0 , y0 ) then Nd (x0 , y0 ) = Nd (x, y)
Consideration of the Color Distribution on the Image Space
So far we have only considered the features of the color distributions in color space and have not considered the relationships between neighboring pixels in image space. As a result, if an input image has a color graded region that covers more than one basic color category, then a pseudo-contour may appear. In this section, we define fuzzy color naming of pixels when pseudo-contours appear in the resulting image, and recalculate the corresponding color values for these pixels. Fuzzy color naming is accomplished by diffusing color names of the pixels. This diffusion can avoid artifacts due to explicit clustering of the color space.
We make a restriction on “diffuse color naming” that pixel naming will not be diffused when the color difference is apparent, even within the same region. This is realized by using anisotropic diffusion [Perona and Malik 1990], where the diffusivity depends on the value of f in . We show an example in Figure 5. Because f in values are large in the upper part of the boundary between blue and green region, the color naming results in this part are not diffused as shown in Figure 5(c). On the other hand, because values of f in in the lower boundary part are small, the color naming results in this part are diffused. Now, pixels near the lower boundary part has probabilities that they belong to both blue and green categories.
For this, the algorithm first performs automatic color naming for each pixel value, merges neighboring pixels sharing the same color name, and generates regions in image space. Next, the algorithm calculates where color naming should be diffused. Pseudo-contours may occur at region boundaries, such as when two adjacent regions have little color differences in the input image yet large color differences in the output image. This may be due to transforming colors in different directions. Thus, the algorithm calculates the integrating probability Pg between each adjacent region, where Pg depends on the color differences at boundary pixels in the input image. If the color differences are large in the input image then Pg will be small, and vice versa. Thus, we calculate Pg for each region-boundary pixel, as shown in Equation 7. 1 − Sigmoid( f in (x, y)), 0, Sigmoid(X) = 1/(1 + e−0.3(X−c) ) Pg (x, y) =
(a) input image
(b) naming result by 3.1
(c) naming result by 4.2
Figure 5: Restriction on diffusion of naming. Finally, for each pixel color value pin , the algorithm recalculates the corresponding color value by using fuzzy color naming result described above. This is achieved by weighted sum of corresponding color values in each color category as shown in Equation 9.
(x, y) is a boundary pixel otherwise
pre f = ∑(cci ∗ pi ), re f
i
(9)
∑ cci = 1
(7) where (x, y) is a position in the image space, f in (x, y) is the color difference at (x, y) in the input image and Sigmoid(X) is a sigmoidlogistic function. Sigmoid function is employed for its smooth change of shape and for avoiding artifacts which may occur by explicit definition of threshold.
i
re f
where cci is the probability that pin belongs to Ci and pi is the re f corresponding point calculated using chi , as shown in Equation 6.
Third, the algorithm calculates the extent to which pixel naming should be diffused. Diffusion should be done until color difference is indistinctive in the output image. Therefore, if the color differences are large in the output image, then the number of diffusion should be large to minimize the visibility of pseudo-contours. On the other hand, if the color differences are originally large in the
5
Results
In this section, we show that our method is capable of generating a variety of color transforming results.
95
(a) input image
(b) by our previous method [Chang et al. 2002; Chang et al. 2003]
(c) by our new method
Figure 6: Automatic color naming result. Figure 6 shows an automatic color-naming result that demonstrates the effectiveness of the color-naming experiment described in Section 2. Figure 6(a) shows an input image and Figure 6(b) shows our prior automatic naming result [Chang et al. 2002; Chang et al. 2003] for comparison. Each pixel is shown in color according to the automatic naming results for each pixel color. In Figure 6(b), low saturation regions, such as those in the shadows and the skies, are named as chromatic colors. On the other hand, as shown in Figure 6(c), colors with those lower saturation are classified into achromatic colors correctly by using our new experimental results. This enables to prevent transferring shadows from black to green, road from gray or white to green, and so on.
Figure 8(f), (g) and (h). Moreover, our color transformation method can preserve the color variety in each color region because our algorithm calculates a matching color value for every pixel, instead of only replacing each segmented area with one color. For example, grass textures in Figure 8(a) have a variety of yellowish colors. In the resultant images, Figure 8(f) and (i), preserve these color variety. Furthermore, the resulting images are perceived as being natural, despite the large color transformations, because each pixel’s color remains in the same basic color category. Figure 9 shows the result of applying our algorithm to the examplebased NPR (non-photorealistic rendering) method. Many studies on NPR [Gooch and Gooch 2001] have investigated methods for recreating brush strokes [Haeberli 1990; Meier 1996; Saito and Nakajima 1996; Litwinowicz 1997; Hertzmann 1998] and realistically simulating paints and materials [Curtis et al. 1997; Mario C. Sousa 1999; Mario C. Sousa 2000]. In the real world, there are a variety of styles that artists can use, and the chosen styles depend on personal preference and the situation. For achieving a variety of NPR styles, a method for example-based NPR was proposed [Hertzmann et al. 2001]. This method is based on feature matching between images, and transfers one’s feature to another. It is very fascinating work and works very well for a wide variety of images, but it uses luminance values as a feature vector in the feature matching process of generating NPR filters. This is because of the “curse of dimensionality” problem. Since color use is also one of the most important features in painting, we combine our example-based color transformation method and Hertzmann et al.’s example-based brush stroke effects. We use the reference image in Figure 9(b) and its blurred image as training data. Figure 9(c) shows the result of only touch-texture transformations by using open source code [Analogies-Web-Page ]. Figure 9(d) shows the result of brush stroke effects after transforming colors. This indicates that the combination of our method and theirs can produce very attractive results.
Figure 7 compares our color transformation result with results from previous works. Figure 7(c) shows the color transformation result by previous work [Reinhard et al. 2001]. This previous work allows manual operations by a user, but here, their automatic method is compared. Figure 7(c) seems as if it is a result of filtering the input image by a yellowish filter. It is because the mean color of reference image is yellowish color. For this reason, color spreads of the input image are moved to yellowish part. As a result, the image looks like the one filtered by yellowish filter. The textures on the woman’s clothes are transferred from reddish color to yellowish one. Although it does not give so much unnatural feelings because it was just textures in this example, but if it was some color regions whose colors are perceptually very meaningful, for example, red apples, strawberries or etc, then this kind of color transformation will cause unnatural feelings. Figure 7(d) shows our previous color transformation result [Chang et al. 2002; Chang et al. 2003]. Because in our previous work, we did not consider the relationship in image space, colors in background, which are defocused and blurred, are transferred discontinuously. This results in pseudo-outlier. An even worse problem is that some parts of hair and webbing strap, which have very low luminance, are named as green and transformed to green. Figure 7(e) shows our new color transformation result. In our new approach, because each pixel color shifts to a different direction according to its basic color category, the resultant image does not cause any unnatural feelings. Moreover, because we also consider the relationship in image space, pseudo-outlier do not appear in the resulting image.
6
Conclusion and Future Work
In this paper, we have developed a perceptual based image processing method to realize an automatic example-based color transformation.
Figure 8 shows various examples of our color transformation algorithm. Figures 8(a), (b) and (c) show input images, and Figures 8(d) and (e) show reference images. Figures 8(f)-(k) show color transformed images. We can see that the colors of the resulting images differ, according to the reference images in Figures 8(f)-(k), because our algorithm can capture the color features of the reference image and transfer it to the input image. For example, reference image1, shown in Figure 8(d), is drawn by dark yellow, light blue, yellowish green and etc. Our algorithm captures those color use, and produces resultant images which are rendered with similar colors, e.g. dark yellow, light blue, yellowish green, etc as shown in
Our psycho-physiological experiment gives us categorical color distribution in color space which takes into account the effects of surrounding colors. The achromatic color categories are expanded compared to conventional color naming experiment. Therefore, in digital color imaging, the obtained color classification data is more suitable to successfully name pixel colors as white, gray, or black which have low saturation. Our color transformation algorithm provides straightforward oper-
96
(a) input image
(b) reference image
(c) by previous work [Reinhard et al. 2001]
(d) by previous work [Chang et al. 2002; Chang et al. 2003]
(e) by our new result
Figure 7: Comparison with previous works. apply our algorithm to video-recoloring. Beyond the clear applications of this work in color transformation, the perceptual categories presented in this paper can be usefully applied in a variety of other areas in digital image indexing where color feature description is important. (a) input image
Our future plans focus on two problems. First, the present work deals with the effect of surrounding color as static when naming colors. However, the effects of surrounding colors would be more complicated. Color might be named by different basic color terms when the surrounding color has significant deviation. We will incorporate these surrounding color effects in order to allow more perceptually correct color naming. Second, we will integrate computational model of the human color vision in our process. For example, when our current system does anisotropic diffusion, it uses the Euclidean color difference in the input image as the key factor for diffusivity. But in fact, it would depend also on the image frequency. We will take into account the image frequency and chromatic color sensitivity function for our process.
(b) reference image
The input images are from Terra Galleria [Terra-Galleria ] and reference images are from the Artchive [Artchive ].
(c) result of adding texture [Hertzmann et al. 2001]
References A NALOGIES -W EB -PAGE. analogies/.
http:// mrl.nyu.edu/ projects/ image-
A RTCHIVE. http://www.artchive.com/. B ERLIN , B., AND K AY, P. 1969. Basic Color Terms: Their Universality and Evolution. University of California Press. C HANG , Y., S AITO , S., AND NAKAJIMA , M. 2002. Color transformation based on the basic color categories of a painting. In ACM SIGGRAPH Conference Abstracts and Applications, 157.
(d) texture+color transformation Figure 9: Result of adding the effect of brush strokes.
C HANG , Y., S AITO , S., AND NAKAJIMA , M. 2003. A framework for transfer colors based on the basic color categories. In Computer Graphics International, 176–183.
ation; just input one reference image. This is achieved by the combination of a perceptual restriction in color space and an evaluation of color difference in image space.
C URTIS , C. J., A NDERSON , S. E., S EIMS , J. E., F LEISCHER , K. W., AND S ALESIN , D. H. 1997. Computer-generated watercolor. 421–430.
This work should have major impact on the field of digital imaging. Because it is an automatic color transformation method, we can also
FAIRCHILD , M. D. 1997. Color Appearance Models. AddisonWesley.
97
(a) input image1
(b) input image2
(c) input image3
(d) reference image1
(e) reference image2
(f) result with (a) & (d)
(g) result with (b) & (d)
(h) result with (c) & (d)
(i) result with (a) & (e)
(j) result with (b) & (e)
(k) result with (c) & (e)
Figure 8: Experimental results. G OOCH , B., AND G OOCH , A. 2001. Non-Photorealistic Rendering. A. K. Peters.
P ERONA , P., AND M ALIK , J. 1990. Scale-space and edge detection using anisotropic diffusion. In IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, 629–639.
H AEBERLI , P. 1990. Paint by numbers:abstract image representation. 207–214.
R EINHARD , E., A SHIKHMIN , M., G OOCH , B., AND S HIRLEY, P. 2001. Color transfer between images. In IEEE Computer Graphics and Applications, vol. 21, 34–41.
H ERTZMANN , A., JACOBS , C. E., O LIVER , N., C URLESS , B., AND S ALESIN , D. H. 2001. Image analogies. In ACM SIGGRAPH, 327–340.
ROSCH H EIDER , E. 1972. Universals in color naming and memory. In Journal of Experimental Psychology, vol. 93, 10–20.
H ERTZMANN , A. 1998. Painterly rendering with curved brush strokes of multiple sizes. In ACM SIGGRAPH, 453–460.
S AITO , S., AND NAKAJIMA , M. 1996. Automatic production of hand-painted images. In The Journal of the Institute of Television Engineers of Japan, vol. 50, 1528–1535.
I CHIMURA , N. 1997. Region segmentation method using both feature space and image space based on robust clustering. In IEICE Transaction on Information and Systems, vol. J80-D2, 1752–1763.
T ERRA -G ALLERIA. http://www.terragalleria.com/. U CHIKAWA , K., AND B OYNTON , R. M. 1987. Categorical color perception of japanese observers: Comparison with that of americans. In Vision Research, vol. 27, 1825–1833.
L ITWINOWICZ , P. 1997. Processing images and video for an impressionist effect. In ACM SIGGRAPH, 407–414.
U CHIKAWA , K., K URIKI , I., AND S HINODA , H. 1994. Expression of color appearance in aperture and surface color modes with a category rating estimation method. Springer-Verlag, vol. 78, 41– 51.
M ARIO C. S OUSA , J. W. B. 1999. Computer-generated graphite pencil rendering of 3d polygonal models. In Computer Graphics Forum, 195–208. M ARIO C. S OUSA , J. W. B. 2000. Observational models of graphite pencil materials. In Computer Graphics Forum, 27–49.
W ELSH , T., A SHIKHMIN , M., AND M UELLER , K. 2002. Transferring color to greyscale images. In ACM SIGGRAPH, vol. 21, 277–280.
M EIER , B. J. 1996. Painterly rendering for animation. vol. 30, 477–484.
98
Example-Based Color Stylization Based on Categorical Perception Youngha Chang, Suguru Saito, Keiji Uchikawa and Masayuki Nakajima
(a) input image
(b) by our previous method
(c) by our new method
Figure 6: Automatic color naming result.
(a) input image (b) reference image (c) by previous work
(d) by previous work (e) by our new result [Reinhard et al. 2001] [Chang et al. 2002; Chang et al. 2003]
Figure 7: Comparison with previous works.
(a) input image1
(b) input image2
(c) input image3 (d) reference image1 (e) reference image2
(f) result with (a) & (d)
(g) result with (b) & (d)
(h) result with (c) & (d)
(i) result with (a) & (e)
(j) result with (b) & (e) Figure 8: Experimental results.
(k) result with (c) & (e)
188
