Accessible Image Search Meng Wang Microsoft Research Asia Beijing 100080, P. R. China
[email protected]
Bo Liu
∗
Xian-Sheng Hua
University of Science and Technology of China Hefei 230027, P. R. China
kfl
[email protected]
Microsoft Research Asia Beijing 100080, P. R. China
[email protected]
ABSTRACT
Keywords
There are about 8% of men and 0.8% of women suffering from colorblindness. We show that the existing image search techniques cannot provide satisfactory results for these users, since many images will not be well perceived by them due to the loss of color information. In this paper, we introduce a scheme named Accessible Image Search (AIS) to accommodate these users. Different from the general image search scheme that aims at returning more relevant results, AIS further takes into account the colorblind accessibilities of the returned results, i.e., the image qualities in the eyes of colorblind users. The scheme includes two components: accessibility assessment and accessibility improvement. For accessibility assessment, we introduce an analysisbased method and a learning-based method. Based on the measured accessibility scores, different reranking methods can be performed to prioritize the images with high accessibilities. In accessibility improvement component, we propose an efficient recoloring algorithm to modify the colors of the images such that they can be better perceived by colorblind users. We also propose the Accessibility Average Precision (AAP) for AIS as a complementary performance evaluation measure to the conventional relevance-based evaluation methods. Experimental results with more than 60,000 images and 20 anonymous colorblind users demonstrate the effectiveness and usefulness of the proposed scheme.
Image Search, Colorblindness
Categories and Subject Descriptors H.3.3 [Information Storage and Retrieval]: Retrieval Models; K.4 [Computers and Society]: Social issues Assistive technologies for persons with disabilities
General Terms Algorithms, Experimentation ∗This work is performed when the second author was visiting Microsoft Research Asia.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MM’09, October 19–24, 2009, Beijing, China. Copyright 2009 ACM 978-1-60558-608-3/09/10 ...$5.00.
1.
INTRODUCTION
The last decade has witnessed great advance of image search techniques [22]. However, the existing frequentlyused image search engines, such as Google, Yahoo and Microsoft image search, all aim to provide search service for normal viewers and focus on returning as many as ranked relevant results with the users’ queries. Special human groups, such as colorblind users, have not been taken into consideration. In fact, worldwide around 8% of men and 0.8% of women have certain kinds of colorblindness, i.e., having difficulties in discriminating certain color differences. Good search results for a normal viewer may not be acceptable for colorblind users, since many images will not be readily perceived by them. As an example, Fig.1(a) illustrates the top results of query “rose” returned by the Google image search engine, and we can see that these results are all relevant and with high quality. Then, in Fig.1(b) we mimic the view of a colorblind user (i.e.,what the colorblind user will perceive) using the simulation algorithm proposed in [5], and we can see that the qualities of many images have significantly degraded due to the loss of color information, i.e., they cannot be well perceived by colorblind users. As noted by Berners-Lee (the inventor of World Wide Web), “the power of the Web is in its universality, and access by everyone regardless of disability is an essential part” [1]. However, up to now there is no service that accommodates colorblind users in image search, and an associated fact is that many colorblind users cannot efficiently find and enjoy images with these existing search engines. Therefore, an image search scheme that can serve the numerous colorblind users is highly desired. It is worth mentioning that there are already several techniques that can support special users in web search. For example, Google provides an accessible search service, which aims at facilitating visually impaired users (mainly blindness) [4]. It is developed based on several heuristic criteria, such as the page’s simplicity, how much visual imagery it carries and whether it is immediately viable with keyboard navigation, and then analyze whether the page can be easily read by speech synthesis software accordingly. However, accommodating colorblind users in image search will be much more challenging, since it will involve not only the psychological and cognitive study of colorblind users but also image visual analysis technology. Via communicating with a lot of colorblind users, it is found that we can facilitate them in the following two aspects:
(b)
(a)
Figure 1: What is the difference between a normal viewer and a colorblind viewer in image search? (a) illustrates what a normal viewer observe and (b) illustrates what a protanopia (a type of colorblindness) observe. We have marked the images with red border of which the qualities degrade significantly due to the loss of color information. Original ranking list
Accessibility Assessment
Output
Image Corpus
User Customization
...
...
...
Query
Colorblind Simulation
...
Rerank top 100 results Recolor top 50 results
...
(1) Identify and prioritize those images that can be better perceived by colorblind viewers. (2) Provide a method to modify the colors of the images such that they can be better perceived by colorblind viewers. In this work, we introduce an Accessible Image Search (AIS) that can meet the above two requirements. The scheme contains two components, i.e., image colorblind accessibility assessment and accessibility improvement. The colorblind accessibility of an image can now be understood as how well an image can be perceived by colorblind users, and in Section 3 we will further discuss this term in detail. Based on these two components, we can facilitate colorblind users in different ways, such as reranking search results according to accessibility measurements or recoloring the images, as illustrated in Fig. 2. Comprehensive study with a batch of anonymous colorblind users has demonstrated the usefulness of the proposed scheme. This is our primary step to help colorblind users better enjoy the advances of multimedia search technology. The main contribution of this paper can be summarized as follows: (1) Propose an accessible image search scheme. To the best of our knowledge, this is the first integrated scheme to facilitate colorblind users in image search. (2) Propose an image colorblind accessibility assessment approach, based on which we can perform different reranking strategies to prioritize highly accessible images. (3) Propose an efficient image recoloring method which is able to improve the accessibilities of images. It is worth noting that image recoloring is an important topic per se [7][12] and our algorithm can actually be applied in many other applications. (4) Propose a performance evaluation measure for accessible image search that takes the accessibilities of search results into account. The organization of the rest of this paper is as follows. In Section 2, we provide a short review on the related work. In Section 3, we introduce the AIS scheme, including accessibility assessment, accessibility improvement and the per-
Accessibility Improvement
Figure 2: The schematic illustration of AIS.
formance evaluation measure. Experimental results are presented in Section 4. Finally, we conclude the paper in Section 5.
2. 2.1
RELATED WORK Color and Colorblindness
Colors are perceived by humans with their cones absorbing photons and sending electrical signal to the brains [17]. According to their peak sensitivity, the cones can be categorized into Long (L), Middle (M ) and Short (S), which absorb long wavelengths, medium wavelengths and short wavelengths, respectively. Consequently, light is perceived as three members: (l, m, s) where l, m, and s represent the amount of photons absorbed by L-, M - and S−cones, respectively. More formally, color stimulus (Si ) for a light can
Original Images
Protanopic View
High Accessibility
Middle Accessibility
Low Accessibility
Figure 3: Several typical examples with high, middle and low colorblind accessibilities for protanopic users. We see that the qualities of several images are well preserved while several others significantly degrade due to the loss of color information. proposed an image recoloring process named Daltonize, which first increases the red/green contrast in the image and then use the red/green contrast information to adjust brightness and blue/yellow contrast. Iaccarino et al. [10] have proposed a simple recoloring method to improve the accessiwhere φ stands for power spectral density of the light, lL , bility of web pages. Yang et al. [15] proposed a method lM , and lS indicate L-, M -, and S- cones. which changes a monochromatic hue into another hue with Colorblindness, formally known as color vision deficiency, less saturation for dichromats. Rasche et al. formulate the is caused by the deficiency or lack of certain type of cone. recoloring task as a dimensionality reduction problem, i.e., Dichromats are referred to as those who have only two types how to map the colors in a 3-dimensional space into a 2of cones, and they consist of protanopes, deuteranopes, and dimensional space that can be recognized by colorblind viewtritanopes which indicate the lack of L-cones, M -cones and ers [12]. Huang et al. [7] proposed an image recoloring algoS-cones, respectively. Protanopes and deuteranopes have rithm that keeps both the discriminative abilities of colors difficulty in discriminating red from green, whereas tritanopes and the naturalness of the image. In [9], Jefferson proposed have difficulty in discriminating blue from yellow. In this a document recoloring algorithm. It first selects a reprework we focus on protanopia and deuteranopia as most dichrosentative set of colors from the source document, and then mats belong to these two types, but our methods can also changes these colors while preserving their differences, and be easily extended to deal with tritanopia. finally it performs interpolation operation for other colors. Significant research works have been put on simulating In [8], Jefferson et al. provided an interface to support the colorblindness [24]. Brettel ea al. [5] proposed a method interactive recoloring implementation for colorblind viewers. that transforms colors from RGB space to LMS (long, medium, In [30], Huang et al. proposed a generalized histogram equalshort) color space based on cone response and then modiization method to re-map the hue components of images in fies the response of the deficient cones. This algorithm is HSV color space. Wakita et al. [14] proposed an optimizawidely adopted by colorblindness simulation systems such tion approach which simultaneously takes into account the as VisCheck [2] and IBM aDesigner’s low vision mode [3]. contrast, consistency, distinguish-ability and naturalness of Obviously, to facilitate colorblind users, we first have to rethe mapped colors. veal what they have seen. Thus, the colorblind simulating Several encouraging results have been reported in these algorithms form the basis of our work. works. However, most of them implement an optimiza2.2 Efforts on Accommodating Colorblindness tion process to accomplish the color mapping and thus need large computational costs. So they can hardly be applied in Several efforts have been dedicated to helping colorblind practical large-scale application. In our proposed recoloring users better perceive and enjoy visual documents, such as method, we simply perform several color rotation operations web pages and images. In 2005 and 2008, a workshop named Computer Vision Applications for the Visually Impaired(CVAVI) and it is thus much efficient. Empirical results will show that it even outperforms the existing optimization-based methhave been organized with two top conferences in computer ods. vision society [25][26]. However, most of the research foIt is worth noting that recoloring can only improve the cuses on recoloring images or web pages such that they can colorblind accessibilities of images in a certain degree, i.e., be better perceived by colorblind viewers [23], whereas how the images will not be as good as those that perceived by to help them find more useful and accessible images or web normal viewers even after recoloring, as 1D color informapages, which is also critical and meaningful for these users, tion has lost in the colorblind view [12]. Therefore, in AIS receives much less attention. Kovalev conducted a study on we simultaneously provide the accessibility-based reranking image retrieval for colorblind users [11], but it only investiand recoloring-based accessibility improvement techniques, gates several effects of colorblindness in image retrieval and and their combination can provide a series of services for do not provide any solution or service for colorblind users. colorblind users. About image/webpage recoloring, Dougherty et al. [2] be computed as the integration over the wavelengths λ: Si = φ(λ)li (λ)dλ, i = L, M, S (1)
3. ACCESSIBLE IMAGE SEARCH In this section, we introduce our AIS scheme. First, we provide a definition of the colorblind accessibility of an image, and then we propose an objective accessibility assessment approach. After that, we introduce the accessibility improvement method, i.e., the image recoloring algorithm. We will show what services we can provide based on these two components. Finally, we introduce a performance evaluation measure for AIS. It is worth mentioning that in fact AIS should simultaneously take into account relevance and accessibility of search results. But there are already many research efforts focusing on relevance [27] (as well as several other related criteria, such as diversity [28] and typicality [29]), and thus in this work we mainly focus on accessibility. Our scheme can also be easily integrated with the relevance improvement methods [27][28][29] or extended to compromise relevance and accessibility.
Color DistinguishǦability Loss Color DistinguishǦability
(a) SVR Model
Training Data
3.2 Accessibility Assessment Obviously a crucial component in AIS is the automatic assessment of image accessibility. Based on the evaluated accessibilities, we can adopt different reranking techniques to prioritize the highly accessible images, as illustrated in Fig. 2. The accessibility assessment task is different from general image quality assessment [20], since in accessibility assessment we can focus more on color information in comparison with edge and texture. Here we introduce two
Feature Vector
Accessibility Score
BlockǦWise Color Moment Extraction
(b)
3.1 What is Image Colorblind Accessibility In Merriam-Webster dictionary, Accessibility is explained as the capability of being used or seen. However, this term actually has more specific meaning: “Accessibility is a general term used to describe the degree to which a product is accessible by as many people as possible. It is often used to focus on people with disabilities and their right of access to entities, often through use of assistive technology” [18]. A typical example is “web accessibility”, about which W3C has provided a guideline in order to makes web pages accessible to everyone [1]. Analogous to the definition of web accessibility, we straightforwardly define an image’s colorblind accessibility as the degree of how well the image can be perceived by colorblind users. Given the context of helping colorblind users in this work, in the rest of our paper we will replace colorblind accessibility by accessibility for short. Typically, the accessibility of an image involves two factors: the quality of the original image and the information loss in colorblind perception. In this work we focus on the investigation of the second factor considering two facts: (1) we find that actually nowadays most of the results returned by popular image search websites are with high quality (such as the images used in our experiments); and (2) image quality assessment [20] [21] is a challenging issue per se and it is difficult to find an accurate and robust algorithm for real-world application. We illustrate several web images with different colorblind accessibilities and their deuteranopic views in Fig. 3. We can see that several high-quality images will have low accessibilities due to the loss of color information. We have to emphasize that accessibility is a very subjective measure. But common judgment still exists. For example, most users will agree that in Fig. 3 the colorblind views in the first row have better accessibilities than those in the last row.
Accessibility Score
Figure 4: (a) In analysis-based accessibility assessment approach, the accessibility score of an image is defined based on the loss of color distinguishabilities for a colorblind viewer; (b) In learningbased accessibility assessment, the accessibility score of an image is predicted by a model learned from labeled training data.
methods for accessibility assessment, as shown in Fig. 4, one is an analysis-based method and the other is a learningbased method. The analysis-based method takes advantage of the prior knowledge that the colorblind accessibility problem is caused due to the loss of color information. Therefore, we estimate the accessibility of an image based on the loss of color distinguish-ability. The learning-based method belongs to a different approach. It learns the accessibility of images through a regression model with a labeled training set. In experiments we will compare these two methods in terms of both performance and computational efficiency.
3.2.1
Analysis-based Accessibility Assessment
We assume that the accessibility is decreasing with the color information loss between the original image and its colorblind view, which is defined as
Loss(x, π(x)) =
n−1 n 1 ∆(ci , cj ) − ∆(π(ci ), π(cj ))2 n2 i=1 j=i+1
(2) where ∆(ci , cj ) indicates the difference between colors ci and cj , and π(ci ) represents the colorblind view of ci . It measures if the difference between color pairs has been preserved in the colorblind view. We adopt the CIE94 color difference for ∆(., .) [16], which is a weighted Euclidean distance in LCH (luminance, chroma, hue) color space. For the sake of computational efficiency, we equally quantize the original RGB color space into Q bins, and denote by ni the number of samples that belong to the i-th bin. Thus Eq. (2) turns to
Loss(x, π(x)) =
Q Q−1 1 ni nj ∆(ci , cj )−∆(π(ci ), π(cj ))2 n2 i=1 j=i+1
(3) Based on the information loss measurement, we define the colorblind accessibility of an image as Accessibility(x) = 1 − Loss(x, π(x))
(4)
3.2.2 Learning-based Accessibility Assessment In learning-based accessibility assessment, each image is represented by a d-dimensional feature vector, i.e., it is assumed that the accessibility score of an image can be predicted with these features. Then we collect a set of training data T = {(x1 , y1 ), (x2 , y2 ) ..., (xl ,yl )}, where yi indicates the ground-truth accessibility score of xi . As shown in Fig. 4 (b), a model is learned from these training samples, and the accessibility scores of the other images can be directly predicted by this model. Here we adopt Support Vector Regression (SVR) model [31][13]. It learns a non-linear fitting function by a linear maximum-margin learning machine in a kernel-induced feature space. The fitting function takes the form Accessibility(x) =< w, Φ(x) > +b
(5)
where Φ(.)is a mapping from Rd to a Hilbert Space H, and < ., . > denotes the dot product in H. Then the soft-margin SVR is formulated as follows 1 (ξi + ξi∗ ) w2 + C 2 i=1 ⎧ ⎨ yi − < w, xi > −b ≤ ε + ξi < w, xi > +b − yi ≤ ε + ξi∗ subject to ⎩ ξ , ξ∗ ≥0 i i
Figure 5: The re-coloring process is accomplished by two color rotation steps in CIELAB domain.
3.3
Accessibility Improvement
We propose an efficient image recoloring algorithm to improve the accessibilities of images. Different from the traditional recoloring algorithms that optimize the color mapping functions, the proposed method simply performs several color rotation operations in CIELAB domain to accomplish the recoloring. The method consists of two steps, i.e., local color rotation and global color rotation, as illustrated in Fig. 5. For local color rotation, we adopt a method similar to the one proposed by Huang et al. [7], of which the basic idea is to map the information of a∗ into the b∗ axis since the color information in a∗ gets lost significantly for protanopia and deuteranopia. We rotate the color which has the included angle θ with respect to the a∗ axis by an angle φ(θ), i.e.,
l
minimize
(6)
With the help of Lagrange multipliers, the dual of the above problem can be obtained
− 12 li,j=1 (αi − αi∗ )(αj − αj∗ ) < Φ(xi ), Φ(xj ) > l −ε i=1 (αi + αi∗ ) + li=1 yi (αi α∗i ) (7) where α is a vector with components αi that are the Lagrange multipliers. Since the mapping Φ(.) only appears in the dot product, we need not to know its explicit form. Instead, we can define a kernel K(., .) with K(xi , xj ) =< Φ(xi ), Φ(xj ) > to accomplish the mapping from the training data space to the Hilbert Space H. More details about SVR can be found in [31][13]. In this work, we adopt the block-wise color moment as the low-level features for its capability to simultaneously capture the statistical and spatial distribution of colors in images. For each image, we average split it into 25 blocks, as shown in Fig. 4 (b), and then extract 9D color moments in LAB color space from each block (the mean, variance and skewness of l, a and b components respectively). For the SVR model, we adopt RBF kernel. In experiments we will show that the accessibility scores predicted in this way correlate well with subjective perception. maximize
⎤ ⎤⎡ ⎤ ⎡ L 1 0 0 L ⎣ a ⎦ = ⎣ 0 cos(φ(θ)) − sin(φ(θ)) ⎦ ⎣ a ⎦ b 0 sin(φ(θ)) cos(φ(θ)) b ⎡
(8)
Such a rotation has three advantages: (1) the image after color rotation has the same luminance as the original image; (2) colors with the same hue in the original image still maintain the same hue after color rotation; (3) the saturation of the original colors is not altered after color rotation. Huang et al. [7] define φ(θ) as φ(θ) = φmax 1 −
|θ| π/2
γ (9)
They use different parameters φmax and θ for left and right planes and there are six parameters involved in all. They use Fletcher-Reeves conjugate-gradient method to find the optimal values of the parameters and the optimization process needs intensive computation. Here we scale up the method by reducing the number of parameters and simplifying the parameter decision process. Intuitively, the parameter φmax should be more sensitive than the parameter γ since it controls the range scope of rotated colors, and in practical experiments we have also empirically validated this fact. Thus we only keep the parameter φmax by simply setting the parameter γ to 1. We perform opposite operations in left and right planes, which can help balance the blue and yellow hues in the re-colored images. Therefore, there is only one
parameter φmax now in the function φ(θ) ⎧
⎪ ⎪ φmax 1 − |θ| if − π2 ≤ θ < π2 ⎨ π/2
φ(θ) = |θ − π| ⎪ ⎪ if π2 ≤ θ < 3π ⎩ −φmax 1 − 2 π/2
(10)
Then, we select the parameter φmax by grid search from a predefined candidate set, of which the criterion is to maximize the diversity of the colors on b∗ axis, i.e., φmax = argminφ∈S
i
(bi − bj )2
(11)
j
It is equivalent to maximizing the variance of the b components of the colors. After local color rotation, we then further adopt a global color rotation to refine the result. We regard each color in the image as a sample in a∗ -b∗ plane, and perform 2-dimensional Principle Component Analysis (PCA) to extract the major component. We then rotate the colors such that the major component is consistent with the discriminative orientation of colorblind users. The discriminative orientation stands for the orientation of the 1dimensional surface on which the colors can be best distinguished. It can be calculated that the normals of the approximating orientation is (0.99, 0.14). Denote by θd and θm the included angle of the discriminative orientation and the major component with respect to the a∗ axis. Then we can derive that the rotation angle θr is ⎧ ⎨ θd − θm − π, θd − θm + π, θr = ⎩ θ −θ m d
if θd − θm > π if θd − θm < −π else
(12)
The global rotation can be formulated as ⎤⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 0 L 1 0 0 T (L) ⎣ T (a) ⎦ = ⎣ 0 cos(θ(r)) −sin(θ(r)) ⎦ ⎣ a ⎦+⎣ a ⎦ 0 sin(θ(r)) cos(θ(r)) T (b) b b (13) where a and b are the mean values of the a and b components respectively. In fact, if we suppose that the colorblind simulation π is equivalent to projecting colors into the discriminative orientation, we can prove that the above equation which maximizes is an optimal global rotation 2 2 i,j (π(T (ai )) − π(T (aj )) + π(T (bi )) − π(T (bj )) ), i.e., optimizing the discriminative abilities of the colors perceived by colorblind users on a∗ − b∗ plane. Although the two steps both perform color rotation, they have different impacts. In the local color rotation step, there is a stretch effect on the color space such that the space near b* will be condensed, whereas the global color rotation just rolls the colors while keeping the relative position of each color. ⎡
3.4 Services in Accessible Image Search The accessibility assessment and improvement are off-line processes and they are implemented in the back-end. With these two components, we can provide a series of services for colorblind users. We illustrate several typical applications as follows: (1) Rerank top K search results based on accessibility scores; (2) Rerank top K search results based on the combination of original ranking scores and accessibility scores;
(3) Improve the accessibilities of the originally searched images (i.e., recoloring them) without changing order; (4) Rerank search results after improving images’ accessibilities; (5) Improve the accessibility of an individual image. Users can accomplish it with simple operation, such as moving the mouse on the image and then click “improve accessibility”. These choices can be customized and selected according to users’ preference1 .
3.5
Evaluation Measure for AIS
Obviously the existing performance evaluation measures for information retrieval, such as Average Precision (AP) and NDCG [19], all focus only on relevance. So, we have to provide a complementary performance evaluation measure for AIS to estimate its effectiveness in searching accessible images. Here we modify the existing AP evaluation to obtain a new measure named Accessibility Average Precision (AAP), but it is worth noting that we can also extend other measures to AIS, such as NDCG [19] which takes into account the different levels of relevance in comparison of AP. The AAP measurement at M (AAP @M , i.e., the AAP measure of the top M results) is defined as
AAP @M =
M 1 yi M Gmax i=1
i r=1
i
gr
(14)
where yi is the binary relevance label of i-th sample (i.e., yi = 1 if the sample is relevant and otherwise yi = 0), gr is the groundtruth of the r-th sample’s accessibility score, and Gmax is the maximum value of the accessibility score which is used to normalize the AAP measurement into [0, 1]. Based on the AAP measure, we can easily obtain a Mean Accessibility Average Precision (MAAP) measurement by averaging the AAP measurements of multiple concepts as an overall evaluation. It is the coordinate of the existing Mean Average Precision (MAP) measurement. Thereby, the performance of AIS can be evaluated with two measures, e.g., MAP and MAAP.
4.
EXPERIMENTS
We conduct our experiments using 65,443 images collected from a popular commercial image search engine. We first select 68 queries and then collect the top images for each query. The queries are:Party, Cat, Panda, Earth, Dogs, Snakes, Cartoon, Backgrounds, Ronaldinho, Horses, Women, Dragons, Spider, Car, Fish, Boy, Ghosts, Live, Youtube, Birds, Animals, Flowers, Angel, Turtles, People, Heart, Frogs, Chocolates, Cake, Starts, Baby, Beach, Wolves, Weather, Batman, Email, Hairstyles, Trees, Lion, Children, Hawaii, Food, Tiger, Waterparks, Indians, School, Sports, Military, Bees, Medical, Plants, Pigs, Cow, Disney, Flags, Rose, Baseball, Football, Games, Police, Fruit, , Nodia, War, Jesus, Golf, Maps, Cowboys, and Hotels. It is worth noting that, instead of selecting the queries specifically to fit our algorithms, we 1 Of course simply providing many customizable options may not be a good choice for users. The design of the options and a friendly user interface is crucial for real-world application. But in our current stage we will only focus on the performance of reranking and recoloring, whereas more specific studies on the customizations of these services are left for our future work.
AnalysisͲBasedMethod
LearningͲBasedMethod
0.8
CorrelationCoefficient
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
QueryID
Figure 6: Pearson correlation coefficients between our accessibility measurements and the subjective scores.
Images Accessibility Scores
0.990
0.974 0.957
0.92
0.732 0.694
0.661
0.554
0.399
0.282
0.282
0.250
Figure 7: The evaluated accessibility scores of the images illustrated in Fig. 3. We can see that the scores are well consistent with subjective perception. have chosen a set of diverse and representative queries as a picture of real-world image search. For simplicity, we use 1 ∼ 68 to denote the IDs of these queries, respectively. There are 20 anonymous protanopic/deuteranopic users in total 3 (18 male and 2 female) participating in the labeling and evaluation tasks. These participants come from the two largest cities of China, and they are from different backgrounds, including students, teachers, editors, etc.
4.1 Evaluation of Accessibility Assessment First, we evaluate the consistency of our accessibility measurement with subjective test. Three colorblind viewers are involved in the subjective labeling. Every image is manually given a subjective accessibility score among 0, 0.5 and 1 by each volunteer. These three scores indicate low, medium and high accessibility, respectively. Several typical examples with different accessibility scores have been shown in Fig. 3. We compare the two different accessibility assessment methods proposed in Section 3.2. For analysis-based accessibility assessment, the parameter Q is empirically set to 4096. For learning-based accessibility assessment, we regard the images collected from the queries 1 to 20 as training data. The images of the other 48 queries are used for testing. Since the training and testing images are collected from different queries, the over-fitting effect can be avoided. The parameter ε in SVR model is empirically set to 0.01, and the tradeoff parameter C and the radius parameter σ are tuned by 5-fold cross-validation. We average the scores provided by the three volunteers and then compute the Pearson correla3 Actually the perceptions of protanopic and deuternaopic viewers are very close [5][12]. They come from the two largest cities of China. Most of these viewers are not aware of which type they belong to, and they only know they are red-green colorblind. So we will not distinguish these two types of colorblindness in our study.
tion of the averaged subjective accessibility scores and our accessibility measurements. Figure 6 illustrates the results. From the results we can clearly see that, although the analysis-based method is intuitive, the learning-based method performs much better. This can be attributed to the fact that the SVR model built on block-wise color moment features can implicitly capture the high-order relationships between accessibility scores and the statistical and spatial distribution of colors via kernel mapping. For example, it may be learned from training data that the central blocks should be more important than boundary blocks. The mean correlation coefficients obtained by the learning-based and analysis-based methods are 0.471 and 0.169, respectively. Considering there are only three scales for the manually labeled scores, 0.471 is a good result and we can conclude that the learning-based accessibility assessment method can achieve rather consistent results with the perception of colorblind viewers (Figure 7 illustrates the measured accessibility scores of the images in Fig. 3, and we can see they are reasonable). For a typical image with size 320 × 240, the learning-based and analysis-based accessibility assessment methods cost about 1 and 2 second, respectively (Pentium4 3.0G CPU and 2G memory). Therefore, the learning-based method is superior in terms of both performance and computational efficiency.
4.2
Evaluation of Reranking Strategy for Accessible Search
To evaluate the accessibility-based reranking approach, we adopt the AAP measure introduced in Section 2.2. In addition, we also illustrate the AP measurements before and after reranking to investigate the impact of the reranking on search relevance. Based on our conclusion in the last section that the learning-based accessibility assessment is superior to analysis-based method, we adopt the accessibility scores
AAP@300BeforeRerank
AAP@300AfterRerank
AP@300BeforeRerank
AP@300AfterRerank
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2
QueryID
Figure 9: The comparison of AAP@300 and AP@300 before and after reranking. Tree
the original ranking scores and the accessibility scores to compromise relevance and accessibility if the relevance after reranking is a concern. The existing relevance improvement methods can also be further integrated [27].
4.3 Flowers
Before reranking
After reranking
Figure 8: The top results for query Tree and Flowers before and after reranking perceived by protanopic viewers. We can see that the top images after reranking have better accessibilities.
obtained by the first approach. We rerank the top 300 images for each query with the accessibility scores in descending order, and then we compute the AAP measures before and after reranking. Figure 8 illustrates the top images for the queries tree and flowers before and after reranking. Here we have mimicked the perception of protanopic viewers using the simulation algorithm in [5]. The detailed results for different queries are illustrated in Fig. 9, from which we can clearly see the improvement of AAP after reranking. The MAAP increases from 0.59 to 0.65 after reranking. This indicates that the AIS scheme can successfully identify and prioritize the images with better accessibilities. The figure also illustrates the AP@300 measures before and after reranking. We can see that the reranking has only slightly degraded the search performance in terms of relevance. The MAP measure degrades from 0.74 to 0.72 after reranking. In fact, as previously mentioned, we can also choose to combine
Evaluation of Accessibility Improvement Algorithm
We now evaluate the accessibility improvement approach introduced in Section 3.3. To reduce the workload of colorblind labelers, we only randomly select 90 images from each of the 68 queries, and obtain a set of 6120 images in this way. We empirically set the parameter candidate set S (see Eq. (11)) to {−π/3, −π/6, 0, π/6, π/3} to achieve a compromise between recoloring performance and computational cost. Figure 10 illustrates several examples, including original images, their protanopic views and the recolored results (to better demonstrate the effectiveness of the recoloring approach, we have further added two Isharaha plates that are out of the web image dataset). We compare our recoloring algorithm with two existing methods: (1) Optimization-Based Color Rotation (OBCR) [7]; (2) Generalized Histogram Equalization (GHE) [30]. We choose these two methods because the existing studies have shown their superiority over many other re-coloring methods. Each recolored result is given a subjective accessibility score using the method introduced in Section 4.1 with three colorblind labelers. In the labeling process, the results produced by all algorithms are shuffled and blended to generate a fair comparison. Figure 11 illustrates the average accessibility scores of the original images and the recolored results using the three algorithms. From the results we can see that the three recoloring methods all achieve better accessibility scores than the original images. This demonstrates the effectiveness of the recoloring approach. Among the three recoloring methods, the proposed algorithm performs the best. Our method also has advantage in computational cost. In addition, our method only needs 1.5 seconds to process one image and this is faster than the other two algorithms. Therefore, the superiority of the proposed method is evident considering both performance and computational efficiency.
0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Baseline
OBCR
GHE
Ourmethod
Figure 11: The average accessibility scores of the images after implementing different recoloring methods. 3 2.5 2 1.5 1 0.5
(a)
(b)
(c) 0
Figure 10: (a) Original Images. (b) Perceived images by protanopic viewers. (c) Perceived images by protanopic viewers after accessibility improvement. We can see that the recoloring approach can increase the contrast of images and help colorblind viewers better distinguish several details in the images. The two Isharaha plates, which originally cannot be read by the colorblind viewers, can also be recognized after recoloring.
4.4 User Study Finally, we conduct user study with all the 20 colorblind viewers to evaluate the usability of AIS. In the accessible search scheme, we recolor all the images and then rerank the top 300 images for each query based on the evaluated accessibility scores. First, we regard the original search and the accessible search as two schemes, and then ask the users to freely choose queries and observe the results. The users are asked to rank the two schemes using “>”, “>>” and “=”, which mean “better”, “much better” and “comparable” respectively. To quantitate the results, we convert the ranking information into ratings. We assign score 1 to the worse scheme. The other scheme is assigned score 2, 3 and 1 if it is better, much better and comparable than this one, respectively. The average rating scores and the variances are illustrated in Fig. 12. We can clearly see the preference of users towards the accessible search scheme. We also perform an analysis of variance (ANOVA) test [6]. The F-statistic of scheme factor is 64.08 and it can be derived that p < 0.000001. This indicates that the difference of the two schemes is significant. The F-statistic of user factor is 1.0 and it can be derived that p > 0.5, and this indicates that the difference among users is statistically insignificant. Then we conduct a more detailed study. For each query, the users are provided with the original search results and the accessible search results. The two ranking lists are shuf-
AIS
ORI
Figure 12: The mean ratings of Accessible Image Search (AIS) and Original (ORI) schemes.
fled and blended, and each user then selects the preferred list from the pair. The percentages of different preferences are illustrated in Table 1. From the table we can see that most users prefer the accessible search results. A small part of users have also chosen the original results. This can be attributed to two facts: one is that for several queries the top results in the original ranking list already have high accessibilities (we can see in Fig. 9 that the AAP measures of several queries can hardly be improved after reranking), and the other is that the relevance of several queries slightly degrade after reranking and some users feel uncomfortable for this. However, considering the diversity of the queries and the anonymous users, these results are already sufficient to demonstrate the usefulness of the AIS scheme. The results will also be better if we provide the reranking and recoloring as the additional options that can be chosen by the users.
5.
CONCLUSION
This paper describes an AIS scheme that serves colorblind users. Different from the traditional image search that aims to return more relevant images to users, the accessible images search takes into account the accessibilities of images. The scheme includes an image accessibility assessment component and an accessibility improvement component. Based on the two components, we can provide a series of services for colorblind users, such as prioritizing the images with high accessibilities in the search results and recoloring images to improve their accessibilities. Experiments and user study have demonstrated the effectiveness of the scheme. Relevance and accessibility are both important for colorblind users in image search. Since the relevance issue has already been studied for decades, in this work we have focused
Table 1: User study of the preference of accessible image search Query Animals Flowers Angel Turtles People Heart Frogs Chocolates Cake Stars Baby Beach Wolves Weather Batman Email Hairstyles Trees Lion Children Hawaii Food Tiger Waterparks Indians School Sports Military Bees Medical Plants Pigs Cow Disney Flags Rose Baseball Football Games Police Fruit Nokia War Jesus Golf Maps Cowboys Hotels Average
Prefer accessible search results 60% 85% 65% 75% 50% 65% 65% 65% 70% 70% 50% 40% 50% 65% 50% 65% 45% 80% 65% 60% 65% 40% 55% 70% 55% 60% 75% 80% 80% 70% 65% 55% 55% 35% 80% 55% 65% 85% 60% 50% 20% 45% 70% 60% 45% 80% 65% 80% 61.04%
Neutral 25% 5% 20% 15% 25% 10% 20% 15% 15% 25% 45% 15% 45% 15% 35% 30% 30% 15% 25% 20% 20% 20% 20% 20% 25% 25% 25% 10% 20% 30% 20% 30% 15% 50% 15% 40% 25% 5% 25% 35% 45% 40% 30% 20% 45% 15% 25% 10% 24.27%
Prefer original results 15% 10% 15% 5% 25% 25% 25% 20% 15% 5% 5% 45% 5% 20% 15% 15% 35% 5% 10% 20% 15% 40% 25% 10% 20% 15% 0% 10% 0% 0% 15% 15% 30% 15% 5% 5% 10% 10% 15% 10% 35% 15% 20% 20% 10% 5% 10% 0% 14.69%
on accessibility. In fact, many existing techniques for improving search performance in relevance can be extended to accessibility. For example, analogous to the relevance feedback which is an intensively studied topic in image search, we can develop an accessibility feedback technique that can improve the accessible search performance based on several feedback results from users. We will further study these issues in our future work.
6. ACKNOWLEDGMENTS We would like to thank Dr. Ke Colin Zheng for his instructive suggestions.
7.
REFERENCES
[1] Web Accessibility Initiative. http://www.w3.org/WAI/. [2] Vischeck. http://www.vischeck.com. [3] Accessibility research: aDesigner. http://www.research.ibm.com/trl/projects/acc tech/adesigner.htm [4] Google accessible search. http://labs.google.com/accessible/ [5] H. Brettel, F. Vienot, and J. Mollon, “Computerized simulation of color appearance for dichromats”, Journal of the Optical Society of America, vol. 14, no. 10, 1997 [6] R. A. Fisher, “Statistical methods for research workers”, Macmillan Pub Co, 1970 [7] J. -B. Huang, Y. -C. Tseng, S. -I Wu, S. -J. Wang, “Information preserving color transformation for protanopia and deuteranopia”, IEEE signal processing letter, vol. 14, no. 10, 2007 [8] L. Jefferson and R. Harvey, “An interface to support color blind computer users”, in Proceedings of ACM SIGCHI, 2007 [9] L. Jefferson and R. Harvey, “Accommodating color blind computer users”, in Proceedings of ACM Conference on Computers and Accessibility, 2006 [10] G. Iaccarino, D. Malandrino, M. D. Percio, and V. Scarano, “Efficient edge-services for colorblind users”, in Proceedings of International World Wide Web Conference, 2006 [11] V. A. Kovalev, “Towards image retrieval for eight percent of color-blind men”, in Proceedings of International Conference on Pattern Recognition, 2004 [12] K. Rasche, R. Geist, and J. Westall, “Re-coloring images for gamuts of lower dimension”, in Proceedings of Eurographics, 2005 [13] A. J. Smola and Bernhard Scholkopf, “A tutorial on support vector machine”, Statistics and computing, vol. 14, no. 3, 2004 [14] K. Wakita and K. Shimamura, “Smart color: disambiguation framework for the colorblind”, in Proceedings of ACM Conference on Computers and Accessibility, 2005 [15] S. Yang and Y. M. Ro, “Visual content adaptation for color vision deficiency”, in Proceedings of International Conference on Image Processing, 2003 [16] CIE Technical Report, “Industrial colour-difference evaluation”, 1995 [17] Wandell, Foundations of Vision, Sunderland, MA:Sinauer, 1995 [18] Wikipedia. www.wikipedia.org/wiki/Accessibility [19] K. J¨ arvelin and J. Kek¨ al¨ ainen, “Cumulated gain-based evaluation of IR techniques”, ACM transactions on Information Systems, vol. 20, no. 4, 2002 [20] Z. Wang and A. C. Bovik, “Modern image quality assessment”, Morgan & Claypool Publishers, 2006 [21] Z. Lu, W. Lin, X. Yang, E. P. Ong, and S. Yao, ”Modeling visual attention’s modulatory aftereffects on visual sensitivity and quality evaluation”, IEEE transactions on Image Processing, vol. 14, no. 11, 2005 [22] M. S. Lew, N. Sebe, C. Djeraba, and R. Jain, “Content-based multimedia information retrieval: state of the art and challenges”, ACM transactions on Multimedia Computing, Communications and Applications, 2006 [23] M. Song, D. Tao, C. Chen, X. Li, and C. -W. Chen, “Colour to Grey: Visual Cue Preservation”, IEEE Transactions on Pattern Analysis and Machine Intelligence, to appear [24] R. M. Evans, An introduction to color, Wiley, New York, 1948 [25] IEEE CVPR 2005 Workshop on Computer Vision Applications for the Visually Impaired, http://www.soe.ucsc.edu/ manduchi/CVAVI/ [26] ECCV 2008 Workshop on Computer Vision Applications for the Visually Impaired, http://www.ski.org/Rehab/Coughlan lab/General/CVAVI08.html [27] Y. Jing and S. Baluja, “PageRank for product image search”, in Proceedings of International World Wide Web Conference, 2008 [28] L. Kennedy and M. Naaman, “Generating diverse and representative image search results for landmarks”, in Proceedings of International World Wide Web Conference, 2008 [29] M. Hua, J. Pei, A. Fu, X. Lin, H.-F. Leung, “Efficiently answering top-k typicality queries on large databases,” in Proceedings of VLDB, 2007 [30] J. -B. Huang, S. -Y. Wu, and C. -S. Chen, ”Enhancing color representation for the color vision impaired”, in Proceedings of ECCV Workshop on Computer Vision Applications for the Visually Impaired, 2008 [31] V. Vapnik. The nature of statistical learning theory, Springer, New York, 1995
