Describe full reference image quality by image similarity. Quantize the quality of a reproduction relative to a reference. Scheller Lichtenauer et al. (EMPA, TUD ...
Learning Image Similarity Measures from Choice Data Matthias Scheller Lichtenauer, Peter Zolliker, Ingmar Lissner, Jens Preiss, Philipp Urban
May 7, 2012 at CGIV in Amsterdam, Netherlands
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
1 / 18
Describe full reference image quality by image similarity
Quantize the quality of a reproduction relative to a reference
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
2 / 18
‘ From Error Visibility to Structural Similarity ’ Wang, Bovik, Sheikh & Simoncelli, 2004
Compare reference image X to reproduction Y For corresponding sliding windows x and y, SSIM(x, y ) = [l(x, y )]α1 [c(x, y )]α2 [s(x, y )]α3 Structural Similarity (SSIM) features per window: l(x, y ): lightness (absolute level) c(x, y ): contrast (variation) s(x, y ): structure (cross-correlation)
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
3 / 18
In practice, SSIM is applied to the luminance channel. . . SSIM was tested on high spatial frequency distortions like noise, blur, compression
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
4 / 18
. . . but in gamut mapping, chromatic distortions occur Chromatic distortions are often of low spatial frequency
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
5 / 18
SSIM for color images: Apply it to all color channels?
Bonnier & al. (CIC 2006) applied SSIM to each channel in IPT to evaluate spatial gamut mapping algorithms. Their conclusion: ‘ . . . we compared the results of the experiment with Image Quality Metrics and found that none presented a strong correlation with observer’s Z-scores. ’
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
6 / 18
Extend SSIM with chromatic features Considering here projections of pixel colours to (a , b ) plane in CIELAB
}| q { qz ∆C(x y ) = a + b a +b s ∆H(x y ) = (a | ({zx y)} | a ) +{z (b b )} ∆C Radial
,
,
2 x
2 y
2 x
x
y
2
2 y
x
y
Euclidean
1 χ (x, y ) = 0
h(x, y ) =
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
,
2
Radial
1 c4 ∆C(x,y )2 +1
∆C(x, y )2
1 0
2
1 c5 ∆H(x,y )2 +1
∆H(x, y )2 Learning Image Similarity
CGIV 2012
7 / 18
Extend SSIM with chromatic features How to integrate features
Wang et al.
Our approach
Combine first the features for each window, then average across the image
Average first across the image per feature, then combine features
Identify areas where images differ (difference maps)
Identify features by which images differ
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
8 / 18
Evaluation: paired comparison - match to sample Which transformed image better represents the reference?
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
9 / 18
Evaluation: no reference yet for chromatic distortions
Chroma database A new reference database for chromatic image distortions Modeled distortion up to now: mapping colors to smaller gamut A dozen studies from Z¨ urich, Gjovik, M¨ unchen, Jena Reference and transformed images in sRGB Over 50’000 paired comparisons available Expert observers as well as lay observers
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
10 / 18
Evaluate by accuracy of choice prediction (hit rate) Use cross-validation method to generate distributions
Hit rate distribution for 95 reference images
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
IPT
DIN
0.9 Hit Rate
Set 1 2 3 4 5 6 7 8 9 10
Features used RGB IPT DIN ID l,c,s l,c,s χ h l,c,s χ h x xxx xxx xxx x x xxx x xxx x xxx xxx x x xxx x xxx x
0.8 0.7 0.6
1 Feature Sets
2
3
4
; compare l,c,s Learning Image Similarity
5
6
7
8
9
10
to l,c,s,χ,h
CGIV 2012
11 / 18
Significance analysis Mean hit rates are significantly better with χ and h on gamut mapping data
Differences of mean hit rates Conjoint 2 Conjoint 1
l,c,s compared to l,c,s,χ,h
Individual Image Gamut
Considering only linear models
Local Contrast
1000 cross validation runs with 10% hold out data
Basic Mixing 6 Mixing 5
>
Differences 0 mean that hit rates are better with χ and h
Mixing 4 Mixing 3 Mixing 2
Error bars: 99% interval
Mixing 1 −2
−1
0 IPT
1
2 DIN
3
4 LAB
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
5
6
LHL
Learning Image Similarity
CGIV 2012
12 / 18
Resum´ e
Conclusion Chromatic information is not negligible in full reference image quality Open questions Performance of SSIM+χ + h with other distortions? Correlation lightness to chroma in gamut mapping? Memory colours?
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
13 / 18
Memory colours: use abstract images
Picture by Zentrum Paul Klee, Bern und Abteilung f¨ ur Medientechnologien, Universit¨at Basel Creative Commons for research in gamut mapping
Paul Klee, polyphon gefasstes Weiss, 1930, 140 Feder und Aquarell auf Papier auf Karton 33,3 x 24,5 cm
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
14 / 18
Thank you for your attention!
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
15 / 18
A closer look at SSIM features Consider the role of the constant c1 near zero in L , a and b respectively
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
l(x, y ) =
2µx µy + c1 µ2x + µ2y + c1
c(x, y ) =
2σx σy + c2 σx2 + σy2 + c2
s(x, y ) =
σxy + c3 σx σy + c3
Learning Image Similarity
CGIV 2012
16 / 18
Evaluation: Why using hit rate?
Likert’s method Normalization models needed for variance in answers Paired comparison Argumentation on raw data, no normalization needed
; hit rate allows use of classification models
Cross validation or resampling yields hit rate distributions Compare hit rate distributions to compare models
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
17 / 18
Corrigendum
One reference in the paper as published was not correctly resolved. Wherever [?] appears, it should refer to reference [23].
Scheller Lichtenauer et al. (EMPA, TUD, FSU)
Learning Image Similarity
CGIV 2012
18 / 18
