ABSTRACT. We propose a new content-aware image resizing scheme, Stream. Carving, which is based on the well-known seam carving method. Our algorithm ...
Proceedings of 2010 IEEE 17th International Conference on Image Processing
September 26-29, 2010, Hong Kong
STREAM CARVING: AN ADAPTIVE SEAM CARVING ALGORITHM Daniel Domingues, Alexandre Alahi and Pierre Vandergheynst Ecole Polytechnique F´ed´erale de Lausanne (EPFL) ABSTRACT We propose a new content-aware image resizing scheme, Stream Carving, which is based on the well-known seam carving method. Our algorithm may introduce larger seams in the retargeted image, i.e. seams with a width larger than one pixel, that we call “streams”. The resulting holes are then recovered using an inpainting method. Our retargeting algorithm is also more related to human perception by exploiting an adaptive importance map that merges several features like gradient magnitude, saliency, face, edge and straight line detection. Our approach induces an increase in the quality of the retargeted image when compared to the original seam carving method and provides similar or better results than other actual image retargeting techniques. Index Terms— Image retargeting, seam carving, inpainting 1. INTRODUCTION In the last decade, the amount of images and videos available from the internet had a significant increase. Moreover, the continuous development of new display devices (e.g. mobile phones, notebooks) induces a constantly growing consumption of media content. The devices may have different resolutions and aspect ratios and a resizing of the image might be needed to enable its display on a particular device. The use of primitive image resizing methods, such as letterboxing, scaling or cropping, causes the user dissatisfaction as the first solution does not allow the use of the full screen potential of the device and the others may change the image content. This situation induced the first works on image retargeting. This concept was introduced a few years ago, its role is to better resize media content in order to allow the users to exploit the full potential of their equipment, without noticing any decrease in the quality of the displayed content. In this paper, we present Stream Carving, a new content-aware image resizing procedure based on seam carving [1, 2], which is the most popular image retargeting method. The first objective of our approach is to improve the results obtained by the seam carving algorithm. This goal is reached with the insertion of “streams”, i.e. seams larger than one pixel, in the retargeted image and the use of a new importance map that is more related to human attention. This paper will be composed as follows: in Section 2, the related work on content-aware image and video resizing will be presented, the features of our algorithm will be explained in Section 3 and the results of our retargeting method will be compared to other algorithms in Section 4, before concluding remarks in Section 5. 2. RELATED WORK An important work on image retargeting is the one from Avidan and Shamir [1] who introduced the concept of Seam Carving. A seam is an 8-connected path of pixels with one pixel per row (respectively column) in the case of a vertical (respectively horizontal)
978-1-4244-7994-8/10/$26.00 ©2010 IEEE
901
(a) (b) (c) (d) Fig. 1. Comparison of 3 different retargeting methods: (b) Scaling, (c) Improved Seam Carving [2], (d) Our proposed retargeting algorithm (Stream Carving). Original images (a) respectively from [2], [1] and http://www.telegraph.co.uk.
seam. Based on an importance map, the least energy seam is found in the desired direction in order to remove or duplicate it in the image. Any energy function may be used with the seam carving method, but gradient magnitude is usually associated to it. Rubinstein et al. [2, 3] proposed some improvements to the original method, e.g. forward energy was used instead of backward energy to compute the cost matrix M . Forward energy adds to the latter, the values CL . CU and CR , which represent the computation of the gradient in a grayscale version of the image (I) after the seam insertion: CL (i, j) = |I(i, j + 1) − I(i, j − 1)| + CU (i, j) CU (i, j) = |I(i, j − 1) − I(i − 1, j)| CR (i, j) = |I(i, j − 1) − I(i + 1, j)| + CU (i, j)
(1)
The cost matrix using forward energy becomes: ⎧ ⎨ M (i − 1, j − 1) + CL (i, j) M (i, j − 1) + CU (i, j) M (i, j) = ESC (i, j) + min ⎩ M (i + 1, j − 1) + C (i, j) R (2) where ESC is the energy used for the importance map. In [3], scaling and cropping may be applied jointly with seam carving in order to achieve the best retargeted image. Hwang et al. [4] proposed a change to the conventional seam carving energy map: instead of using gradient magnitude alone, they combine it with the output of a saliency detection algorithm and a face detector. Their energy map is therefore more related to human perception; however, the saliency map that they use is not the actual
ICIP 2010
Fig. 2. Composition of our Importance Map.
(a)
(b)
(c)
(d)
(a) (b) Fig. 3. Insertion of 20 vertical pixels in one image, using: (a) 20 seams, (b) one “stream”. Original image from [2]. state-of-the-art: the approach proposed by Achanta et al. [5] outperforms other saliency detection methods. The latter [6] presented the results of seam carving using their saliency map instead of gradient magnitude. The resulting energy function is the L2 -norm of the difference, in the Lab colorspace, between the average color value of an image (Iμ ) and its Gaussian filtered version, using a n × n kernel: ESaliency (x, y) = Iμ − In×n (x, y)2
(3)
The resulting importance map is more related to human perception and presents therefore, in many situations, better results than gradient magnitude. However, their saliency map [5] may also fail sometimes, leading to undesirable results, as explained in Section 4. Seam carving is not the only image retargeting approach and there exist many other methods. Another solution may be an image warping based approach, where the original image is considered as a grid mesh and the retargeting process consists in computing the new locations of the points from the grid. Two retargeting methods based on these principles are proposed in [7] and [8]. Pritch et al. [9] presented an image retargeting technique which obtains more visually pleasing results; however, their method may completely change the content of an image, e.g. people or objects may be completely removed or duplicated in the image. The method from Pritch et al. [9] does therefore not always perform in a content-aware manner. 3. OUR IMAGE RETARGETING ALGORITHM Our image retargeting algorithm relies on the Improved Seam Carving algorithm [2] with the following improvements: Stream Carving. In the case of a seam duplication, we allow to add a “stream” instead of a seam: a seam is a path of one pixel width and a stream may have several pixels width. This choice is motivated by the usual locations of the seams in an image: they are usually in easily reproducible areas and may be enlarged without causing any visual distortion. The difference between a seam and a “stream” is illustrated in Fig. 3. Our algorithm checks if the least energy seam crosses any important edge or a face. In the negative case, a “stream” may be added. It may have a width of up to 5% of the corresponding image dimension (width or height). This percentage should then decrease with the number of seams that were previously found. When
902
(e)
(f )
Fig. 4. Retargeted images using: (c-e) Improved Seam Carving [2], (d-f) Stream Carving. (b) Our resulting importance map. Original images (a) from [6] and [3]. a “stream” is inserted, the remaining hole cannot be recovered with a simple seam duplication. We use the inpainting scheme from Betalmio et al. [10]. Bi-directional search. It may be observed that when only a one-dimensional change is needed to retrieve the retargeted image, the seam carving algorithm only looks for seams in one direction. In our proposed approach, we search for the least energy seam in both width and height dimensions. We reason in terms of aspect ratios and not anymore in terms of dimensions, i.e. when the desired target ratio is larger than the input ratio, vertical seams may be added to the image and horizontal seams may be removed: targetRatio =
inputW idth + addSeams inputHeight − removeSeams
(4)
In the case of a decrease of the aspect ratio, the opposite happens. This process allows our algorithm to always look for the best seam in two directions and to possibly use “streams” more often. After having a retargeted image with the desired aspect ratio, it may then be homogeneously rescaled to retrieve the desired output dimensions. Importance Diffusion. Similarly to [11], we apply importance diffusion in the case of a seam removal, by adding a penalty term to the energy map on the neighbors of the pixels through which a seam went. Importance Map. Similarly to [4], instead of employing gradient magnitude, we use an energy map more related to human per-
ception, which is illustrated in Fig. 2. There is firstly a Decision Process, which chooses to use the saliency map from [5] or not, as the latter may lead to wrong results, i.e. it does not always detect the right image regions as salient. The Decision Process consists in a simple comparison: (a,b)B S(a, b) xH,yW S(x, y) < (5) |H| × |W | |B| where H = {0, . . . , height − 1}, W = {0, . . . , width − 1}, B = {(a, b) | (a ≤ 0.01 × |H|) ∨ (a > 0.99 × |H|) ∨ (b ≤ 0.01 × |W |) ∨ (b > 0.99 × |W |)}, and S is the saliency map. Eq. 5 checks whether the average of the saliency map is smaller than the average of the values from its borders, which represent 1% of the image width or height. In the positive case, the saliency map from [5] is used. Otherwise, it is replaced by gradient magnitude. Usually, the border pixels from an image are unimportant, as they may contain sky and background content. After the Decision Process, the outputs of a Canny edge detector (C), a Hough line detector (H) and a face detector (F ) [12] are added to its output (D). In terms of ranges, we have a continuous operator D{0, 1, . . . , 255} and three binary ones C, H, F {0, 255}. All of the four features have the same weight in the resulting importance map. However, it may be noticed that a face and a straight line will also be outlined by the Canny method and by gradient magnitude when the latter is used. There is therefore a hierarchy in the features from the importance map: the most important ones are faces and straight lines, followed by the Canny edge detection, and the gradient or the saliency map. This hierarchy is similar to human perception, as a subject will firstly notice distortions in faces and straight lines before taking care of edges and salient regions. The whole algorithm procedure is as follows: an importance map is firstly computed from the original image and our Stream Carving Algorithm is then applied. The algorithm stops when an image with the desired target ratio is obtained. The resulting image will then be homogeneously rescaled to the desired dimensions, if needed. In order to avoid the introduction of noticeable artifacts to the image, our algorithm may switch from “stream” carving to scaling. Formally, this situation may happen, when the average weight of the last seam is more than 0.75 times the average of the importance map and one of the dimensions needs be scaled by less than 15% more than the other. The first criterion was tested empirically, the factor 0.75 may be justified by the fact that a seam should normally have a smaller weight than the average of the importance map. The second criterion helps avoiding a noticeable distortion due to inhomogeneous scaling.
(a)
(b)
(a)
4. RESULTS The first retargeting methods that should be compared to Stream Carving are the ones that are most similar to it, namely Improved Seam Carving [2] and the Saliency-based Seam Carving [6]. Relatively to Improved Seam Carving [2], Figs. 1, 4, and 5 show that Stream Carving outputs more visually pleasing results. This improvement is mostly due to the use of “streams” that avoid seams to go through the important regions of the image. The reasoning in terms of ratios and our importance map also have an influence on the results. Fig. 4 shows that our Decision Process chose to use the saliency map from [5] and the important content is therefore better preserved in the retargeted image. Nevertheless, the saliency map alone may not keep all the important edges from the original image, especially when they are straight lines. It may also completely remove a large region from the image, causing noticeable artifacts.
903
(c)
(d)
(b)
(c) (d) Fig. 5. Retargeted images using: (b) Improved Seam Carving [2], (c) Saliency-based Seam Carving [6], (d) Stream Carving. Original images (a) from [2], [1], [9], [6], and http://2.bp.blogspot.com/.
Examples of both of these drawbacks, are illustrated in Fig. 5. These observations justify the use of edge and straight line detection fea-
(a) (b) (c) (d) (e) (f ) Fig. 6. Comparison between 5 different image retargeting algorithms: (b) Improved Seam Carving [2], (c) Non-homogeneous Videoretargeting [7], (d) Optimized Scale-and-stretch [8], (e) Saliency-based Seam Carving [6], (f) Stream Carving. Original images (a) and results (b-d) from [9].
tures in our importance map. The complete removal of an image area might be due to a “wrong” saliency detection and our Decision Process tries to avoid such situations. Based on the results from Figs. 1, 4, and 5, it is clear that Stream Carving represents an interesting alternative to Improved Seam Carving [2] and Saliency-based Seam Carving [6]. An objective quantitative metric does not exist to compare retargeted images. We therefore present qualitative results with respect to state-of-the-art methods. It may be noticed that the retargeted images obtained by Stream Carving usually better preserve the aspect ratio of the important objects and areas from the original images. Fig. 6 shows a comparison between many retargeting algorithms.
[4]
[5]
[6]
5. CONCLUSION [7] Stream Carving has been presented in this paper. The use of “streams”, the bi-directional search, and the adaptive importance map leads to visually more pleasant results when compared to Improved Seam Carving [2] and Saliency-based Seam Carving [6]. Our algorithm constitutes therefore an interesting choice for contentaware image resizing. As it was the case for many retargeting techniques, an evolution of our method could be to take the temporal dimension into account and therefore become able to retarget video content as well.
[8]
[9] [10]
6. REFERENCES [1] S. Avidan and A. Shamir, “Seam Carving for Content-Aware Image Resizing,” ACM Transactions on Graphics, vol. 26, no. 3, pp. 10, 2007.
[11]
[2] M. Rubinstein, A. Shamir, and S. Avidan, “Improved Seam Carving for Video Retargeting,” ACM Transactions on Graphics-TOG, vol. 27, no. 3, pp. 16–16, 2008.
[12]
[3] M. Rubinstein, A. Shamir, and S. Avidan, “Multi-operator Me-
904
dia Retargeting,” ACM Transactions on Graphics (TOG), vol. 28, no. 3, pp. 23, 2009. D.S. Hwang and S.Y. Chien, “Content-Aware Image Resizing using Perceptual Seam Carving with Human Attention Model,” in 2008 IEEE International Conference on Multimedia and Expo, 2008, pp. 1029–1032. R. Achanta, S. Hemami, F. Estrada, and S. S¨usstrunk, “Frequency-tuned Salient Region Detection,” in IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2009. R. Achanta and S. S¨usstrunk, “Saliency Detection for Contentaware Image Resizing,” in IEEE International Conference on Image Processing (ICIP), 2009. L. Wolf, M. Guttmann, and D. Cohen-Or, “Non-homogeneous Content-driven Video-retargeting,” in IEEE International Conference on Computer Vision (ICCV), 2007. Y.S. Wang, C.L. Tai, O. Sorkine, and T.Y. Lee, “Optimized Scale-and-Stretch for Image Resizing,” in International Conference on Computer Graphics and Interactive Techniques. ACM New York, NY, USA, 2008. Y. Pritch, E. Kav-Venaki, and S. Peleg, “Shift-Map Image Editing,” in ICCV’09, Kyoto, Sep-Oct 2009. M. Bertalmio, A.L. Bertozzi, G. Sapiro, et al., “Navier-stokes, fluid dynamics, and image and video inpainting,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Citeseer, 2001, vol. 1. S. Cho, H. Choi, Y. Matsushita, and S. Lee, “Image Retargeting Using Importance Diffusion,” in IEEE International Conference on Image Processing (ICIP), 2009. P. Viola and M. J. Jones, “Robust Real-Time Face Detection,” International Journal of Computer Vision, vol. 57, no. 2, pp. 137–154, 2004.
