Jian Guan and Guoping Qiu. School of Computer Science, The University of Nottingham. Jubilee Campus, Nottingham, NG8 1BB, UK. {jwg, qiu}@cs.nott.ac.uk.
Interactive Image Segmentation using Optimization with Statistical Priors Jian Guan and Guoping Qiu School of Computer Science, The University of Nottingham Jubilee Campus, Nottingham, NG8 1BB, UK {jwg, qiu}@cs.nott.ac.uk
Abstract. Interactive image segmentation is important and has widespread applications in computer vision, computer graphics and medical imaging. A recent work has shown that interactive figure ground segmentation can be achieved by computing a transparency image using an optimization framework, where user interactions are used to supply constraints for solving a quadratic cost function with a unique global minimum, which can be efficiently obtained using standard methods. In this paper, we introduce statistical priors as constraints to solve the optimization problem. We show that for some images, the statistical priors can provide good enough constraints to automatically obtain satisfactory figure ground segmentation results. For more difficult cases, we use the segmentation result of the statistical priors as a starting point for interactive figure ground segmentation. We show that segmentation results obtained based on statistical priors can be effectively employed to guide user interaction thus helping to reduce users labor in the interaction process. We also present a new effective adaptive thresholding method for making binary (hard) segmentation based on the computed continuous transparency image. Another contribution of this paper is the extension of the optimization based interactive figure ground segmentation framework to interactive multi-class segmentation, where user can provide multi-class seed pixels instead of just foreground background 2-class seeds, for segmenting the given image into the desired number of regions by performing a one-shot optimization operation, which again has a unique global minimum and can be obtained by solving a large system of linear equations. We present various experimental results, including segmentation error rates on an online image database with human labeled ground truth, to show that our method works well and has direct applications in areas such as interactive image editing.
1 Introduction Foreground background segmentation has wide applications in computer vision (e.g. scene analysis), computer graphics (e.g. image editing) and medical imaging (e.g. organ segmentation). Fully automatic image segmentation has many intrinsic difficulties and is still a very hard problem. In many applications, such as image editing in computer graphics and organ segmentation in medical imaging, semi-automatic and interactive approaches, where human operators provide strong priors for the computational algorithms to perform segmentation, can not only overcome the inherent tech-
nical difficulties of fully automatic image segmentation, but may also be desirable because the operators in many of these applications may want to be able to control the segmentation process and results. There have been increasing activities in the research community to develop interactive semi-automatic image segmentation techniques [2, 3, 9, 11, 12]. In [2], the authors presented an interactive image segmentation technique based on graph cut. Users labeled seed pixels which indicating definite background and foreground were used as strong priors for segmenting images into figure and ground. In [3], the authors showed that graph cut based segmentation algorithms could be implemented very fast. In [12], the authors presented a segmentation given partial grouping constraints method. User inputs were used as bias to a natural grouping process, and the authors formulated such biased grouping problem as a constrained optimization problem that propagates sparse partial grouping information to the unlabelled data by enforcing grouping smoothness and fairness on the labeled data points. They used the normalized cut criterion and solved the optimization problem by eigendecomposition. In [9], the authors presented an interactive image foreground extraction method that was computationally based on graph cut of [1] but the authors introduced a simpler user interaction technique to reduce user efforts in the interaction process and an iterative model updating procedure to improve accuracy. In [11], an interactive foreground background segmentation method was introduced in the context of image matting. The authors used Belief Propagation to iteratively propagate user labeled pixels to the unlabeled pixels. A recent work [13] has developed an optimization based figure ground segmentation technique, where a transparency image was computed by optimizing a quadratic cost function with user supplied linear constraints. The optimization problem has a unique global minimum and can be solved efficiently by standard numerical methods. In this paper, we introduce statistical priors as constraints to solve the optimization problem. For some images, the statistical priors can provide good enough constraints to automatically obtain satisfactory figure ground segmentation results. For more difficult cases, user interaction is necessary. In such cases, we use the segmentation result based on the statistical priors as a starting point for interactive figure ground segmentation, and as a guide to help users to place the constraints in the correct locations to generate the desired results. In this way, the statistical priors not only guide the user but also help reducing users’ labors in the interaction process. We have also developed a new method to make binary (hard) segmentation based on the computed continuous transparency image. Another contribution of this paper is the extension of the optimization based interactive figure ground segmentation framework to interactive multi-class segmentation, where user can provide multi-class seed pixels instead of just foreground background 2-class seeds, for segmenting the given image into the desired number of regions by performing a one shot optimization operation, which again has a unique global minimum and can be obtained by solving a large system of linear equations. The organization of the paper is as follows. In Section 2, we present the framework of optimization based figure ground segmentation and describe solutions based on statistical priors and user interaction. In section 3, we present experimental results and demonstrate the possible applications of our method. In section 4, we extend the op-
timization based figure ground segmentation framework to interactive multi-class image segmentation and present preliminary results.
2 Interactive Figure Ground Segmentation using Optimization For a given color image I(z), where z∈(x, y) is the co-ordinate vector, there is a corresponding (hidden) transparency image α(z), where 0≤α(z)≤1 is called the alpha matte in computer graphics [4]. We consider each pixel as being generated as an additive combination of a proportion α(z) of foreground color with a proportion 1-α(z) of background color. For a definite background pixel, we have α(z) = 0 and for a definite foreground pixel, we have α(z) = 1. For pixels that are between foreground and background, we have 0
