A DISCRETE FIREFLY ALGORITHM FOR GEOMETRIC IMAGE INPAINTING B. Fergani1, H. Bensuici2, H.S.E.Naanaa3, M. K. Kholladi4 MISC Laboratory. Constantine2 University. Algeria & University of El-Oued. Algeria 1
[email protected] , 2
[email protected],
[email protected],
[email protected]
ABSTRACT Image inpainting is the process that reconstructs the image’s structure by matching each damaged contour with its corresponding one in such a way to obtain an image visually plausible. The search space of candidate solutions increases dramatically as the number of contours increases, which makes the search of the optimal solution by a traditional deterministic method infeasible. A good choice to solve this problem is the use of a metaheuristic, which is in the present paper a discrete version of the Firefly algorithm. So, in the present paper, a Discrete FFA (DFFA) is used to find close contours using their curvatures and values abscise of curve’s end point. The advantage of using curve’s end point is to add spatial information of the location of damaged contour. Experiment results show the efficiency of the DFFA for the geometric image inpainting. Index Terms—Image inpainting, Geometric inpainting, Damaged contours, Contour matching, Discrete FF algorithm.
1. INTRODUCTION Image processing has been widely investigated for many years; it is used in order to improve digital images’ quality, to extract some useful information from it or to reduce image storage space. Image processing tasks can be mainly grouped into three categories: image enhancement and restoration, measurement extraction and image compression. Image enhancements techniques aim at improving the image’s quality such as histogram equalization. Image restoration techniques’ purpose is to reverse the degradation effect in order to bring life to damaged images, such as image denoising. Measurement extraction refers to extract some useful information from the original image such as segmentation and edge detection. Image compression involves reducing the amount of memory needed to store a digital image. Image inpainting, is a fundamental problem in image processing. Its aim is to reconstruct lost areas from their surroundings following three main steps. The first one consists of locating the damaged region, a manual location is used. The structure of the damaged image is then reconstructed, called geometric image inpainting. After that, a synthesis of texture is applied to obtain an inpainted image. In order to reconstruct the image’s structure, damaged contours, contours corrupted by the damaged area, are first detected using a contour detector and position information relative to the damaged area. Then, damaged contours couples to be matched are found. After that, found couples are joined using a curve fitting technique. On the other hand, the DFF algorithm is a discrete version of the conventional Firefly algorithm. The Firefly algorithm is among the most powerful algorithms for optimisation; it is very efficient in finding the global optima with high success rates [1, 2]. Yang [1] shows that FFA is superior to both PSO and
GA in terms of both efficiency and success rate. Saibal et al [2] show that FFA tends to be better than PSO; especially if the functions have multi-peaks. It is potentially more powerful in solving noisy non-linear optimisation problems. Because many real-life problems are discrete (or combinatorial) optimisation problems, the DFFA was proposed. The search space of contours to be matched increases dramatically as the number of contours increases. That makes the search of the optimal solution by a traditional deterministic method infeasible. A good choice to solve this problem is the use of an optimisation algorithm. In this paper, a discrete version of FF algorithm is used to solve the geometric image inpainting. Contours corrupted by the damaged area are first detected using a contour detector and position information relative to the damaged area. Then, each damaged contour is joined with its corresponding one using the proposed DFF algorithm and an interpolation technique so that the inpainted image seems reasonable to the human eye. The rest of this paper will be organized as follows. In section2, we present basic concepts related to image inpainting and the DFF algorithm. Then, the DFF algorithm used to inpaint the structure of damaged images, which is our contribution, is detailed in section 3. Our experimental results and some analysis are presented in section 4. Finally, a conclusion is drawn up.
2. BASIC CONCEPTS In this section, basic concepts related to image inpainting and the discrete Firefly algorithms are presented.
2.1. Image inpainting Image inpainting is defined as the art of rebuilding damaged regions and filling in missing data. The aim behind using its techniques can be defined as follow [3]. The reconstruction of images containing missing regions consists of filling in these parts in such a way that the inpainted image looks visually plausible (Figure 1). Basically, the corrupted image is represented by in case of grayscale images) and a mask M: Ф → {0, 1}. The inpainting algorithm fills in the target region Ω from the source region where M (p) = 0 (p is a pixel belonging to Ф). The boundary ∂Ω, representing the area between the two regions, plays a major role in deciding the intensities to be put in Ω. Image inpainting algorithms require three main steps. The first one consists of locating the damaged region such as spots and gaps, or inserts a damaged known region. The second step consists of reconstructing image’s structure, called geometric inpainting. The last step is the synthesis of texture to obtain an
inpainted image. In this paper we are only interested in geometric image inpainting.
1. 2.
3.
All fireflies are unisex so that one firefly will be attracted to other fireflies regardless of their sex. Attractiveness is proportional to their brightness, thus for any two flashing fireflies, the less bright one will move towards the brighter one. The Attractiveness is proportional to the brightness and they both decrease as their distance increases. The brightness of a firefly is affected or determined by the view of the objective function. For a maximization problem, the brightness can simply be proportional to the value of the objective function. Other forms of brightness can be defined in a similar way to the fitness function in genetic algorithms or the bacterial foraging algorithm.
Figure 1 Inpainting the hole left by removing objects [13] Image Inpainting has been widely investigated in many applications. Its applications include, among others: damaged old paintings and photographs restoration [5], by recovering small cracks [6], scratches [7, 4], red – eye [8], spots, etc. ; removing undesired objects or writings from photographs (Figure 1); filling in spatiotemporal holes (left by the removal of dynamic objects from videos or movies) [9]; removing visual redundancy inherent in order to compress images [10]. Another interesting application is recovering blocks of missing data caused by wireless image transmission [11] or by image zooming and super resolution [12]. Existing methods of image inpainting can mainly be classified into two categories [3, 14]: Geometric methods and Exemplarbased texture synthesis methods. The geometric method, also known as diffusion based method, tries to fill in the missed region by diffusing the image information from the source region into the target one at the pixel level. This method was first introduced by Bertalmio et al [15], where the image inpainting algorithm is founded on the theory of Partial Differential Equation (DPE) and variation methods. The main purpose behind the first proposed geometric method is to propagate the Laplacian of the image in the Isophotes directions (lines of constant intensity) [15]. Geometric methods have achieved excellent results in filling in the non-textured or relatively small missing regions. However, they tend to introduce smooth effects in the textured region or larger missing regions [16]. The second category of image inpainting methods was first introduced by Criminissi et al [17]. Exemplar-based texture synthesis methods preserve linear structure by reordering the synthesizing process according to the priority of pixels. This method uses patch-based structure synthesis to replicate the texture in the image to fill in the gap(s); it searches the most similar texture using the Sum of Squared Differences (SSD). The Exemplar- based texture synthesis method is capable of propagating both linear and two-dimensional textures into the missing region. The present research paper focuses on the first category of image inpainting techniques which is the geometric method. Recent geometric methods try to formulate the image inpainting as an optimisation problem [18, 19] which is the case of the present work. The DFF algorithm is used here to match each damaged contour with its corresponding one in such a way to obtain an inpainted image visually plausible.
The FFA assumes that populations of n candidate solutions for an optimisation problem are agents of type firefly. These agents are vectors of dimension d representing the problem variables. Each solution x= (x1, x2… x d) T is evaluated by a fitness function , that represents its quality. Each agent glows proportionally to its quality which, together with his attractiveness (β), dictates how strong it attracts other members of the swarm. Two other user defined parameters are the maximum attractiveness value (β0) and the absorption coefficient (γ) that determines the variation of the attractiveness with increasing distance from communicated firefly. The Firefly algorithm is among the most powerful algorithms for optimisation but it was designed to optimize continuous parameters. Because many real-life problems are discrete (or combinatorial) optimisation problems, the discrete version of the Firefly algorithm was proposed. Discrete firefly algorithm has been applied for solving the travelling salesman problem [21, 22]. Another interesting application of DFFA is to solve the manufacturing cell formation problem [23].
3.
THE PROPOSED DISCRETE FFA FOR GEOMETRIC IMAGE INPAINTING
As stated above, this work’s aim is to inpaint damaged images’ structure. This is done by finding first close contours, which is an optimisation problem, using a discrete version of firefly algorithm. Couples of matched contours are then joined using an interpolation technique.
3.1. Problem formulation The problem of geometric image inpainting can be formulated as follow. Inpainting the structure of the damaged image is equal to reconstruct the structure of the lost area(s), which is done by matching N corrupted contours in such a way to obtain an inpainted image that looks natural for human eyes. Figure 2 presents a damaged image with one lost area; corrupting four contours into eight (N=8) damaged contours numbered from 1 to 8. The purpose of geometric image inpainting is to match each contour with its corresponding one, which means in the example of Figure 2 to join contours as presented in Figure 3.
2.2. Discrete Firefly algorithm (FFA) The firefly algorithm (FFA) was proposed by Yang at Cambridge University in 2007. The FFA uses three main basic rules [20]: Figure 2 Damaged image
Contour number
8
6
Matched contours
1
2
The advantage of ends’ abscises value is to add spatial information of the location of damaged contours that allows us to match close contours with close curvatures’ values together.
5
2
1
3
4
Matched contours
4
5
Matched contours
7
Matched contours
3
10
Figure 3 Matched contours
6
7
8
9
Figure 5 Damaged image with ten contours
3.2. Representation of Solutions
9
10
1.2143
1.7027
79
89
70
53
8 1.1786
1.0714
80
7
41
1.1429
71
6 1.6429
1.1429
5
90
4
2.1842
3
53
7
2 1.1071
4 1
Curvature’s value Ends’ abscise
1
44
6
8
Contour number
1.464
A solution for the contours-matching problem (matching corrupted contours) is represented as an array of integer permutation of numbers from 1 to N, where N is the total number of contours, as illustrated in Figure 4. In this representation, a firefly represents one solution and an element of the array represents a contour number.
Figure 6 Ends points Xc
Figure 7 Curvature and Ends’ abscise of each contour
2 3
5
Matched contours Using curvature’s values (1-6) or (6,10) (2-7) (3-4) or (3-8) or (8,9) (4-3) or (4,8) (5-10) or (6,10)
Desired contour matching
2
7
8
6
5
3
1
(1,6) (2,7)
4
(3,8)
Figure 4 Solution representation
(4,9) (5,10)
Matched contours using curvature values and Curvatures’ ends abscises
(1, 6) (2, 7)
(1,6) (2, 7)
(3, 4)
(3,8)
( 4,3 ) (5 , 10)
(4,9) (5,10)
Figure 8 Couples of matched contours
3.3. Objective Function Contours matching problem is basic to inpaint the image’s structure; it is formulated in this work as an optimisation problem. The challenge is to formulate an objective function that is capable of finding couples of contours to be matched satisfying the connectivity principle of human perception (close contours). Figure 5 presents a damaged image containing ten contours numbered from 1 to 10. In order to inpaint the image’s structure, contours must be matched as illustrated in the first column of Figure 8. If the objective function minimizes only the distance between matchedcontours’ curvatures presented in Figure 10 (using curvatures’ values presented in Figure 7) the obtained couples of matched contours are presented in the second column of Figure 8, which is not the desired result. The fourth column of Figure 8 presents couples of contours found using an objective function minimizing both the distance between matched- contours’ curvatures and the difference between ends’ abscises of matched contours. End abscises are abscises of points relating damaged contours and damaged area (Figure 6). Figure 9 presents differences between values of ends’ abscise of matched contours.
Matched Contours using curvature values
1
2
3
4
5
6
7
8
9
10
1 2 3
9 27
9 18
27 18 -
36 27 9
46 37 19
3 12 30
9 0 18
26 17 1
35 26 8
45 36 18
Matched contours Using Curvatures’ ends abscises 1-6 2-7 3-8
4
36
27
9
-
10
39
27
10
1
9
4-9
5
46
37
19
10
-
49
37
20
11
1
5-10
6
3
12
30
39
49
-
12
29
38
48
6-1
7
9
0
18
27
37
12
-
17
26
36
7-2
8
26
17
1
10
20
29
17
-
9
19
8-3
9
35
26
8
1
11
38
26
9
-
10
9-4
10
45
36
18
9
1
48
36
19
10
-
10-5
Figure 9 Couples of matched contours using only ends’ abscises
1 2 3 4 5 6 7 8 9 10
1
2
3
4
5
6
7
8
9
10
0.5372 0.3214 0.3214 0.7199 0.1786 0.3929 0.2857 0.25 0.2384
0.5372 0.0358 0.0358 1.0771 0.5358 0.0357 0.0715 0.1072 0.5956
0.3214 0.0358 0 1.0413 0.5 0.0715 0.0357 0.0714 0.5598
0.3214 0.0358 0 1.0413 0.5 0.0715 0.0357 0.0714 0.5598
0.7199 1.0771 1.0413 1.0413 0.5413 1.1128 1.0056 0.9699 0.4815
0.1786 0.5358 0.5 0.5 0.5413 0.5715 0.4643 0.4286 0.0598
0.3929 0.0357 0.0715 0.0715 1.1128 0.5715 0.1072 0.1429 0.6313
0.2857 0.0715 0.0357 0.0357 1.0056 0.4643 0.1072 0.0357 0.5241
0.25 0.1072 0.0714 0.0714 0.9699 0.4286 0.1429 0.0357 0.4884
0.2384 0.5956 0.5598 0.5598 0.4815 0.0598 0.6313 0.5241 0.4884 -
Matched contours Using curvature’s values 1-6 2-7 3-4 4-3 5-10 6-10 7-2 8-3 or 8-4 or 8-9 9-8 10-6
Figure 10 Couples of matched contours using only curvature’s values The objective function f, used in the proposed algorithm, is represented by Equations (1), (2) and (3) below. The curvature c, given by equation (3), computes the curvature of a damaged contour. The term of equation (1) represents the difference between curvatures of two contours i and i+1. The term of equation (1) represents the difference between ends’ abscises of two ends points and associated to two contours i and i+1, where and .
Damaged image
Locate the damaged region
Contours detection Compute mean curvature, ends’ abscises for each contour
(1) Where
(2) (3)
Population initialization
Compute distance, attractiveness
3.4. Matching process Figure 12 illustrates steps of discrete firefly algorithm used to resolve the contours matching problem. Initialisation. A solution representation for the contours matching problem is a permutation as illustrated in figure 11. Here, a firefly represents one solution. It is just like a chromosome that represents an individual in genetic algorithm. In this representation, an element of array represents a contour. It corresponds to mean curvature C and to curvatures’ ends abscises . Each firefly generates an initial solution randomly. Ends’ abscises
45 1
28 2
70 3
47 4
30 5
73 6
Individual
3
1
4
5
6
2
Curvature
1 2
2 1.3
3 0.8
4 2.2
5 1.1
6 1
Figure 11 Individuals Initialization A discrete firefly algorithm (DFFA) population is represented as individuals . Each individual is randomly initialized from 1 to N which is the number of contours. Since an element of individual represents a contour, its number must be unique in an individual. Indexes of and represents contours.
For each particle
Y
Stop criterion is met
e s
Join contours according to the best particle
N
Visualise the result
o Evaluate the objective functionirefly Move the firefly
Figure 12 DFF algorithm for the contours matching
Distance. For contours matching problem, distance between two fireflies and can be defined as the number of different arcs between and . In figure 13, three arcs 4-5, 5-2, and 2-6 in firefly do not exist in firefly . Hence, the number of different arcs between firefly and firefly is equal to three (a=3). 3
1
4
5
2
6
3
1
4
2
6
5
Figure 13 Two different fireflies and . Attractiveness. In the original FFA, the main form of attractiveness function can be any monotonic decreasing function , is the distance between two fireflies, is the attractiveness at =0, and γ is a fixed light absorption
coefficient [21]. In the discrete firefly algorithm used in this paper, attractiveness is calculated as follow. (4)
A=
the image’s structure. High threshold gives us too much contours, low threshold gives less contours. The challenge is to adjust Canny’s threshold in order to have only contours related to the damaged area.
Movement the movement of a firefly attracted to another brighter firefly is determined using permutation. Mutation is done A times between elements of the less bright firefly , movement is done on different arcs, see Figure 14 and Figure 15.
(5)
3
1
4
5
2
6
3
1
4
2
6
5
Figure 18 Effect of Canny‘s threshold Figure 19 illustrates the results of the proposed algorithm presented above. The first column presents input damaged images. The second column shows inpainted images.
Figure 14 Before movement of fireflies 3
1
4
5
2
6
3
1
4
5
6
2
Figure 15 After movement of fireflies
4. EXPERIMENTAL RESULTS Our discrete firefly algorithm for contours matching uses parameters which can be classified into two main categories: inpainting- related parameters and DFFA parameters. Inpainting parameters are the size of the damaged area, the total number of damaged contours and the input values of Canny detector. Optimisation parameters are the DFF algorithm parameters’, the number of generations, the population size, etc. Figure 18 presents values of parameters used to inpaint test images presented in Figure 16. The fourth figure of Figure 16 is taken from [19].
Figure 16 Test images DFF Algorithm Population size Number of generations
Values 10 100
Figure 17 Parameters’ Values To validate the proposed algorithm, the set of damaged images presented above in Figure 16 is used. Adjustment of Canny’s thresholds depends on the input image of the algorithm. Figure 18 illustrates the effect of modifying Canny’s threshold on
Figure 19 Image inpainting results
5.
CONCLUSION
Image inpainting, is a fundamental problem in image processing, which aims at reconstructing completely-lost areas from their surroundings, for example damages caused by spots, gaps, etc. In order to rebuild lost regions’ structure, we need first to detect contours of the damaged image using an edge detector such as canny detector. Then, close damaged contours are found. Finally, matched damaged contours are joined using an interpolation technique. The search space of candidate solutions increases dramatically as the number of contours increases, which makes the search for the optimal solution by a traditional deterministic method infeasible. Many image inpainting applications need to much large number of contours such as 1. Damaged historical documents, they contain several damaged area to rebuild and several contours to match. 2. Geographical maps contain several contours, especially if the damaged area(s) corrupts level curves. A good choice to solve this problem is the use of an optimisation algorithm, which is in the present paper a discrete version of FF algorithm. In this paper, DFFA is used to find close contours using their curvatures values and the mean ordinate values. The experimental results, presented above, confirmed the effectiveness of the DFFA for geometric image inpainting problem. For the future research, the DFF algorithm will be applied to inpaint the texture of damaged images. KHNOWLEDGEMENT The authors would like to thank Dr. A. DRAA for his help. REFERENCES [1] Saibal K.P, C.S Rai, A.P Singh. “Comparative study of firefly algorithm and Particle swarm optimization for noisy non-linear optimization problems”. I.J. Intelligent Systems and Applications, 10, 50-57, 2012. [2] Yang, X.S. “Firefly Algorithms for Multimodal Optimization”. In: Watanabe, O., Zeugmann, T. (eds.) SAGA 2009. LNCS, vol. 5792, pp. 169–178. Springer, Heidelberg, 2009. [3] Bugeau. A. and M. Bertalmio. “Combining Texture Synthesis and Diffusion for Image Inpainting”. VISAPP 2009- Proceedings of the fourth International conference on Computer Vision Theory and Applications, Portugal, 2009. [4] Bertalmio. M. “Processing of Flat and non-Flat Image Information on Arbitrary Manifolds using Partial Differential Equations”. Doctoral thesis, University of Minnesota, 2001. [5] Stanco. F, G. Ramponi, A. de Polo. “Towards the Automated Restoration of Old Photographic Prints: A survey”. IEEE Region 8 EUROCON, VOL B, PROCEEDINGS: COMPUTER AS A TOOL. Pages: 370-374, 2003. [6] Laccetti. G, L. Maddalena, A. Petrosino. “Removing Line Scratches in Digital Image Sequences by Fusion Techniques”. Image Analysis and Processing ICIAP 2005, PROCEEDINGS Book Series: LECTURE NOTES IN COMPUTER SCIENCE Volume: 3617 Pages: 695-702, 2005.
[7] Cornelis. B, T.Ruzic , E. Gezels ,A. Dooms , A. Pizurica , L. Platisa , J. Cornelis ,M.Martens, M. De Mey, I. Daubechies. “Crack detection and inpainting for virtual restoration of paintings: The case of the Ghent Altarpiece”. Signal Processing. Volume 93, Issue 3, pp, 605-619, 2013.
[8] Yoo. S, R-H. Park. “Red-eye detection and correction using inpainting in digital photographs”. IEEE Trans. Consumer Electronics 55(3): 1006-1014, 2009. [9] Granados. M, J. Tompkin, K. Kim, O. Grau, J. Kautz, C. Theobalt. “How Not to Be Seen- Object Removal from Videos of Crowded Scenes”. Euro graphics, volume 31, Number 2, 2012. [10] Liu. D, X. Sun, F. Wu, S. Li, Y-Q. Zhang. “Image Compression with Edge- based Inpainting”. IEEE transactions on circuits and systems for video technology. Volume 17, No.10, 2007. [11] Rane. SD, G. Sapiro, M. Bertalmio. Structure and texture filling-in of missing image blocks in wireless transmission and compression applications. IEEE Transactions on Image Processing. Volume 12. N° 3, pp. 296-303, 2003. [12] Chan. T and J. Shen. “Mathematical models for local non texture inpaintings”. SIAM J. on Appl. Math., vol.62, pp.10191043, 2002. [13] Shen. B, W.Hu, Y. Zhang, Y-J. Zhang. “Image Inpainting via Sparse Representation”. ICASSP, 2009. [14] Zhang, Q., and J. Lin. “Exemplar based Inpainting using Color Distribution”. Journal of information science and engineering, volume 28, 2012, 641-654, 2012. [15] Bertalmio. M, Sapiro. G, V. Casselles, and C. Ballester. “Image Inpainting”. Computer Graphics, SIGGRAPH, pp.417-424, 2000. [16] Fergani. B, M. K. Kholladi. “On the Analysis of Parameters’ Effect in PDE- based Image Inpainting”. Journal of Theoretical and Applied Information Technology. Vol.47.No.3. ISSN: 1992-8645.E-ISSN: 1817-3195, 2013. [17] Criminissi. A, P. Pérez, and K. Toyama. “Region Filling and Object Removal by Exemplar based Images Inpainting”. IEEE transactions on image processing. Vol.13, pp. 1200-1212, 2004. [18] Hameed. S.M, N.J. Kadhim and M.A. Othman. “Image Inpainting based on Particle Swarm Optimization”. Iraqi Journal of Science, Vol.50, No.2, PP.231-235, 2009. [19] Wang. Z, Q. Wang, C S. Chang, M. bai, Z. Sun, T. Yang. “Image Inpainting Method based on Evolutionary Algorithm”. International journal of Digital Content Technology and its Applications. Volume 5, Number 4, 2011. [20] Yang, X.-S. “Firefly Algorithm, Lévy Flights and Global Optimization”. In, Research and Development in intelligent Systems XXVI (Eds M. Bramer, R. Ellis, M. Petridis), Springer London, pp. 209-218, 2010. [21] Jati. G.K. and S. Suyanto. “Evolutionary Discrete Firefly Algorithm for Travelling Salesman Problem”. International Conference on Adaptive and Intelligent Systems (ICAIS 2011), Austria. Springer, September 2011. [22] S.N. Kumbharana, G. M. Pandey. “Solving Travelling Salesman Problem using Firefly Algorithm”. International Journal for Research in science & advanced Technologies. Issue-2, Volume-2, 053-057. ISSN 2319-2690, 2013. [23] Sayadi M. K., A.H. Hafezalkotob, S.G.J. Naini. “Fireflyinspired algorithm for discrete optimization problems: an application to manufacturing cell formation”. Journal of Manufacturing Systems, 32:78-84, 2013.
