c Allerton Press, Inc., 2011. ISSN 8756-6990, Optoelectronics, Instrumentation and Data Processing, 2011, Vol. 47, No. 3, pp. 226–232. c S.A. Bibikov, R.K. Zakharov, A.V. Nikonorov, V.A. Fursov, P.Yu. Yakimov, 2011, published in Avtometriya, 2011, Original Russian Text Vol. 47, No. 3, pp. 25–33.
ANALYSIS AND SYNTHESIS OF SIGNALS AND IMAGES
Detection and Color Correction of Artifacts in Digital Images S. A. Bibikova, b , R. K. Zakharova , A. V. Nikonorova , V. A. Fursovb , and P. Yu. Yakimovb a
Samara State Aerospace University named after Academician S. P. Korolev, Moskovskoye sh. 34, Samara, 443086 Russia b Image Processing Systems Institute, Russian Academy of Sciences, ul. Molodogvardeiskaya 151, Samara, 443001 Russia E-mail:
[email protected] Received March 1, 2011
Abstract—A problem of automated retouching of point artifacts in the pre-press process is considered. A new algorithm of detection and localization of multiple point flares is proposed. The algorithm is based on using the so-called conjugate indicator. A scheme for constructing learning rules for tuning the system for different types of artifacts is developed. An example illustrating the proposed algorithm performance on a real image is given. Keywords: digital processing of images, point flare, color correction, recognition, identification. DOI: 10.3103/S8756699011030046
INTRODUCTION There is significant recent demand for new information technologies of color correction of digital images in various applications. The problem of color correction is usually solved by professional proof-readers with the use of standard tools, e.g., Adobe Photoshop for image processing. Unfortunately, engineering and computational capabilities of such systems are rather limited, and correction of color images takes a long time: “manual” processing of an image with a volume of about 30 megapixels may require up to 10 hours (depending on the character and intensity of distortions). Such times are significant even for prepress processes and are absolutely inadmissible in video monitoring and video surveillance systems, because real-time decisions have to be made. Therefore, the problem of constructing an information technology for detecting and eliminating distortions with a high degree of automation is rather important. An automated technology of color correction of a shadow from a frame, formed because different parts of the canvas are located at different distances from the light sources, was considered in [1]. One of the most frequent types of distortions in images are flares. The flare is understood here as an area with elevated brightness, which appears in the course of image registration owing to specular or diffuse reflection of light from the object surface. Depending on the size and character of reflecting surfaces, the flares may have various shapes and sizes. In particular, flares on mirror surfaces (e.g., windows in buildings and cars) usually have large sizes and a typical “star” shape. In this paper, we consider only small-size flares (with an area of 16–36 pixels) that appear in photographing works of art for making reproductions. Such flares are induced by reflections from oil-based paint strokes and are visible in the digital image as a large number of small bright points. To distinguish them from other types of flares, they will be called the multiple point flares or artifacts. Algorithms of detection, localization, and color correction of multiple point flares, based on processing with a sliding window, were considered in [1, 2]. To make a decision about the presence of a flare in a window of a given fixed size, we have to calculate the characteristics of the absolute and relative (compared with its neighborhood) brightness for each position of the flare. The problem is that the sizes of point flares 226
DETECTION AND COLOR CORRECTION OF ARTIFACTS IN DIGITAL IMAGES
227
can differ within wide limits even in one image. Therefore, to improve the quality of artifact correction, it is necessary to divide the set of point artifacts into classes (according to their types and sizes), and the information technology should include a procedure for recognition of artifacts. In recognition learning, a learning (reference) set of flares of the corresponding type and size should be formed for each class. In this work, we made an attempt to work our some general recommendations and approaches to constructing decision rules and procedures and to create an integral technology of detection, localization, and color correction of flares. In particular, an algorithm of recognition and localization of multiple point flares with the use of the so-called conjugate indicator is proposed. A scheme of constructing learning rules for tuning the system to different artifact types and sizes is proposed. An algorithm of color correction based on identification of a polynomial model is described. An example illustrating the operation of the proposed algorithm on a real image is presented. 1. FORMATION OF A SYSTEM OF FLARE SIGNS To solve problems of detection, localization, and recognition of the flare type, it is necessary to choose a system of signs. We assume that color images are defined in the RGB color space, and the three functions of brightness corresponding to these components in the domain of each pixel are defined as fR (x, y), fG (x, y), and fB (x, y). Let us consider a total brightness function f (x, y), which is a weighted sum of the functions mentioned above: f (x, y) = 0.3fR (x, y) + 0.59fG (x, y) + 0.11fB (x, y);
(1)
the coefficients of this function are specified empirically [3]. Let W1 be an image area covered with the flare, and let W0 be its certain neighborhood. Let us consider the membership function I0 (x, y) = 1, (x, y) ∈ W1 , (2) / W0 . I0 (x, y) = 0, (x, y) ∈ In accordance with the above-given description of the mechanism of emergence of point artifacts, we assume that there is a point flare in the domain W1 if the following conditions are satisfied: max
(x, y) ∈ W1
f (x, y) = f ≥ f1 ,
M
f (x, y)
M
f (x, y)
(x, y) ∈ W1 (x, y) ∈ W0
(3)
= M ≥ M1 ,
(4)
I0 (x, y) = S ≤ S1 ,
(5)
(x, y) ∈ W1
D(W1 ) = D ≤ D1 .
(6)
Inequalities (3) and (4) impose constraints on the minimum absolute brightness and on the relative brightness, respectively. Here, M f (x, y) is the mean value of the brightness function and D(W1 ) is the domain diameter, i.e., the quantity determining the degree of domain compactness. In [4], these signs were used for automatic detection of flares with fixed threshold values of the quantities f, M, S, and D. Flares with different sizes (too large and/or too small) often remained undetected. Another frequently encountered problem is the fact that several points with satisfied conditions (3)–(6) are usually encountered in a certain neighborhood, and it is not clear which point is the central point of the flare. If the domain is constricted by setting more rigorous thresholds, some flares are missed. If the thresholds are too low, the number of points suspected to be flare centers increases. Moreover, flares with different sizes of the domains W1 and W0 can be encountered in an immediate vicinity of each other. Thus, using the general information technology, one can only identify some domains “suspected” to contain a flare at the initial stage. In this case, it is possible to use a simplified detection scheme based on checking only one condition, i.e., condition (3). After that, the problem of localization and recognition of the flare type should be solved for each set of points before color correction proper is started. OPTOELECTRONICS, INSTRUMENTATION AND DATA PROCESSING
Vol. 47
No. 3
2011
228
BIBIKOV et al.
2. CONSTRUCTION OF THE RECOGNITION ALGORITHM To construct the recognition procedure, it is reasonable to introduce unified notations for signs. In particular, we introduce the vector xi = [x1 (i), x2 (i), x3 (i), x4 (i)]T
(7)
of dimension N × 1 (in the case considered, the vector dimension is 4 × 1), where x1 = f (i), x2 = M (i), x3 = S(i), and x4 = D(i). In the general case, solving the recognition problem consists in designing a decision function g: Rn → {0, 1, 2, . . . , k}, which makes a decision for each vector xi about its belonging to a certain class. In this case, we have to solve simultaneously the problems of determining the class (flare type) and its localization in a certain domain of image points identified as a set of points suspected in terms of the flare. We assume that we have K vectors of the form (7), which are classes of reference flares that differ by their geometric and brightness characteristics. We assume that M vectors are specified for each class so that an N × M matrix can be composed for each kth class: Xk = [x1 (k), x2 (k), . . . , xM (k)],
k = 1, . . . , K.
(8)
Let us introduce the so-called conjugate indicator with a subspace stretched on the vectors of signs of object images from a given class: xT Xk [XTk Xk ]−1 XTk x . (9) xT x Here, x is the vector of signs of an unknown image presented for establishing the closeness to the kth class and Xk is an N × M matrix (8). Assuming that an N × N matrix is formed for each kth class, Rk =
Qk, R = Xk [XTk Xk ]−1 XTk ,
(10)
we construct a decision function g(x). The vector x is the image of the sought object (flare) and belongs to the mth class, i.e., g(x) = m (m = 1, 2, . . . , K) if Rm = max Rk (i, j),
(11)
k, i, j
where Rk (i, j) =
xTi, j Qk xi, j xTi, j xi, j
.
(12)
Thus, to construct the recognition procedure, we have to construct an N × N matrix Qk by Eq. (10) with the use of matrix (8) for each expected type (class) of flares at the learning stage. It should be emphasized that the maximum value of the conjugate indicator Rk (i, j) is sought over all classes and all points of the domain suspected for the presence of a flare. 3. DETERMINATION OF PARAMETERS OF DECISION RULES As was indicated above, we have to construct a matrix Qk of size N × N for each type (kth class) of flares at the learning stage and to choose threshold values of f, M, S, and D involved into inequalities (3)–(6). For this purpose, it is necessary to define several (in the case considered, no more than four) reference flares of each type. It is difficult to formalize this part of work, and it is usually done manually for each particular image. When image fragments containing one reference flare of a certain type are defined, it is necessary to find their central points and their boundaries and to select the sizes of the domains W1 and W0 . The central point of the flare is determined by using the window model [3] presented in the form of two nested squares with the common center (x0 , y0 ). The domain sizes are specified by the user on the basis of approximate estimates of the size of a flare of this type. The flare is expected to be in the inner square. Possible inaccuracy in estimating the domain sizes at this stage is irrelevant, because the central point is found by using the values of relative brightness (4) OPTOELECTRONICS, INSTRUMENTATION AND DATA PROCESSING
Vol. 47
No. 3
2011
DETECTION AND COLOR CORRECTION OF ARTIFACTS IN DIGITAL IMAGES f (xi , yi )
M (x0 , y0 ) =
xi , yi ∈ W1
229
−1 f (xi , yi )
(13)
xi , yi ∈ W0
calculated in the half-tone image. The central point is assumed to be the point (x0 , y0 ) for which M (x0 , y0 ) in the identified fragment containing one reference flare reaches the maximum value. The reference flare boundary is determined by using the so-called contrast function [1]. In a polar coordinate system defined by the radius-vector oz with the origin at the point (x0 , y0 ), the flare can be defined as a domain whose set of boundary points z ∗ is a set of points with the maximum values of the contrast function [1] F (z ∗ ) = max f (zi ) − f (zi ) , (14) ∀z
zi < z ∗
zi > z ∗
calculated in the direction of the radius-vector oz being rotated around the origin o, in the case considered, around the point (x0 , y0 ) by an angle [0, 2π]. The flare boundary in this model can be presented as a certain function constructed by means of interpolation of the set of boundary points. If the boundary has a complicated shape that can be hardly described in a certain parametric family of functions, it may turn out to be more convenient to describe it as a table of coordinates of boundary points. The values of the brightness function at boundary points of the reference flare allow us to set a threshold for constructing the membership function (2). After it becomes known, we can also determine threshold values for f , S, and D and refine the sizes of the domains W1 and W0 . To refine the sizes of the domains W1 and W0 , the sizes of the nested squares with a common center (x0 , y0 ) are chosen from a certain specified series of admissible types and sizes in a manner to reach the maximum relative brightness function (12): −1 M (x0 , y0 ) = max . (15) f (xi , yi ) f (xi , yi ) ∀ W0 , W1
xi , yi ∈ W1
xi , yi ∈ W0
If the sizes of the windows of all reference flares are determined, a vector of signs (7) can be formed for each of them, and the vectors of signs of each type and size can be arranged into a corresponding matrix Xk (8). Let us recall that, as the vector xi has the size 4 × 1, the number of such vectors in the matrix Xk cannot be greater than four, i.e., no more than four learning examples can be defined for each flare type. The number of classes for each image is determined by the range of variation of the characteristics of flares encountered in it and by the requirements of the needed recognition reliability. Taking into account the matrices Xk formed, we then calculate the matrix Qk for each class (flare type) by Eq. (10), which is used in the decision rule (11) at the stage of flare localization and recognition. 4. COLOR CORRECTION OF FLARES To replace the color of pixels in the area of the point flare, an algorithm based on cloning of the color and texture from the close vicinity of the flare to the flare domain was used in [5, 6]. After that, the Gaussian smearing was applied in a domain slightly greater than the flare domain, which made it possible to avoid sharp boundaries between the retouched area and its neighborhood. In the case considered, following [7, 8], replacement of the flare pixels is performed through identification of parameters of the formal model with the use of information about the color in the close vicinity around the flares. We assume that the color is defined in the RGB space. The problem of color correction of the flare domain is solved for each color channel independently with the use of identical algorithms. For simplicity, we consider only the color correction in the channel R. The colors in the flare domain and in its neighborhood are assumed to be presented by a certain model, for instance, by a polynomial model R(xi , yi ) = c1 x2i + c2 yi2 + c3 xi yi + c4 xi + c5 yi + c6 .
(16)
To construct the algorithm for determining the values of the color at the flare points, we solve the problem of identification of the vector C = [c1 , c2 , . . . , c6 ]T of the parameters of model (16), based on reference values of the color R(xi , yi ) [4], where (xi , yi ) are points from the flare neighborhood. To increase the number of reference points, the flare neighborhood is extended by means of dilatation. To solve the problem of identification on the basis of the actual values of the color coordinates at the reference points, we use a linear (in terms of its parameters) model R = XC + ξ, OPTOELECTRONICS, INSTRUMENTATION AND DATA PROCESSING
(17) Vol. 47
No. 3
2011
230
BIBIKOV et al.
where
⎡
fr (x11 , y11 )
⎢ ⎢ ⎢ fr (x12 , y21 ) ⎢ R=⎢ ⎢ .. ⎢ . ⎣ 1 1 fr (xI , yI )
⎤ ⎥ ⎥ ⎥ ⎥ ⎥; ⎥ ⎥ ⎦
⎡ ⎢ ⎢ ⎢ C=⎢ ⎢ ⎢ ⎣
c1
⎡
⎤
⎥ c2 ⎥ ⎥ ⎥ .. ⎥ ; . ⎥ ⎦ c6
⎢ ⎢ ⎢ ξ=⎢ ⎢ ⎢ ⎣
ξ1
⎥ ξ2 ⎥ ⎥ ⎥ .. ⎥ ; . ⎥ ⎦ ξI
⎡
⎤
(x11 )2
⎢ ⎢ 1 2 ⎢ (x2 ) ⎢ X=⎢ ⎢ .. ⎢ . ⎣ (x1I )2
(y11 )2
x11 y11
x11
y11
1
(y21 )2
x12 y21
x12
y21
1
.. .
.. .
.. .
.. .
.. .
(yI1 )2
x1I yI1
x1I
yI1
1
⎤ ⎥ ⎥ ⎥ ⎥ ⎥; ⎥ ⎥ ⎦
ξ are errors associated with inaccuracy of approximation in the model. The algorithm for determining the sought estimates of parameters is constructed as follows. At the first step, using the least squares method, we find the usual estimate [9] cˆ = [XT X]−1 XT R.
(18)
After that, we calculate the weight matrix G = diag[g1 , g2 , . . . , gN ] consisting of N diagonal elements gn = g0 /(ε + |ξn |) (n = 1, N ), where ξn is the nth component of the vector ξ, and g0 is determined from the normalization condition N
gn = N.
n=1
Here, ε is the number definitely smaller than the smallest value of ξn , which is introduced to avoid the possibility of division by zero. Using the thus-obtained weight matrix, we repeat the calculation of the estimates of the transformation function parameters by the method of the least weighted squares: cˆ = [XT Gi X]−1 XT Gi R.
(19)
As a result, the estimates obtained approach those of the method of the smallest absolute values [10, 11] with a high degree of approximation. 5. INFORMATION TECHNOLOGY AND EXPERIMENTAL RESULTS The information technology used for experimental verification of the above-described algorithm was constructed in the following manner. First, local areas suspected for the presence of flares are identified. The absolute brightness characteristic (3) is used to detect points suspected to belong to a flare. This procedure forms a binary image with a large number of identified local areas, each containing a set of points suspected to belong to a flare. After that, the values of the indicators Rk (i, j) obtained on the set of points belonging to an individual closed local domain suspected to the presence of flares are “checked.” The decision about the flare type and coordinates of its central point here is made in accordance with the decision rule (11). All identified local domains are subjected to this procedure to identify all flares in the image. For each thus-found artifact, satisfaction of the conditions of flare existence (3)–(6) is verified. When the coordinates of the centers of all flares and their types (sizes) are found, their color correction (retouching) is performed with model (16) whose parameters are determined by solving the problem of identification described in Sec. 3. Figure shows an example of using this technology to processing a fragment of a real image. Figure 1a shows the initial image containing a large number of artifacts in the form of point flares of different sizes. Figure (b) shows the result of processing with fixed parameters of algorithm tuning. The sizes of the windows W1 and W0 were set to be 4 × 4 and 8 × 8, respectively. Figures (c) and (d) show the images obtained by using the algorithm of localization and recognition with the conjugate indicator. Figure (c) illustrates the image obtained by defining two classes (with window sizes W1 = 4 × 4, W0 = 8 × 8; W1 = 6 × 6, W0 = 12 × 12). Figure (d) shows the image obtained by defining three classes (with window sizes W1 = 4 × 4, W0 = 8 × 8; W1 = 6 × 6, W0 = 12 × 12; W1 = 8 × 8 and W0 = 14 × 14). OPTOELECTRONICS, INSTRUMENTATION AND DATA PROCESSING
Vol. 47
No. 3
2011
DETECTION AND COLOR CORRECTION OF ARTIFACTS IN DIGITAL IMAGES
(a)
(b)
(c)
(d)
231
Figure.
CONCLUSIONS The information technology of detection, recognition, localization, and correction of point artifacts (flares) in digital images, which is described in this paper, is implemented in the form of a software system for prepress preparation of reproductions of various works of art [4]. The main advantage of the algorithm developed is the possibility of reaching a high degree of automation of the digital image correction process. Automated processing of real images usually ensures results that do not require additional manual processing. In developing and describing the algorithm and in arranging experiments, we primarily meant the problem of color correction of artifacts in digital images of paintings. Therefore, the technology described in this paper will most probably find applications in pre-press preparation of images in polygraphy, in creating virtual museums, etc. At the same time, these results may also be applied in other fields. In particular, the proposed technology can be used for image processing in systems of remote probing of the Earth, elimination of artifacts in images in video monitoring and video surveillance systems, etc. This work was supported by the Russian Foundation for Basic Research (Grant No. 09-07-00269-a).
OPTOELECTRONICS, INSTRUMENTATION AND DATA PROCESSING
Vol. 47
No. 3
2011
232
BIBIKOV et al. REFERENCES
1. S. A. Bibikov, A. V. Nikonorov, and V. A. Fursov, “Information Technology of Correction of Shadow Distortions in Color Digital Images,” Komp. Opt. 34 (1), 124–131 (2010). 2. S. A. Bibikov, V. A. Fursov, and A. V. Nikonorov, “Shadow Artifacts Correction on Fine Art Reproductions,” in Proc. of the IMTA 2010 in Conjunction with VISIGRAPP 2010 (Angers, France, May 17–21, 2010), pp. 3–12. 3. D. Judd and G. Wyszecki, Color in Business Science and Industry (Wiley, New York, 1975). 4. S. A. Bibikov, A. V. Nikonorov, V. A. Fursov, and P. Yu. Uakimov, “Studying the Effectiveness of the CUDA Technology in the Problem of Distributed Pre-Press Preparation of Digital Images,” in Scientific Service in the Internet: Scaling, Parallelism, and Effectiveness, Proc. All-Russia Supercomputer Conf. (Izd. Mosk. Gos. Univ., Moscow, 2009), pp. 204–207. 5. V. A. Fursov and A. V. Nikonorov, “Distributed Algorithm of Correction of Point Artifacts in Color Images,” in Parallel Computations and Control Problems, Proc. 4th Int. Conf., Moscow, October 27–29, 2008, p. 1087. 6. A. V. Nikonorov and V. A. Fursov, “Distributed Computational Environment of Correction of Color Images,” in Proc. XVth All-Russia Conf. “Telematika 2008,” St. Petersburg, June 23–26, 2008, pp. 88–89. 7. S. A. Bibikov and V. A. Fursov, “Color Correction on the Basis of Identification of Models from Test Fragments of Images,” Komp. Opt. 32 (3), 302–307 (2008). 8. S. A. Bibikov, “Identification of Parameters of the Model of Color Correction of Point Flares,” in Mathematical Modeling and Boundary-Value Problems, Proc. 7th All-Russia Conf., Part 4: Information Technologies in Mathematical Modeling (Samara State Tech. Univ., Samara, 2010), pp. 19–21. 9. Ch. L. Lawson and R. J. Hanson, Solving Least Squares Problems (Society for Industrial Mathematics, 1987). 10. V. I. Mudrov and V. L. Kushko, Methods of Processing Measurements: Quasi-Reliable Estimates (Radio Svyaz’, Moscow, 1983). 11. V. A. Fursov, Identification of Models of Image Formation Systems, Based on a Small Number of Observations (SGAU, Samara, 1998).
OPTOELECTRONICS, INSTRUMENTATION AND DATA PROCESSING
Vol. 47
No. 3
2011
