adaptive image sensing and enhancement using the ... - CiteSeerX

ADAPTIVE IMAGE SENSING AND ENHANCEMENT USING THE CELLULAR NEURAL NETWORK UNIVERSAL MACHINE MÁTYÁS BRENDEL AND TAMÁS ROSKA

Analogic and Neural Computing Systems Laboratory, Computer and Automation Institute Hungarian Academy of Sciences, P.O.B. 63., H-1502 Budapest, Hungary SUMMARY As an attempt to introduce Interactive, Content Dependent Adaptive (ICDA) image processing a simple but powerful active image sensing and two image-enhancement methods are introduced via adaptive CNN-UM sensorcomputers. Thus the method ICDA can be used for adaptive control of image sensing and for subsequent on-line or offline image enhancement as well. The algorithms use both intensity and contrast content. The image sensing technology can be realized with the current CNN-UM chip [1],[2]. Our first image enhancement method is also executable on this chip, but it is more suitable for the Adaptive Cellular Neural Network Universal Machine (ACNN-UM) architecture [3]. Some results of simulator and chip experiments and an adaptive extended cell is presented. Our second, dynamical image enhancement method is planned to be executable on a multi-layer, complex cell CNN architecture [3]. In [15] a 3-layer architecture is described which is capable to realize the main part of the second enhancement method. The main issues of our paper are as follows: the novel outlook of the ICDA framework, 3 new methods for two key application-area of CNN-UM, the notion of “regional” adaptive computing, the novelty of application of equilibriumcomputing in the third method. However, the key novelty of our work is not just a new method and a new realization: by combining sensing and computing, dynamically and pixelwise, a new quality becomes practical.

1. INTRODUCTION In recent years, a revolution has been apparent in the field of sensors. Considering sensor arrays, it is an important progress that high resolution and locally programmable arrays are available. A fast parallel device is needed for programming the sensor in real-time mode for adaptive sensing. Cellular Neural Networks (CNN) are parallel-computing, analog arrays, which are suitable for most of the computation needed. Adaptive sensing is one of the ideal applications for CNN-type sensorcomputers (see [3] and [4]), where both the 2D parallel architecture and the high speed are utilized, when the CNN Universal Machine (CNN-UM) architecture is used. Another progress can be recognized in the development of video cameras and electronic photo machines. Cheap devices are available commercially and adaptive devices are emerging. Intelligent solutions with different hardware and software are implemented. The CNN Technology technically provides for a cheaper, more powerful, smaller and less power consuming solution. A CNN-UM based visual microprocessor chip could be built into an electronic photo machine or video camera and can be adequately programmed. This device is able to capture images or video frame-sequences much more adaptively and may be more adjustable than other solutions. In adaptive sensing and image enhancement computation time is significant, hence using a 2D, analogic, parallel computer, like the Cellular Neural Network Universal Machine (CNN-UM [1],[2]) may be important. It is significant to use operations that are executable on currently available CNN-UM chips or which are likely to be available in the near future [3], [15]. The adaptive CNN-UM architecture has been introduced in [3] and [4]. Among others, plasticity and variable resolution in space and time as well as the complex-cell are handled in this architecture. By using this hardware, unlike in “smart sensors”, stored programmable spatiotemporal computing is performed in the sensing-computing loop, interactively. The key novelty here is the local, interactive, content dependent adaptation of sensors and algorithms. Our research in adaptive image processing lies in the framework of Interactive Content Dependent Adaptive (ICDA) image processing. The method is interactive, because sensors or the operators are locally adjusted (sensors are required to be programmable). It is content dependent,

since the programming depends on the image content. Finally, it is adaptive, as programming is carried out in order to make sensing more adaptive to regional changes. Due to improper or uneven lighting conditions, important information may be lost during sensing. Adaptive sensing, in our case the adaptive control of the parameters of the sensor can solve this problem. Parameter computation is based on the information available during the exposure; this technique may reduce information loss. Integrated sensor-computers provide for a new and unique capacity as they dynamically control the sensors, based on interactive computation. Another problem is that the acquired image may be improper for human visual perception. A kind of visualization can solve this problem, which means adaptive image enhancement in our case. This kind of problem can already be solved off-line, since the image is already acquired. Several methods like amplitude scaling, contrast modification and various kinds of histogram modifications are available at this time [7]. The difference between adaptive sensing and adaptive image enhancement is that the former must be controlled in cooperation with sensing, while the latter is used after sensing. Hence image enhancement cannot restore information lost during sensing (e.g. because of saturation). The task of adaptive sensing is to acquire proper image content, while the goal of enhancement is to produce an excellent result for human perception. The importance of image enhancement is great and the applications are wide. The most important area is medical imaging, where a wide range of the image enhancement methods are applied. For similar methods see [26]and [27] here reviews of several earlier methods can also be found. The most common method for image enhancement is histogram equalization. CNN techniques have already been used for this task [5]. In this work several methods were described. More complex methods, which can only be realized in the future; and indirect realizations or simplifications, which can be realized in the near future. The authors of that paper are confronted with the same problem as we in this paper: hardware limitations and efficiency of the method. The methods described in [5] are in complexity similar to our methods: second order cells, image multiplication is used. However, our work addresses the problem from a widely different viewpoint: contrast and intensity equalization rather than histogram equalization. We do not address histogram equalization explicitly, but our method partially equalizes the histogram as well. The simplest histogram equalization methods are global, which is not sufficient for our purpose. For a comprehensive description of many methods and their drawbacks see for example the first two chapter of [22]. Adaptivity means in our case that information are treated not global, neither local, but regional. We call methods “global”, if they use attributes of the whole picture (for example global mean intensity, global histogram) to compute modification of one pixel. This is usually not sensitive to regional changes. The nomination “local” denotes methods, where only attributes of the one pixel, which is to be computed is considered, or only a small neighborhood (3x3-9x9). For example histogram, mean, contrast or standard deviance in the local neighborhood. These methods consume much resource, and the computation may not take into account the information, which is on the picture in a larger region. For example the regional change of the background is lost. This is not sufficient, since a typical object in a natural image is larger than a local neighborhood. By our definition, regional method means that the computation involves a region of the picture larger than a 9x9 neighborhood , but very distant pixels are not involved, or with a practically 0 weight. Thus the modification of the pixel is practically influenced by its region, and the actual regional attributes determine the computation. In this way the typical regional changes in a natural picture are handled adequately. The “shape” of the region, which influence the computation of one pixel is also adaptive, due to the “logic” of the method. Certainly there are such adaptive methods in similar areas, connected to our work, like adaptive histogram equalization [9], adaptive contrast enhancement [21] adaptive multi-scale gain control

2

[22] and adaptive noise filtering [24],[25]. However, these methods are computationally intensive. Some adaptive methods are simpler and they belong to the group, which we named “local”, and not “regional”. This is a simpler kind of adaptivity, like finite neighborhood histogram modification.. The fact that an interpolating technique has been proposed in [6] indicates the significance of the drawback of high resource demand. There is no need of interpolation if the adaptive CNN-UM is used, since parallel computation enables individual adaptation for each pixel. The novelty of our method is as follows: it overruns global and local (locally adaptive) methods, with a quasi “large-neighborhood”, regional computation. The problem computational demand is solved by the CNN hardware and thus it is not only a theoretical possibility. The method resembles and involves many aspects of other methods. Resembles to histogram adaptation, to contrast enhancement, contrast stretching, multi-scale analysis, adaptive smoothing and adaptive unsharp methods. It does not realize all the advantages of these methods but involves many benefits of them. In the algorithms the radius of the adaptation can be controlled by the time or gain of diffusion, thus all intermediate cases between full global and local equalization are dynamically available. For example, if the range (time) of the diffusion is short, then the adaptation occurs for a small region, therefore equalization will be very strong. On the other side, if the diffusion is very large, then the intensity and contrast is modified based on almost global features. This means that practically only the medium contrast and intensity is adjusted and there is only a small and flat regional component of the enhancement. In section 2 general theoretical notions are introduced and the image quality measure, we use in this paper. In section 3 adaptive sensing is developed. Section 4 and 5 deals with adaptive image enhancement, section 4 presents the one-step method and section 5 the dynamical method. Section 6 presents the possible hardware realizations of the methods and finally in section 7 the chip experiment results are presented. There is also a short conclusion at the end of this paper. 2. MEASURING IMAGE QUALITY A grayscale image is represented by the intensity distribution in a matrix I, on the range [-1,1], according to the CNN convention (black=1, white=-1). The image is assumed to be sampled in space, i.e. I is a matrix of form I∈[-1,1]MxN, where MxN is the size of the image. The intensity of pixel (x,y) is I(x,y). Simple and complex measures may be developed for describing the quality of an image. We are interested in a simple method, which describes point to point the worth of an individual pixel in the image. For achieving adaptivity the measure should also involve the neighborhood of the pixel. It is evident to base the measure on some important features of the image. The two main features are intensity and contrast. Other properties can also be involved, but now we deal with these two most important and simplest one. The method can be improved further as the advance of CNNtechnology makes realization of more complex algorithms rewarding. Figure 1 illustrates how the intensity and contrast influences the quality of the image.

3

a) original

b) intensified intensity

mean of intensity measure Ιµ=0.3226 mean of contrast measure Χµ= 0.0071 Shannonian information Sµ= 4.7764 c) intensified contrast

mean of intensity measure Ιµ=0.5723 mean of contrast measure Χµ=0.0062 Shannonian information Sµ=5.2254 d) intensified intensity and contrast

mean of intensity measure Ιµ=0.3576 mean of contrast measure Χµ=0.0113 Shannonian information Sµ=4.9175

mean of intensity measure Ιµ=0.6592 mean of contrast measure Χµ=0.0162 Shannonian information Sµ=5.0517

Figure 1 The same image with different average contrast and intensity. a) shows an image with low contrast and intensity b) shows an image, where the intensity is stronger, but the contrast is roughly the same. Image b) is better than image a), because the intensity is stronger. Image c) shows an image, where the contrast is stronger, but the intensity is roughly the same as in image a). Image c) is better than image a), because the contrast is stronger. The result of comparison of image c) and b) is not so obvious, it depends on the subjective importance of contrast versus intensity. However, image d) is clearly better than any image before, both contrast and intensity is here the strongest. It is interesting that the information is not so high.

As it is shown, both intensity and contrast is an important characteristic, considering image quality. The pictures also demonstrate (b and c) that the importance of intensity and contrast is a question, which should be solved by appropriate weighting of these features. The measures illustrated are the following ones. The mean absolute intensity:

∑ abs( I ( x, y ))

Ιµ =

x, y

MN

where Ιµ is the intensity measure, and the sum is computed over the entire image area. Similarly the mean absolute contrast is computed as:

∑ abs(C ( x, y ))

Χµ =

x, y

MN

where Χµ is the mean absolute contrast and C(x,y) is the contrast of pixel (x,y). The contrast of the pixels can be measured by a simple template, which will be described later (see Table 1).

4

Another measure will also be used by presenting the images. This measure only serves for analysis, it is not apart of the current algorithm. The measure is known as Shannonian information content:

∑ pi log pi Sµ = − i

MN

where pi is the number of occurrences of the the ith color (intensity) in the picture. It is known that the higher this measure is, the more information the image contains. This measure also describes the lower limit of the size by compressing this image by the most effective general coding method. A complex measure as the weighted sum of these measures can be computed from the given data in this paper for each image. Since we do not know an objective criterion to chose the weights, we do only give the separate measures. The mean intensity and contrast measure (Ιµ and Χµ) describes the quality of the image overall. The local contrast and intensity (C(x,y) and I(x,y)) describes each pixel separately. For adaptivity a measure is needed, which is between this two extremities. We need to measure this features neither global, nor local, but regional. This can be achieved by using a regional operator instead of the ∑. We use a diffusion operator D, which is easy to realize via CNN and involves the required properties (this is a common known operator, see Table 1). Actually the D operator turns out to be a weighted sum. Using the diffusion operator the measure describes how good the average square contrast and intensity are in a certain regional neighborhood of the pixel. The two features are weighted and added instead of the absolute value, square value is used, since it is easier to implement. The exact formulation is as follows: Q ( x, y ) := c1D ( P 2 ( x, y )) + c2 D (C 2 ( x, y )) (1)

where Q(x,y) is the measure of quality, c1 and c2 are parameters, I(x,y) is the intensity of the pixel, C(x,y) is the contrast of the image at pixel (x,y) and D is some diffusion operator (regional averaging). 3. ADAPTIVE SENSING It is assumed that an image-capturing device is available, in which the exposure time, gain, shutter-speed, brightness or other parameters can be adjusted pixel-wise. The idea is to adjust this parameter (or more parameters) so that the image quality is high for each pixel. Thus a picture with a quite even and proper quality can be captured. For example the method for controlling exposure time is described as follows: (i)

Capturing with a continuous exposure, which can be stopped at each pixel by a command

(ii)

The image quality is measured real-time and pixel-wise

(iii)

The exposure of the pixel is stopped, if proper quality is achieved

This method acquires an image, which is satisfactory in each region, but over-saturation is avoided. The method can be formulized as follows: Continue exposure of (x,y) while Q(x,y)