2011 IEEE Conference on Sustainable Utilization and Development in Engineering and Technology (STUDENT) The University of Nottingham, Semenyih, Selangor, Malaysia. 20-21 October 2011.
Highly Optimized OpenCV Based Cell Phone S. Sankaraiah1 Faculty of Information Technology Multimedia University Cyberjaya,, Selangor, Malaysia
[email protected]
Abstract: - The new advancement in technology made cell phone as a smart phone with highly computing capabilities and low processing power. Currently, achieving real-time image processing routines such as convolution, edge detection, thresholding and some of complex media applications is a challenging task with the smart phone, because of limited memory and power constraints. An open software framework, Symbian OS is used in cell devices to provide a good starting point for developing the multitasking kernel, integrated with communication protocols, data management and graphical user interface for reducing the total development time. To resolve the problems faced by image processing applications in cell phones, new application environment is developed. This environment provides the resources available in the operating system which runs on the hardware with complex image processing libraries. These papers presents the capturing of an image from the printed media, applies to image processing algorithms to scaling and translation of the image, wipe out the noise, background subtraction, image rotation and detect edges. The developed software chops the comics strip along border into individual units, store on the permanent memory and and display on the handset. Implementation of image processing algorithm using Open Source Computer Vision Library (OpenCv), with Visual C++ Integrated Development Environment (IDE) and application Graphical User Interface is designed using Qt. Images are preprocessed with several image processing techniques, such as, RGB to Grey scale conversion, smoothening using Gaussian Filter, Histogram equalization, Detect Border (cumulative sum) and Set Region of Interest (ROI) crop. This developed software integrated in cells by the cross compilation of Qt and the OpenCV software for an Operating system. The result utilized the CANNY method in finding edges of the image using OpenCV. The integrated approach succeeded in cell phone by detecting the edges, chopping the images to required format of individual units, then stored it using OpenCV and finally displayed on a cell phone without any loss of image information. The impacts of memory latency, synchronization and the data management overhead are also studied and presented in this paper. Keywords: OpenCV, Qt, Image Processing Library, Region of Interest, Image Processing, Smart Phone, Embedded Platform.
978-1-4577-0444-4/11/$26.00 ©2011 IEEE
Deepthi.R.S2 Manipal Centre for Information Science Manipal University Manipal, Karnataka, India
[email protected]
I. INTRODUCTION The problems faced by the development of image processing applications on cell phones are with limited memory to store the images and the intermediate calculations. The operating system runs on the hardware do not provide complex image processing libraries. Cell phones do not contain arithmetic units with dedicated for performing floating point operations in most applications [1]. The mobile devices with cell animation and applications which include smaller display resolution, low quality image capture are the key challenges [2]. Most of the present smart cell phones from Nokia use an ARM9 processor as symbian operating system with internal memory 64MB to 128MB [3]. The application requires minimum symbian 3rd edition which provides support for the Qt libraries. Music, movies, books are outdated technology, future looks on comic technology. There are some technologies are available to read digital comics, but they are not up to the mark for 3D comics etc. OpenCV is a open source library, which is widely used in image processing applications to accomplish real-time credit for embedded appliances [4]. OpenCV library is written in C, with better portability. This can easily compiled for lot of embedded structures. OpenCV library includes with fundamental image functions: logical, arithmetic. It also contains object detection and object tracking. Qt is a crossplatform application. Qt permits developers to write applications once, then those applications can be installed on desktops and embedded operating systems without rewriting the source code. This paper describes how efficiently image processing applications can be run on cell platform. The results are obtained show without loss of image quality. The remainder of this paper is structured with five sections. Section 2, tells previous works on cell platform. In Section.3, the simulation environment and the experimental methodology to evaluate the dynamic library functions are presented. In Section.4, the implementations of cell application using Qt are discussed in detailed. In Section 5, the extensive simulation results, analysis and the performance of the image quality are
47
presented. Section 6 consists of the conclusion and the future work suggested. II.PREVIOUS WORKS ON CELL PLATFORM Since OpenCV library has many useful functions, which is widely used in many applications. Willow Garage’s personal robot was developed an image Recognition system with OpenCV library [5]. Stanley also used OpenCV library for his image processing engine. J.-P. Farrugia et al. used GPUCV library, this needs additional computational resources, more rapid processing and also uses huge power [6]. Even though GPUCV library is set of OpenCV image tasks, these can’t used in embedded platforms such as cells, automotives and robots, because of power constraints. In Masayuki Hiromoto et al. paper gives embedded applications, such as cell, robotics. This requires special embedded processors to achieve image recognition within system’s size and less power [7]. Clemens Arth et al. was developed the specialized DSP based enhanced system for image recognition [8]. To overlay these issues on embedded applications, we propose optimized OpenCV implementation for image applications for cell platforms. III. METHODOLOGY AND SIMULATION ENVIRONMENT In this section the tools and methodology to implement and evaluate the dynamically loading IPP with OpenCV libraries are detailed. The Intel Integrated Performance Primitives (IPP) is used to collect automatically OpenCV functions, to optimize the processors on the assembler level. So the task is to make those image processing libraries available on the development environment. This iss performed with the image processing applications in real time. The computations on the processing are modeled as cycle-accurate. The experimental results focused on the more complex comic images. We have used QT SDK by Nokia v210.05for the implementation of image application. This is developed by using OpenCV 2.0.0.CMAKE 2.8 GUI for cross compiler OpenCV 2.0.0 for Microsoft Visual studio C++ 2008 Express edition to get the image processing libraries. The simulation is conducted with Visual studio 2008 on Dell platform. Dell Laptop built with Intel Core2 Duo CPU T5750 operated on Windows XP OS, running at 2.0GHz with 32KB L1 D-Cache, 32KB L1 I-Cache 2MB L2 cache with 8-way set associative and 2GB RAM. Then using Qt, cross compiled images loads into Nokia cell operated with Symbian OS to display images without loss of quality of image. IV. IMPLEMENTATION OF CELL PLATFORM USING OpenCV The image processing algorithms are implemented using OpenCV. OpenCV focused on real-time functions to produce high efficiency with more computational resources. OpenCV is written in C with optimized manner, so this could benefit of multi-core architectures. The library is modified
with C language according to our cell platform for image processing application on Windows. Figure1 shows the proposed implementation of image processing application on cell. Create frame
HOME
START
CAMERA Open
Capture
Not ok
DISPLAY THUMBNAILS
VALIDATE
Select
READ FILE
Ok DISPLAY
PROCESS
FRAME
IMAGE
Save Figure1: Implementation of image processing on a cell
Opening of the application it will display home screen which has four options: A. Create a strip The application will start the camera and initiates it according to the .config file and has the standard buttons which are default to the handset, which basically includes capture button, settings and cancel button. The user can take picture by pressing capture. When the user takes picture the application will start all the image processing steps. If it succeeds the images will be stored in the specified directory in .config file and displays the images. If user press cancel button it opens the main screen. If the user press settings the camera settings will be opened and changed values will be written to .config file B. O pen a strip The application goes to the default directory and checks for the images and displays as list/thumbnails. So, the user can select the image. When image is selected it will display in full screen. It also has a button for going back to the image selection screen. C. Help Opens a file in the text edit box which is only readable, where instructions for using the application is given D. Exit: Exit the application. The Figure2 shows the proposed flow of the image processing application on a system. In this, first it starts with image capture, that captured image would be stored as temporary image in temporary buffers. Temporary image converted to grey scale using RGB to grey scale converter. Grey scale image smoothened by using Gausian filter. Once smoothens the image, that equalized by using image histogram and so on.
48
The detailed steps explanation has given with individual blocks using figure 2. Capture Image
Save as temp image
RGB to Grey scale
Smoothen Gaussian Filter
Histogram equalization
One of the simplest operations in image processing is blurring an image. Blur, is a process done to reduce noise. As this can be achieved in different ways, this paper demonstrates one method called Gaussian blur. This is achieved by convolving the image with a Gaussian filter. The libraries, OpenCV includes a gaussian filter that can be applied to an image by calling the cvSmooth function. Convolution of image with Gausian function is a Gausian blur. Based on visuality circle, convolution gives clear representation to reproduce the bokeh effect [9]. Fourier transformed Gausian function gives Guasian function only. High frequency components minimized by using the Gausian blur. Hence, Gausian blur can be treated as low-pass filter. While applying low-pass filter, its mask waits for neighboring values in the image, which results on decreasing the difference between these values. Therefore this procedure smoothes sharp edges in the objects and causes effect of blurriness. If the given image is sharp, practically it may take a minimum of about 3– 5 iterations of low-pass filter to obtain a noticeable blurred image [10]. The Figure 4 shows the algorithm of Gausian blur with OpenCV.
No Is OK? Yes
Detect Border (cumulative sum)
Get temp image
Set ROI crop and save Figure 4: Algorithm Gaussian Blur with OpenCV
End Figure 2: The flow of Image processing application on Cell
A. Capturing Image The Figure3, demonstrates the algorithm of opening a new camera capture device to capture one frame, then open new window to display the result with the use of OpenCV libray.
Figure 3: Image Capture and display with OpenCV
B.
Grey Scale Grey scale gives the value of every pixel separate sample. Grey scale holds only intensity information. Grayscale images are also called as monochromatic, which denotes chromatic variation with absence. C. Gaussian blur
D. Image histogram An image histogram acts as a graphical representation of the tonal distribution in a digital image. Image histogram gives plot for individual tonal value with number of pixels. Image histogram increases the total contrast of lot of images. Image histogram enhances the global contrast of images. This technique helps in images having background and foregrounds. Also this technique could give better picture details of x-ray images. The great benefit of this technique is, it could be straightforward and an inverttible operator. E. Edge detection Detecting edges are important tools for image processing. Edge detection performs fundamental preprocessing step to image processing applications. Edges map have a significant role in application such as image categorization, image registration, feature extraction, and pattern recognition. An edge detector can be defined as a mathematical operator that responds to the spatial change and discontinuities in gray levels of pixels set in an image [11]. Edge detection algorithms can be generally categorized as
49
parametric models, template matching, or gradient-based algorithms [12]. Most of edge detection operators operate on grayscale images since color image components impose some sort of complexities in detecting edges [13]. In grayscale images, an edge can be detected by investigating the discontinuity in gray levels [14,15]. However, in color images there won’t be clear and defined color edges. Although color images carry more information and have the potential of having more hidden and accurate edges [16,17]. Novak and Shafer found that 90 percent of edges are the same in both grayscale and color images, leaving 10 percent of edges to be detected through the use of color information [18]. The limitation factor in detecting all the edges in gray level images is due to the fact that different colors have different hue values but they share the same intensity value. Some of the common edge detection techniques for color images utilize fusion technique of edges detected at different color space using traditional edge detectors. There are two fusion methods used to find edges [19]. The first fusion method finds the gradient and the edge map at each of the three RGB levels, then fuses the resultant edge maps output. The second method fuses the gradients at each RGB component, and then finds the edge map of the final fused gradient. To make edge detection better manner, Canny used certain procedure [20]. The main consideration is image with edges should be taken care without missing any edges. Another consideration is edge points should be localized. Also should consider single edge has to single response.
data type is defined in CXCORE. Raw pixel data also included with more descriptive fields. These are width, height and depth of images. V. EXPERIMENTAL RESULTS AND DISCUSSIONS This section gives the detailed explanation of experimental results. In first stage, the simulation is conducted using OpenCV 2.0.0. The compilation of OpenCV libraries for visual studio C++ 2008 has done using CMAKE 2.8 GUI. The image of color to grey scale conversion is shown in figure 6. The image after smoothening is shown in figure 7. Histogram equalization achieves with effective manner widening the large amount of recurrent intensity values. By knowing the histogram equalization functions, actual histogram could be recovered with the help of image histogram. The image after histogram equalization is shown in Figure. 8.
F. Region of interest ROI is chosen based on subset of samples surrounded by dataset, which recognized for specific function. The ROI is set to the coordinates after calculating the cumulative sum and the ROI is cropped and saved to the permanent memory. ROI is a non-squared space of image, and for further processing it will splits as object. The Figure 5 shows the selection of subimage Using ROI
Figure 6: RGB to GreyScale image Conversion
. n Figure 5: Selection of subimage using Region of Interest (ROI) Figure 7: The image after smoothening
In figure 5, the ROI is present in upper left portion of image. When ROI located, then OpenCV functions would be functioned on the specific location. This provides beneficial, suppose if we have interest to collect an entity from an image or do pattern inside the sub-image. Within image, Region of Interest should present. G.OpenCV Images Store Using IplImage C structure, images swill be tored using OpenCV. IPL is inheritance of OpenCV version. IplImage
50
Region of Interest defined by using cvSetImageROI( IplImage* img, CvRect rect ). Where img will be source image. Here, rect will be area inside source image. The Figure 11 shows the image after ROI cropping.
Figure 8: The image after histogram equalization
In order to remove noise, edge detector smoothens the image. High spatial derivatives find image gradient to highlights regions. Hysteresis reduces the gradient array. Hysteresis is also tracks suppressed pixels. Two thresholds of the hysteresis tells based on the magnitude that has an edge or non-edge. If hysteresis lies in between the 2 thresholds, then it treated as zero. Suppose gradient is above T2, with path from pixel from pixel, then it can’t be treated as zero. To find the gradients, edge detection algorithm performs differentiation on an image. This derivative of image function determines the direction of the gradients. The gradients such as vertical, horizontal, diagonal up, diagonal down. Edge detection algorithm checks the gradient with the direction until the amplitude less than the lower threshold value. Edge detection algorithm also smoothens local non-maximums within the region of edges. The Figure 10 shows the image after the edge detection.
Figure 10: The image after ROI cropping
The simulated images using OpenCV 2.0.0. GUI with Visual Studio C++ and cross compiled by using Qt SDK by Nokia v210.05 as a cell platform. After cross compiling with the Qt the complete output of processed image is shown in Figure 11.
Figure 9 : Image after the edge detection Figure 11: Qt Creator Implementation using Qt SDK by Nokia v210.05
51
VI. CONCLUSION AND FUTURE WORK In this paper, we presented a method to capture an image from the printed media, then by applying the image processing algorithms to scaling and translation of the image, wipe out noise, do background subtraction, image rotation and detect edges. The developed software is able to slice or chop the comics strip along border into individual units, stored on the permanent memory and displayed on the handset.The result utilized the CANNY method for finding edges of the image using OpenCV. The integrated approach succeeded in detecting the edges, chopping the images to required individual units and to store it using OpenCV. This developed software can be easily integrated in cells by the cross compilation of QT and the OpenCV software for an Operating system. The future work can be extended to the computationally intensive applications of real-time video processing on cell platform using OpenCV and Qt.
[12]. Linear Gaussian blur evolution for detection of blurry images E. Tsomko1 H.J. Kim1 E. Izquierdo2 [13]. X. Wang and J. Jian-Qiu, "An edge detection algorithm based on Canny operator". IEEE Proceedings of the seventh International Conference on Intelligent Systems Design and Applications (ISDA2007),623-628, 2007. [14]. E.J. Wharton, K. Panetta , and S. Agaian, "Logarithmic edge detection with applications"., Journal of computers, 3(9), 2008. [15]. S. Wesolkwoski and Ed. Jernigan, "Color Edge Detection IN RGB using Jointly Euclidean Distance and Vector Angle". University of Waterloo,Waterloo Canada, Vision Interface, pp. 9-16, 1999 [16]. A. Koschan, "A Comparative Study on Color Edge Detecton", IEEE Proceedings of ACCV'95, Singapore, pp. 547-478,1995 [17]. JOHN CANNY “A Computational Approach to Edge Detection” IN IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 679-698, 1986.
REFERENCES [1] Khana, “The future of Cell Application Storefronts”, Wireless Expertise market report, 2009 [2] Fitzek and Reichert, “Cell Phone Programming and its Application to Wireless Networking”, 2007 [3] Jurgen Scheible and Ville Tuulos, “Cell Python: Rapid prototyping of applications on the cell platform”, 2007 [4] http://sourcefouge.net/projects/opencvlibrary. [5] Willow Garage, Inc., Willow garage. http://www.willowgarage.com/ (August 2009). [6] J.-P. Farrugia, P. Horain, E. Guehenneux, Y. Alusse, “GPUCV: a framework for image processing acceleration with graphics processors”, Proceedings of the ICME, 2006. [7]. Masayuki Hiromoto, Kentaro Nakahara, Hiroki Sugano,Yukihiro Nakamura, and Ryusuke Miyamoto, “Aspecialized processor suitable for adaboost-based detection with haar-like features,” Proceedings of CVPR, 2007. [8]. Clemens Arth, Florian Limberger, and Horst Bischof,“Real-time license plate recognition on an embedded DSP-platform,” Proeedings of CVPR, June 2007. [9]. S. Dikbas, T. Arici, and Y. A1tunbasak, "Chrominance Edge preserving Grayscale Transformation with Approximate First Principal Component for Color Edge Detection", IEEE Proceedings of ICIP Vol.11, pp.261-264,2007. [10] K. Panetta, S. Qasi, and S. Agaian, "Techniques for detection and classification of edges in color images", SPIE Proceedings, Vol. 6982, pp. 69820W69820W-11 , 2008. [11]. H.M. Merklinger, “A Technical View of Bokeh,” Photo Techniques,1997
52
