Implementation of Webcam-based Touchscreen Hoang Minh Phuong, Nguyen Tien Dzung, Thang Manh Hoang
'
Faculty of Electronics and Telecommunications, Hanoi University of Science and Technology, Hanoi, VIETNAM ,
[email protected] Abstract-This paper presents implementation of webcam-based touchscreen where the main tools used in this work is image processing techniques. Two cameras are employed for acquiring sequence of images. Touching object is recognized and its location is identified. Experiment is carried out to demonstrate and verify the effectiveness of proposed method.
Keywords- innovative touchscreen; image recognition.
I.
INTRODUCTION
Nowadays, touchscreen is widely used in many screen based display devices such as the personal digital assistant (PDA), satellite navigation devices, mobile phones, and video games, etc. While the demand is increasingly growing and the cost for touchscreen in the display device is required as low as possible, conventional touchscreens are more expensive in compared with the screens without touch function due to the presence of touching detectors. There are several types of touchscreen technologies [1] such as resistive touchscreen, surface acoustic wave, capacitive sensing, coded LCD, etc. This paper describes a novel method to realize a touchscreen using webcams. That provides a potential to utilization in inexpensive screen-based display devices, and it can be add the touching function to nonnal screens. The approach allows recognizing any touching objects such as a finger, pen, etc. The experimental result will show both the operation and perfonnance of the system. The paper is organized as follows. In section II, we briefly define the touchscreen using image processing. In section III, we discuss the algorithm of image recognition. The implementation and precision are given in Section IV. Finally our work is summarized in Section V. II.
THE PROPOSED WEBCAM-BASED TOUCHSCREEN
The system consists of a personal computer (PC) connecting with a screen and two webcams. The PC is for image acquisition from webcams and for locating a touching object on the screens. As depicted in the Fig. 1, one webcam (WC2) is placed in front of screen to detennine relative location of touching object, and the other (WCI) is at the side of screen to detect whether the object touches the plane of screen. The resolution of webcams is required no too high, and we use 0.7 M pixels and l.3M pixels for WCI and WC2, respectively. And, the PC should be chosen as high speed as to response almost real time under heavy load of image processing. For convenient, touching object assumed is a finger in the following description.
978-1-4244-7057-0/10/$26.00 ©2010
IEEE
(b) Figure I.
III.
(a), (b) position of webcams IMAGE PROCESSING
In our method, acquired sequences of images from webcams are used for identifYing the finger and touch action. The procedure is as shown in Fig. 2. WCI and WC2 capture sequential images. Firstly, the sequence of images from WCI is to detennine the distance from finger tip to the plane of screen using motion detection method by comparing between two successive frames as depicted in Fig. 3, the current frame h(x,y), and the previous one 1; (x,y). The difference is an image which is computed as follow:
201
f(x,y) fz(x,y)- h(x,y) =
( 1)
by three stages, i.e. of preprocessing with the technique of Color Edge Detection using Gradient [6], intermediate processing with motion-based detection [6] and pattern recognition using template matching [7]. Template matching technique allows us to significantly reduce computational complexity in compared with other previous one of Prediction Update framework such as the Kalman filter [2], particle filters [3] , and others proposed by Isard and Blake [4], Martin [5]. Then, a projective transformation is applied to five points on the image at the time of touching action. A.
Color edge detection
Gradient-based color edge detection [6] is useful for feature extraction. Let c represent an arbitrary vector in RGB color space at the point with coordinates (x, y) by using the notation:
F
rCR(X'Y)1 r X'Y 1
c(x,y) = co(x,y) cB(x,y)
=
R( ) G(x,y) B(x,y)
(2)
For a scalar function,j(x, y) , the gradient is a vector pointing in the direction of maximum rate of change of f at coordinates(x, y). Let r, g, and b be unit vectors along the R, G, and B axis ofRGB color space and define the vectors.
Figure 2.
oR oG oB b u= - r+-g+ Ox Ox ox
proposed algorithm
(3)
oR oG oB v=-r+-g+-b oy oy oy
Due to the constraint of time response, there is no image preprocessing applied to images from WC 1.
(4)
Let the quantities gxx,gyy and gxy be defined in terms of the dot product of these vectors, as follows:
(5) (a)
(b)
gyy
T
v.v =vv =
=
oR
Oy
2
oG
+-
Oy
2
oB
+-
2
Oy
oR oR oG oG oB oB gxy =u.v=u v=--+--+-ox Oy ox Oy ox Oy
(6)
T
(7)
R, G, B, gxx,gyy,gxy are the functions of x and y. Using this notation, it can be shown that the direction of maximum rate of change of c(x, y) is given by the angle: (c) Figure 3.
(d)
(a) the previous frame; (b) the current frame; (c) difference between (a) and (b); (d) binruy image.
(8) Secondly, the purpose of WC2 is to determine the location of the finger tip in the area of screen. Finger tip is recognized
And, the value of the rate of change at (x, y), in the direction of B, is given by:
202
F(B) =
{�[
]y I
(gxx +gJY)+ (gxx -gJY)cos2B +2gJY sin2B
(9) Because tan(a)
=
tan(a ± 1%') , if 80 is a solution to Eq.
(9), so is Bo±1t12. Furthennore, F(8)= F(8+1r), so F
needs to be computed only for values of B in the half-open interval [0, 1r). Fig. 4 illustrates the stage of color edge detection.
(a)
(a)
(b)
(c)
(d)
Figure 4. (a) the previous frame- when the finger ' s near the screen; (b) the current frame- finger touches on the screen; (c) Color Edge Detection for the current frame; (d) Color Edge Detection for the previous frame B.
Motion detection
C.
Finger tracking based on template matching
(b)
Motion detection on a sequence of images from WC2 is realized as done with one from WCI. Moving objects at this context can be the moving finger or the effects of illumination variation. Let say illumination variation to be noise. A morphological operation is applied to reduce that noise with the number of pixels less than 20. After these operations, we obtain some moving candidates representing for the finger. In fact, we have only one intent finger.
After identifYing moving objects on successive frames, we can further recognize the finger that touches the screen based on technique of template matching. In practice, users often touch the screen by the finger tip. Hence, we mainly concentrate on the position their finger not whole parts of hand. The pattern dataset of 60x80 images with finger-like shapes is created. Then, the specific pattern in the dataset is chosen and used for matching with sub-images in the image. The correlation is calculated as follows:
(c) Figure 5. Motion detection from a sequence of images of WC2: (a) the difference between images Fig. 4(a) and Fig. 4(b); (b) binary image; (c) Image after morphological operation
203
•
r
=
m n r==� =========;=======�
( ��
(Amn
_A)2)(�� -s/)
•
(Bmn
(10)
where A and B are matrices of the same size, A and B are the average values of A and B.
Pentium(R) Dual CPU E2I60 @ 1.8GHz with the Windows XP operating system and LOG Ram
2 Delux webcams with the captured format and frame rate are: WCI: 320x280, 20 frames per second; WC2: 640x480, 20frames per second.
Ones have first manually initialized 2 points on the WCI and 4 points on WC2 to determine the scope of screen.
The sub image with the highest correlation coefficient among all sub images is determined. Coordinate of fingertip is mapped to the relative one on the screen, which is used for considered the cursor coordinate. In general case, images in the dataset should be large to enable to recognize various shapes of objects like a pen, a stick etc. However, this number shouldn't be too large for respondent time.
..
'The .,bdivision Ii"" 0 oolv us..! mn IBe inverse mo�9 method porometer is sello Quod,••• pproUnotioo.
Subdivisioo une Dually lOOD< • 1
VeOi
