An application of computer vision for programming

1

An application of computer vision for programming computer numerical control machines A E Eladawi1 , E S Gadelmawla2 *, I M Elewa2 and A A Abdel-Shafy2 Engineering Science Department, Faculty of Petroleum and Mining, Suez Canal University, Suez, Egypt 2 Production Engineering and Mechanical Design Department, Faculty of Engineering, Mansoura University, Mansoura, Egypt

1

Abstract: Generation of the part programs, or tool paths, for products to be manufactured by computer numerical control (CNC) machines is very important. Many methods have been used to produce part programs, ranging from manual calculations to computer aided design/ manufacturing (CAD/CAM) systems. This work introduces a new technique for generating the part programs of existing products using the latest technology of computer vision. The proposed vision system is applicable for two-dimensional vertical milling CNC machines and is calibrated to produce both metric and imperial dimensions. Two steps are used to generate the part program. In the first step, the vision system is used to capture an image for the product to be manufactured. In the second step, the image is processed and analysed by software specially written for this purpose. The software CNCVision is fully written (in lab) using Microsoft Visual Cþþ 6.0. It is ready to run on any Windows environment. The CNCVision software processes the captured images and applies computer vision techniques to extract the product dimensions, then generates a suitable part program. All required information for the part program is calculated automatically, such as G-codes, X and Y coordinates of start-points and end-points, radii of arcs and circles and direction of arcs (clockwise or counterclockwise). The generated part program can be displayed on screen, saved to a file or sent to MS Word or MS Excel. In addition, the engineering drawing of the product can be displayed on screen or sent to AutoCAD as a drawing file. Keywords: computer vision, edge detection, feature extraction, computer numerical control

NOTATION

1

ACA ADF CCD CCW CMA CW EDA FFE PFE PPA

Computer numerical control (CNC) technology has been one of manufacturing’s major developments in the past 50 years [1]. Generating the part program, or tool path, for the product to be manufactured by CNC machines is considered very important. Therefore, many techniques have been developed to generate CNC part programs, for example using developed software and computer aided design/manufacturing (CAD/CAM) systems [2–5]. Machine vision is concerned with utilizing existing technology in the most effective way to provide a degree of independency in specific applications [6]. In fact, the human visual system is extremely complex and cannot be simulated at all, but some of its properties can be simulated. Building a computer vision system that emulates the essential functionality of the human system may seem a deceptively small objective to the uninformed, but in fact it takes a great deal of work

arc calculation algorithm arc direction factor charge coupled device counterclockwise contour manufacturing algorithm clockwise engineering drawing algorithm final feature extraction primary feature extraction part program algorithm

The MS was received on 24 February 2003 and was accepted after revision for publication on 20 May 2003. *Corresponding author: Production Engineering and Mechanical Design Department, Faculty of Engineering, Mansoura University, 35516 Mansoura, Egypt. B03303

Paper

# IMechE 2003

B03303

INTRODUCTION

Proc. Instn Mech. Engrs Vol. 217 Part B: J. Engineering Manufacture

2

A E ELADAWI, E S GADELMAWLA, I M ELEWA AND A A ABDEL-SHAFY

[7]. Computer vision has been recently used in many applications in the field of production engineering. These applications include sheet metal cutting [8], lace cutting [9], feature recognition [10], reverse engineering [11], surface texture [12], dimensional measurements [13–15], defect analysis [16], tool wear assessment [17] and classification of machined features [18]. The process of extracting objects from an image using computer vision requires two steps: capturing the image and analysing it. Many hardware tools are available for capturing images, such as video cameras, digital cameras, scanners and frame grabbers with charge coupled device (CCD) cameras. The CCD cameras and the frame grabbers are suitable for industry applications because they are very sensitive and give highly accurate results. Analysis of the image requires software capable of performing various image analyses such as edge detection, feature recognition and feature extraction [19–21]. In this work, a unique vision system is introduced for generating CNC part programs for existing products that could be produced on two-dimensional CNC vertical milling machines [22]. 2

PROPOSED SYSTEM

Figure 1 shows a photograph of the proposed vision system. The system consists of two main parts, hardware

Fig. 1

and software. The hardware is represented by the capturing system and the PC computer, while the software is represented by the introduced software (CNCVision). 2.1

Capturing system

By referring to Fig. 1, the capturing system consists of three items. The first item is the backlighting table (1), which is a lighting box with a diffusing surface at its front, and is used to produce backlighting for the product to be captured (2). The second item is a CCD colour video camera (4) and a set of lenses with different focal lengths. The camera is carried by a camera holder (3). The third item is a 24 bit/pixel (ELF VGA) frame grabber video card (5), which is installed in the PC computer (6) and connected to the CCD camera. Capturing software (7) is provided with the frame grabber to acquire images and save it to files with different file formats. 2.2

CNCVision software

The CNCVision software is an extension and development of the DMVision software [13], which was used to extract the product dimensions from captured images. It is fully written in lab using Visual Cþþ 6.0. It features many image processing and computer vision

Photograph for the proposed vision system


B03303

# IMechE 2003

AN APPLICATION OF COMPUTER VISION FOR PROGRAMMING CNC MACHINES

algorithms to extract the product dimensions from the acquired images. The CNCVision software is a Win 32 application that can be executed in any Windows environment. It is customized to generate CNC part programs for existing products to be manufactured on

3

two-dimensional vertical milling machines. The main algorithms of the CNCVision software are described in section 4. 2.3

Output of the proposed system

The inputs for the CNCVision software are the captured images of the products to be manufactured. After applying various image processing and computer vision algorithms to the images, three outputs can be obtained, as shown in Fig. 2. The first output is a full CNC data sheet, which can be displayed on screen, saved to an ASCII file or sent to MS Word/Excel. The second output is a full CNC part program, which can be fed to a CNC machine or CNC simulator software, CNCEZ for example. The third output is a complete AutoCAD drawing for the product with the actual dimensions. 3

PROCEDURES OF WORKING

The product to be manufactured is set on the backlighting table, and then an image is acquired and saved to a BMP file using the capturing software provided with the frame grabber. The BMP file is then opened by the CNCVision software and converted to a greyscale image. Several image processing and computer vision techniques are applied to the image, and then the CNC data sheet, the CNC part program and the AutoCAD drawing can be obtained. Figure 2 shows the main working procedures of the proposed system. 4

ALGORITHMS OF THE CNCVision SOFTWARE

The following are the main algorithms used to build up the CNCVision software. 4.1

Fig. 2 Main procedures of the proposed system B03303

# IMechE 2003

Image processing and calibration algorithms

Several image processing algorithms are used to separate objects from the background and to extract the object features. These algorithms include converting colour images to greyscale images, extracting the grey level histogram, calculating a suitable threshold, generating a binary image, detecting edges of the product and extracting different features (lines, arcs and circles). The processing section in Fig. 2 shows the sequence of these algorithms. The edge detection algorithm extracts the pixel coordinates of the boundary edges and stores them into an array called the Edge array. The primary feature extraction (PFE) algorithm manipulates the Edge array to create a series of horizontal and vertical lines and stores them into an array called Feature1. The final feature extraction (FFE) algorithm manipulates the Feature1 array to extract lines, arcs and circles Proc. Instn Mech. Engrs Vol. 217 Part B: J. Engineering Manufacture

4


and stores them into an array called Feature2. The Feature2 array is used by other algorithms (mentioned below) to generate the CNC part program and the engineering drawing for the product. The system is calibrated for both metric and imperial dimensions. Therefore, the extracted dimensions can be calculated in either image units (pixels) or absolute units (millimetres or inches). The image processing algorithms and the calibration algorithm were explained in detail in previous work related to the DMVision software [13]. 4.2

The FFE algorithm extracts the coordinates of the two end-points for lines and arcs as shown later in Fig. 8. For circles, both points are the same. In order to obtain information about arcs, another point on the arc should be obtained. This point is obtained from the Edge array, which is created by the edge detection algorithm. The arc calculation algorithm (ACA) takes the point in the middle between the two end-points of the arc. In Fig. 3, the points (x1 , y1 ) and (x2 , y2 ) represent the start point and the end-point of the arc respectively. The point (x3 , y3 ) represents the point obtained from the Edge array. From these points, the ACA calculates the centre point of the arc from the equations ð y3 y1 Þ½ðx2 x1 Þðx2 þ x1 Þ þ ð y2 y1 Þð y2 þ y1 Þ ð y2 y1 Þ½ðx3 x1 Þðx3 þ x1 Þ þ ð y3 y1 Þð y3 þ y1 Þ cx ¼ 2½ðx2 x1 Þðy3 y2 Þ ð y2 y1 Þðx3 x2 Þ

ð1Þ ðx2 x1 Þ½ðx3 x1 Þðx3 þ x1 Þ þ ð y3 y1 Þð y3 þ y1 Þ ðx3 x1 Þ½ðx2 x1 Þðx2 þ x1 Þ þ ð y2 y1 Þð y2 þ y1 Þ cy ¼ 2½ðx2 x1 Þð y3 y2 Þ ð y2 y1 Þðx3 x2 Þ

ð2Þ where cx and cy are the x and y coordinates of the arc centre point respectively. The radius R of the arc is calculated from the equation qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi R ¼ ðx1 cx Þ2 þ ð y1 cy Þ2 ð3Þ

Using the ADF to calculate the direction of arcs


I ¼ cx x 1 ,

J ¼ c y y1

ð4Þ

Circular interpolation movements in CNC machines can be specified either clockwise (CW) or counterclockwise (CCW) by G-Codes G02 or G03 respectively. The arc direction is calculated using the arc direction factor (ADF), which calculates the position of the second point (x2 , y2 ) relative to the line connecting the start point and the end-point of the arc. The ADF is calculated from the equation ADF ¼

Arc Calculation Algorithm (ACA)

Fig. 3

The values of I and J are calculated from the equation

ð y1 y2 Þðx3 x1 Þ ðx1 x2 Þð y3 y1 Þ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi2 ðx3 x1 Þ2 þ ðy3 y1 Þ2

ð5Þ

If the ADF is less than zero, then the point (x2 , y2 ) lies to the left (looking from the start point to the end-point) of the line connecting the end-points of the arc. Hence, the arc direction is clockwise (CW). On the other hand, if the ADF is greater than zero, the point (x2 , y2 ) lies to the right of the line connecting the end-points of the arc and the arc direction is counterclockwise (CCW). If the ADF is equal to zero, the three points are collinear. 4.3

CNC part program algorithm (PPA)

This algorithm deals with the automatic generation of CNC data sheets and CNC part programs. Any CNC part program can be divided into three phases: program set-up, material removal and system shutdown. The program set-up data includes the positioning system (absolute or incremental), the type of units (metric or inch), coolant and clamps (on/off), spindle direction (CW/CCW) and the initial feed. These data could be specified through a user-friendly dialogue box as shown in Fig. 4. The system shutdown data are generated automatically according to the specified data for the program set-up. The material removal phase requires specifying the cutting conditions for each contour. Therefore, the

Fig. 4 Program set-up settings dialogue box B03303

# IMechE 2003


5

Table 1 Main G-Codes used by the CNC PPA G-Code

Objective

Used for

G00

Rapid movement

Tool movement from the zero point to the start point of the first contour Tool movement from the end-point of the last contour to the start point of the next contour

G01 G02 and G03

Linear interpolation Circular interpolation

G41 G42

Left compensation Right compensation

All extracted lines All extracted arcs according to its direction (CW/CCW) Holes with a diameter larger than the tool diameter Features (lines, arcs and circles) of all internal contours Features (lines, arcs and circles) of the external contour

contour manufacturing algorithm (CMA), explained next, deals with this phase. Features extracted by the FFE algorithm are classified into lines, arcs and circles. The CNC PPA calculates all necessary information for the CNC part program automatically from these features. This information includes the type of tool motion (G00, G01, G02, G03), X and Y coordinates for cutter movements, the radius (I, J) for circular interpolation and the cutter compensation (left or right). Features of all contours are extracted from top to bottom in a CCW direction as shown later in Fig. 7. The tool path is generated in the same manner. Therefore, the cutter compensation is considered to the right (G42 in this situation) for all objects of the external contour. On the other hand, the cutter compensation is considered to the left (G41 in this situation) for all objects of the internal contours. The main G-Codes used by the CNC PPA are given in Table 1.

4.4

Contour manufacturing algorithm (CMA)

If the captured image contains more than one contour, the contour manufacturing algorithm (CMA) marks all objects of each contour by a number corresponding to its contour. These numbers are used to differentiate between different contours and to enable the cutting condition parameters for each contour to be specified separately through a user-friendly dialogue box called contour manufacturing settings. Figure 5 shows the contour manufacturing settings dialogue box for the product shown later in Fig. 7. For this product, the contour list contains five contours. The external contour is always displayed first, and then the internal contours. The cutting condition parameters can be specified for each contour separately by selecting the contour number from the contour list and entering the parameters through the properties area. The selected contour will be displayed with a different colour in the preview area. The cutting condition

Fig. 5 Contour manufacturing settings dialogue box B03303

# IMechE 2003


6


Fig. 6 Contour manufacturing conditions dialogue box

Fig. 7 Acquired image for a simple product

parameters include speed, feed, tool number, tool compensation, depth of cut and tool approach. All contours can be manufactured using the same cutting conditions by checking the same settings check box. When more than one set of cutting conditions is used, the assigned cutting conditions

of all contours can be displayed by clicking the contour list button to display the contour manufacturing conditions dialogue box, as shown in Fig. 6. In addition, the cutting conditions can be saved and loaded through the save settings and the load settings buttons.

Table 2 Cutting condition parameters used to generate the CNC part program Contour list

Speed (r/min)

Feed rate (mm/min)

Tool number

Compensation (mm)

Depth of cut (mm)

Approach (mm)

1 2 3 4 5

1200 1000 1000 1000 1200

0.5 0.5 0.5 0.5 0.5

1 2 2 2 1

5 3 3 3 5

2 2 2 2 2

1 1 1 1 1


B03303

# IMechE 2003


7

Fig. 8 List of the extracted features

4.5

Engineering drawing algorithm (EDA)

The engineering drawing algorithm (EDA) is responsible for creating the engineering drawing from the extracted features. After calculating all information for the

extracted features, the EDA creates an AutoCAD script file including all necessary commands to draw the engineering drawing for the product. The script file could be opened from AutoCAD by executing the SCRIPT command and specifying the script file name.

Fig. 9 CNC data sheet dialogue box B03303

# IMechE 2003


8


After opening the script file by AutoCAD, it is easy to make any modifications to the drawing and it is easy to draw the dimensions of each object. 5

VERIFICATION OF THE PROPOSED SYSTEM

The accuracy of the proposed system was tested [13] and confirmed accurate results. Two methods were used to check the validation of the proposed system. In the first method, some simple products were manufactured in the Mansoura University Faculty of Engineering workshop and used to check the whole system. In the second method, the validity of the CNCVision software was checked by some complex images drawn by image processing software, typically Paint Shop Pro. 5.1

Verification for actual products

Figure 7 shows a captured image for one of the products tested by the proposed system after applying an edge enhancement algorithm to remove the noise around the edges. The vertical arrows in the figure indicate the starting points for the extracted contours, while the curved arrow shows the direction of extraction. The cutting condition parameters used to generate the CNC part program are shown in Table 2. A list of the extracted features for this product is shown in Fig. 8, and the CNC data sheet is shown in Fig. 9. Table 3 shows the generated CNC part program. The sharp corners marked by þ in Figs 7 and 9 will not be produced by the generated CNC part program; instead, these corners will be produced as curves with radii equal to the tool radius. In addition, because the positive direction of the Y axis is down in the proposed system, the generated CNC part program will produce the product exactly as if it is mirrored about the X axis. 5.2

Verification for complex shapes

To check the validity of the CNCVision software with complex shapes, many images with different shapes were drawn as products using the Paint Shop Pro (PSP) software. Figure 10 shows one of these images opened by the CNCVision software. The CNC part programming was created and tested by the CNCEZ simulator and the results were completely right. Figure 11 shows the engineering drawing extracted from the image shown in Fig. 7 after it was sent to AutoCAD. 6

DISCUSSION

As shown in Table 3, the whole CNC part program is obtained, including the tool movement for all contours. The arc direction and the values of I and J were calcuProc. Instn Mech. Engrs Vol. 217 Part B: J. Engineering Manufacture

Table 3

Generated CNC part program tested by the CNCEZ simulator

% :1000 N5 G90 N10 G21 N15 G42 R5.000 N20 G17 N25 G80 N30 M6 T1 N35 M3 S1200 N40 G0 X117.939 Y12.808 N45 Z1 N50 G1 Z-2 F0.500 N55 X11.659 Y53.442 N60 Y143.542 N65 X43.947 N70 G2 X69.059 Y167.833 J24.292 N75 G1 Y194.775 N80 X238.120 N85 G3 X268.614 Y164.300 J-30.475 N90 G1 Y49.467 N100 G1 X117.939 N105 G0 Z1 N110 M6 T2 N115 M3 S1000 N120 X206.730 Y34.008 N125 G1 Z-2 F0.400 N130 G41 R3.000 N135 G3 X206.730 Y34.008 I22.373 N140 G0 Z1 N145 X143.052 Y41.075 N150 G1 Z-2 F0.400 N155 G41 R3.000 N160 G3 X143.052 Y41.075 J15.183 N165 G0 Z1 N170 X47.086 Y88.333 N175 G1 Z-2 F0.400 N180 G41 R3.000 N185 G3 X47.086 Y104.233 J7.950 N195 G3 X96.414 Y88.333 J-7.950 N200 G1 X47.086 N205 G0 Z1 N210 M6 T1 N215 M3 S1200 N220 X130.944 Y95.400 N225 G1 Z-2 F0.500 N230 G41 R5.000 N235 Y143.100 N240 X229.152 N245 Y95.400 N250 X130.944 N255 G0 Z1 N260 G40 N265 X0.000 Y0.000 N270 M5 N275 M30

lated for all arcs. In addition, changing the tool and moving it from contour to another contour was achieved. The generated part program was tested by the CNCEZ simulator and no errors occurred, so it could be applied to vertical milling CNC machines to reproduce the original product. By obtaining the engineering drawing as an AutoCAD drawing, the drawing could be edited and modified for further extensions in the product shape. The lighting technique plays an important factor in the accuracy of the results. Perfect images with sharp edges could give B03303

# IMechE 2003


9

Fig. 10 Bitmap image opened by CNCVision

Fig. 11 B03303

# IMechE 2003

AutoCAD drawing for the image shown in Fig. 7 Proc. Instn Mech. Engrs Vol. 217 Part B: J. Engineering Manufacture

10


an accuracy of not less than 99.85 per cent of the original dimensions, as mentioned in the DMVision software [11].

7

CONCLUSIONS

Generation of CNC part programs is very important in the field of production engineering. A unique vision system is introduced as a new technique for programming vertical milling CNC machines. The system could be useful in the field of reverse engineering for reproducing existing products. Complete CNC data sheets and part programs could be obtained for existing products by capturing images for the products and then using the CNCVision software to generate the part programs. In addition, engineering drawings for the products could be obtained as an AutoCAD drawing from the captured images. Care should be taken while capturing images because the accuracy of the system is affected by the quality of the acquired images. The system was verified and proved to give good results for most captured images.

REFERENCES 1 Nanfara, F., Uccello, T. and Murphy, D. The CNC Workbook: an Introduction to Computer Numerical Control, 1995 (Addison-Wesley). 2 Gadelmawla, E. S. Development of programming system for numerically controlled milling machines. MSci. thesis, Mansoura University, Mansoura, Egypt, 1991. 3 Azza, F. B. and Mahrous, A. An introduction to CAD/ CAM system for automatic generation of NC part program. Engng Res. J., 2000, 2(67), 53–65. 4 Mansour, S. Automatic generation of part programs for milling sculptured surfaces. J. Mater. Processing Technol., 2002, 127(1), 31–39. 5 Choy, H. S. and Chan, K. W. A corner-looping based tool path for pocket milling. Computer Aided Des., 2003, 35(2), 155–166. 6 Awcock, G. and Thomas, R. Applied Image Processing, 1995 (Macmillan Press, Basingstoke and London). 7 Vernon, D. Machine Vision—automated Visual Inspection and Robot Vision, 1991 (Prentice-Hall International (UK) Limited). 8 Venkat, A., Sam, A. and Yean, C. Intelligent CNC cutting of sheet metal parts using machine vision. Int. J. Ind. Engng Applic. and Practice, 1994, 1(4), 305–314.


9 Sherkat, N., Martin, J. and Thomas, P. Real-time vision for automatic lace cutting. Proc. SPIE, Int. Soc. Opt. Engng, 1994, 2183, 322–333 (Society of Photo Optical Instrumentation Engineers, Bellingham, Washington). 10 Tuttle, R., Little G., Corney, J. and Clark, D. Feature recognition for NC part programming. Computers in Industry, 1998-4, 35(3), 275–289. 11 Huang, C. and Motavalli, S. Reverse engineering of planar parts using machine vision. Computer and Ind. Engng, 19944, 26(2), 369–379. 12 Gadelmawla, E., Elewa, I., Soliman, H., Koura, M. and Maksoud, T. Assessment of surface texture using a uniquely featured computer vision technique. In Society of Manufacturing Engineering 4th International Conference on Machining and Grinding, Troy, Michigan, May 2001, MR01-233. 13 Gadelmawla, E. S. and Elewa, I. M. On-line measurement of product dimensions using computer vision. In Proceedings of 9th IMEKO Symposium on Metrology for Quality Control in Production—Surface Metrology for Quality Assurance, Cairo, Egypt, 2001-9, Vol. IV, pp. 1–10. 14 Zscherpel, U. and Bellon, C. Wall thickness estimation from digitized radiographs. In 7th European Conference on NonDestructive Testing, 1998-5, Vol. 3, No. 8. 15 Hirayama, K., Yokota, S., Hashimoto, K., Nakamura, M., Nakamura, K., Imanaka, T. and Katoh, M. Measurement of a flaw height by radiographic testing. In Proceedings of 14th World Conference on Non-Destructive Testing, New Delhi, India, 1996-12, Vol. 3, pp. 1313–1316. 16 Lehr, C., Feiste, K., Stegemann, D. and Liedtke, C. Three dimensional defect analysis using stereo radioscopy based on camera modeling. In 7th European Conference on Non-Destructive Testing, 1998-5, Vol. 3, No. 8. 17 Kurada, S. and Bradley, C. A machine vision system for tool wear assessment. Tribology Int., 1997, 30(4), 295–304. 18 Hummel, K. E. Coupling rule-based and object-oriented programming for the classification of machined features. In Proceedings of ASME Conference on Computers in Engineering, Anaheim, California, 31 July–4 August 1989, pp. 409–418. 19 Dizenzo, S., Cinque, L. and Leviaidi, S. Run-based algorithms for binary image analysis and processing. IEEE Trans. Pattern Analysis and Mach. Intell., 1996, 18(1), 83–88. 20 Canny, J. A computational approach to edge detection. IEEE Trans. Pattern Analysis and Mach. Intell., 1986, PAMI-8(6), 679–698. 21 Wang, Y. I., Wang, X. J. and Shenghua, Y. E. Study on vision recognition, position, and orientation in a flexible automatic coordinate measuring system. Proc. SPIE, Int. Soc. for Opt. Engng, 1996, 28(99), 517–526. 22 Eladawi, A. E. An application of computer vision techniques for programming CNC milling machines. Master’s thesis, Mansoura University, Mansoura, Egypt, 2001.

B03303

# IMechE 2003