Computers in Industry 56 (2005) 997–1004 www.elsevier.com/locate/compind
Machine vision for feedback control in a steel rolling mill Paul O’Leary Chair of Automation, University of Leoben, Peter-Stunner-Tunner-Strasse 27, Leoben, Austria Received 1 December 2004; received in revised form 31 March 2005; accepted 31 May 2005
Abstract This paper presents an application of an image processing system in the monitoring and control of the hot-rolling of steel bars. A robust implementation both with respect to camera packaging and data transmission has been choosen. A gigabit ethernet network is used for all communication within the system, whereby a switch with integrated fire-wall is used to distribute the data and provide connection to the outside world. The image processing algorithms use statistical estimates of confidence to ensure robust operation under extreme conditions. Feedback-control requires a sampling rate approximately 10 times higher that the bandwidth of the system which is to be controlled [K.J. Astrm, T. Hgglund, PID Controllers: Theory, Design, and Tuning, Instrumental Society of America, 1995 [1], Jukka Lieslehto, Wap application for pid controller tuning, in: Proceedings of the 2000 IEEE International Symposium on Computer-aided Control System Design, vol. September 25–27, Ancorage, Alaska, USA, 2000, pp. 168–172 [2]]. The system demonstrates the acquisition and processing of 500 frames per second in real time, this is commensurate with 50 Hz bandwidth presently possible for hydraulic systems. This opens the door to real-time feedback control using digital image processing. # 2005 Published by Elsevier B.V. Keywords: Machine vision; Active snakes; CMOS cameras
1. Introduction The successful application of machine vision in harsh industrial environments requires many issues to be considered: (1) robust image acquisition, which can withstand the harsh environmental conditions associated with industrial production plant;
E-mail address:
[email protected]. 0166-3615/$ – see front matter # 2005 Published by Elsevier B.V. doi:10.1016/j.compind.2005.05.023
(2) mechanisms for the transmission of the images which are not susceptible to the electro-magnetic disturbances which are associated with large electrical drives; (3) theoretically well founded image processing algorithms, which enable the integration of apriori knowledge. In particular the question of confidence interval in the measurement data should be addressed; (4) access to and the generation of input/output signals to control machines and plant (IEC-61131 compatible);
998
P. O’Leary / Computers in Industry 56 (2005) 997–1004
(5) coordination of multiple measurements, since it may require data from different sources to make the desired decision with respect to quality control; (6) integration of the results in an enterprise quality control system. A machine vision application in a steel rolling mill is presented; whereby, all the above issues are addressed. The rolling mill at hand is a reversingmill with six stands, three horizontal and three vertical; one stand is shown in Fig. 1. The mill is primarily for hot-rolling of flat-blocks. The vision system should fulfill two main tasks: Fig. 2. Example of catastrophic failure due to looping.
(1) Discrepancies between the rolling speed and thickness reduction is consecutive stands can lead to looping of the material. Catastrophic failure occurs when the magnitude of the looping exceeds certain limits: this leads to a complete interruption of production and commonly requires the rolls to be exchanged, Fig. 2 shows a catastrophic failure. A method for the measurement of the inter stand material geometry have been published [3]. However, the difficulty of the measurement in the present application is exasperated by the cooling water, which flows over the material during rolling. This obscures locally the view of the material.
Fig. 1. One of the six stands, showing the rolling gap with red hot glowing material.
The vision system should be capable of measuring the onset of looping in real time, determine its magnitude and send suitable signals to the mill-control system so as to prevent catastrophic failure. (2) The correct operation of the system controlling the position of the rolls can only be initiated when it is known that the material is between the rolls. Presently, the material is detected by the force it exerts when it enters the rolling-gap. The control system has a lag-time before it can correctly control the thickness of the material; consequently, the ends of each rolled steel-bar do not fulfill the required specifications. It is a further goal of the image processing system to measure the position and speed of the material, and given this data to predict the timepoint when the material will enter the rollinggap. This should reduce the amount of material at the ends of the steel-bar, which must be scrapped. Both the above tasks need to be performed in real time, whereby, the hydraulics have a bandwidth of approximately 50 Hz. Several hundred images per second must be acquired and processed, to ensure a stable control system. A closer look at the material in the rolling gap (see Fig. 3), reveals some of the problems associated with this environment: e.g. cooling water and steam can obscure a clear view of the material. A correct and reliable measurement is required despite these conditions.
P. O’Leary / Computers in Industry 56 (2005) 997–1004
999
Fig. 3. Hot material in the rolling gap. Note the cooling water and steam impairing the view of the material.
2. Image acquisition and communication concept The images which need to be acquired have an unusual aspect ratio of approximately 10:1 (see Fig. 3). This has lead to the selection of a CMOS camera (JAI CV-A33), with which the speed of acquisition can be increased by reducing the number of rows, which are acquired. The camera has 640 480 pixels and can acquire 117 full frames per second. In this application a region of interest with 640 64 pixels is used, theoretically this enables the camera to acquire over 850 such frames per second. However, a frame rate of 500 per second has been chosen; this corresponds to 10 times the bandwidth of
the servo-hydraulic system, more was not considered necessary. The CMOS camera has a camera-link1 interface, which was considered unsuitable for the transmission of data over longer distances in an industrial environment. Consequently, a camera-link to gigabit ether-net converter was selected, this enables the use of standard ether-net cabling (1000-Base-TX, SSTP double shielded twisted pair RJ45) over distances of 100 m at 1 gigabit/s. Optical fibers can be used and are recommended (1000-Base-SX) for longer distances. The use of ether-net for the transmission of images, leads to a consistent communications concept based on standard components: switches, routers, etc. (see Fig. 4). The system includes a gigabit switch with
Fig. 4. Ether-net based communications concept. All the components associated with the image processing system are behind the fire-wall, protecting them from external interference.
1000
P. O’Leary / Computers in Industry 56 (2005) 997–1004
image processing [4–6]. The evaluation of the image can be considered in two major sections: (1) Detection and tracking of the leading edge. (2) Determination of the vertical displacement of the material. 3.1. Detection of the leading edge
Fig. 5. Camera housing, containing the camera, camera-link to gigabit ethernet converter and power supply adapter. This provides a IP-67 protected image acquisition system, requiring a single 24 V power supply and an ether-net connection.
integrated fire-wall, all components associated with the image processing system are behind the fire-wall. Whereas, the external ports are used to provide connectivity to the plant database and for remote servicing. This concept enables the simultaneous routing of image data from multiple cameras to multiple computers for processing. The IEC 61131 compatible industrial IOs are implemented using an ether-net bus-coupler with a TBase 100 interface. The computer responsible for the communication to the IOs has two ethernet connections: a T-Base 100 for IO data and a T-Base 1000 for image data. This together with the speed of the bus provides sufficient bandwidth and avoids access clashes, ensuring a real-time response. The camera, a camera-link1 to gigabit ether-net converter, and the necessary power supply adapters, are packaged in an IP67 housing (see Fig. 5). In this manner only a industrial standard 24 V power supply and an ether-net CAT-5 connection are required. Furthermore, standard IP-67 connectors are available for ether-net.
3. Image processing The optical conditions generated by the environment and, in particular, the cooling water, require the application of robust numerical methods during
The regions at the left and right hand sides of the image are observed to detect the entry of the rolling material. Both sides of the image must be evaluated since this is a reversing mill, i.e. the material can enter from both sides. The leading edge is then tracked across the image. A statistical approach has been applied to the detection of the leading edge. The concept is to determine if the probability density of the intensity of the pixels in a particular column is significantly different from the distribution of the image noise. This is calculated in the following steps: (1) The intensity vector ic(y) of the column y to be considered is determined: ic ðyÞ ¼ iðx; yÞ;
8x
(1)
where i(x, y) is the intensity map of the image. (2) The column data is made mean free: (2) icm ðyÞ ¼ ic ¯ic ðyÞ: (3) The standard deviation sim of the image noise is determined. (4) The percentage of pixels np in ic(y) lying outside the 3sim boundary is determined. (5) A column is deemed to have systematic content (not just noise) if np exceeds a given threshold, this corresponds to a specific confidence interval [7]. The position of the leading edge xle is considered to correspond to the last column which has a systematic content, note: a continual progress of the block is imposed, i.e. the leading edge is not permitted to retract. This eliminates errors due to temporary occlusion of the leading edge due to the cooling water. The result of this calculation is visualized in Fig. 6 for three different positions of a block. The results of this method have proven to be very robust.
P. O’Leary / Computers in Industry 56 (2005) 997–1004
1001
Fig. 6. Estimation of the leading edge of a block for three different positions, the mono-chromatic portion of the figures corresponds to the original un-processed image. The red marking is proportional to the number of pixels lying outside the 3sim boundary. The vertical blue line is the predicted position of the leading edge.
3.2. Tracking of the vertical displacement The second main task is to determine and track the vertical position of the material. This problem is made considerably more difficult by the cooling water and steam, which lead to total occlusion of parts of the material. Furthermore, only limited computational time is available due to the speed of the application. The solution chosen uses: (1) Temporal extrapolation from one image to predict where the material is to be found in the next image. This limits the area of the image, which needs to be processed and consequently reduces the time required for the numerical computation. (2) Spatial approximation and interpolation with splines [8]. This ensures a robust prediction of the material position even in areas of occlusion. The algorithm proceeds as follows: (1) Start-up phase6 The program waits until the leading edge has passed the first 20 columns within the image. The first moment of the pixel intensity is calculated for each of the 20 columns: Pymax kiðx; kÞ ymp ðxÞ ¼ Pk¼1 (3) ymax k¼1 iðx; kÞ This value is used as the first prediction of the vertical position of the material.
A first estimation of the coefficients C, for the splines is determined by spreading them evenly in x and assigning the mean value ymp, for n control points: ixle Ci ¼ ; y¯ ðxÞ (4) n mp where xle is the position of the leading edge. The pp-form [9,10] splines are used here with evenly distributed break-points and with the coefficients Ci. B-splines are more commonly found in the literature, however, the pp-form is numerically more efficient [7]. There is a closed form relationship between the pp- and B-forms. (2) Running phase The y-gradient of the image up to the leading edge is determined: diðx; yÞ ; 8 x 2 ½1; . . . ; xle (5) dy Starting from the predicted position: the first maximum and minimum of the gradient which exceeds a given value is selected as an estimation of the upper and lower edges respectively: ry ðx; yÞ ¼
yt ðxÞ ¼ maxfry ðx; yÞg; and 8 x 2 ½1; . . . ; xle
8 y 2 ½ym ; . . . ; 1
yb ðxÞ ¼ maxfry ðx; yÞg;
8 y 2 ½ym ; . . . ; ymax
and 8 x 2 ½1; . . . ; xle
(6)
(7)
The mean of the top and bottom positions is calculated as an initial prediction of the vertical
1002
P. O’Leary / Computers in Industry 56 (2005) 997–1004
Fig. 7. Three different positions of the material during the first pass. The mono-chromatic portion is the original image. The red and green dots correspond to the locally detected upper and lower edges of the material respectively. The blue circles correspond to the spline approximation and the solid blue line to the spline interpolation. The solid blue line is the predicted vertical position ym(x) of the center line of the material. The predicted end of the material and the vertical position at five points are presented numerically (here the number corresponds to pixels). Note: the blue line extends beyond the predicted end of the material, this is an extrapolation and is used for the detection of the material in the next image.
position of the material: yt ðxÞ þ yb ðxÞ ; 8 x 2 ½1; . . . ; xle (8) 2 Predictions are eliminated from ymp(x) for which the intensity in the corresponding column is below a predetermined threshold. This eliminates the evaluation of portions of the image where occlusion has occurred. A nonlinear least mean square optimization of the pp-spline coefficients is performed to determine the best approximation. The coeffiymp ¼
cients are initialized at the previous values of Ci. There are only small changes in Ci from image to image since there is only little motion in the elapsed time. This makes the fitting numerically efficient since, Ci, are initialized close to their final values. The error vector for optimization is the difference between the spline interpolation vector ym(x) and the values ymp(x): eðyÞ ¼ ym ðxÞ ymp ðxÞ
Fig. 8. Three different positions of the material during the second pass (see Fig. 7).
(9)
P. O’Leary / Computers in Industry 56 (2005) 997–1004
The final value of the spline interpolation ym(x) is used as the prediction of the vertical position of the material. 3.3. Discussion of the results The algorithms were tested on multiple image sequences each consisting of 3000 frames. The image sequences used were acquired in real-time at the rolling mill, but were processed off-line. The result of this algorithm are presented for a first (Fig. 7) and second pass (Fig. 8). These have been chosen since they represent the rolling of thick and thin material respectively. The danger of looping is considerably higher for thin material as can be seen in Fig. 8. The algorithm produces stable results under all tested conditions. In Fig. 8, considerable looping can be seen and a stable prediction of the vertical position of the material even in areas where the cooling water has lead to total occlusion. The light from the spontaneous emission of the material is also reflected from numerous portions of the rolling stand (see Fig. 3). In some cases the gradient of the light intensity at these reflections is steeper than at the edge of the material itself. The detection of the top and bottom edges starting from a prediction of the position of the material prevents the erroneous detection of an edge at a reflection (see Fig. 7).
4. Conclusions A robust system for image acquisition, suitable for use under harsh industrial conditions (IP-67 protection) has been demonstrated. The gigabit ether-net based transport of image data has proved stable and robust in an industrial environment. It brings many advantages with it: for example, standard switches, bridges, etc., can be used to route the data to the required processing node. Furthermore, it supports a bi-directional communication with the camera enabling the configuration of the camera under working conditions. The communications concept supports multiple cameras and multiple processing nodes. Statistical techniques have been used during image processing to estimate the confidence interval asso-
1003
ciated with each measurement. These techniques have proven to be very robust, even the temporary occlusion of the material by cooling water has not disturbed the measurement. The use of statistical techniques are considered to be a key issue for the implementation of future image processing systems in such environments. The rate of 500 frames per second opens the door to real-time feedback control using digital image processing.
Acknowledgements Much of the original work on image acquisition was performed by Andreas Mundler and Markus Leitner during a feasibility study, images of the equipment were produced by Christian Sallinger. I wish to thank Boehler Edelstahl, Kapferberg, Austria for their support in this project and for enabling the measurements in their plant.
References [1] K.J. Astrm, T. Hgglund, PID Controllers: Theroy, Design, and Tuning, Instrument Society of America, 1995. [2] Jukka Lieslehto, Wap application for pid controller tuning, in: Proceedings of the 2000 IEEE International Symposium on Computer-aided Control System Design, volume September 25–27, Ancorage, Alaska, USA, 2000), pp. 168–172. [3] D. Sollander, S. Linder, Idc-interstand dimension control. ABB Technik, No. 1., 2000. [4] S. Milan, V. Hlavac, R. Boyle, Image Processing, Analysis and Machine Vision, 2nd ed., Brooks Cole Publishing Company, London, 1999. [5] B. Jaime, Digitale Bildverarbeitung, Springer, Berlin, 2002. [6] C. Russ John, The Image Processing Handbook, 2nd ed., CRC Press, Inc., Boca Raton, 1995. [7] W.H. Press, S.A. Teukolsky, W.T. Vetterling, B.P. Fannnery, Numerical Recipes in C, 2nd ed., Cambridge University Press, Cambridge, 1995. [8] P.C. Yuen, G.C. Feng, J.P. Zhou, A contour detection method: initialization and contour model, Pattern Recognition Letters 20 (1999) 141–148. [9] C. de Boor, Efficient computer manipulation of tensor products, ACM Transmission Mathematical Software 5 (1979) 173–182. [10] C. de Boor, Spline Toolbox: for use with MATLAB, 3.0 ed., Math Works Inc., Natick, 2000.
Paul O’Leary was born in 1960 in Ireland, he studied Mathematics and Electronics Engineering at Trinity College, did his Masters
1004
P. O’Leary / Computers in Industry 56 (2005) 997–1004
Degree in Electronic Engineering at the Philips International Institute in Eindhoven, Netherlands and his PhD at the University of Pavia, Italy. He worked from 1984 to 1987 in the design of Analogue-Integrated Circuits at ITT Intermetall in Freiburg, BRD. In 1987 he moved to Austria where he worked at AMS as head of the Analogue Integrated Circuits Gropu. In 1990, He founded the Institute for Chemical and Optical Sensors at Joanneum Research in Graz Austria. He received the chair of Auto-
mation at the University of Leoben, Austria in the fall of 1995 and was director of the Christian Doppler Laboratory for Sensor and Measurement Systems from 1996 to 2003. He is a fellow of the Christian Doppler Research Society. He is interested in metric vision for automatic quality control and the automation of industrial processes. The area of geometric surface inspection has been at the centre of his research in recent years.
