ARDIS: Knowledge-Based Dynamic Architecture for Real-Time Surface Visual Inspection D. Mart´ın1 , M. Rinc´ on2 , M.C. Garc´ıa-Alegre1, and D. Guinea1 1
2
Industrial Automation Institute, CSIC, 28500 Arganda del Rey, Madrid, Spain {dmartin,maria,domingo}@iai.csic.es Dpto. de Inteligencia Artificial, ETSI Informtica, UNED, C/Juan del Rosal 16, 28040 Madrid, Spain
[email protected]
Abstract. This work presents an approach to surface dynamic inspection in laminated materials based on the configuration of a visual system to obtain a good quality control of the manufacturing surface. The configuration task for surface inspection is solved as a Configuration-Design task following the CommonKADS methodology, which supports the proposed knowledge-based dynamic architecture (ARDIS). The task is analysed at the knowledge level and is decomposed into simple subtasks to reach the inference level. All the generic knowledge involved in the surface inspection process is differentiated among environment, real-time, image quality and computer vision techniques to be integrated it in ARDIS. An application has been developed to integrate four operation modes relating to visual system configuration and specific surface inspection. An example is shown for configuring a stainless steel inspection system and another one for wood inspection.
1
Introduction
The use of visual systems for inspecting surface defects has always been present in the laminated materials industry. Nevertheless these systems still present several drawbacks, such as reusing, as they are designed for a particular surface inspection application and do not offer the possibility of changing either the objectives or the inspection necessities. Nowadays, some authors propose specific surface inspection systems focused on a particular inspection task but lacking the formulation of a general purpose framework for surface inspection systems. The computer vision wizards assist to select manually the inspection type without using configuration parameters and integration of domain knowledge [4]. The injection of surface inspection knowledge of the human experts is an essential issue in the configuration process of any visual inspection system. For example, the change of the surface thickness or lighting in a production line J. Mira et al. (Eds.): IWINAC 2009, Part I, LNCS 5601, pp. 395–404, 2009. c Springer-Verlag Berlin Heidelberg 2009
396
D. Mart´ın et al.
implies a variation which is not considered in current inspection systems and need knowledge to adjust the surface inspection process. In case of surface thickness change, the distance between the camera and the surface changes and the camera gets out of focus generating blurred images, thus knowledge is necessary to solve it like human experts do. Therefore, current visual systems are designed for a specific application and are not ready to hold unexpected variations. This fact entails a high cost as each new visual inspection task has to be redesigned from the surface defect analysis to the overall inspection system by a human expert. The solution here proposed points to reusing the generic knowledge on surface visual inspection. To this aim generic knowledge is previously differentiated in types. This would allow changing or reconfiguring the components related to the application that vary, in the production line, due to changes on either environment camera, surface or defect type. The increase of the inspection systems and its complexity requires analysing complex tasks related to surface visual inspection to support surface quality control. The approach has to use the knowledge of the human experts to infer solutions (system configurations) to specific surface inspection problems (stainless steel or wood defects among other defects). Cognitive architectures are defined as knowledge based models that assist to organize data, information and inferences to solve complex tasks [5]. Firstly, current work deals with the complex task of visual inspection of laminated surfaces in industrial environments with a high variability in lighting, reflectance and real-time defect detection conditions. Secondly, the generic expert knowledge on surface visual inspection and the deep knowledge on the inspection preferences, restrictions and defect characteristics that exhibit the “line inspectors”, has to be used in the design of a visual inspection system. The remainder of this paper is organized as follows: In Section 2, we provide an overview of the ARDIS architecture. In section 2.1, we explain system requirements that compose the dynamic knowledge. Section 2.2 gives some guidelines of the generic domain knowledge on surface inspection and its types and influences among them. Section 2.3 points out the dynamic configuration using ARDIS architecture that has been proposed in the former section. Section 3 performs a number of experiments in a variety of different situations to obtain empirically some parameters from images for evaluating the configured system. Section 4 shows the SIVA II application tool that integrates four operation modes to achieve better overall performance for system configuration task. Finally, conclusions are discussed in Section 5.
2
ARDIS a Cognitive Architecture
A dynamic cognitive architecture is proposed, namely ARDIS [2], to integrate expert knowledge for real-time defects detection which offers an adaptive behaviour for detecting different defects or inspecting different type of surfaces. To this aim the ARDIS architecture is designed so as to be able to configure on
ARDIS: Knowledge-Based Dynamic Architecture
397
real-time a visual inspection system, adapted for each specific type of laminated surface and defect. Accounting for the characteristics of the industrial environment — under rigid quality control regulations, different and random types of defect to be inspected, specific real-time conditions and suitable computer vision techniques — the configuration of a visual inspection system requires a detailed analysis. Consequently, the ARDIS architecture has been designed following the CommonKADS methodology [6], considering real-time surface dynamic inspection as a complex task of the paradigm of design-configuration. The three terms relating to the ARDIS architecture are defined as follows: – The architecture term, for referring to a model which is decomposed in different functional modules. – The dynamic term, meaning that the inspection system depends on the environment, real-time, image quality and computer vision techniques requirements of the specific inspection application. – The cognitive term, as it analyses at the knowledge level the inspection problem of surface defects in laminated materials. The method used to solve the configuration task is Propose-&-Revise and is displayed in figure 1, showing the diagram of subtasks and inferences. This method shows how the ARDIS architecture operates in three steps: 1
INP UT Surface Inspection System Requirements
Configure Initial Skeleton
System Initial Skeleton (Initialisation)
Fuzzy or Usual Rule-Based Configure Initial Skeleton
2 Provide Global Prefe. & Cons.
Propose Initial Skeleton extension
Global Preferences
Fuzzy or Usual Rule-Based Compose Actual Skeleton current system skeleton extension
Global Constraints
Fuzzy or Usual Rule-Based Provide global Prefe. & Cons.
Surface Inspection System Global Configuration
3
O UT PU T
Fuzzy or Usual Rule-Based Modified System
Modify System
Global Repair
Fuzzy or Usual Rule-Based Selected System Repairs
Interpreted Image Fuzzy or Usual Rule-Based verified System
Propose System Repairs
Repair Global list
Global Constraint Violation
Verify System
Optimum Global System (Control Variable)
Fuzzy or Usual Rule-Based Global Repairs
Select System Repairs
Fig. 1. Subtask and inference diagram of the method Propose-&-Revise to solve the system configuration task
398
D. Mart´ın et al.
The first is the “initialization process”, which generates an initial skeleton of the visual inspection system that behaves as a seed for its subsequent extension into a more complex skeleton. The initial configuration is fast and allows the system to inspect with this basic configuration until the overall system is defined. The second is the “extension of the initial inspection skeleton” which configures an inspection system that considers all the specific requirements of the inspection application. The configuration is slower but the system is set more accurately as all the inspection information on the application and system components are available. The third is the “revision process” of the overall inspection system, configured in the former step. The revision will be explained in section 3. The next three subsections give a brief description of the requirements (dynamic roles — white rectangles in figure 1 —) and the generic domain knowledge on surface inspection (static roles — grey rectangles in figure 1 —) and how they are utilized in the ARDIS architecture to dynamic system configuration. 2.1
Dynamic Knowledge: Requirements
The dynamic knowledge that exhibits the “line inspectors” is used in the configuration of a visual inspection system. The configuration process starts with the selection of the requirements by the “line inspector” where the requirements have been previously differentiated into global, initial, environment, real-time, image quality and computer vision techniques to be used in the ARDIS architecture. Next, the differentiated requirements are divided into preferences and restrictions. For instance, environment requirements selected by “line inspector” and used in the inferences of the subtask “Propose Environment Skeleton Extension (PESE)”} — PESE is a subtask of “Propose Initial Skeleton extension”, see figure 1 — to inspect stainless steel or wood are shown in table 1. 2.2
Static Knowledge: Domain Knowledge on Surface Inspection
The generic domain knowledge on surface inspection is composed of different types of generic knowledge. This differentiation of the type of knowledge makes possible to distinguish the knowledge used on each inference, making easier the specification of components and its interactions. Thus, the designed architecture has been provided with all the necessary functionalities to configure a specific complete system of visual inspection where knowledge is partitioned to be ease reused. Table 1. Example of environment requirements selected by line inspector ENVIRONMENT REQUIREMENTS USED IN THE INFERENCES OF THE SUBTASK Propose Environment Skeleton Extension Stainless Steel Wood ProductionLineSpeed=1 m ProductionLineSpeed=0.1 m s s DefectOrientation={longitudinal} DefectOrientation={random} IlluminationType={GreenLaserSource} IlluminationType={ConventionalLight}
ARDIS: Knowledge-Based Dynamic Architecture
399
Table 2. Generic knowledge related to environment GENERIC KNOWLEDGE RELATED TO ENVIRONMENT USED IN THE INFERENCES OF THE SUBTASK Propose Environment Skeleton Extension DiffuseIllumination = {true, false} ExpositionTime = {long, medium, short} CameraSensorGain = {automatic, minimum, medium, maximum} PixelsPerDefect = {4pixels(2x2), 9pixels(3x3), ManualSelection} Camera-IlluminationRelativePosition = {0deg , 45deg, 90deg, ManualSelection} CameraPosition-InspectionPlane = {OverLaminate, UnderLaminate}
The generic domain knowledge on surface inspection is differentiated on global, initial, environment, real-time, image quality and computer vision techniques. Accordingly, the knowledge is distinguished to each inference and the knowledge can be reused. The generic knowledge, for instance related to the environment that will be used and reused in the configuration process, is presented in table 2. The system configuration process requires knowledge about the relationships between knowledge types that point out the influence among them. The influence graph is described in figure 2. As an example, the environment knowledge is related to real-time knowledge, such as image acquisition rate or lighting components can influence real-time components in one or another way. So, during the configuration step of an inspection system if an environment component is configured this has to be taken into account in the configuration of the real-time components. 2.3
Dynamic Configuration Using ARDIS Architecture
The dynamic configuration process is based on (i) ARDIS architecture, (ii) generic domain knowledge on surface inspection and (iii) “line inspector” knowledge who selects the requirements of the inspection system. This dynamic configuration process allows the inspection of each specific surface in a production line. Consequently,
Fig. 2. Relations between domain knowledge types of surface inspection
400
D. Mart´ın et al.
Fig. 3. Stainless steel images with micro oxide defects
Fig. 4. Wood images with knots and others defects Table 3. Example of the knowledge base which allows configuring the system KNOWLEDGE BASE USED IN THE INFERENCES OF THE SUBTASK Propose Environment Skeleton Extension IF ProductionLineSpeed is high THEN ExpositionTime is short IF IlluminationSystem is ExtLaserIllumination THEN CameraSensorGain is auto IF IlluminationSystem is ExtLaserIllumination THEN Camera-IllumiRelPos is 45deg IF IlluminationType is GreenLaserSource THEN ImageChannel is Green
it is possible to inspect stainless steel, wood, paper and plastic, among others in the same production line. Next, we choose stainless steel and wood to show how it is possible to configure two inspection systems based on the former premises. The figure 3 shows a set of images of stainless steel where it is necessary to inspect micro residual oxide scale defects on cold stainless steel strip. Figure 4 displays a new laminate material, wood, where knots and others defects have to be inspected in the same production line. The wood and stainless steel images have been acquired by an experimental system based on green laser illumination — diffuse lighting technique for surface inspection which consists of a green laser diode — that allows inspecting from micro to five millimetres defects, as the acquisition system uses magnification 1 [2]. The configuration process, accounts for the former requirements (table 1) and domain knowledge (table 2), which are codified by means of crisp and fuzzy rules. Using these rules and following the control structure of the ARDIS architecture [2] it is possible to configure the components of the surface inspection system. Thus, the solution is a configured inspection system which depends on the type of surface and defect to be inspected. The table 3 shows a set of crisp rules which have been used to configure the environment components of the inspection system.
ARDIS: Knowledge-Based Dynamic Architecture
3
401
Parameters from Images for the Architecture Revision Process
The aim of the “revision process” is to validate the configuration of the system by means of the acquired images. The analysis of the images and the numeric parameters for the revision process of the configured inspection system has been accomplished with the Image Processing Toolbox of MATLAB [3]. Parameters of environment (E), image quality (IQ), real-time (RT) and computer vision techniques (CVT) are differentiated and extracted from the original images. Once the revision parameters are obtained from images it is necessary to perform its evaluation. The evaluation of these parameters will generate the additional dynamic knowledge that is required to establish the revision process of the configured system through the acquired image. The knowledge, extracted in such a way, will be injected in the revision process by means of a set of crisp and fuzzy rules. In our architecture, we have selected the Mean-Shift algorithm [1] as a generic segmentation technique, due to its adaptive capacity to the different kind of images that can be presented for inspection. Mainly based on this segmentation method, Top-Hat filtering and grey-level co-occurrence matrix, the parameters chosen to evaluate the E, IQ, RT and CVT properties are the following (Table 4 summarizes the results obtained over two inspection configuration scenarios: stainless steel and wood): 1. Revision parameter of the environment (E): the lighting non-uniformity of the image is used to measure the influence of the environment in the image. The lighting non-uniformity is calculated by “Top-hat filtering” which is the equivalent of subtracting the result of performing a morphological opening operation — with a flat structuring element — on the input image from the input image itself. The numerical value of the environment revision parameter for each image is set as the difference between the standard deviation of the input image and the standard deviation of the filtered image. 2. Revision parameter of the Real-Time (RT): the computation time — in seconds — of the Mean-Shift algorithm is used to estimate a Real-Time parameter in the image, using the following Mean-Shift parameters: SpatialBandWidth(8), RangeBandWidth(4) and MinimumRegionArea(50). 3. Revision parameter of Image Quality (IQ): the image contrast property is used as Image Quality revision parameter and is calculated through the grey-level co-occurrence matrix of the image. This measure quantifies the intensity contrast between a pixel and its neighbour over the whole image. 4. Revision parameter of the segmentation technique (CVT): the Mean-Shift technique is used as the segmentation technique and the evaluation is based on the number of Mean-Shift regions located by the algorithm.
402
D. Mart´ın et al.
Table 4. Mean and standard deviation of the evolution of the variable lighting, MeanShift computation time, contrast and number of Mean-Shift regions
Lighting non-uniformity Computation time of the Mean-Shift Contrast Mean-Shift regions
4
Stainless Steel Mean Std 2.7623 3.4442 10.8926 0.8618 16.8680 7.5442 2249.6 801.7189
Wood Mean 24.3636 11.9844 4.7306 100.5714
Std 12.1547 1.0373 2.7405 82.6975
The SIVA II Interface Tool
The SIVA II tool is developed as a friendly and complete interface. SIVA II has been developed and implemented for configuring and operating each surface inspection system and designed according to the premises of ARDIS architecture. The main window of the application is displayed in figure 5. Four operating modes can be selected: – Mode 1. “Operation in the production line”. This mode is selected when the configuration of the inspection system is finished to show the inspection results. This mode is used by the inspection operator to validate the inspection results of the configured system and to decide between cleaning or refusing the laminated material if defects are present. – Mode 2. “Automatic configuration of the system”. This mode automatically configures the surface inspection system under the requirements selected by the “line inspector”, aided by the acquired image. This mode is used by the line inspector who knows the defect type and the specific requirements to carry out the defect inspection.
Fig. 5. SIVA II application main window
ARDIS: Knowledge-Based Dynamic Architecture
403
Fig. 6. (a) Window (left) for automatic configuration of the system. (b) Window (right) with inspection results obtained by the configured system.
– Mode 3. “Semi-automatic configuration of the system”. This mode configures automatically the surface inspection system, but allows changing manually the components that have been automatically configured in the former mode. This mode is utilized by the line inspector and allows adjusting the configured components that require a change. – Mode 4. “Add knowledge (in SIVA II)”. This mode is used only by an expert to inject knowledge that will be used later within the frame of the ARDIS architecture. The role of the expert in the process is previous to the system configuration; however he can introduce or change knowledge at any time for reconfiguring the surface inspection system. Finally, figure 6a displays the window of the application for automatic configuration of the system (Mode 2) and figure 6b displays the window with inspection results obtained by the configured system (Mode 1).
5
Conclusions
A knowledge-based architecture has been proposed to configure a specific surface inspection visual system to operate in dynamic environments. The architecture is instantiated to solve a specific surface inspection problem in a production line of laminated materials based on the requirements of the specific system and the identified generic knowledge on surface inspection. The ARDIS architecture allows real-time changes due to the capability to reconfigure a new inspection system in the production line whenever the inspection goal changes. The structure stages of the proposed architecture offers a high flexibility for adaptation to different specific inspections. The domain knowledge is organized so that it is possible to access to the specific knowledge used in each inference. The inferences use reusable generic knowledge
404
D. Mart´ın et al.
on surface inspection which has been differentiated in several types related to the environment, image control, real-time and computer vision techniques. Moreover, revision parameters have been obtained from images to the revision process of the configured system. Finally, an application (SIVA II) has been proposed and demonstrated to configure, either automatically or semi-automatically, each visual surface inspection system.
Acknowledgements The authors thank Acerinox, Malaga University, IMSE-CNM (CSIC), TCC and Anafocus, for fruitful cooperation. The work has been supported by Acerinox and TCC S.A. companies under the “Visual detection system and residual oxide scale classification in stainless steel laminates” project.
References 1. Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 603–619 (2002) 2. Mart´ın, D.: Arquitectura din´ amica para inspecci´ on visual de superficies en tiempo real. Ph.D thesis, UNED, Madrid, Spain (2008) 3. Matlab, The MathWorks Inc. Natick, MA, United States, http://www.mathworks.com/ 4. NeuroCheck. Industrial Vision Systems. Version 5,1,1065 [SP9], NeuroCheck GmbH (2006) 5. Rinc´ on, M., Bachiller, M., Mira, J.: Knowledge modeling for the image understanding task as a design task. Expert Systems with Applications 29, 207–217 (2005) 6. Schreiber, G., Akkermans, H., Anjewierden, A., de Hoog, R., Shadbolt, N., de Velde, W.V., Wielinga, B.: Knowledge Engineering and Management: The CommonKADS Methodology. The MIT Press, Cambridge (2000)
