Object-Oriented Image Processing in Smalltalk: Using Complex Operator Objects Michael Klupsch, Wolfgang Eckstein Forschungsgruppe Bildverstehen (FG BV) Informatik IX, Technische Universit¨at M¨unchen Orleansstr. 34, 81667 Munich, Germany E-mail: fklupsch;
[email protected]
ABSTRACT In this paper we present concepts for modelling image processing operators within an object-oriented system like the Smalltalk environment. The operators, which are described in a knowledge-base and receive their functionality from a large, proven, and well-established C library, are integrated into the Smalltalk environment in order to provide this environment with extensive image processing capabilities. Within the Smalltalk class hierarchy the operators are modelled as independent objects — as tools used for creating, manipulating, and evaluating images, regions, and features on a highly abstract level. Besides this functionality, the operator objects represent knowledge about the corresponding image processing procedures. This information attached to the operators is profitable not only for being questioned by the user, i.e., the software developer, but also for being evaluated by tools and applications based on operator objects. Moreover, this knowledge is used for the automatic generation of appropriate Smalltalk operator classes that build the interface to the underlying C library. In addition to the implemented representation of the operators, we will discuss the kind of representation, the scope, and the use of the operator knowledge. Keywords: Image Processing, Object Orientation in ISAS, Smalltalk, User Interface Design, Software Engineering, Knowledge Base 1. INTRODUCTION Among many others, image processing is a disciplin within computer science that produces a lot of complex data which are linked by means of more or less complicated algorithms. Big efforts are being made to find appropriate data structures to ease the handling of this data and hence to reduce the risk of errors. Due to their great advantages in modelling complex data, object-oriented languages and systems like C++ or Smalltalk are increasingly used. On the other hand the tools used for processing this data, like image processing libraries, are often no less complex than the handled data itself. Some of them, like
Khoros/Cantata [8], KBVisionTM [1], and the HORUS system [11] [3] [4], comprise several hundred routines on different abstraction levels, and many industrial vision tasks can already be solved by using such libraries. Usually these routines are optimized with regard to speed and efficiency. However, handling them directly is often quite laborious and the search for an appropriate operator sequence with suitable parameters is difficult and timeconsuming. This leads to the requirement to find suitable structures for these operators as well. These representations should pass on information relevant to the user (including knowledge about the operators) and hide details of the implementation. We will show how it is possible to comply with this demand, by using the object-oriented programming language Smalltalk and the HORUS system, which includes an image processing library as well as a knowledge-base with facts about the image processing operators. In contrast to other approaches, like the IUE [7], presenting objectoriented technology for image processing, our approach does not try to implement all data types relevant to image processing, but it concentrates on the highly abstracted object-oriented modelling of iconic data and image processing operators. For instance, data types for a direct access and manipulation of image matrices, like pixels, iterators, or neigbourhoods are not integrated into the system. Thus, the architecture is kept easy to grasp and handle even for novice users. However, the extendability of the system, according to the needs of the user, or by separately provided class libraries for special tasks, is not restricted by this. The proposed system aims primarily at supporting the user of image processing operators, but not the developer of new operators. This is made to separate different levels of software development: the providing and the use of image processing tools. 2. RAPID PROTOTYPING OF IMAGE PROCESSING APPLICATIONS IN SMALLTALK The presented system is based on the programming language Smalltalk. This is sensible because Smalltalk stands out due to the following features.
compilied Smalltalk C code environment
Smalltalk image processing application Integration
ST tools
Smalltalk system classes
Smalltalk HORUS classes (iconic objects + operators)
Access Automatically generated code
Smalltalk HORUS interface (C)
virtual Smalltalk machine
HORUS
HORUS library
knowledge-base
Figure 1: System architecture
Smalltalk is a mature object-oriented language which consistently translates object-oriented principles into practice. Owing to its mostly intuitive and very lean syntax, Smalltalk is easy to learn and to handle. An interactive programming environment is an integral part of the Smalltalk system [5]. Editors, browsers, searching tools, object inspectors, a GUI builder, and debuggers are its most important components. In combination with the fact that Smalltalk is an interpreted language, very short turn-aroundcycles are received. Smalltalk applications are almost hardware independent. A virtual machine represents the bridge between Smalltalk objects and a particular hardware platform. As a consequence of this, good portability is achieved. There is a high number of quasi-standard objects defined in the so-called Blue-Book [6]. This class hierarchy provides a wide range of useful objects and is supported by all Smalltalk suppliers. There is no partitionbetween system and user objects. All system classes are as modifyable and extendable as user-defined classes are, including all the interactive tools of the programming environment. The user-defined objects are integrated into the system at the moment of their definition. Smalltalk objects are persistent. The environment contains mechanisms for reconstructing the last system state even after a power outage or other system failure. Smalltalk became widespread and is used in many industrial projects [9].
Because of these features Smalltalk is predestined for complex rapid prototyping tasks. While developing image processing applications, a basic problem is to find an appropriate operator sequence with suitable parameter values for extracting some characteristic features in images. A lot of unknowns have to be tuned, which is a very complex and hence time-consuming and error-prone task. Within the Smalltalk system the developer is particularly supported by facilities which allow him to inspect interim results, develop and execute methods stepwise, interupt a running program, change the code, and restart the program or single expressions at any time without the need to change the environment. Further user support is given by the HORUS system providing wide knowledge about the operators (see section 4). The following list summarizes the concepts of the presented system; Figure 1 illustrates its architecture.
Software developers with little or no experience in image processing should be enabled to use a large image processing library for integrating useful functions into own applications. For this reason the interface to the library is kept simple and intuitive. Suitable object-oriented data structures within a clear architecture are provided for iconic data as well as for the image processing tools. They enable the user to work with highly abstracted objects which hide details confusing or unimportant to the user. The developer is supported by numerous tools and extensive operator knowledge, which is always and in an intuitive way accessible. By this, the user receives also a platform, where she/he can interactively investigate any image data, without programming at all.
With Smalltalk, the presented system offers the power and flexibility of a full programming language. Many standard objects and interactive tools, which can be modified and extented, expand the opportunities of the user. By this, further investigations of the results, which are supplied by the image processing operators, can be performed, and it is possible to develop whole applications within the same environment.
This approach to call a image processing function is recommendable in many cases, because the syntax is simple and intuitive and it supports polymorphism. Data objects can directly ‘be asked’ to modify themselves and to answer the result. They know the correct operator they have to call and thereby encapsulate the way they work, e.g., wether a HORUS operator or any other method body is to be performed.
The proposed architecture is open for further developments both of the image processing library and the development tools. This is achieved by a uniform and general modelling of the image processing operators, which supports, apart from a uniform access to the operators, the automatical generation of the Smalltalk code defining the operator objects.
However, by hiding the concrete implementation of an operator, the operator as an individual entity is hidden and all information concerning it as well. So it is not possible to handle the operator as an entity representing all dimensions of a complex tool and independently of any data. This has some important disadvantages. In general, by using complex operations it is not neglectable what operator actually is used, which features it has and how it works. The user often needs to know these special features, or at least some of them, for working efficiently. Default values, value ranges, and particular implementation or process dependent restrictions belong to this information. Additional information is discussed in section 4. In the method approach it is taken for granted that the user already has the knowledge or experience to handle the operator correctly and efficiently, or that a special help tool exists, which provides the required knowledge. This, however, may cause consistency problems, and the splitting of objects into a functional and a knowledge part contradicts object-oriented principles.
3. THE REPRESENTATION OF IMAGE PROCESSING OPERATORS IN SMALLTALK There are two basic ways for integrating functionality into an object-oriented language. Usually methods1 are defined for providing the involved data objects (e.g., images or regions) with the desired features and abilities. On the other hand, for more complex tasks it is advisable to create special tool objects, e.g., a graphics context or image renderers as they are provided in Smalltalk. In this section we will discuss both approaches concerning their suitability to represent image processing operators in the Smalltalk development environment. Operators as methods The common way for adding a function to the Smalltalk system is to define a method for (one of) the related object(s). This approach reflects the data driven viewpoint on many problems. Supposing that there is a class HImage representing image objects, we can define a mean filter operator as a method HImage mean: returning the smoothed image and having the mask size as a parameter. In general, Smalltalk method names have the following form:
j
ReceiverClass methodname ReceiverClass class methodname with methodname ::= unary binary keywords ::= identifier unary binary ::= ‘+’ ‘-’ ‘*’ ‘/’ ‘= 30. ˝Intersection of three regions˝ intersect := region1 & region2 & region3.
Operators as tool objects As shown in the previous section, methods are not sufficient to represent complex operations within an objectoriented system. By developing image processing applications it is, as a rule, not sufficient to know only all the operator names and their syntax. To be able to work efficiently, the developer needs much more information (see section 4). From the developers point of view, HORUS is a large toolbox and its operators are complex tools for processing (iconic) objects. This is also the way they have to be modelled in an object-oriented language: as intelligent tool objects representing both a special functionality and knowledge about the process, e.g., by being equipped with a detailed ‘personal manual’. These operator objects can be passed as parameters, they are able to manage their own resources, and several instances of one operator can be created. To some if their parameters, constant values can be assigned in order to reduce the complexity of these operators. This approach is strictly object-oriented and guarantees a uniform strategy for handling all operators. A similar approach, modelling filters in electrical circuits as C++ functors, is proposed by Coplien [2]. Besides, this objects allow to establish a network or graph of several operator instances as the most appropriate kind of representation for operator sequences in a graphical development tool, for instance. Figure 2 gives an example of a graphical representation for a simple image processing sequence. This example also shows that an operator is a concrete tool for an abstract task and that the operator knows something about alternatives and, e.g., successors for improving the results. Such operators are interchangeable as a whole, in fact the operators are able to exchange themselves for an alternative operator they know. In a development tool the operators are able to answer inquiries by the user as well as by the system concerning themselves, their use, their behaviour, and their parameters. So they
HMean
SmoothedImage edge detection
HSobelAmp
alternative HGauss
alternative
HMedian
alternative HRoberts
HPrewittAmp
EdgeImage segmentation
HThreshold
Region polygon approximation
successor
HSkeleton
HFetchPolygon
Polygon
Figure 2: Graphical representation of an image processing sequence with various operator instances may, e.g., influence widgets for parameter setting in a way that only allowed values can be passed to the operators as parameters. There is another important aspect in our approach. If each operator is an individual, it can be located anywhere in a distributed system providing special services. Once published in such a system, all connected objects have transparent access to the operator objects. There are several research activities in developing standards and implementations of distributed systems (e.g., CORBA [12] and HP Distributed Smalltalk). However, an investigation of these is beyond the scope of this article. Implementation: Figure 3 shows the operator class tree. The aim to get a uniform way for handling all operators is achieved by creating the abstract class HOperator. This class is derived from the Smalltalk root class HObject. It is the superclass of all operator objects and defines most of their abilities and generic behavior: the interface to the C library, mechanisms for inquiring the operator knowledge, access to the parameters and creating and handling of operator instances. This reflects a general and uniform operator model, which is an important requirement for uniform access, exchangeability of operators and highest possible code reuse. The uniform operator model is not restricted to the provided (atomic) HORUS operators but covers user-defined operators (molecules) as well. This provides the user the
Object
System Classes
Dictionary
Programmed Classes HOperator do: default man inputValues primHorusError
HInfoEntry
ProcName
on: hOperator InfoDictionary
do doWith: in: out inputTypes
inputArray outputArray inputConstVar
InputTable
HParamEntry
OutputTable
HOperatorEntry
on:name:type: name
HOCAtom
type
HOCMolecule Programmed Classes Automaticly Generated Classes
HOCFilter
HOCSegmentation
HOCMorphology
HOCMorphologyRegion
HOCFilterSmoothing
HOCMorphologyGreyvalues
HThreshold HMean
initialize do: image:minGrey:maxGrey:
HErosion1
HOpening
initialize do: region:structElement:iterations:
initialize do: image:maskExtent:
initialize do: region:structElement:
Figure 3: Class hierarchie of the operator objects
opportunity to create new operators in Smalltalk and handle them in the same way as the provided operators. This is useful, e.g., for combining often used operator sequences or even a whole developed program to one single operator. Therefore, on the next higher hierarchie level two classes are derived from HOperator by splitting the tree into atomic HORUS operators (HOCAtom) and compound operators (HOCMolecule). The class HOCAtom defines the access to the HORUS C library which is the same for all atomic operators. In order to organize the about 600 HORUS operators, one or two intermediate class levels are inserted combining similar or related operators into groups (compare Figure 3). These classes are automatically defined as well as the actual operator classes, using the information from the operator knowledge-base. The operator class objects constitute the leaves of the hierarchy tree. Each operator is represented by one class object. This class object manages the particular functionality and knowledge of the operator, such as the name of the C routine (ProcName) or information about the input parameters (InputTable). For constructing an operator graph, several instances of one operator class, for example with different parameter values, can be created. Furthermore, this approach offers the opportunity to extend the behaviour of operators. For example, a special
error treatment for particular operators can be installed b y overloading the primHorusError-method which is defined identically for all operators in HOperator. Another opportunity is to derive new operators from existing ones. They can, e.g., preprocess some data or hide special parameters from the user. Figure 4 shows an example of splitting a
HOpenFramegrabber initialize do: name:fGWidth:fGHeight:extent:startPosition:squarePixels:field:bits:spaceOrThresh:gain:generic:device:port:
MyVigraPixGrabber initialize do: extent:
MyProgResGrabber initialize do: extent:
Figure 4: Extending the operator class hierarchy by deriving new classes very complex and powerful open framegrabber operator into several smaller operators which are much easier to handle.
Data adaptation between C and Smalltalk One final important task of the operator objects, which shall be discussed here, is the data transformation. In Smalltalk, objects are handled: images, points, rectangles, filenames, etc. The C library, however, can handle only basic C types: integer, double, character, pointer, and tuples of these. Before calling the C function, the Smalltalk
Smalltalk operator aHImage key
aPoint
C operator ObjType
int[]
y
int
int[]
x
int
aHPolygon
HPolygon x: y:
Input objects
Output objects
Figure 5: Example for the data conversion carried out within the do: method of a Smalltalk HORUS operator objects have to be broken down to basic types and the return values must be assembled to points, rectangles, and so on. Figure 5 illustrates this task. The code performing this is part of the individual do: method of the operator classes. It is automatically created using information from the operator knowledge-base (compare section 4). The operator methods of the iconic objects, as described in the first part of this section, use this do: method as interface to the C library too. 4. THE OPERATOR DATABASE An integral part of the system is a knowledge-base, the so-called operator database, an extensive collection of information about the operators. These operator descriptions are used for two basic tasks, both serving the aim to increase the reusability of the system. Its first task is to enable the automatic generation of interface code for including the C library into the Smalltalk system. By a special parser-compiler tool, the operatordatabase is scanned and transformed into an internal representation. The compiler uses this information for writing a Smalltalk and a C file, from which the interface is built. The following pieces of information are used for the interface: an external operator name, the name of the C procedure to call, its parameter names and types, and a category name for combining similar or related operators into groups (e.g., edge filters, smoothing filters, segmentation operators, etc.). Furthermore, information is needed to reflect object-oriented qualities. For this purpose, firstly, objects can be declared, for which the operator is defined as a method (compare the first part in section 3). Secondly,
the semantics of the parameters with rules for the transformation between basic C parameters and complex objects are defined (see Figure 5). The automatic interface generation guarantees that new operators can be integrated into the system quickly and without any effort. The same strategy is also used for integrating HORUS into other host languages (C, C++, Lisp and Prolog) as well as for generating the reference manuals (in PostScript and in HTML format). The second main task of the database is to provide knowledge about the use of the operators. This knowledge must be available at system runtime and callable by the host language. The main attention lies on supporting rapid prototyping of image processing applications. During the program development, e.g., within a Smalltalk workspace or while using a graphical tool, the developer needs information for optimizing the search for promising operator sequences and suitable parameter values. Such pieces of information concerning the parameters are default values, ranges or a list of useful values, certain restrictions, properties of the return values, and a textual description. Beyond this, information about the complexity, alternative operators, common predecessors and successors, and keywords is stored, as well as a short and a detailed operator description, examples, references, and special hints about effects and restrictions using the operator. Additionally, this information can be used by ready-made applications during their runtime for dynamically influencing the course of the program, for example by modifying parameter values in a way specific for this special operator, or even by exchanging operators for other alternative operators in order to adapt changed conditions. In Smalltalk the access to knowledge concerning one operator is carried out by asking the class object of this operator, i.e., by sending a message to it. Every operator object holds a special dictionary, an instance of HOperatorEntry, who manages the available information and answers the inquiries about the operator. Information concerning a special parameter can be inquired from a HParamEntry object, also a dictionary, which manages the parameter knowledge and is stored for each parameter either in the InputTable or OutputTable array. The dictionaries HOperatorEntry and HParamEntry get their information via particular HORUS operators which can access the operator database. Information once inquiried is stored persistently within the corresponding dictionary. The following examples show different messages for accessing operator knowledge and illustrate how they can be used: ˝Open a detailed manual page containing all information available about the edges operator:˝ HEdges man. ˝Create an instance of the edges operator, setting all con-
trol parameters with the default values:˝ edgeOp := HEdges default. ˝Give an example for the use of the edge operator:˝ HEdges example. ˝Inquire all known alternatives of the edges operator, create the default instances of them, perform them with the input image and collect the results within the new collection altResults:˝ altResults := (HEdges infoAbout: #alternatives) collect: [ :op | op default doWith: inputImage ] ˝Answer for all input parameters the possible or recommended values:˝ HEdges inputTable values. ˝Answer if more than one value for the first input parameter can be passed to the operator:˝ HEdges inputTable first infoAbout: #multivalue.
5. SUMMARY AND PROSPECTS In this paper we presented concepts for the integration of image processing operators from a complex C library into the object-oriented language Smalltalk and its interactive programming environment. By this, three aims could be achieved. Firstly, the object-orientated programming language Smalltalk could be extended with powerful tools for image processing. For this, a reasonable object-oriented representation for operators was presented and compared with other kinds of representation. Secondly, a comfortable development environment for rapid prototyping of image processing applications is obtained. Finally, the extended Smalltalk system already represents a complete image processing platform. Within this platform an immediate investigation of any iconic data is possible without the need of changing the environment between development, test, and working phase. The proposed system has been completely implemented R class hierarchy. and is integrated into the VisualWorks It has been used for about one year and is, as the underlying HORUS system, continually developed further. Besides tools for textual programming of image processing applications, a graphical programming editor was developed; this tool is implemented in Smalltalk and uses the suggested HOperator objects for the representation of the HORUS image processing operators [10]. 6. REFERENCES [1] Amerinex Artificial Intelligence, Inc., “General Support Tools for Image Understanding”, Amerinex Artificial Intelligence, Inc. technical report, 1992.
[2] James O. Coplien, “Advanced C++ programming styles and idioms”, Addison-Wesley Publishing Company, 1992. [3] W. Eckstein, G. Lohmann, U. Meyer-Gruhl, R. Riemer, L. Alamirano-Robles, J. Wunderwald, “Benutzerfreundliche Bildanalyse mit : Architektur und Konzepte”, Mustererkennung 1993, 15. DAGMSymposium, L¨ubeck, Germany, 1993.
HORUS
[4] W. Eckstein, C. Steger, “Interactive Data Inspection and Program Development for Computer Vision”, SPIE conference 2656: Visual Data Exploration and Analysis III, San Jose, Ca, USA, 1996 [5] Adele Goldberg, “Smalltalk-80: the interactive programming environment”, Addison-Wesley Publishing Company, 1984. [6] Adele Goldberg, David Robson, “Smalltalk-80: the language”, Addison-Wesley Publishing Company, 1989. [7] Image Understanding Environment, Overview, February 1995.
Program,
IUE
[8] K. Konstantinides, J. R. Rasure, “The Khoros Software Development Environment For Image And Signal Processing”, IEEE Transactions on Image Processing, Vol. 3, No. 3, pp. 243-252, May 1994. [9] R. Laguna de la Vera, “Warum boomt Smalltalk in kommerziellen Großprojekten?”, in OBJEKTspektrum, Nr. 3, 1995, SIGS conferences, M¨unchen, Mai/Juni 1995. [10] Uwe Meyer-Gruhl, “Konzepte f¨ur eine wissensbasierte visuelle Sprache zur Bildanalyse”, Diss. Institut f¨ur Informatik der Technischen Universit¨at M¨unchen, M¨unchen, 1996. [11] B. Radig, W. Eckstein, K. Klotz, T. Messer, J. Pauli, “Automatization in the Design of Image Understanding Systems”, Proc. 5th International Conference IEA/AIE, Paderborn, Germany, 1992. [12] Ben-Natan Ron, “CORBA: A Guide to the Common Object Request Broker Architecture”, McGraw-Hill, New York, 1995.
