An Application Framework for Image Processing Marcel Worring
Jan-Mark Geusebroek Benno Mosterd Richard van Balen Department of Computer Systems Faculty of Mathematics and Computer Science University of Amsterdam e-mail :
[email protected]
Abstract
In this contribution we consider the user interface aspects of building image processing applications. The nature of such applications poses special demands on the construction of the interface which are not present in common user interface generation tools. The design is based on a Publish and Subscribe event mechanism giving high exibility in the construction of applications on the basis of a set of independent modules. The developed system is easily portable to other platforms and includes a what you see is what you get visual editor for very easy interface construction.
1 Introduction Image processing has reached the point where toolboxes containing several routines for image manipulation, analysis, and visualization are available (e.g. [3]). In end-user applications only a limited set of image processing routines from such systems is required and such routines have to be combined into modules with a limited set of parameters. Any end-user should have an easy way of seeing the e ect of di erent parameter settings to nd the ones which are optimal for the speci c problem at hand. Furthermore, the system should allow for di erent methods of visualization to see the various aspects of the images and the results. As the application system will often be used by inexperienced users the system should be geared towards the speci c application domain to reduce the amount of super uous information. In the past, image processing applications have either been constructed within the image processing toolbox environment, which is not designed for end-user application use, or constructed using a generic user interface generation tool. Such tools in general do not include images as an integral part, (e.g. TkTcl [2] and XVT). As our applications are all based on C the use of C++ based frameworks like ET++ [4] is not appropriate. The above frameworks are based on callback mechanisms making the use of dynamically changing multiple servers dicult as names and parameters are usually xed. So when adding a server, either the system has to be recompiled or the system must be based on an interpreter (like TkTcl) which makes it slower. Multiple servers are often needed in image processing. As an example consider the task of visualization in a 3D-imaging application. Then one needs di erent types of viewers like showing the set of subsequent 3D-slices, volume rendering, and stereo to reveal di erent aspects of the image. All visualization windows should be updated when the image changes. Furthermore, the use of viewers will vary with time. First, one uses simple viewers 1 and at a later stage the use of high quality
viewers comes into play. We will introduce a system including this functionality, which can be used in interpreted as well as compiled mode. Apart from the speci c needs of image processing a number of demands are to be satis ed in any proper Graphical User Interface (GUI). A clear GUI/application separation is required to allow for the incorporation of image processing modules into existing user interface environments. The GUI itself should have a high degree of modularity such that it becomes easy to extend the functionality or replace existing interface parts with application dependent equivalents. The system should be portable between such varying systems as Unix, Windows and Macintosh and on all platforms the GUI should have the native look to which the end-users are already familiar. Furthermore, the system should be generic and extensible in a simple and consistent way. Finally, it should be simple to use for both the programmer as well as the end-user.
2 The application framework From a conceptual viewpoint the user interface consists of two parts. First, the objects which constitute the visible items on the screen and second the mechanism with which the objects communicate i.e. how can objects perform actions on the basis of changes of other objects? We discuss the latter problem rst.
2.1 The publish and subscribe event mechanism
Our event mechanism is based on abstract events [1]. It is a high level event mechanism which is independent of the windowing system on which the user interface is implemented. Every object in the system is required to publish all its changes. For example a button will publish that it is pushed and an image will indicate that his pixel data has changed. A slider will indicate its new value whenever it is adjusted. Zero or more subscribers can subscribe to any other object to receive all changes published by that speci c object. The publishing object is unaware of this. In fact, the publisher is not allowed to assume the existence of any subscriber or assume an ordering in the processing of events. This allows for the development of modules which are totally independent of other parts of the system. Hence, giving a modular program with inherent parallelism. In principle any subscriber will be noti ed of all changes from the object it is subscribed to. However, when many di erent changes might occur the subscriber can also decide to receive noti cation of publishes from the publisher of a speci c type only.
2.2 The objects
In the design and implementation of interface objects three layers are distinguished. 1. The generic layer: this is the abstract high level de nition of objects. Here we can Set/Get certain generic features like size and position. Objects can also be destroyed at this level. 2. The item speci c layer: this is the layer where the Create function is de ned and the Destroy functionality is mapped to the proper item speci c function and where item speci c features are implemented. At this level all window-events are transformed to publishes. 2
PVP
IMAGE
recalculate new position
pan view delete
shift old position recalculate
Figure 1: Illustration of the Picture View Port (PVP) and how it performs a pan operation. 3. The window-system speci c layer: at this layer the mapping is made from the features of objects to the speci c windowing function able to perform the corresponding action. We have developed abstract high-level de nitions for common interface items like buttons, sliders, menus etc. which are all publishing objects as described earlier. A high level object which is very important in image processing, but not available in common systems, is the Picture View Port. This viewport is a rectangular area on the screen providing a view of the image data. The PVP has a bu er in which the display data is kept. It is independent from the data of the image being viewed. The PVP calls a user de ned routine which maps the pixels in the viewed image to the pixels in the display bu er. When the picture is larger than the viewport on the screen only part of the image is displayed. The PVP has a width, a height, a center, and a ( oating point) zoomfactor, which can all be adjusted dynamically. When panning, zooming or enlarging the size of the PVP, these parameters are adjusted and the data in the display bu er has to be recalculated. This is done in an intelligent way i.e. the PVP lets the user de ned routine recompute only the part which is not already available in the display bu er. This is done using non-overlapping rectangular areas. Hence, the PVP can handle large images without the usual communication overhead to the windowing system. The method is illustrated in gure 1. One can subscribe a PVP to changes of a speci c image. Hence, one can perform automatic redisplay when the image being viewed has changed. As a separate display data bu er is used, several PVPs can be used to display one image for example one showing a global overview and one showing a detail of the image. Apart from PVPs one can also subscribe a histogram display to an image change, allowing for automatic updates of the histogram. 3
2.3 Application framework tools
A set of the above objects can be combined to construct general tools like a worksheet, an image viewer with sliders and buttons to perform the zooming and panning, interactive measurement tools etc. All the objects within the tool will communicate using publish and subscribe. The tool itself publishes changes which might be relevant to others and the tool is made an application framework object with all the generic characteristics. The publishes of the tool will be high level. Publishes of objects within the tool will be handled locally and these are for example used to do automatic geometry management; for example when a window is enlarged a publish will be generated and objects within the window can react by changing their appearance and position accordingly.
2.4 The visual editor
We have built a what you see is what you get visual editor for generating a user interface using building blocks from the application framework. Of course the editor itself is also expressed in the application framework. The editor is generic and any item which is added to the application framework can also automatically be used in the visual editor. The editing result is stored in a simple internal format. Previously built interface parts can be loaded as part of a new interface. When a nal application interface is reached the internal format can be transformed into C-source code automatically. This can then be compiled to achieve optimal performance. This has the added advantage that the resulting code is less error-prone than user written code. One can also select objects on the screen and change their parameters. This can be done either by manipulation the visual object on the screen (enlarging windows, changing the text of a button etc.) or in a con gure window. The visual object and the parameters in the con gure window are coupled and changes will be re ected directly in both representations. Bindings between objects are also de ned visually by selecting a publisher and a subscriber using the mouse. When the objects are selected the user is able to de ne the speci c message. With the message information can be passed to the subscriber; for example a worksheet can generate a publish of the type new input and pass the input itself as data. These bindings are also added automatically to the C-source code when the GUI is transformed.
2.5 Modularity
The layered structure of the application framework provides a high degree of modularity. An important consequence of its use is the clear separation achieved between the (image processing) application and the GUI. In fact a GUI can be replaced completely as long as it reacts in the proper way to the publishes issued by the image processing package. However, due to the layered structure, total replacement of the GUI can be achieved in a gradual sense and can be performed at di erent levels. It also allows for partial replacement.
windowing system level: porting the system to a di erent windowing platform only requires a remapping of the basis application framework objects to the appropriate windowing system functionality or widgets de ned in the windowing system. object level: new user de ned objects are easily added to the set of existing objects. 4
publish/subscribe handler
IMAGE filtering enhancement measurement etc
AFW TOOLS image viewers, help, afw editor worksheet, Izoom, Imeasure AFW WINDOWING SYSTEM
Figure 2: Overview of the system. It shows how the image processing library communicates with the application framework using the publish and subscribe mechanism. It further illustrates how the windowing system acts as a low level for the application framework. It further shows the relation between application framework tools and application framework objects. tool level: new tools can be constructed easily using the visual editor or by a program. As long as the publishes remain the same the tool can replace a previously de ned tool. In fact in some cases the tool might be mapped to a windowing system widget directly whereas on another platform it might require extensive programming. The whole system is exempli ed in gure 2.
3 Applications Currently we have an implementation of the application framework built on top of TkTcl [2]. There are plans to implement the framework on the Windows and Macintosh platforms as well. As a testcase we have implemented an alternative user interface for the commercially available image processing environment SCIL image [3] using the application framework. Commands in this system can be issued via a worksheet or via menus and dialogues. In both cases one can use shorthands and a command expander will generate valid C-code with proper syntax and defaults lled in. A C-interpreter then issues a call to the image processing library. This functionality is achieved using a table driven approach where one adds an (image processing) routine to the GUI by adding an entry in a command description le. The interface is shown in gure 3 and its basically equivalent to the existing interface. However, the modularity of the code using the framework is much higher (see gure 4). The framework is further used to generate a user interface for a sophisticated package for visualization of 3D confocal microscope images. The system is shown in gure 5.
4 Conclusion We now systematically consider the demands raised in the introduction and show how the developed framework ful lls them. As the whole framework is publish and subscribe based we can attach multiple servers to one task, which might run in parallel. Furthermore, it makes it easy to extend the system with new modules. In fact because of the 5
Figure 3: The user interface for SCIL image as expressed in the Application Framework using TkTcl.
command description file
C-interpreter
succes or error report
command expander new input publishes from
image library call
worksheet or
menu/dialog generator
AFW library calls
menu/dialog
Figure 4: Implementation of SCIL image using the application framework. Note the modularity of the system. Further note that the worksheet is not present at it is de ned as a tool in the application framework. 6
Figure 5: User interface for a visualization for 3D confocal microscopy. Note that a number of tools are used which are also present in the standard SCIL image interface. However, tools like an image browser are currently not present in the latter.
7
publish/subscribe mechanism there is a clear separation of image processing functionality and GUI. The objects de ned in the framework are high-level. Hence, they are not based on some speci c platform and are therefore portable to any system having the required functionality. Another consequence of the high-level de nition is the native look we will achieve. Objects are created in a layered structure and are therefore generic to the largest extent possible. This makes them simple to use for the programmer. The set of objects can easily be extended with new user-de ned objects. An important contribution of the system is the Picture View Port which makes images an integral part of the GUI. Due to the layered structure of the application framework it provides a high degree of modularity and an existing GUI can gradually be transformed into a new one. The visual editor makes generation of applications, given the proper set of image processing routines, very simple and quick. For the end-user, systems are simple to use because they can be geared to their speci c needs with minor e ort. The current system lacks an Application Graphics Framework. Such a framework should contain publishing objects like lines, circles etc. This is topic of further research. We further foresee a large extension of the toolbox with general interface functionality on top of the application framework items.
References [1] R.D. Hill. Supporting concurrency and synchronization in human-computer interaction - the Sassafras UIMS. ACM Transactions on Graphics, 5(3):179{210, 1986. [2] J.K. Ousterhout. Tcl and the Tk toolkit. Addison-Wesley, 1994. [3] R. van Balen and A.W.M. Smeulders. ScilImage: an environment for collaborative use and development of image processing software. In Proceedings of the ICPR, Computer Vision and Image Processing, Jeruzalem, 1994. [4] A. Weinand, E. Gamma, and R. Marty. Design and implementation of ET++ a seamless object-oriented application framework. Structured Programming, 10(2):63{ 87, 1989.
8
