Bachelor Thesis Visual Programming of image ...

Daniel Senff (s0514457)

March 20, 2009

Visual Programming of image processing algorithms using ImageJ

Bachelor Thesis Visual Programming of image processing algorithms using ImageJ

Daniel Senff s0514457

FHTW Berlin / University of Applied Science Internationaler Studiengang Medieninformatik International Media & Computing Wintersemester 2008/2009

1


March 20, 2009


2


March 20, 2009


Introduction..................................................................................................................6 Abstract....................................................................................................................6 Motivation................................................................................................................6 Basic..............................................................................................................................7 Visual Programming................................................................................................7 Workflow.....................................................................................................................................8 Vertices........................................................................................................................................8 Connections.................................................................................................................................9 Visual Programming Languages.................................................................................................9 Deutsch limit..............................................................................................................................10

Workflow paradigms..............................................................................................11 Linear workflows........................................................................................................................11 Pipeline concept..........................................................................................................................11 Non-linear workflows................................................................................................................12

Digital Image Manipulation...................................................................................14 Destructive image manipulation................................................................................................15

Programming language..........................................................................................16 Frameworks............................................................................................................17 ImageJ.....................................................................................................................17 Concept/Analysis........................................................................................................18 Goal.........................................................................................................................18 Requirements.........................................................................................................19 Previous Implementations....................................................................................20 Image Editing............................................................................................................................20 Compositing...............................................................................................................................22

Shake.............................................................................................................22 Fusion............................................................................................................23 Nuke...............................................................................................................25 Blender..........................................................................................................25 Sound Design.............................................................................................................................26

Visual Programming Framework..........................................................................26 Requirements of visual-programming frameworks..................................................................26 Existing frameworks..................................................................................................................27 VisualAp.....................................................................................................................................28 3


March 20, 2009


ImageJ macro interface.........................................................................................30 ImageTypes................................................................................................................................31

Workflow processing..............................................................................................31 Algorithm...................................................................................................................................31 Macro generation.......................................................................................................................33

Desktop frameworks..............................................................................................33 Implementation..........................................................................................................34 Interfacing ImageJ.................................................................................................34 Interfacing VisualAp..............................................................................................35 Changes to VisualAp..................................................................................................................35 Object structure.........................................................................................................................37

Graph implementations.........................................................................................37 UnitElement..............................................................................................................................38 Input..........................................................................................................................................39 Output........................................................................................................................................39 Parameters.................................................................................................................................40 Connection.................................................................................................................................41

Delegates management..........................................................................................41 Visual workspace...................................................................................................42 Graph representation................................................................................................................42 Graph Interaction......................................................................................................................43

Document handling...............................................................................................45 Result..........................................................................................................................45 Project evaluation..................................................................................................45 Development potential..........................................................................................47 Loosing VisualAp.......................................................................................................................47 Tighter ImageJ Integration.......................................................................................................48 Workflow improvements...........................................................................................................49

Summary....................................................................................................................50 Attachments................................................................................................................51 Appendix 1..............................................................................................................51 Appendix 2..............................................................................................................51 Appendix 3.............................................................................................................53 Content of CD.............................................................................................................58 Sources........................................................................................................................59 4


March 20, 2009


Acknowledgment First and foremost, I would like to thank Professor Kai-Uwe Barthel, for allowing me to write my thesis on this topic and for his constant support throughout the development of the application. I would also like to thank David Fichtmüller. I was very excited when I first used node-based applications and he brought me the idea, to write my Bachelor-Thesis about it. He was also one of my proof-readers. Thank you! Petra Pillat, my former English teacher, I am very grateful that you found the time to smooth out my language a bit, even though the technical writing is very hard. Friedrich Maywald, for testing the program and giving feedback. Actually I can thank everyone, who tried it and gave me report. Finally I would like to thank my family and friends for their patience and support. The past few weeks I was not available as often as I liked, so given the circumstances, I hope you forgive me.

5


March 20, 2009


Introduction Abstract Automated image processing, while being a very powerful tool for experienced users, has a steep learning curve. Taking a step back from editing a single image, lets you see the taken sequence of operations, that processes this image. To view the workflow abstract from the image is not easy. The often applied techniques of Macros or Plugins require a lot of knowledge and are therefore hard to use. This thesis introduces to Visual Programming, a way to build node-based workflows, which describe the operation process visually. In this thesis I will present the theory of Visual Programming. This will include some basics of the underlying Graph theory, will analyze different Workflow Concepts, how they can be abstracted using the Visual Programming approach and how alternative workflows can be constructed. We will take a look at previous implementations of such node-based workflows. Based on the theory, we will construct an application which builds on this premise as a new means for Macro Editing in the graphical application ImageJ. The analysis will outline the goal of the application and its requirements. This will also include a look of the Frameworks the program will be constructed upon. The Implementation will cover solutions to the biggest issues and particular problems, that needed to be sorted out for the successful realization of the application. This chapter will review the intricacies of the development of this program. The Result will be evaluated at the end, including a small outlook on the potential for future development.

Motivation I was involved in a video project which used eyeon Fusion for post-production. Fusion is a Visual-effects-tool, which uses a node-based approach. You got your workspace, a set of filters and you could combine them by simple drag and drop. This way of editing effects was very intuitive and the data structures were very comprehensive, since the graph conveyed visually the way the filters would process 6


March 20, 2009


each frame. This approach was very flexible and allowed for filter-combinations, that would not be possible in traditional image manipulation interfaces. I liked this way of editing and was looking for graphic software which would adopt this workflow, but by 2007 only a few high-priced commercial applications were available. This sprung the idea to pick up the topic for this thesis and to develop an own image manipulation program with a node-based editing workspace.

Basic Visual Programming The theory of Visual Programming is more than 30 years old. The idea is to develop a graphical interface, which allows to construct program logic by actually drawing graph structures. Functions are defined in graph nodes and connections symbolize how data is passed along. This approach has the goal to construct procedures purely visually without actually writing any source code. The theory of Visual Programming is discussed in the academic community for decades and especially during the 1990s many publications dealing with various aspects of this topic have been released. An in-depth analysis and evaluation of the topic is not part of this paper, which deals with the practical implications and the construction of a node-based application. A detailed synopses of the papers released and the historic changes in perspective on the topic could be the basis for another thesis.[1] However, Visual Programming developed as generic term for node-based data structures used to process information and data. A subcategory is flow-based data processing, which only deals with the internal logic for data processing, but does not rely on visual representation. The dataflow describes the passing of information along the graph. Data is input in a source, “flows” through the graph network and is transformed along the way in the processing units. Visual Programming adds a graphical representation to the flow-based processing. It is often referred to as node-based representation or node-based data processing as the visual graph is constructed of Node-elements, which represent the processing 7


March 20, 2009


units. This graph describes a procedure and how data are processed. Node-based Interaction means data manipulation due to changes in the visual representation as a graph, with which the user actually interacts on screen. Workflow The Workflow denotes a sequence of consecutive performed tasks. This happens implicit by the user or consciously in a explicitly programmed fashion. The latter could be in form of a Macro or otherwise automated script. When we talk about a process - a chain of actions – taken by the user, we can present it in a graph. A workflow is an abstraction of the performed operations as such. Like every graph it consists of vertices and edges. The vertices symbolize one processing step, the edges display data dependencies between the processors. Graphs can be either linear describing a simple consecutive sequence of nodes or in the shape of a tree data structure. Trees are acyclic graphs.[5] The workflow has a beginning - a source, for example an image. It can have any number of processing steps and depending on the implementation an ending - a sink. Processing units receive their input data, process it and return the result to the next processors connected on the output along the path. This is the principle of the Dataflow.[4] Vertices Vertices or Nodes are part of the graph and represent one instance of a processing unit. They contain the algorithm instructions - the processor, which will be applied to the data. Vertices are connected by edges or arcs. These connect to pins, differentiated in input-pins and output-pins. Inputs receive data, while Outputs pass the processed result to the next connected units. This is displayed in the schematic of Figure 1.

8


March 20, 2009


Figure 1: Schematic display of a unit-element.

Input and Outputs have different properties. Inputs only allow one connection at a time while Outputs allow several connections and therefore allow for branches. Pins in general have a defined DataType. This can be different on Inputs and Outputs. How different types are compatible is a matter of implementation. Connections Data is passed from one Vertex to another via connections - often also called edges or arcs. The result on the output becomes the input of the consecutive unit. In the graph, this relation is symbolized by the connections between the units. Connecting edges are simple - between two pins only one connection is allowed. These are directed and data can only flow in one direction. The Dataflow requires this data dependencies illustrated by the edges. Data is processed along this graph. [5] Visual Programming Languages The goal behind Visual Programming Languages is, to provide a fully graphical programming language to visually construct programs. Yet decades of research have not lead to any groundbreaking solution. There are several reasons, despite the amount of research, some features available in modern textual programming languages are missing even a theoretical specification. As of 2004 Iterators and data 9


March 20, 2009


structures were not possible to translate into Dataflow-compatible form. While simple program structures can be created by the means of modern Visual Programming Languages, they fail for complex applications, let alone operating systems. For this reasons Visual Programming has not been a viable alternative, despite its interesting approach. Instead the principles have been transferred to other appliances and found fruition especially for signal and image processing. Technically, workflows still describe a program, although it is not an executable, but rather a procedure for data processing. This is basically Dataflow processing. This can be realtime or on-demand. In realtime, the data streams through the workflow and returns instantaneous results, for example for graphical previews or data streaming. [4, 6] Deutsch limit The Deutsch limit is attributed to Peter Deutsch. During a talk about Visual Programming, Deutsch commented, that the problem of Visual Programming lies in the limited number elements the graph can be constructed of without either being incomprehensible or not displayable on screen. If the graph has too many elements, it looses clarity and the dataflow is not immediately recognizable anymore. This also applies to the actual graph representation which – by his statements – does not allow more than 50 elements on a screen. While theoretically more elements are possible, the screen would be too cramped for practical use. Large program structures would grow beyond recognition quickly. In this respect textual programming languages are said to be in advantage to visual languages. This is in so far correct, that they rely on modularization to split-up large source code into smaller pieces, Object-Oriented Programming being one approach to this purpose. Similar modularization in visual environments can be achieved by grouping and collapsing of functionality. Another thought for condensing the space and the complexity of the workspace is to apply ZUI1-functionality, making the graph zoomable and unfold detail on zoom-in or loss detail on zoom out.[1] 1 ZUI is an abbreviation for Zoomable User Interfaces and is a concept for large program interfaces to display information by zooming into the program. 10


March 20, 2009


Workflow paradigms Linear workflows Linear working paradigms assume one local data-document for example an opened image, that is created and edited. One always see only the current state of this image and has a set of tools and operations to manipulate it. Changes by user input are updated in the current state and displayed immediately. Typing on the keyboard adds characters to a text-document; in image manipulation, dragging the brush over the canvas results in a drawn line; calling a color-correction filter, will change the look of the image. A taken action results in a direct consequence on the image. Its state is updated with the changes. This paradigm is popular, because it resembles very exactly how work is performed on a physical object. In the digital world it is easy to save the steps that were taken to get the final results. The editing process can be recorded and the single operations can be saved. This enables the possibility of Undo-concepts. If we think the editing process as a linear sequence of operations, it is possible to step back and forth to restore previous states of the document. There are many different undo-concepts, which is beyond the scope of this paper. All have in common, that they inherit the property of looking at and focussing on the document at hand. Many graphical applications apply layer concepts for compositing images. This will not cover this in this paper along with special solutions applied by some programs. Pipeline concept The Pipeline is the first abstraction to the linear working paradigm. The main focus of the work shifts away from the actual image to the meta process of Image Editing. As the workflow can be broken down to individual steps. These steps can be represented in an one-dimensional graph as depicted in Figure 2. The image itself is not changed anymore, but instead the operations taken in each step are defined to produce the desired outcome at the end. In a meta sense the user does not edit his image, but defines how the image is edited for him/her. The picture is not in the center of attention anymore, the process of editing is. This is a shift of 11


March 20, 2009


paradigm.

Figure 2: Schematic of a Pipeline

The pipeline concept describes a linear sequence of processing operations that are performed on the image loaded in the Source-Unit. The picture becomes just one element among many in the graph. Additional actions can be inserted in the pipeline at any position resulting in a change of the results of consecutive nodes. The advantages of the Pipeline concept are evidently for experienced users, who can reconstruct the image editing process easily by looking at the pipeline. The complexity of this approach for the common user has increased, because it demands knowledge of the processing operations performed by the application on the image. One has to know what operations he can use, how they apply on the image and in what way he/she can control and configure them. In order to work effectively he needs to be able to understand the actions performed by the constructed graph beforehand and has to be able to estimate the desired outcome. The transition from document-focused editing to a pipeline-based process-environment makes it necessary to be aware of the current working state and to understand every state in the chain to reconstruct the result achieved. Knowledge about every image produced in each operation needs to be taken into consideration. Good UI implementations can guide in this process to elaborate on cause and effects. Non-linear workflows The Pipeline is just the first step of abstraction. While it gives a more freedom in editing and reconstructing workflows, it has the same big disadvantage. Processing is still only performed along a linear one-dimensional graph. All operations are in sequence. It doesn't allow branching or merging, as it is depicted in Figure 3. For

12


March 20, 2009


compositing or other operations, which require multiple inputs or which deliver multiple outputs, linear processing is no option, because every operation only allows one input. Special cases can be constructed, where the the second input becomes a parameter in the processing unit, but this has its own limits and is no replacement for branching non-linear workflows.

Figure 3: Schematic of a branching workflow. Image data comes from the same original Source-Unit, and branches to two independent Display-Units.

Non-linear workflows allow for this bigger flexibility. The graph is not just onedimensional, but multi-dimensional and allows for complex compositions. Operations in the workflow become nodes in the graph and can be connected as defined by their abilities. Again, increasing the possibilities results in higher complexity for the user and the application. The transition from Pipeline to non-linear is not as steep once the step back from looking at the image to looking at the workflow is taken. After all, a multidimensional graphs can be reduced to a one-dimensional pipeline. The advantage is the visual representation of complex workflows. Graphs can visualize sequences of operations, which performed by hand could be extensive. Using conventional state-focused workflows, branching is difficult. To branch a image editing by hand, the image has to be duplicated and editing continues in a different file. Merging a branch would be equivalent to copying the image back to the original file. The images would lose their common history and changes could not be performed afterwards. This is the problem of Destructive Image Manipulation.[2]

13


March 20, 2009


Digital Image Manipulation Although the concepts of Visual Programming and Flow-based data processing apply to any kind of digital data, this thesis will focus on the Digital Image Manipulation. This genre allows to illustrate the topic and to give a basic theme for the actual implementation. Digital Image Manipulation is a computer based technique of processing images in a digital environment. Digital images can exist in two kinds of data. It is distinguished between Vector Graphics and Pixel Graphics. Vector Graphics describe image-data as mathematical equations. Geometrical primitives such as points, lines and splines define the image discretely. The paper will not deal with Vector Graphics even though they can be used to compliment data-flow-processing of pixel-graphics - for example as computational parameters. The paper instead will focus on Pixel Graphics. These digital images are based on a fixed-size raster. Every position in the raster stores a color value. The picture is quantized to this raster and stored as an array of integer values. The amount of colors is also quantized to a certain range. Although it can be debated if grayscale-values are actual colors, it can be regarded as the most simple Color model. It only allows brightness values stored in just one channel. The bit-depth defines the number of brightness levels. One channel with 8 bit allows 256 levels, 32bit allow for 4,294,967,296 levels. The color range is defined by the Color model, which describes the amount of colors available, the Color Gamut2, and the technical implementation. Color models have to define a color discretely and objectively. To achieve this, colors are quantized by as coordinates in a 3D color space stored. These coordinates can be stored in different ways. Depending on the model colors are composed of 3-4 channels. Every Color model is limited in the range of colors it can represent by the way the color is derived from these Color spaces. In computing the most common model is RGB, which allows to store 3 channels for red, green and blue. The amount of possible values in all channels is defined by the 2 The Color Gamut the complete set of colors, that can be represented by a Color Model. The Gamut can only be a subset of the the whole Color Space. 14


March 20, 2009


number of bits reserved for a single channel. Over the years RGB had several technical implementations with increasing bit-depths. Today, R8G8B8 is the most widespread. Every channel has 8 bit reserved for 256 values. Professional graphic applications begin to use RGB with bit-depths up to 32 bit per channel. RGB is one example of a Color model using 3 channels. Another well-known Color-model is CMYK, which is mostly used for printing and stores colors in 4 channels. Digital images can be processed by the means of filters or processes. These filters apply mathematical operations on the pixel data which result in a change of the looks of an image. Filters can achieve all kinds of purposes. Color adjustment, blurring, sharpening and edge-finding are just a small selection of possible operations. Because of the different Color models, not all filters can be applied to every model. Either explicitly or implicitly the image data would require conversion between formats. Destructive image manipulation Linear document-focused editing has the big disadvantage of only seeing the latest state of the document after applying filters. After saving to file, intermediary steps of the editing process get lost. This involves the editing history and performed actions as well as filter settings. This work can only be restored by doing it a second time manually. Even then it requires to actually remember the right steps in correct order with the correct operation settings to get exactly the same result. Different values for these settings can result in varying results. Having a meta-history for parameter changes is not covered in this paper. Non-destructive image manipulation avoids such situations by giving the user the possibility to restore any previous state of the image. This goal can be implemented in different ways with various success. Most editing application have a session-based history. Session-based means, that the protocol of operations is only for this user session and is not saved with the file. This offers limited ways to go back and forth on a linear protocol. Going back and changing the image in an earlier state removes the history of all following actions. They need to be redone by hand in the same sequence as before. This linear approach 15


March 20, 2009


does not convey the freedom of true non-destructive data manipulation. The Pipeline concept is non-destructive and supports the necessary abstraction of operations and data. Not the final result is saved, but the whole setup of operations. The original file is not changed and based on the workflow the final resulting image can be edited later on.

Figure 4: Different states of an image during a workflow.

Visual node-based interfaces are also non-destructive by nature. This advantage combined with the comprehensive representation of operations makes the Visual Programming approach interesting for professional applications of high complexity.

Programming language The programming language of choice was Java developed by Sun Microsystems, Inc. Java is an object-oriented programming language. It is Open Source since 2006 and in constant development by the community and Sun. New versions of the language are released every 2-3 years, which include an extended API and often new programming features. One of the major advantages is its extensive development API3. Java offers welldocumented solutions for the most-common programming use cases. Reusing and building upon the original classes is explicitly welcomed and ensures good compatibility with future Java versions. Another big advantage of Java is the big community that has developed of the past decade and which actively continues the development on Java. Thanks to the community the language is well-documented and a lot of knowledge is available. 3 The API is the complete documentation on all Objects-classes available in a framework. It documents the use, the methods and sometimes even example usage. 16


March 20, 2009


Many projects embrace Open Source and invite to participate or to reuse code for new projects. Existing projects can be the base for new endeavors and for new innovations.

Frameworks Despite of a rich API, which may already offer a lot of the functionalities that are required for the development of an application, it is still up to the programmer to layout the structure of the project and combine all components. Frameworks allow the programmer to set up a scaffold for his project, in which he can work. The framework helps finding solutions to common problems by offering interfaces in which the individual problems can be resolved very efficiently by the programmer. Frameworks do not offer complete solutions, but they offer a path to implement solutions on your own by guiding the development. Frameworks can be minimal and offer only interfaces4, which the programmer has to serve on his own. They can also offer functionalities, that are expected to be used or which require a default implementation. The purpose of Frameworks is to give guidelines for the development and to offer as much of the functions as possible by default.

ImageJ ImageJ5 is an image processing and analysis software mostly used for medical and biological analysis. It is developed by the National Institute of Health in the US. It has been in development for over 12 years, Version 1.0 was released in March 1999, the current version is 1.41. ImageJ features a well-documented and powerful macroand plugin-interface, which allows to write own image processing filters. Around ImageJ a lively community developed, which over the years created an extensive collection of third-party add-ons for the program. Macros in ImageJ are scripts for automatic image processing. These scripts have an 4 Empty interfaces only consist of the class and method declarations, but do not include any definitions. 5 Website of ImageJ: http://rsb.info.nih.gov/ij/ 17


March 20, 2009


own syntax. Opened images are identified by their name or their ID and macros call filters which are applied to the selected image. The PlugIn-interface registers installed PlugIns, which can be called by the Macro. The syntax allows to pass arguments to the filter, when it is called.

Concept/Analysis Goal The Visual Programming approach is not new and it has been implemented in a couple of programs in different genres. Graphics manipulation is an area where complex operations can be condensed to simple steps and complex manual workflows can be translated to rather simple graphs. Based on the theory in the last chapter, the goal was to design and implement an application, which would use a Node-based User Interface. The application would consist of two parts. Firstly, the visual workflow representation and interaction for constructing a visual dataflow. And secondly, the actual data processing logic to execute the editing operations defined in the workflow. Dataflow processing can be applied to many different genres of information processing, this application would be specialized on image data processing, in particular in ImageJ, which serves as foundation for the business logic. Based on nodes, the user would be able to construct workflows, which will be transformed in a way, that ImageJ can interpret and process the elements of the flow. The business logic would be applied in the background and be hidden from the user as much as possible. The focus should be on the workspace setting up the workflow and should not require any technical or theoretical knowledge about Visual Programming or the particular implementation. The working title for the project and the application was ImageFlow, as a combination of ImageJ and Flow-based data processing. Since the application is a bridge between a Flow-based User Interface and ImageJ, the title would be very descriptive.

18


March 20, 2009


Requirements Required for achieving this goal was finding the individual components, which would have to interact together to fulfill the desired function. To focus on the work on the user experience on the final program, it should be based on an existing Visual Programming framework. Available frameworks were to be assessed and then decided, if one is suitable or if it would be necessary to construct something from scratch. It was clear, that the application would need to be based on a graphics editing library. ImageJ is a very powerful image editing tool with a wide variety of filters, plugins and macros. ImageJ is stable and in constant development for many years. Due to it's well-documented Plugin-API, past experiences with the system and the large amount of processing-filters, ImageJ was chosen as the basis for the image processing logic included in the final program. It would be required to see, how a Visual Programming interface could be implemented for the program. Since the goal is to create a fully applicable application, it would need a proper program structure. While simple programs can be written in Java very easily, with increasing complexity common problems arise when dealing with Resources, Threads and Actions. Therefore the structure needs to be well-planed and if possible be based on a common framework. The program itself should not be a prototype, but end up being an end-user application, which is ready to use. This includes a good usability with a set of features, which could be helpful for the day-to-day usage. This would involve being able to load and save the workspace. Another idea could be clipboard-functionality. The design was another criteria, while not looking to intrusively, information should be conveyed in a clean and elegant look. At last, security features needed to be included. This mainly relates to graph security and ensuring, that errors in the graph are delivered to the user early and in a way, that mistakes can be minimized.

19


March 20, 2009


Previous Implementations Before outlining the specific analysis the components used in the workflow applied in the application, we take a look at previous implementations of Dataflow-based processing. Since the first concepts were developed 30 years ago, several programs have been based upon this concept. Often these applications are very specialized, rich of features to convey to they user and deal with complex tasks, which Node-based Interaction tries to convey in a simple interface. The interfaces often behave quite similar irrespective of of the data that are being processed in the flow. Since this paper can not take a look at all applications ever produced, we select a fair number from different fields of expertise. Image Editing Khoros is a signal and image software environment developed by Khoral Research Inc for Unix in 1993. It is a programming environment for processing modules and algorithms, which can be combined by the user to achieve the desired results. Khoros itself manages the flow execution, the visual representation is done by cantata, a GUI for Khoros based on X11. Cantata allows to actually draw workflows, which can be interpreted and executed by Khoros. Naked light6 is a non-destructive image editing tool for Mac OS X, developed in 2008. Editing is achieved by combining elements in a workflow. The visual representation is different to other approaches. Visually it is closer to the Pipelineconcept, however the theoretical concept of Visual-Programming is applied.

6 Website of Naked light: http://www.naked.la/light/ 20


March 20, 2009


Figure 5: Khoros processing environment with cantata-GUI © Universität zu Karlsruhe

Figure 6: Screenshot of a workflow in Naked light © naked.la, 2008

Apart from Khoros/cantata and Naked light Visual Programming is not wide-spread in single Image processing.

21


March 20, 2009


Compositing Image Editing only involves single image processing. Though the basics are similar with Compositing, the latter has more specialized techniques towards animation, visual effects or video editing. Shake Shake7 by Apple is a Compositing software developed for large commercial projects. It was designed to interface with the video editing tool Final Cut. Shake allows large scale visual effects production. It is available for Mac OS X and Linux. Visual effects are one of the most-promoted use-cases. Node-based Interaction is used in the user interface to compose the single elements of a scene to one resulting frame. Visual effects are constructing in branches and merge at the end with the final image. The workflow basically sets up an image-processing program, which calculates one frame. By running this setup on a complete video, a batch-process is started which renders all frames in a predefined manner. A major difference to Image Editing is the influence of the time-dimension. Filter-parameters can be changed over time, while the graph remains the same at any moment.

7 Website of Shake: http://www.apple.com/shake/ 22


March 20, 2009


Figure 7: Screenshot of the user interface of Shake. © Apple Inc: http://www.apple.com/shake/

Fusion Having to deal with time influences the design of the application a lot. Eyeon Fusion8 works very similar to Shake. For time-related parameter manipulation it comes with several solutions. Every graph-element has a number of parameters that can be defined in the GUI. These values are constant throughout the scene. Animations can require predefined changes of the values and so Fusion gives the user several ways to accomplish these changes by presenting 3 different time-editing modes9.

8 Website of Fusion: http://www.eyeonline.com/Web/EyeonWeb/Products/fusion5/fusion5.aspx/ 9 Digitial Fusion Interface Short Summary: http://www.eyeonline.com/web/eyeonweb/products/fusion5/fusion5_interface.aspx 23


March 20, 2009


Figure 8: Fusion Timeline editing mode

The first is a Command-line interface, which allows actually programming of values and functions. Very advanced users can achieve very special results on the commandline, but it is not very visually. A better representation is the non-linear Timelinemode depicted in the screenshot in Figure 8. Selected workflow elements and filters are displayed and edited on a timeline. The most effective parameter editing interface is the Spline-Editor shown in Figure 9. Selected parameters are displayed as splines. Constant values are a simple horizontal lines. Value-changes display as curves on the diagram. Along the x-axis is the time, the value-range is displayed on the y-axis and suits the selected parameter. Splines can be edited by mouse interaction, by defining key-frames and dragging the spline. This representation is very well-conceived and practical.

Figure 9: Fusion Spline-Editor, Parameters can be edited in relation to time, by editing splines.

24


March 20, 2009


Fusion 5 is available for Microsoft Windows XP and Vista and Linux. has been on the market since 200510. Fusion 6 has been announced. Nuke Nuke11 was originally developed by D2 Enterprise, later in 2005 bought and continued by The Foundry. At times, parts of the software were licensed under LGPL and free of charge, but the later version were commercial-oriented. The Foundry released Nuke 5 in February 2008. Nuke also offers a Node-based User Interface for compositing visual effects, very similar to the past two examples. Nuke supports stereoscope combined with a 3D-System, so working on 3D-movies is one promoted use-case. Since its release Nuke has become popular in the visual-effect community. Blender So far all software discussed has been proprietary and commercial. Blender12 is an Open Source 3D-animation program, which includes 3D-modeling, animation, editing and compositing. In version 2.43 the compositing interface received a major rework and offers node-based compositing. Interesting in Blenders approach of the visual representation is that the parameters of the processing unit are displayed on the unit-element itself. It doesn't require an extra window and once the settings are saved, the node can collapse to require a minimum of space.

Figure 10: Part of a workflow in in Blender's compositing editor. 10 Press release: Fusion 5 availability announcement, August 02, 2005 http://eyeonline.com/Web/EyeonWeb/Press/DisplayArticle.aspx?articleid=196 11Website of Nuke: http://www.thefoundry.co.uk/pkg_overview.aspx?ui=CBC2593A-2C9F-4EF9-84BE-C198B0171453 12 Website of Blender: http://www.blender.org/ 25


March 20, 2009


Sound Design In sound-design a popular application is Reaktor13 by Native Instruments. It is a program for sound synthesis to create new sound producers, which could be used as sources for samples or even as instruments for composing tools. Reaktor lets the user create his own instruments by the use of a node-based workflow. Wave generators create input sounds, that are processed in sound processor nodes. Branching and merging is supported and can be used to create sound producers in all variations.

Visual Programming Framework Requirements of visual-programming frameworks An important early consideration for the conception of the application was the question, whether the development can be based on an existing visual-programming framework or if the development of a new system would be required. A new system was one option, which in the long-run may have enabled more freedom, but would have resulted in a much greater complexity. The outline of the graph-structure, the graph representation and interaction would have caused a massive amount of planning, design and implementation. The focus would have moved away from achieving a viable end-user application towards a well-organized framework. The preference was to find a framework, which offers a lot of the basic functionality to extend and recycle. The review did not only involve true frameworks, but also open source applications, which implement the desired functions. This were the internal graph implementation, representation in a graphical user interface and mouse interaction to build the workflow in the GUI and thereby create the internal graph. Actual dataflow-processing was not required, since the image-manipulation would be done by ImageJ using the Macro-interface. Despite this, reviewing different methods was part of the analysis of the available frameworks. More formal requirements involved the programming language and the license. Depending on the platform and the performance required by applications, the 13 Website of Reaktor: http://www.native-instruments.com/index.php?id=reaktor5_us 26


March 20, 2009


programming language is chosen. Applications with high demands on performance are developed in languages closer to the hardware like C or C++. Java, is often preferred for programs that are cross-platform and feature a wide availability with lower demands on performance. As the ImageFlow-project was laid out for Java, it was decided to use Java 1.5 as the Java-version of choice. Java5 was released in 2005 and was succeeded in 2007 by Java6. However for example due to Apple's policy to not update the Java-Environment on older versions of Mac OS X, Java5 is still widely in use, so that a change to Java6 could exclude a lot of users. For the later release of the software, the license was an important criteria, as it was supposed to be available as Open Source and thereby should not contain any proprietary code. To effectively build on the framework or application selected as a basis, the source code had to be available for review. Existing frameworks An early part of the analysis and preparation for the project was research into already existing visual-programming frameworks based on the defined requirements. Since one of the major requirements was to use an Open Source visual-programmingengine programmed in Java, this analysis only takes a quick look into assorted frameworks and programs that were taken into consideration. Filtering licenses and programming language revealed 3 candidates. Many projects were discarded because of proprietary licensing or because projects were abandoned or too immature to be used. The last three final candidates were JavaFBP (Java Flow-based-programming), JavaSteps and VisualAp. JavaFBP14 was a promising candidate because it was an actual attempt to construct a working framework for flow-based-processes. By the time of planning the project, it was at a beta-stage. The main-reason why this was not chosen was its use of Java6. Since Java 6 is not available for Mac OS X 10.4 and to enable more backwardscompatibility, the latest version to support should be based Java 5. JavaSteps15 is also an approach to create a framework for flow-based-process14 Website of JavaFBP: http://sourceforge.net/projects/flow-based-pgmg/ 15 Website of JavaSteps: http://sourceforge.net/projects/javasteps/ 27


March 20, 2009


operation, however under a different premise. The frameworks offers the necessary interfaces, the process structure is completely defined by configuration in XML-files. One important part missing in JavaSteps was the implementation of the visual representation of the processes. JavaSteps is a flow-based solution for step-by-step process management, without the presenting these processes visually, the framework only covered half the functionality required. This left VisualAp the last candidate. It promotes itself as a framework, but more exactly is an a flow-based application with a very rich and well-documented plugininterface. Plugins are Java-Beans, which can be read by the program and are interpreted as Filters. These Filters can be added to a canvas as Nodes and connected using simple drag-and-drop actions. VisualAp, was in this respect not just a framework, but a fully-working application, that allowed to build, check and run workflows using official or third-party plugins. Basic mouse-handling was included allowing to select, connect and move units. An implementation of a clipboard, as well as loading and saving of workflows. The application is licensed under Gnu Public License Version 2. It is programmed using Java-Version 1.5. On the one hand, VisualAp was the last candidate, at the same time it was also the most-promising. So it was chosen as a base for the development of the project. VisualAp The application VisualAp has a simple basic structure, which was suitable for extension. The graph consists of Node-objects that implement a couple of interfaces for the graph-interaction. The Node is an abstract class which contains the most general method definitions for selection and positioning. Two classes extend the Node. The NodeText-class specifies a note-object on the graph. The processing units are covered by the NodeBean-class. The NodeBean contains two lists of Pins. These define the input and the outputs of a Node. Two pins are connected by an EdgeObject. These are the basic objects, that construct a larger graph. The Nodes are stored in a GList-Object, which itself is an extended ArrayList with additional functionality. For storing the Edge-objects, there is the Edges-class, which also is an extended ArrayList. This classes are the basis for the graph handling. A simplified

28


March 20, 2009


Class-diagram can be see in Figure 11.

Figure 11: Original VisualAp graph implementation

The visual representation is handled by a GPanel-class, an extension of a JPanel. This rich class, combines the drawing of the graph, mouse-handling and interaction with the nodes and edges. The GPanel has references to the Edges- and GList-objects and serves as a view to these models.

Figure 12: VisualAp with a small example workflow. 29


March 20, 2009


The Node-objects are based on JavaBeans, which are read from a folder. These beans contain meta-information about the node, it's function, the author, as well as name and icons. They contain an Object-class, which contains the methods, that are called, when this bean is used in the workflow. The meta-files contain the definition, how incoming data is handled and by which methods it is processed. On startup VisualAp scans its plugin-directory, reads all JavaBeans and creates a list of Delegate-objects. This Delegates store the information on the single beans and are used to create the actual Node-instances in the graph. The list of delegates shows what nodes are available for inserting on the graph. VisualAp has a full implementation on reading and writing workflows to XML. The format however is very specialized on the JavaBean-plugin solution.

ImageJ macro interface One of the prominent features of ImageJ is its powerful Macro-Interface. Macros describe a script-language, which can call filters on selected images. The goal for the application is to construct a visual workflow, which is translated into a viable ImageJ-Macro. The transition of the workflow to the Macro-Script defines the workflow as a Visually Transformed Language. The original Macro-ScriptLanguage of ImageJ is a traditional non-Visual Programming language. ImageFlow will be build on top of that to mimic visual-programming behavior and representation. On execution of the workflow, the visual representation is translated into a linear format and executed by ImageJ.[3] The Macro-language allows to call filters and plugins which are integrated in ImageJ. Each filter is associated with a command, which calls it for execution and passes necessary parameters. Since ImageJ does not have an interface to read all filters and the necessary meta information about inputs, outputs, parameters and ImageTypecapabilities, this information would be need to be available by other means, for example by defining XML-configuration files for every filter. While this is more work to map all of ImageJ's functions, it is a clean description without compromises. Certain functionalities of ImageJ like Stacks or Region of Interest are not of concern in the first version. 30


March 20, 2009


ImageTypes As explained in the basics, Images may store Color data in different ways. ImageJ supports 5 different ImageTypes16, which differ from each other in the number of channels and bit-depth. Not every filter in ImageJ supports every ImageType. Every plugin has their capabilities hardcoded and ImageJ displays error messages when trying to call a PlugIn on an unsupported ImageType. Because this checks happen as late as during execution of the Macro, ImageFlow would need to check the ImageTypes passed by every Units during graph construction.

Workflow processing The Workflow will be constructed in a visual workspace. In contrast to Pure Dataflow-Models, the application does not do any image processing itself. When processing the graph, ImageFlow only substitutes variables in the ImageJ-Macrocommandos and writes the Macro-file. As this is only simple text substitution, it is easy on the performance. Because of this, ImageFlow has no kind of Caching or Multithreading, which would be required for Pure Dataflow processing. The workflow is not processed in realtime, but only on call. So, many problems arising from live-execution are not covered in this paper. The actual image processing is done by ImageJ, whose performance has been optimized for macros. [4] Algorithm The graph set up in the workspace may contain inactive branches or isolated units, who have no connection to the graph. Branches become inactive if there is no Display-Unit at some node in the graph. Execution of this branch would therefore do nothing and would only cost performance. These dead elements in the graph need to be filtered before generating the Macro. At the end, the executable path is to be isolated and reduced to its minimum size. For the purpose of removing unused elements from the graph, an algorithm determines, which elements are required and 16 ImageJ ImageTypes: 8-bit grayscale, 16-bit grayscale, 32-bit grayscale, 8bit Color, RGB color 31


March 20, 2009


which can be removed. At this point the non-linear graph in the workspace is transformed into a linear processing order. The algorithm is based on a SelectionSort-algorithm. The algorithm receives the list of all unit-elements in the current workspace and loops over all units as long as this list is not empty. In every step one Node-element of the graph is checked. CommentNode are discarded right away. Unit-Elements have to fulfill certain criteria to be processed. The UnitElement, a processing unit, is being marked with a number. At the start of the algorithm all unit-elements have the mark 0. If the current unit is a Source-unit - meaning, it has no Inputs - then the mark for this element is set and incremented. The unit is removed from the element-list and added to a new list, which contains all final elements. Every unit added to the final list gets this mark. Source-unit's Inputs are checked, if the unit-element connected to each Input has been marked. If all Inputs are marked, this unit itself is marked and moved to the new list. Otherwise the mark on some Input-units is missing and the unit remains in the original list. If the unit is for whatever reason invalid, it is removed from the original list. Unit-elements remain in the original list and are checked until they qualify to be removed. This way the list reduces gradually, until all unit-elements are either added to the new list or discarded. Since this is implemented with a while-loop and the criteria is, that the original list is empty at some point, the implementation needs to ensure, that this condition is met. Therefore good testing and a review of all criteria for which units are either moved or discarded is important. Otherwise infinite loops can cause accidentally. While the order of unit-elements in the original list of units does not matter, the new list sports the correct order of elements for execution. This new list is passed to the MacroGenerator. The algorithm performs with O(n2). For an algorithm this is not the best performance, but since we only deal with a limited number of objects and not with 32


March 20, 2009


heavy data, the algorithm performs quickly. Macro generation The final ImageJ Macro is generated based on the list of units passed on by the cleaning algorithm. This list is in correct order and the MacroGenerator iterates over all elements. In each iteration, the Macro-object containing the abstract macrocommando is read. The parameters are injected into the syntax and the resulting macro-code is combined to the final Macro. This can either be handed over to an ImageJ instance for processing or saved to disk during export.

Desktop frameworks Java has a rich API, which delivers many classes for all sorts of purposes ready to use on every platform. For Desktop applications Java comes with the AWT and the Swing-GUI-framework. Both have their merits and their disadvantages. It leaves a lot of choices and freedom to build a customized application. Small programs can be build very easily, but the bigger the program grows the more important is an outlined structure. As soon as various views, actions and resources are involved programming can become messy without frequent refactorings. Programmers find a personal way to deal with the common repeating tasks and build a rudimentary application framework of their own for their programs, which covers issues they experienced so far and which needs updates with every new challenge. This work is good enough for small to medium sized programs, but requires a lot of maintenance on peripheral program-structure which does not concern the actual program logic. During the outlining process of the ImageFlow application this was an issue, because work on the program was not to be distracted from issues by the surrounding application structure. The decision was made to work within an established Java Desktop Application framework for the general outline of the program. Research revealed a rather surprising situation. Java is established for Server and Web application purposes and consequently a lot of work has been put into 33


March 20, 2009


developing various Web Application Frameworks. The situation for Desktop Frameworks is rather dire. Swing and AWT only give a very loose interfaces on how to create proper Application. The Standard Widget Toolkit17 (SWT) developed by IBM and prominently used in the Eclipse-IDE18 is a framework that encapsulates a lot of the program logic, but itself is quite large and builds upon its very own Widgetclasses. More lightweight was the Swing Application framework19, which is also known as JSR 296. It was first outlined and finally approved in May 2006. Its purpose is to “simplify building of Swing applications to a point where even novice Java desktop developers find the job straight forward.” With only 2 years of development, the framework is stable, but still in development and growth. IDEs like Sun's NetBeans allows by default to build Applications using this framework. Swing Application Framework is planed to be included to the official Java-API in the Release of Java7 in 2010. Because of the strict specification of the framework, major changes are not expected.[7] The Swing application framework was the first choice. The life cycle of the program is outlined by the frameworks interface. Actions are managed using introspection20 and Tasks are dealt as separate threads. Texts are read from Resources and can be localize. Frameworks always have a learning curve, this one was no different, but having a good foundation is important to any application.

Implementation Interfacing ImageJ In cooperation with Professor Kai-Uwe Barthel originated a prototype. This was a proof of concept to assess how the basic graph structure can be translated into an ImageJ Macro. It was possible to create a hard-coded graph, which would be transformed into a 17 Website of the Standard Widget Toolkit:

http://www.eclipse.org/swt/

18 IDE abbreviation of Integrated Development Environment 19 Website of Swing Application Framework: https://appframework.dev.java.net/ 20 Automatic code creation using Annotations. 34


March 20, 2009


macro by a basic MacroGenerator. The macro is passed on to an ImageJ instance and executed. The macro itself is set to run in Batch-Mode. All processes run in the background and only selected results are displayed. Branching outputs, require to duplicate the resulted image and be processed in its own branch. The structure of this prototype was very basic, not fully object-oriented, but served as a foundation for the later application. It was properly refactored over the course of the development.

Interfacing VisualAp Changes to VisualAp While the overall structure of VisualAp corresponded very well with the theoretical class concept of ImageFlow and the structure of the first prototype, there have been things which had to be addressed to make it an actual basis for development. VisualAp itself is an application and offers an interface for plugins, yet it is no framework itself. It does not intend to be a general Visual-Programming framework albeit it could have been one with little effort. To be able to build upon VisualAp and extend its functionality to suit the needs of the ImageFlow-project, it was necessary to perform changes to the original source code. Usually changes to the source code of a library or a framework should be avoided, because custom-changes cause a lot more maintenance in case of software-updates. Since the use of libraries and frameworks is supposed to avoid redundant work by the use of a common API, that could be reused in various projects. Customization or a specialization for a single project is therefore not the best thing to do. However this was necessary be able to use VisualAp at all. Most changes introduced into VisualAp involved the visibility-modifiers for fieldvariables in various classes of the application. Most often objects were either completely private or friendly, so extending these classes without copying the complete implementation would not have been possible. Obviously this defied the 35


March 20, 2009


reason for using an existing code-base in the first place. Where necessary, the modifiers were changed to protected, so that the object could be used in the extended classes as well. This was done to the GPanel, as well as many graph-related classes. Bigger changes were necessary regarding the graph implementation. Despite the good structure, the implementation was very specialized and a lot of functionality was either useless for the project or even harmful and distracting. For example the NodeBean-object proofed to be not abstract enough to build upon it and included many methods related to the JavaBeans. For this reason the AbstractNode was introduced as shown in Figure 13. This AbstractNode-class made it possible to separate the general Node-properties like labels and coordinates from the specialized methods serving the JavaBeans-interface.

Figure 13: Class-diagram of VisualAp's modified graph implementation. The NodeAbstract was added including the most common methods for the graph representation.

The Node-class was one case, which lacked a proper interface definition. The simple split of Node and AbstractNode involved several more changes in the Application, for example in the Glist storing the unit-elements, which had to be able to handle AbstractNodes. This introduced type-casting in many cases.

36


March 20, 2009


Object structure Despite these changes, the basic structure resembled the structure outlined in the application very good. The necessary classes needed to be extended to incorporate the functionality implemented in the prototype. While the prototype was a raw proof of concept, this final implementation got more attention in terms of Object-Oriented Programming, Inheritance and reusable code. The most important Model-classes were developed with according JUnit-TestCases. Over the course of the development this enabled to test the logic quite thoroughly. In a few cases it proofed good practice to lay out the test-cases beforehand and to define the expected outputs before programming the classes. This proofed to be very efficient when working on the graph algorithms. Traversing through the graph to determine the ImageType on Output-pins, loop-checking and the SelectionSort applied to the graph before exporting the data to an ImageJ Macro were developed using test-cases.

Graph implementations The final implementation of the elements of the graph was influenced by the original graph implementation of VisualAp as well as the first conceptual approach of the proof-of-concept prototype. The structure of both was very in sync and good for merging the classes. This process started out with an extension of the element's baseclass from VisualAp and adding the methods and properties of the prototype-classes.

37


March 20, 2009


Figure 14: Class diagram of graph related objects. Blue classes are carried over from VisualAp.

UnitElement The UnitElement is the encapsulation of the abstract concept of an image processor. The UnitElement contains the processor, in this case the macro-commando. This will be run when executing the graph. The class covers the visual representation of a unit, the parameter-management and contains Vector-lists for Inputs and Outputs. The UnitElement is an extended Node from VisualAp. For better code-reusability, two abstract classes are introduced as you can see in Figure 14. AbstractNode was added to VisualAp, for methods, that could be shared between ImageFlow's UnitElement and VisualAp's NodeBean. The second abstract class AbstractUnit contains the common methods that are used by the CommentNode-class and the UnitElement. CommentNode can be used to display notes and comments on the graph. UnitElement has two specialization. SourceUnitElement adds the use of a JFilechooser for adding files to the workflow and can determine their ImageType. BackgroundUnitElement determines its ImageType based on a selected Parameter. The UnitElement stores general information about the graph element like the name, position, icon and the Macro-object. The latter is a special class, which contains the

38


March 20, 2009


ImageJ-Macro-Syntax with variables and methods to replace the variables conveniently with the final values when the graph is processed and converted to a macro. UnitElements of any kind are provided by a special UnitFactory-class. The process of creating UnitElements is described in the chapter about Delegates management. Input Image processing units require a way to receive the image data for processing. The interface for receiving the incoming data is the Input-class. It is based on the Pinclass from VisualAp, but is extended with many more methods for conveniently traversing the graph and checking for different status. For example methods to determine if connecting this input could cause loops or if the ImageType of this input is compatible with the connecting output. The ImageType of the Input is defined in the units XML-description. The format for defining the compatible ImageTypes is based on ImageJ's PlugIn-handling even using the same constants. Every ImageType is mapped on a constant. Adding two constants means, that the Input can receive both ImageTypes. Inputs allow only one incoming Connection. Trying to add a second connection will result in deletion of the first. Output Whereas in VisualAp the Pin-class is used for both inputs and outputs, for ImageFlow it was decided to split those classes. Although both share a very similar interface the definition of the methods is very different. In contrast to the Input, which can only yield one connection at a time, Output-pins allow several outgoing connections. The Output is the element which allows branching the workflow. For this reasons many of the graph-traversing methods are implemented differently than in the the Input. However, the output also contains methods to help determine loops, compare ImageTypes and checking if the connected branches have display-units. The latter query is required for reducing the graph in the workspace to only the subset of 39


March 20, 2009


elements required for displaying a working graph. Outputs have an additional ImageType. SAME defines, that the output is of the same kind as the first Input of the parent UnitElement. When ImageTypes are compared, the method call passed recursively through the graph until a valid ImageType can be found. Parameters Unit-elements can allow Parameters to change their processed result. These parameters can be of different types. They all implement the Parameter-Interface, each type having a specialized class. The Parameter-objects are created by the use of a ParameterFactory-class. In the first version of the program all primitive parametertypes like boolean, integer, double and string are implemented. ImageJ would support more for example the use of Stacks and Regions of Interest, those were not considered for the moment and are a possible extension for future development.

Figure 15: Parameter-Dialog for an ImageCalculator-unit. Unit-Label is the display name of the unit and display toggles this unit as a Display-Unit. Both settings are available for all units.

The Parameters for a unit can be reviewed and changed in a dialog like the one depicted in Figure 15. These dialogs are dynamically created based on the parameters available for the particular unit. The parameters can be changed using common 40


March 20, 2009


widgets like textfields and checkboxes. While this works in most cases, it is not the optimal solution for parameters which expect user input. For example instead of a mouse click on the image, coordinates for certain actions need to be inserted manually. Connection A Connection describes the edge between two pins. It has an implied direction as Input- and Output-pin are explicitly stored. The Connection is extended from VisualAp's Edge-class. A few convenience methods were added, but it works exactly the same.

Delegates management The UnitElement is the class-representation of an actually instantiated element in the graph. To create instances of this object, a Delegate-class is required, which stores the information about the way a new node is to be created. ImageJ does not offer extensive meta-information about its plugins and macro commands. Therefore node-elements needed to be described externally in a unique XML-format attached in Appendix 2. This describes the meta-information and capabilities of this unit. The XML is read during the start of the program into a UnitDescription-object and contains the mentioned meta-information. Based on this description a UnitFactory-class can create new instances of UnitElement-objects. The UnitDescription is stored in the UnitDelegate-class, which represents this type of image processor in the GUI. Every Delegate represents and can create one type of unit.

41


March 20, 2009


Figure 16: Class-diagram for the Delegates management. UnitDescriptions are read from an XML and stored in a UnitDelegate. Based on this description, a Factory can create instances of this Unit.

Visual workspace The visual representation of the graph is handled by the GraphPanel, which itself is an extension of the VisualAp's GPanel. The original class covered most things involving the drawing of elements, connections and allowing mouse interactions including selection, moving and connecting. The extension involved two major changes, that were achieved by overriding the original methods. The first change were of mostly aesthetic nature and changed the drawing style. More important were additions to the graph-interaction. Graph representation The visual representation of the graph structure is drawn by the GraphPanel by overriding the paint()-method. It draws the units and all connections. The executable graph itself is a tree. One workspace can handle multiple trees at a time. Forests, trees, that are not connected which each other, are therefore allowed and can be dealt with. UnitElements are displayed in the color defined in the object and with an optional icon. The unit's label can be used to assign custom names to elements of the graph. 42


March 20, 2009


Figure 17: Simple example workflow. An Image is blurred and blended with its original. To the result noise is added. The final unit will be displayed.

Inputs are symbolized by a square-pin, whereas the arrow signifies Outputs-pins. Graph Interaction The GPanel did not check if actions taken by the user are actually permitted by the graph. A connection between two elements was always allowed. It allowed to connect two input-pins or to input audio data into an image processor. The user could only check his graph afterwards using a Check-Action in the GUI. The graph itself did not display any errors. The GraphPanel was extended to include on-the-fly checks, if user actions are permitted. When trying to connect two elements, it checks for various issues: whether you connect two pins of the same kind, input and output of the same unit, whether the new connection would cause loops like in Figure 18 and whether the ImageType of the output can be handled by the input. The reasons why connections are not allowed are instantly displayed in the graph as tooltips. Trying to add the connection nonetheless will not result in any action. This way, disallowed graph-operations are caught before they can corrupt the workflow.

43


March 20, 2009


Figure 18: Error-tooltip on trying to create a loop.

There is one exception to this rule. The ImageType-check displays problems, but still allows to establish a connection. The reason for this decision is based in the workflow-XML-format. Images included in the graph are saved in the XML with an absolute path. If you want to share the xml-file or you move the file, then the next time you open the workflow in the application would result in the loss of all existing connections, because the image-file can not be found. Without the file the initial ImageType can not be determined, which would cause uncertain connections to be discarded, the workflow would have no connections. To avoid this, connections with undefined ImageTypes are allowed, but will be displayed red with an explanatory message. This can be seen in Figure 19.

Figure 19: If an image is not found an ImageType can not be defined, subsequent edges are marked incompatible.

44


March 20, 2009


Document handling Important for the final application was the document handling. With the departure from image-focused editing to node-based editing workflows, the way to edit and save images has changed. Before, the image was loaded, edited and once finished - saved. The editing workflow was lost at the end. With node-based editing the focus is on the workflow. The image is just the source - one part of many in the process. This workflow is the new focus on which common document handling functions need to be applied, giving the user the possibility to load and save workflows from and to hard-disk. The workflow is saved in a special XML-format. The unit-definitions are included along with the graph-definition. The example workflow of Figure 17 is attached as Appendix 3. This workflow can also be exported to an ImageJ Macro. However, the macro-export involves a loss of information, because unused unit-elements will be stripped from the workflow. A reconstruction of the executable workflow based on a macro could be possible, but is not implemented. However unused elements would be discarded. The import feature merges the current workflow with a second one imported from a file.

Result Project evaluation The first version of ImageFlow was released in March 2009. This was a beta-release, which contained all features of the first stable version 1.0, but was not yet thoroughly tested. The purpose of the beta was to demonstrate the program in the ImageJ Community and receive feedback for the first version. While the program was developed without participation of the community so far, future developments can be based on user feedback and use cases. The application was tested on Microsoft Windows XP and Vista, Mac OS X 10.4, Linux and theoretically every Java Runtime Environment Java 1.5+. Screenshots of the beta release can be seen in Figures 20 and 21. Certain platform-specializations are not yet in place. 45


March 20, 2009


Figure 20: Screenshot of the final ImageFlow application in Mac OS X.

The practical use of the program will increase with the number of ImageJ commands supported. The first version included 20 filters to build a workflow from and documentation on how to create own processor-units. As long as the application is not stable, in production ImageFlow should be used with caution. For other appliances like teaching, it is save enough to use.

Figure 21: Screenshot of the final Imageflow application in Microsoft Windows XP

46


March 20, 2009


Development potential While all planed features for the first version are in place, there is a list of features, that were considered nice-to-have and did not make it into the first build. During the development a few features wishes came up, which were not possible to do for the first release. They can be considered as reference for future development. Loosing VisualAp VisualAp certainly was a good choice to have a basis for constructing the application. Its structure and rich amount of implemented features enabled quick progress and early results. While changes to the original code were necessary the amount was so low, that there were no doubts about taking the right foundation. However, with continued development issues came up, that could be dealt with, but required workarounds and code duplications. Those could have been addressed in a nicer way, if implemented in a fresh start. VisualAp is an application and no framework. It is very specialized for it is own purposes and does not allow for easy customization. Interfaces were created for some basic functionalities, but not consequently. To use it as a full-potential framework, proper interfaces will need to be defined. Building on top of the existing classes worked out, but often it required type-casting which should be minimized. The graph implementation could make more use of java-interfaces to unify as much as possible and to allow abstraction. Even more important would be some attention to the GPanel-class for the graph representation. With modifications it was good to use, but some of the internal implementations did not convince. Especially the drawing of unit-elements and the mouse interaction deserves a review. GPanel is an extension of the default JPanel. The draw-methods are overridden and contain the logic for drawing unit-elements, connections, selection-rectangles and more. Mouse events are used upon this drawing logic. For example a simple mouse click on the panel to pick a Node involves as polling through all unit-elements to see if one fits the mouse coordinates. A pushimplementation for example using MouseEvents could improve the event handling. One possibility could be to build unit-elements as extensions to JComponent, which 47


March 20, 2009


would allow for more better event handling and drawing capabilities. For example with the current GPanel implementation, it is no trivial task to add a simple ToolTip on a unit-element. A possible improvement of the drawing performance using double-buffered drawing of JComponents would be a welcomed side-effect. Tighter ImageJ Integration ImageFlow can be easily integrated in ImageJ as a Plugin by copying its folder to ImageJ's plugin directory. On the next restart it appears in the Plugin-Menu to start from within ImageJ. While this is a simple integration. More complex interactions between ImageJ and ImageFlow can be taken into consideration in future versions. This mostly raises the question, what interactions do make sense. ImageFlow was conceived to work as a stand-alone application. It does not require an ImageJ installation, but sharing installed plugins and macros is a feature that would improve the integration a lot. Future features certainly will be influenced by the way ImageFlow will be received by the community and by the way it will be used. Apart from the integration of ImageFlow as a plugin there are a couple of features, that ImageJ offers, but are not yet supported by the derived application. Certain editing tools are missing from the first version and should be added later. Region Of Interests(ROI) are selected areas on an image-object in ImageJ. They limit the area on which an effect is applied to. Creating and passing ROIs to the unitelements is not yet supported. The most basic ImageTypes of ImageJ are supported, yet Stacks can not be used for the moment. Adding these features also changes the unit-element requirements. More Input-pins could be added for various purposes with different capabilities. This will demand creative ways to convey information for example about the ImageTypesupport of a pin without sacrificing the clear user interface the program exhibits now. After all, the Deutsch limit dictates an upper limit of elements on a workflow and more possible element-types and more cluttered information do not make it easier to use.

48


March 20, 2009


Workflow improvements Building a workflow and understanding the intricacies involved into constructing complex graphs is important to every user and so the learning curve to use the software should be as low as possible. While the program itself is quite basic and tries to hide the complexity from the user, it still takes some time to learn how to use the unit-elements appropriately. The chapter about the Pipeline covers earlier, how the complexity of the system comes from the fact, that the user has to know the filters by hard. He needs to get a feel how to use them appropriately and how to achieve his goals without experimentation. At the moment the graph-construction is a blind process. Either the user has to know what he is constructing, otherwise he will only build the graph using trial-and-error. The current implementation should guide with the worst mistakes, but allowing better preview mechanisms could ease the guesswork for inexperienced users when building their workflows. As for more experienced users, with growing tasks the complexity of workflows grows with the number of units involved. To improve the representation of big graphs dynamic pin-positions could make it more comfortable to build vertical graphs, as the pin automatically is drawn on the side of the unit-element that faces its concurrent element. To reduce the number of units involved in a workflow, Unit-grouping would be an efficient way to collapse subgraphs of a workflow into one unit-element. A zoom-function is another thing, which could help to keep the overview for large graphs. Making units collapsable could also save space, where the current unit design consumes a lot real estate on the screen.

49


March 20, 2009


Summary Visual Programming is a very ambitious field of research, whose practical use is easily underestimated. The actual goal to construct complete programs can not yet be achieved, yet Visual Programming can open up possibilities for special appliances of data processing. Especially in professional applications node-based user interfaces make good use of this concept and allow to convey complex processing structures in an easily understandable workflow. The user experience is the key for a successful system even though the medium aims for professional users. Graph interaction is supposed to be joy, yet not distracting from the tasks at hand. The learning curve is steeper and at first the work may feel more blindly, but the pay off is a fully documented and reusable workflow. ImageFlow was a project, which brought this user experience to ImageJ. It was a good example of developing an alternative user interface for an established application. The project relied on existing source code and extended it to create new functionalities and new possibilities. This involved reviewing and deciding on proper frameworks to lay the basis for an application, as well as smartly using the offered API. Experiences in working with frameworks helps to find common solutions, which can be applied not just on the problem at hand, but whenever the same issue comes up. It helped to identify early mistakes in the design and to reuse code later on. ImageFlow made it possible to experience a full development-cycle beginning with design, implementation, testing and finally release. At the end, it proofed to be a working idea and a successful implementation.

50


March 20, 2009


Attachments Appendix 1

Appendix 1: Class-diagram of all graph-related classes.

Appendix 2 XML-definition for a ImageCalculator-Unit-element: Image Calculator ImageCalculator_Unit.png run("Image Calculator...", "image1=TITLE_1 operation=PARA_STRING_1 image2=TITLE_2 create PARA_BOOLEAN_1"); 0x6e91de Performs a mathematical calculation of two images Math StringArray Add Subtract Multiply Divide AND OR XOR Defines what math should be used to merge both images 51


March 20, 2009

Visual Programming of image processing algorithms using ImageJ 1 32 bit result boolean true create 32 bit result? 32-bit Input1 I1 31 false Input2 I2 31 false Output O -1 false

52


March 20, 2009


Appendix 3 XML-definition for the example workflow in Figure 17: 450 100 Add Noise 4 Add Noise run("Add Noise"); 0x6e91de Adds noise to an image Add Noise.input1 I 31 true Add Noise.output1 O 16 false

53


March 20, 2009

Visual Programming of image processing algorithms using ImageJ 180 50 Gaussian Blur 2 Gaussian Blur run("Gaussian Blur...", "sigma=PARA_DOUBLE_1"); 0x6e91de Performs a gaussian blur Radius double Radius of the gaussian kernel 4.0 Gaussian Blur.input1 I 31 true Gaussian Blur.output1 O 16 false

54


March 20, 2009

Visual Programming of image processing algorithms using ImageJ 30 100 example.png 1 Image Source open("PARA_STRING_1"); 0x9cba92 Opens a new image Image String The source unit needs the path of an image file. /Users/danielsenff/Pictures/example.png der-frust.png.output1 O 16 false 320 100 Image Calculator 3 Image Calculator 55


March 20, 2009

Visual Programming of image processing algorithms using ImageJ run("Image Calculator...", "image1=TITLE_1 operation=PARA_STRING_1 image2=TITLE_2 create PARA_BOOLEAN_1"); 0x6e91de Performs a mathematical calculation of two images Math StringArray Defines what math should be used to merge both images 1 Add Subtract Multiply Divide AND OR XOR 32 bit result boolean create 32 bit result? true 32-bit Image Calculator.input1 I1 31 false Image Calculator.input2 I2 31 false Image Calculator.output1 O 56


March 20, 2009

Visual Programming of image processing algorithms using ImageJ 16 false 30 40 my usual example 1 1 2 1 2 1 3 1 1 1 3 2 3 1 4 1

57


March 20, 2009


Content of CD Visual_programming prototype by Professor Barthel Source code for ImageFlow and VisualAp /Build /doc/

- various documentations to the application

/examples/

- example workflows

/Imageflow_.jar

- executable jar file

/xml_units/

- xml-definitions of processing units

/Documents /Bachelor-Thesis.pdf

- this paper

/JavaDoc/

- JavaDoc generated from the Project

/Source Code /ImageFlow/

- Source Code of the project (Code Revision of first Beta of March 1st, 2009)

/Visual_Programming prototype/ - initial prototype for Macro-Generation by Professor Kai-Uwe Barthel /visualap11branch/ for

- modified source code of VisualAp, this is required compiling ImageFlow

/visualap11original/

- original source code of VisualAp

58


March 20, 2009


Sources 1. Daniel LaLiberte, David McIntyre: “Visual languages”, 1994 http://www.hypernews.org/~liberte/computing/visual.html 2. Philip J. Mercurio, Pixel, “Visualization Tools: University of New Mexico's Khoros”, March-April, 1992 3. Universität zu Karlsruhe – Rechenzentrum: “Signal- und Bildverarbeitung mit KHOROS 2.1”, 1997, http://www.uni-koeln.de/rrzk/software/grafik/imageprocessing/khoros/

4. Wesley M. Johnston, J. R. Paul Hanna, Richard J. Millar: “Advances in Dataflow Programming Languages”, 2004 5. Reinhard Diestel: “Graph Theory” Electronic Edition 2005, Springer Verlag 1997, 2000, 2005 6. Rob Kremer: “Visual Languages for Knowledge Representation”, 1998: http://pages.cpsc.ucalgary.ca/~kremer/papers/KAW98/visual/kremer-visual.html

7. David Flanagan : “The Open Road: Looking Ahead to Java 7”, 9.8.2007: http://today.java.net/pub/a/today/2007/08/09/looking-ahead-to-java-7.html

59