An Overview of ICOLA and PDI: Incremental, Constraint-Based Graphics for Visualization Gregory M. Oster & Anthony J. Kusalik Department of Computer Science University of Saskatchewany September 1995
Abstract
Visualization is valuable in monitoring and debugging programs. The goal of the Wand research project at the University of Saskatchewan is to provide a framework and tools for rapid development of visualization aids for logic programminglanguages. Requirements for the tools are that they be language independent and easily tailorable. ICOLA (Incremental Constraint-based Object Layout Algorithm) is a new graphics system within Wand. ICOLA positions graphical objects according to object declarations and constraints which specify only their positional relationships. Two important features of ICOLA are that it supports both incremental insertions and incremental deletions of objects and constraints, and that it is capable of creating reasonable pictures from highly under-constrained speci cations. This paper surveys (1) PDI, the language implemented by ICOLA; (2) the incremental constraint solution algorithm of ICOLA; (3) a successful implementation in Prolog and C; and (4) results of a performance evaluation of the implementation. ICOLA is shown to successfully meet its objectives: it eciently implements a constraint-based, declarative graphics language geared toward visualization of program execution. Low-level graphics constructs are rendered unnecessary. Other highlights of the language are its simple constructs and the avoidance of absolute (coordinate) positioning.
1 Introduction Visualization is the portrayal of information in a graphical or pictoral form. The usefulness of visualization comes from the ability of humans to more rapidly obtain information from pictures than from text. In spite of this, however, text-based tools still prevail in many areas of computer science, including logic programming environments. ICOLA (Incremental Constraint-based Object Layout Algorithm) is a graphics engine in a visualization framework, called Wand, for monitoring and debugging logic programs. It was developed as part of ongoing research in logic programming environments [KuOs93a, KuOs93b, KuOs93c]. The major contribution of ICOLA is the incremental object layout algorithm. This new algorithm is capable of producing layouts from highly under-constrained picture descriptions, and without requiring the speci cation of absolute object positions. The algorithm supports not only the insertion of new objects and constraints, but their deletion as well. Heuristics are used to obtain consistent and \pleasant" (aesthetically reasonable) drawings despite under-constraint. The language implemented by ICOLA provides a basic, though powerful, set of capabilities. The constructs in the language are geared toward program visualization, and in particular, visualization of logic program execution. The language allows the speci cation of objects within a picture, constraints amongst objects, and attributes of objects. Constructs for absolute positioning of objects are absent from the language; relative positioning is provided instead and proves sucient. y Full postal address is: Department of Computer Science, 57 Campus Drive, University of Saskatchewan, Saskatoon, Saskatchewan, S7N 5A9, Canada. The authors can be also reached by email, via addresses
[email protected] and
[email protected], respectively.
Oster & Kusalik
2
Earlier versions of ICOLA and PDI have been designed and implemented [KuOs93c, Oste93]. The system and language described in this paper are the most well-developed and complete. The addition of features has not meant a reduction in eciency, however, as the current version of ICOLA out-performs earlier ones. Even though PDI was originally designed for the visualization of logic program execution, the language is general enough to support other graphics applications as well. Morton, for example, has successful used PDI and ICOLA as the \back end" of ADL [Mort95], an automatic software engineering diagram layout system. The remainder of this paper is organized as follows. Section 2 describes the Wand framework and earlier versions of ICOLA, and brie y surveys relevant background areas. Section 3 describes the basic concepts of PDI, the language implemented by ICOLA. In Section 4 the incremental constraint solution algorithm is shown, with its implementation discussed in Section 5. Section 6 presents results of performance evaluations of the system. The paper concludes with Section 7, which also suggests some topics for related, future research. Due to space limitations, a complete description of PDI and ICOLA cannot be given here. This paper presents only an introduction and overview. A more complete description | including a treatment of semantics, for example | is available elsewhere [Oste95].
2 Background and Motivation
ICOLA and its implementation draw on research in many other areas. This section brie y discusses these, including visual languages, constraint satisfaction systems, and declarative graphics. As well, the Wand framework is described, along with visualization in logic programming and a brief history of the development of ICOLA and PDI.
2.1 Visual Languages
A visual language is a language for computer interaction which is pictorial in nature [HeMa90]. Helm and Marriott [HeMa90] give four requirements for a picture speci cation (visual) language: (1) it must be expressive enough to encode the relationships between the pictures and their meaning; (2) it must allow highlevel abstractions similar to those people use for describing pictures; (3) it should have a formal, declarative semantics; and (4) the generation and recognition algorithms should be eciently implementable. Helm et al. [Hel+90] note that the meaning of a picture produced by a visual language is generally not dependent on the absolute position, size, or orientation of the objects in the picture. Instead, more abstract relationships, like relative position, are important. Visual languages exist in many di erent application areas. For example, Hirakawa and Jungert [HiJu91] discuss a visual information processing language which is used for de ning and retrieving information in an image database. Their language can be used via icons. Holt [Holt90] describes a visual programming language which is capable of representing variables and functions on those variables.
2.2 Constraint Satisfaction Systems
A natural way to specify the relative positions of objects is to use positional constraints (e.g. \above", \below", \touching"). Such informal constraints can be transformed into a constraint satisfaction problem (CSP) where variables represent the locations of objects and mathematicalformulae represent the constraints. The solution of such a problem de nes the absolute positions of the objects in the speci ed picture. Over-constrained and under-constrained systems of constraints are problematic. Approaches to deal with over-constrained systems include constraint relaxation, as in the system of Helm et al. [Hel+92]. Underconstrained systems may be solved through the use of heuristics, as in the COOL (COnstraint-based Object Layout) system [KaKa91]. Other solvers, like CHIP [Din+88], simply present the solution to an underconstrained system as a set of symbolic expressions. Incremental constraint satisfaction systems maintain a set of variables and constraints evolving over time. Each increment (change) is combined with the previous solution, or internal information used in arriving at that solution, to produce a new solution. Incremental techniques for constraint satisfaction have proven quite useful in many systems [Fre+90, Men+93, Stuc91a, vanH90]. They are considered superior for
Oster & Kusalik
3
applications where the user interactively speci es the constraints, and the application is expected to react quickly [Hel+92]. An increment to a system of constraints is typically understood to be the addition or insertion of constraints or variables. However, in this work an increment is simply a change in the system of constraints, and includes both the insertion and deletion (removal) of variables or constraints.
2.3 Graphical Considerations
E ective application of constraint satisfaction technology to graphical applications requires more than simply determining coordinates for objects. A number of issues in the areas of graphics and diagram layout must be considered. For example, Helm et al. [Hel+92] note that for interactive graphics-based tools constraint solvers must be incremental, the user feedback must be fast, and the rate of feedback should be high. The \Principle of Least Astonishment" [Hel+93] is also very important. This principle states that the transitions between consecutive incremental solutions should be smooth, such that the user is easily able to track the movement of the objects. Rowe et al. [Row+87] also acknowledge that stability between successive drawings is extremely important. The ndings of Rowe et al. come from rst-hand experience with DAG, an interactive browser for directed graphs. In general, the graphical presentation of information requires care that the images used present the information in a clear, consistent, and unobtrusive way.
2.4 Declarative Graphics
Declarative graphics systems specify picture layout without reliance on side-e ects. Many logic programming systems have some sort of \declarative" graphics capabilities. Examples include Graphics Manager (GM) in SICStus Prolog [SICS94], ProXT and ProXL in Quintus Prolog [Quin91], as well as the graphics facilities in Wild LIFE [At+94]. Such facilities di er in the level of abstraction of the graphical constructs, as well as the degree to which the constructs are strictly declarative (e.g. avoidance of side-e ects). Helm and Marriott [HeMa90] also discuss the declarative speci cation of graphics in the context of logic programming. Their declarative picture speci cation language combines ideas from constraint satisfaction and grammarbased formalisms. Haarslev and Moller [HaMo90] give a declarative formalism for specifying graphical layout. Their \boxand-glue" approach is similar to the layout techniques used in TEX [Knut84] and is extended for use in specifying graphical user-interfaces. Johnson and Harrison [JoHa91] present PRELOG, a tool for Presenting and REndering LOGic speci cations of interactive tools. PRELOG uses ideas from temporal logic to create a temporal, declarative graphics engine.
2.5 Constraint-based Graphical Tools
Constraint-based graphical tools are used in logic-program visualization, user interface construction, VLSI layout, and other applications. For example, ThingLab II [Fre+90] is an interactive user-interface construction tool which uses constraint satisfaction techniques. The DeltaBlue algorithm found in ThingLab II shows the power of incremental methods. Roach [Roac84] describes a constraint-based symbolic layout language RPL (Rectangle Placement Language) for VLSI diagram layout. In addition to positional constraints such as above and leftof, RPL also supports restraints, which resemble user-de ned, high-level constraints on the layout. The COOL system [KaKa91] allows pictures to be speci ed as objects and constraints. COOL also supports constructs for laying out a collection of objects in a speci c pattern. Cruz [Cruz93] presents the U-term language for declarative speci cation of pictures. The system is object-oriented, and makes use of absolute coordinates.
2.6 Wand and the Origins of ICOLA
A signi cant amount of work is being done in logic programming to provide tools for visually monitoring and debugging logic programs. Systems such as MUST [SvSu90], TPM [EiBr88], and VisAndOr [Car+93] have
Oster & Kusalik
4
already proven to be quite useful. Unfortunately, the tailorability of these systems is often limited. Systems used for sequential Prolog tracing cannot typically be used for parallel Prolog tracing, for example. Within the parallel Prolog realm, a tracer designed for only AND-tree computations cannot produce OR-tree traces. The nature and presentation of information obtained from the traces is also typically preset, and not easily changed. The Wand framework addresses these shortcomings. The main goal of Wand [KuOs93b, KuOs93c, Oste95], a research project at the University of Saskatchewan, is to provide a set of tools for rapid prototyping of visualization and debugging aids for logic programming languages. Design goals for the tools are that they be language independent and easily tailorable.
meta-interpreter language system run-log data stream
transformation spec.
transformation engine
internal knowledge-base configuration spec.
display engine
visual display
visualization tool
Figure 1: Overview of the Wand framework. The overall design of Wand is found in Figure 1. A meta-interpreter generates trace information from the computation being monitored or debugged. A transformation speci cation controls the mapping between trace information and an internal knowledge base. The rules for converting the data in the internal knowledge base into a graphical form are contained in a con guration speci cation le. The con guration speci cation language is a constraint-based \display description language", similar in style to de nite clause grammars. The terminals of this grammar are basic graphical constructs provided by the graphics or display engine. The grammar rules are used as productions, with their arguments instantiated to information from the knowledge base. Finally, the display engine generates a visualization for the user. As part of Wand, ICOLA (Incremental Constraint-based Object Layout Algorithm) [KuOs93c, Oste95] was developed to provide high-level graphics capabilities. ICOLA can produce images incrementally from graphical object speci cations and constraints among those objects. The language implemented by ICOLA is PDI (Picture Descriptions for ICOLA). Figure 2 shows how ICOLA ts within Wand. (Dashed lines show expansion of detail in the gure.) While the PDI language constructs are easily produced by hand, they are expected to be machine-
Oster & Kusalik
5 internal knowledgebase
configuration spec.
visual display
display engine
database info. terms
configuration spec.
grammar rules
grammar engine
constraintbased display description (PDI)
constraintbased declarative graphics engine
visual display
user interface and control
constraintbased display description
preprocessor
postprocessor
visual graphics engine
constraint network solver (in C)
display
Prolog + graphics engine ICOLA
Figure 2: ICOLA within the Wand system. generated. In the Wand framework, for example, the implementation of a Wand tool will hide PDI from the user. Any application which needs high-level constraint-based graphics capabilities and can produce a stream of PDI constructs is capable of using the facilities provided by ICOLA.
3 Picture Descriptions for ICOLA PDI, the language implemented by ICOLA, is a visual information processing language with linear constructs in the terminology of Chang [Chan87]. That is, objects in the language have an inherent visual representation, and the language constructs can be represented using text. The language uses positional constraints to describe the relationships between locations of objects in the picture. Attributes are used to associate non-positional properties with objects. The following notational conventions are used in presenting PDI: typewriter font is used for actual language constructs; Small Caps font, for object classes; and italics, for emphasis. Language statements have the syntax of ground Prolog terms.
Oster & Kusalik
6
3.1 Objects and Selector Functions
Table 1 lists the types of graphical objects in PDI. A class hierarchy exists for these objects. All objects are in the universal class, Obj. The Obj class is comprised of two subclasses, BB and Line. The latter includes line and arc objects. These objects are distinguished because they are positioned by ICOLA after other objects have been positioned. Objects of class BB all have a rectangular bounding box1. Class BB is further divided into subclasses, but additional detail of the class hierarchy, while de ned, is not necessary for this discussion.
Type
box oval point line triangle bitmap arc complex
text
Table 1: Descriptions for the objects in PDI. Description an extensible rectangle. an extensible ellipse. a single \dot". a straight line between two objects. an extensible triangle. a user-de ned bitmap. a curved arc between two objects. a \grouping" object. It can be used to combine a number of objects. Any operation or constraint applied to a complex object applies to all members of the complex object as a unit. a \text string".
To declare that an object exists in the (output) picture, it is only necessary to specify its type and an associated name. Names are simply Prolog atoms. An example of an object declaration is: object( parent node, box ).
The only exception to this is for complex objects, where the declarations group(: : : ) and end group(: : : ) surround the speci cation of the body of a complex object. An example of a group object declaration is: group( child branch ). object( child node, box ). object( child label, text ). end group( child branch ). complex
In general, the body of a object consists of object, constraint, and attribute declarations. Complex objects may be nested. Each level of nesting determines a level of name scoping. (Object names must be unique within each local scope.) A special complex object called root denotes the entire picture. All objects are automatically contained within the root object. A feature of complex objects is that their de nition can be \re-entered" or extended. Thus a constraint that the textual label child label in the object child branch (above) should be inside the rectangular object child node is easily added to the de nition of that complex object via: group( child branch ). constraint( inside, [ child label, child node ] ). end group( child branch ).
Selector functions allow the explicit speci cation of the left, right, top, or bottom sides of an object's bounding box. Selector functions can be used with any object in class BB. The left side of the bounding box of object node 2 is speci ed by left(node 2), for instance. The other sides of the bounding box are speci ed similarly. 1
A bounding box is a rectangle which is just large enough to entirely surround an object.
Oster & Kusalik
7
3.2 Constraints
Constraints in PDI specify positional relationships among the objects. Table 2 gives a list of the constraints with a brief description for each. The constraints in ICOLA are divided into two categories { basic and aliased. Basic constraints are fundamental in the language.
Constraint
left of right of horizontal distance above below vertical distance connects inside touching aligned
Table 2: Constraints in PDI. Category Description basic one object is to the left of another. aliased one object is to the right of another. basic left of, by at least a speci ed amount. basic one object is above another. aliased one object is below another. basic above, by at least a speci ed amount. basic two object are connected by a line or arc. aliased one object is contained inside another. aliased one object borders on another in a speci ed axis, such that the objects have no space between them. aliased align the speci ed edges of two objects so that they share a common horizontal or vertical axis.
Aliased constraints map to one or more basic constraints. Thus, they are not strictly required for providing
the full functionality of the system. They do, however, make certain relationships (like \inside" and \touching") much easier to specify. Aliased constraints are automatically re-expressed in terms of basic constraints by the pre-processing stages of ICOLA. For example, the constraints constraint( right of, [ left( node 2 ), right( node 1 ) ] ). constraint( touching, [ bottom( main label ), top( aux label ) ] ).
are re-expressed as
constraint( left of, constraint( vertical constraint( left of, constraint( left of, right of
[ right( node 1 ), left( node 2 ) ] ). distance, [ 1, bottom(main label), top(aux [ left( main label ), right( aux label ) ] [ left( aux label ), right( main label ) ] left of
label) ] ). ). ). touching
As can be seen, the constraint maps to a single constraint, while maps to three constraints. Certain constraints expect selector functions as values. For convenience, ICOLA provides default selector functions if any of those selector functions are not present. left of is one such; for example, constraint( left of, [ right( node 1 ), node 2 ] ).
is taken as
constraint( left of, [ right( node 1 ), left( node 2 ) ] ).
In general, when a constraint is speci ed between two objects, the default selector functions are chosen so that the constraint is applied to the entirety of each object. To achieve consistent behavior, aliased constraints are converted to basic constraints before the default selectors are applied.
3.3 Attributes
Attributes are non-positional object properties, such as the colour of a rectangle or whether a line has an arrowhead. All attributes have default values assigned when the object is declared. Attributes can be modi ed. The \colour" attribute for object child node, for example, is easily changed to \red" via: attribute( child node, colour, red ).
The new attribute value replaces the previous one. Values for attributes are of various types. For instance, the colour attribute takes the name of a colour (such as \red" or \blue"), while the text attribute takes a string as in
Oster & Kusalik
8
attribute( main label, text, "Succeeding Goal" ).
Certain attributes, like colour and visible, apply to most objects. Other attributes, like filled or text, only apply to speci c object types. The root object has a number of special attributes which control default properties of the entire picture. The most important of these is the default distance attribute which speci es (default) inter- and intra-object distances.
3.4 Insertions versus Deletions
All the previous examples of PDI statements have assumed that the objects and constraints being declared are being added to (inserted into) the picture speci cation. For syntactic completeness, an insert construct is de ned. Thus object( parent node, box ).
is taken as
insert object( parent node, box ).
The two forms can be used interchangeably. PDI and ICOLA also support constraint and object deletion. To remove a constraint or object a delete construct is used. For example, delete object( parent node, box ).
eliminates the rectangular object named parent node from the constraint system (and picture). Removing a constraint involves a similar statement; for example, delete constraint( left of, [ right( node 1 ), node 2 ] ).
Defaults are applied to deletions before a constraint is removed. For example, the statement above actually results in deletion of the (equivalent) constraint constraint( left of, [ right( node 1 ), left( node 2 ) ] ).
Aliased constraints can also be deleted. ICOLA will expand the aliased constraints into the component basic constraints and remove those. An object can only be removed if it is not (or no longer) involved in any positional constraints. Accordingly, when removing complex objects, all constituent objects and constraints inside and related to that object must rst be explicitly deleted. A positional constraint can be removed only if it exists in the current system of constraints. Thus, it must have been previously inserted and not already been removed.
3.5 Additional Information and Commentary
PDI also contains control constructs which control the ICOLA constraint engine. The most important of these is reset which re-initializes the solver, removing from the internal system all objects and constraints encountered since the previous reset. Control constructs in a picture description can be ignored when determining its meaning. All \white space" and Prolog comments in a PDI speci cation are ignored by ICOLA. Picture descriptions in PDI can become quite large. Fortunately, PDI is only an intermediate language and not meant to be input directly by the user. PDI speci cations were generated by hand in this work solely for testing and demonstration purposes. This section has presented only an introduction to PDI. More detail is available elsewhere [Oste95]. For example, a semantics has been de ned for the language. Also, scoping and object naming are, in fact, more sophisticated than might be suggested by the examples here. Another example of a speci cation in PDI is given in Figure 3 of Section 4.5.
4 Incremental Constraint Solution Algorithm Given a picture speci ed with PDI, absolute coordinate positions must be determined before an actual image can be generated for a user. The positional values must satisfy the constraints imposed by the picture speci cation. ICOLA performs this function.
Oster & Kusalik
9
4.1 Increments: Insertions and Deletions
As mentioned in Section 2.2, an increment encompasses both insertions and deletions. Even though ICOLA supports incremental deletion of variables and constraints, variables and constraints cannot be removed (from the current constraint system) arbitrarily. For a constraint to be deleted, for example, it must have been explicitly inserted earlier. This requirement ensures that the behavior of a deletion is intuitive and well-de ned. As well, the deletion of a constraint does not necessarily mean that the negation of that constraint is true; e.g., the deletion of the (previously inserted) constraint a > 10 does not imply that a 10 holds. Deletions always result in a less speci c (or equally speci c, if the deleted constraint was redundant) constraint system. A design goal of ICOLA is that the image produced after a deletion will be exactly the same as the image produced if the original insertion and deletion were never in the picture description. This is termed the cancellation property in this paper. As will be shown, this property is achieved.
4.2 Notation and Foundations
ICOLA operates on a set of variables and a set of constraints. The variables are of the form aleft , aright , atop, and abottom for some object a. These variables represent the left, right, top, and bottom edges, respectively, of the object. All variables represent absolute coordinate positions and have the non-negative integers as their domains. The value 0 corresponds to either the left-most or upper-most point of the graphical display, depending on whether the coordinate is in the horizontal or vertical dimension, respectively2 . Since the domain for each variable is in nite, the maximum size of an image is theoretically in nite. Constraints are formulae of the form x + y, where 0. Constraints are induced by both object and constraint declarations (except from the declaration of Line objects and the connects constraint). For example, the PDI statement object( a, box ).
induces several formulae, including roottop + 0 atop atop + 20 abottom abottom + 0 rootbottom where 20 is the default intra-object distance. Similar formulae are also added for aleft and aright . In general, these formulae constrain the object to be inside the boundaries of the group to which it belongs. The statement constraint( left of, [ left( a ), right( b )]).
corresponds to aleft + 20 bright . In this case, 20 is the default inter-object distance. V denotes the set of all variables in the current system of constraints. C denotes the current set of formulae (constraints). A solution, S, to the set of constraints is a set of variable assignments for which all the formulae are true. Let G be a weighted, directed graph where nodes in G correspond to variables in V , and each (x + y) 2 C corresponds (one-to-one) to an edge in G of weight from node x to node y. A variable y depends directly on x if y 6= x and there is an edge in G from x to y. A variable y depends on x if y 6= x and there is a path in G from x to y. Hence, dependency is the transitive closure of the \depends directly" relation. y depending on x means that x must have its value determined before y. It also means that the position signi ed by x is left of, or above, y (according to whether the constraint is in the horizontal or vertical dimension, respectively). This notion of dependency is critical to the design of ICOLA. In particular, the value of variable y will not be used in any computation until the value of each variable x on which it depends (i.e. variable x for all (x + y) 2 C) is known to be correct.
4.3 Consistency Check
Prior to attempting to determine a solution given an increment, ICOLA rst performs a consistency check { a check that the increment will not lead to an inconsistent system of constraints. The constraints C are
2 Thus the coordinate system has origin in the top left corner of the display, with increasing values corresponding to points lower or to the right.
Oster & Kusalik
10
consistent if the graph G (induced by C) is acyclic. Otherwise it is the case that the position corresponding to some x is left of (or above) some y (di erent from x), while simultaneously that y is left of (or above) x. In such a case, there can be no solution. A constraint or object deletion can create no cycles. Deletions therefore never lead to inconsistency. Given the insertion of a constraint, say x + y, a consistency check is performed by searching for a path from y to x in the graph G. If the path exists, then the increment would result in a cycle in the graph, and the increment is not allowed. If no path is found, the addition of x + y to C will not result in an inconsistency, and the increment can be allowed. We require that the system of constraints be consistent before and after each increment.
4.4 Solving an Under-Constrained System
Once the system of constraints (given the increment) is known to be consistent, it is then necessary to nd a solution. Since the constraint system in ICOLA is highly under-constrained, however, there are an in nite number of individual solutions (a family of solutions) and, in general, the only nite representation of such a solution is a symbolic one. A symbolic solution could be provided by applying algebraic (symbolic) identities and reductions to the formulae in C, and presenting a reduced set of formulae. A symbolic solution is inadequate, however, as drawing an image corresponding to the picture description requires actual numeric values for the coordinates of the objects. These problems with under-constrained systems are typically avoided through the application of heuristics. In COOL [KaKa91], for example, under-constraint results in two additional constraints being added. These constraints place the average position of the objects at the centre of the picture or sub-picture. ICOLA determines the actual numeric values by rst nding a subsuming set of constraints. However, this reduced set is also symbolic, and is therefore still inadequate. Heuristics, driven by aesthetics and common sense, are then applied to the subsuming set of constraints. These heuristics are: 1. An object's right edge will be as far left as possible. Similarly, an object's bottom edge will be as close as possible to the top of the layout. 2. Objects will tend to be as small as possible. The rst heuristic has priority over the second. This is necessary so that the constraint solver produces a unique solution.3 The realization of these heuristics within ICOLA is accomplished by a breadth- rst-like traversal through graph G. The result is that for any object a, the values of variables aright and abottom are minimized, after which the values of aleft and atop are maximized. ICOLA achieves its goal of having the cancellation property for increments (see Section 4.1). This is because the solution is over all the constraints, there is no prescribed order in which the constraints must be considered, and the heuristics impose a unique solution. There is no di erence, therefore, whether a constraint is inserted and then deleted, or never inserted at all. The only restrictions, therefore, are that objects be declared before they are referenced in a constraint, and that objects and constraints are inserted before they are deleted.
4.5 Incremental Constraint Satisfaction
Like most incremental systems, ICOLA maintains an internal state. This internal state contains a set of variable assignments where each variable has the lowest possible value such that the constraints are still satis ed. When determining a new solution, ICOLA combines the increment with the previous solution and the internal state. The result is a new solution to the new system of constraints and a new internal state. ICOLA attempts to maximize its re-use of both the previous solution and the internal state. The goal is to change only those portions of the solution and the internal state which require updating. Consider the (annotated) image in Figure 4(a). This image corresponds to the picture description (read from left to right, top to bottom) in Figure 3. The incremental deletion of the constraint delete constraint( left of, [ left( c ), right( g ) ]).
results in that image being transformed into the (annotated) image in Figure 4(b). The goal of the algorithm in this situation, therefore, is to only re-compute the positions of objects g and h. 3
Oster [Oste95] provides an example and detailed explanation.
Oster & Kusalik
11
object( a, box ). constraint( left of, constraint( left of, constraint( left of, constraint( left of, object( g, box ). constraint( left of, constraint( left of, constraint( above, [ constraint( above, [
[ [ [ [
a, b, c, d,
[ f, [ g, a, f a, h
b c d e
object( b, box ). object( c, box ). object( d, box ). object( e, box ). object( f, box ). constraint( left of, [ left(c), right(g) ]). object( h, box ). constraint( left of, [ h, e ]). constraint( above, [ a, g ]).
]). ]). ]). ]).
g ]). h ]). ]). ]).
Figure 3: Picture description for the incremental deletion example. a
f
b
c
d
g
h
(a)
e
a
b
c
f
g
h
d
e
(b)
Figure 4: Change in the picture resulting from the deletion of constraint( left of, [ left(c), right(g) ]). The following discussion examines in more detail the steps of the algorithm. The increment (deletion) above is used to illustrate the algorithm's behavior at each step. A deletion is speci cally chosen since deletions are the less frequently supported operation in incremental constraint solvers. The steps in the algorithm are as follows: 1. Identify if the internal state must be updated or if a new solution needs to be computed. If an inserted constraint is not implied by the internal state, or a deleted constraint is not redundant with respect to the internal state, then the internal state must be updated. While the internal state may not need to be updated, it still may be necessary to update the solution. If an inserted constraint is implied by the previous solution, the (new) solution is the same as the previous. Similarly, if a deleted constraint is redundant (implied) then the previous solution is re-used. Otherwise a new solution must be computed. For our example, the deleted constraint is not redundant with respect to the internal state, nor the previous solution (for the system of constraints corresponding to Figure 4(a)). Hence, it is necessary to update both the internal state and the solution. 2. Determine the variables which may require updating. For the insertion of a constraint x + y, y must be considered. Fortunately, most other variables will still satisfy the constraints. For example, for all variables p such that p + q 2 C, if none of these variables p need be changed, then neither does q. Such variables q (other than y) need not be examined. As well, if the deleted constraint is x + y, then only those variables which depend on y may have to change. They are the only ones which need be considered. In the example in Figure 4(a), the variables gright , hleft , rootright , hright , eleft , and eright are identi ed as possibly requiring updating. 3. Find the set of subsuming constraints for the left-to-right positioning (similarly for top-to-bottom) and satisfy the rst heuristic. For some variable y, for example, the following (e ective) constraints might exist: 20 + 50 y, 30 + 80 y, 70 + 20 y. The subsuming constraint is 30 + 80 y, or 110 y. According to the rst heuristic y is chosen to be
Oster & Kusalik
12
equal to 110. In general, for each variable y 2 V , the value of y must be max(fx + j (x + y) 2 C g). This ensures that all variables are assigned the lowest possible values which still satisfy the constraints. This updating is also performed on the internal state. In our example, only gright , hleft , and hright are changed in the new solution and the internal state. Variables eleft and rootright are updated, but remain unchanged. Further optimizations in this step indentify that eright does not require updating. 4. Determine which variables are subsequently a ected by the modi cations made in step 3. Of concern are those variables where the right-to-left (similarly, bottom-to-top) positioning may be incorrect. As in step 2, most variables will not require updating. Obvious candidates for updating are the variables changed in step 3, as well as any variables on which they depend. Further, if the constraint removed was x + y, then any variables on which x depends are also candidates.4 Fortunately, not all these candidates actually need to be updated. For example, those which represent bottom or right edges of objects need not be considered. As well, if p is a candidate variable and for all constraints p + q 2 C no such q is changed, then p need not change. In our example, the candidates identi ed in this step are: hleft , cleft , gleft , rootleft , aleft , bleft , and fleft . 5. Find the set of subsuming constraints for the right-to-left positioning (similarly for bottom-to-top) and satisfy the second heuristic. In general, the value of each variable x, where x represents the left or top edge of an object, must equal min(fy ? j (x + y) 2 C g): The above ensures that these variables are given the maximumpossible value allowed by the constraints and by the current assignments to variables representing the right or bottom edges of objects. Thus, in our example only gleft is actually changed. Variables hleft , cleft , and rootleft are updated, but remains unchanged. Further optimizations in this step identify that variables aleft , bleft , and fleft do not require updating. Other optimizations are present (e.g. as shown by the example in step 3), but space limitations prevent their description. The general ICOLA algorithm is independent of whether the increment is a deletion or an insertion. At a more detailed level, the only steps which need to take into account the type of increment being performed are steps 1, 2, and 4. This generality is born out in the implementation where the code for various steps is generously re-used between the insertion and deletion cases.
4.6 Additional Information
Although objects of class Line are represented by nodes in the graph G, their positions are calculated after objects of class BB. This is because the location of Line objects is determined from the positions of the bounding boxes of the objects at their end points. Only a few straightforward calculations are required to determine the exact location of the line or arc, and heuristics are once again applied to achieve an aesthetically pleasing layout. Discussion of the layout of objects of class Line, plus extensive detail on the ICOLA algorithm is provided elsewhere [Oste95]. That presentation also analyzes the time complexity of the algorithm. Here we mention only that it is at worst O(v + c), where v is the number of variables and c is the number of constraints being considered, but that the optimizations (in reducing the number of candidates, and the number of variables and constraints actually considered) typically result in a lower expected complexity. The quantitative performance results for the algorithm given in Section 6 support this claim. 4
In fact, this case is subsumed by the former.
Oster & Kusalik
13
5 ICOLA Implementation A system for generating pictures speci ed by PDI has been implemented. The system is also called ICOLA, since its heart is the ICOLA algorithm described in Section 4. The system is implemented in Prolog (SICStus Prolog) and C. This section describes that implementation. Constraint Solver Variables and Constraints PDI input
Preprocessor
Abstract Object Descriptions
Constraint Solution Postprocessor
Graphics Primitives
Figure 5: The high-level internal operation of the ICOLA implementation. Figure 5 and the lower portion of Figure 2 give structural and information- ow overviews of the implementation. The constraint solver, the implementation of the ICOLA algorithm, is written in C. PDI statements must be converted to a form acceptable to the solver (pre-processing), and a graphical display must be generated from the constraint solution (post-processing). Both the pre- and post-processors are written in Prolog. The pre-processor parses PDI statements, performs name lookup and dictionary operations, expands aliased constraints and applies defaults (see Section 3.2), handles attribute changes, and translates the picture description into an internal form used by the constraint solver. The post-processor merges the set of object coordinates provided by the solver with abstract object descriptions passed to it by the pre-processor. Using this information, it generates calls to the graphics predicates of GM [SICS94] (a graphics facility in SICStus Prolog). Abstract object descriptions include such information as captured by the attributes of objects. For trial of the implementation, a user interface was added to allow control of the program and viewing of the results. In the pre- and post-processors, dictionaries (in Prolog) maintain the current set of symbolic object labels or identi ers, and a unique numeric identi er for each object. The numeric identi ers are assigned by the pre-processor and used internally by the constraint solver. These identi ers are not available for use in picture descriptions. The pre-processing phase ensures that the input is syntactically well-formed and valid. It determines a unique name for each object, and ensures the existence of referenced objects. As well, the pre-processor is responsible for verifying that the class of an object being deleted matches the class of the object when it was inserted. The domain for variables in the constraint system is the set of non-negative integers. No units are associated with these values. In the implementation, values of these variables (i.e. coordinate values) are mapped to pixel positions in the output display. While the maximum x and y coordinate values for each object are theoretically in nite, the current implementation sets an upper bound of 232 ? 1. This bound is practical given typical workstation graphics capabilities and hardware limitations. Also, a limit of 10,000 objects per picture has been imposed for the current implementation. Experience has shown that this limit is rarely reached. No software limit is imposed on the number of constraints; it is restricted instead by the memory available to store the graph encoding those constraints. Appendix A shows a \screen dump" of the user-interface for the ICOLA implementation. The interface contains such features as buttons (for control), scrollable subwindows (for output images and text), and pop-up windows (for error messages and user-input). While the scrollable main viewing area may only show a portion of the entire image at any time, an \aerial view" always contains the entire image. The scrollable text window contains the constraints de ning the current image. Picture descriptions can be either entered interactively by the user, or read from an input le. A non-incremental or \batch" algorithm is also present in the ICOLA implementation, but is not discussed in this paper. More detail regarding the implementation, such as the heuristics for placing lines, is given elsewhere [Oste95].
Oster & Kusalik
14
6 Performance Results If a constraint solver is to be practical in a visualization application, ecient real-time performance is critical. Performance analysis has therefore been carried out for the ICOLA system [Oste95]. The runtime performance of ICOLA has also been compared to that of a general constraint solution system solving the same (object positioning) problem [Ho 95]. Highlights of those analyses are given here. A primary performance metric for a constraint solver, in general, is the speed with which it solves a set of constraints. In incremental systems, the time required to nd a solution after each increment is the most important. Initial experiments indicated that the time for ICOLA to solve an increment was typically less than the resolution of system clock on the test-bed workstation (a Sun SPARC-10, running SunOS). More thorough investigation determined that an increment typically requires less than 1 millisecond to solve. Morton also reports that only a small portion of time is spent in the constraint solver when a diagram is being generated by his ADL system [Mort95] using ICOLA. Figure 6 illustrates more performance results. Shown is the performance of the ICOLA incremental solver | for increasing numbers of constraints | versus a general-purpose constraint satisfaction tool (the nite domain library of ECLi PSe [Bri+95]) and ICOLA's \batch" (non-incremental) solver. For the \batch" solver, results for solving both the entire picture description only once and nding a solution after each increment are shown. \Average case" picture speci cations were used. The results were gathered on the same test platform (a Sun SPARC-10, running SunOS) to allow comparison of the values obtained. As seen in Figure 6, the incremental algorithm of ICOLA out-performs both the general-purpose constraint satisfaction tool (ECLi PSe ) and the non-incremental algorithm of ICOLA (when used after each increment). It also nearly performs as well as a non-incremental algorithm when only a single, nal solution is required. When only a single, nal solution is required, however, the non-incremental algorithm is the fastest. This not unexpected, since non-incremental constraint solvers require less overhead. 20 Batch after each increment Incremental Batch Eclipse 15
Solution Time (Sec.) 10
5
0 0
200
400 600 Number of Increments
800
1000
Figure 6: Performance of the incremental ICOLA solver vs. other solvers. ICOLA also performs very well on degenerate or pathological picture descriptions. In these picture descriptions, each new constraint forces the re-computation of the position of all objects. For a degenerate situation involving 8001 objects and 15998 constraints, an increment which involves re-solution of all object positions is accomplished in an average of 119.1 milliseconds. The degenerate case served as a benchmark when comparing the current version of the ICOLA algorithm (and its implementation) with precursory
Oster & Kusalik
15
versions. The previous version is typically six times slower (in runtime) on the same examples. The performance results indicate that ICOLA is suitable for \on-line" or interactive user-applications. The non-incremental algorithm is ecient for \batch mode" applications. Since a user can switch between incremental and non-incremental solvers when using the ICOLA implementation, a user can select whichever provides the best performance for the given situation.
7 Discussion and Conclusions ICOLA is an incremental constraint solution algorithm. That is, it uses incremental techniques for updating internal data structures and determines a new solution without the cost of complete recalculation. While many other incremental constraint solvers support only the addition of variables and constraints to the system of constraints, ICOLA supports deletions (again in an incremental fashion) as well. ICOLA is able to produce reasonable pictures despite under-constrained picture speci cations. Finally, the ICOLA algorithm is ecient, both in terms of theoretical complexity and measured runtime performance. This is partially due to the restrictions on the application domain. The language for specifying pictures to ICOLA, PDI, is constraint-based and declarative. It is suciently expressive to quickly specify useful images. PDI's level of abstraction avoids low-level detail (such as absolute size or location of objects), yet it is detailed enough to describe complex pictures. PDI mostly satis es the criteria of Helm and Marriott [HeMa90] for a visual language (see Section 2.1). The only de ciency is that currently only an operational semantics is de ned, rather than a full declarative semantics. Both ICOLA and PDI have undergone several revisions [KuOs93a, KuOs93c, Oste93]. The system and language described in this paper are the most well-developed and complete. In addition, ICOLA has already found use outside the Wand framework. For example, it has already been used in a meta-CASE system for automatic diagram layout [Mort95].
7.1 Future Work
Since ICOLA provides an ecient declarative graphics platform, a logical next step is to utilize it within the Wand framework for generation of execution monitoring and debugging tools. Enhancements can also be made to the ICOLA implementation. For example, certain types of objects (such as arcs) are poorly realized due to limitations of the current graphic output facility (i.e. GM of SICStus Prolog). Use of a better graphics library would rectify these shortcomings. Other possible enhancements to ICOLA include re-use of internal object identi ers and eliminating the current 10,000 object limit. Alternative line-positioning algorithms (for placing objects of class Line) can also be explored. Currently, the performance analysis of ICOLA is being extended. A more accurate average time for solution of an individual increment is being sought, as well as a more thorough comparison with general constraint solution systems.
References [At+94] [Bri+95] [Car+93] [Chan87]
At-Kaci, H., Dumant, B., Meyer, R., Podelski, A., and Van Roy, P. The Wild LIFE Handbook (pre-publication edition), DIGITAL, Paris Research Laboratory, March, 1994. Brisset, P., et al. ECLi PSe (ECRC Common Logic Programming System) Extensions User Manual, Release 3.5, ECRC, West Germany, February, 1995. Carro, M., Gomez, L., and Hermenegildo, M. \Some Paradigms for Visualizing Parallel Execution of Logic Programs". In Proceedings, ICLP'93, (June 21-25, Budapest, Hungary), 1993, pp. 184-200. Chang, S.-K. \Visual Languages: A Tutorial and Survey". In IEEE Software 4, 1 (January 1987), 29-39.
Oster & Kusalik [Cruz93] [Din+88] [EiBr88] [Fre+90] [HeMa90] [HaMo90] [Hel+92] [Hel+93] [Hel+90]
[vanH90] [HiJu91] [Ho 95] [Holt90] [JoHa91] [KaKa91] [Knut84]
16
Cruz, I. F. Using a Visual Constraint Language for Data Display Speci cation. In Position Papers for the First Workshop on Principles and Practice of Constraint Programming, (April 2830, Newport, Rhode Island), 1993, pp. 24-35. Dincbas, M., van Hentenryck, P., Simonis, H., Aggoun, A., Graf, T., and Berthier, F. \The constraint logic programming language CHIP". In Proceedings of the International Conference on Fifth Generation Computer Systems, (December, Tokyo, Japan), 1988, pp. 693-702. Eisenstadt, A. M. and Brayshaw, A. M. \The Transparent Prolog Machine (TPM): An execution model and graphical debugger for logic programming". In Journal of Logic Programming 5, 4, (1988), 277-342. Freeman-Benson, B. N., Maloney, J., and Borning, A. An incremental constraint solver, Communications of the ACM, 33, 1 (January 1990), 54-63. Helm, R. and Marriott, K. \Declarative Speci cation of Visual Languages". Technical Report RC 15813(#70304) 5/31/90, IBM Research Division, T. J. Watson Research Center. Haarslev, V. and Moller, R. A Declarative Formalism for Specifying Graphical Layout. In Proceedings of the 1990 IEEE Workshop on Visual Languages (October 4-6, Skokie, Illinois), 1990, pp. 54-59. Helm, R., Huynh, T., Lassez, C., and Marriott, K. A linear constraint technology for interactive graphics systems, Technical Report RC 16913 (#74966). Also appears in Proceedings of \Graphics Interface '92", (May, Vancouver, Canada), 1992. Helm, R., Huynh, T., Marriott, K., Vlissides, J. \An Object-Oriented Architecture for Constraint-Based Graphical Editing". Advances in Object-Oriented Graphics II, Springer Verlag, 1993. Helm, R., Marriott, K., and Odersky, M. \Building Visual Language Parsers". Technical Report RC 16200(#71795) 10/5/90, IBM Research Division, T. J. Watson Research Centre. Also appears in Proceedings of Computer Human Interaction (CHI) '91 (April, New Orleans, Louisiana), 1991. van Hentenryck, P. \Incremental constraint satisfaction in logic programming". In Proceedings of the Seventh International Conference on Logic Programming, (June 18-20, Jerusalem, Israel), 1990, pp. 189-202. Hirakawa, M. and Jungert, E. An Image Database System Facilitating Icon-Driven Spatial Information De nition and Retrieval. In Proceedings of the 1991 IEEE Workshop on Visual Languages (October 8-11, Kobe, Japan), 1991, pp. 192-198. Ho man, R. H. \A Comparison of Eciency of Specialized versus General Constraint Solvers for a Picture Description Language", Unpublished Report, CMPT 400 Project, University of Saskatchewan, April, 1995. Holt, C. M. viz: A Visual Language Based on Functions. In Proceedings of the 1990 IEEE Workshop on Visual Languages (October 4-6, Skokie, Illinois), 1990, pp. 221-226. Johnson, C. W. and Harrison, M. D. Declarative Graphics and Dynamic Interaction. In EUROGRAPHICS '91, 1991, pp. 195-207. Kamada, T. and Kawai, S. A General Framework for Visualizing Abstract Objects and Relations, ACM Transactions on Graphics 10, 1 (January 1991), 1-39. Knuth, D. E. The TEXbook, Addison-Wesley, Reading, Massachusetts, 1984.
Oster & Kusalik [KuOs93a] [KuOs93b] [KuOs93c] [Men+93] [Mort95] [Oste93] [Oste95] [Quin91] [Roac84] [Row+87] [SICS94] [Stuc91a] [SvSu90]
17
Kusalik, A. J. and Oster, G. M. \Towards a Generalized Graphical Interface for LogicProgramming Development". TR 93-2, Department of Computational Science, University of Saskatchewan, Saskatoon, Saskatchewan, Canada, February, 1993. Kusalik, A. J. and Oster, G. M. \Towards Generalized Visualization Support for Logic Programming". In Proceedings of the Tenth International Conference on Logic Programming, (June 21-24, Budapest, Hungary), 1993, p. 844. Kusalik, A. J. and Oster, G. M. \A Constraint-Based Declarative Display Description Language for Visualization". In Proceedings of the 5th Workshop on Logic Programming Environments, (October 29-30, Vancouver, B.C., Canada), 1993, pp. 59-67. Menezes, F., Barahona, P., and Codognet, P. \An Incremental Hierarchical Constraint Solver". In Position Papers for the First Workshop on Principles and Practice of Constraint Programming (April 28-30, Newport, Rhode Island), 1993, pp. 201-210. Morton, R. D. Automatic Diagram Layout Within Metaview, M.Sc. Thesis, University of Saskatchewan, Saskatoon, Saskatchewan, Canada, June, 1995. Oster, G. M. \Incremental Addition and Deletion of Constraints in Constraint Satisfaction Systems". In Proceedings of the Fifth Annual Graduate Symposium on Computational Science (April 12, Saskatoon, Saskatchewan, Canada), 1993, pp. 285-294. Oster, G. M. An Incremental Constraint-Based Object Layout Algorithm and Its Implementation, M.Sc. Thesis, University of Saskatchewan, Saskatoon, Saskatchewan, Canada, July, 1995. (Available from ftp://ftp.cs.usask.ca/pub/publications/theses/95/oster msc.ps.Z) Quintus X Window Interfaces, Quintus Corporation, Palo Alto, California, 1991. Roach, J. A. \The Rectangle Placement Language". In Proceedings of the ACM IEEE 21st Design Automation Conference (June 25-27, Albequerque, New Mexico), 1984, pp. 405-411. Rowe, L. A., Davis, M., Messinger, E., Meyer, C., Spirakis, C., and Tuan, A. A Browser for Directed Graphs, Software { Practice and Experience 17, 1 (January 1987), 61-76. SICStus Prolog User's Manual, Swedish Institute of Computer Science, KISTA, Sweden, April, 1994. Stuckey, P. J. Incremental linear constraints solving and detection of implicit equalities, ORSA Journal of Computing, 3, 4 (Fall 1991), 269-274. Svensson, C. and Sundberg, J. \MuseTrace | A graphical tracer for Or-parallel Prolog". TR T-90/9003, SICS, Kista, Sweden, 1990.
Oster & Kusalik
A ICOLA User Interface
18
