5.30 Alternate method names for a single method declaration . . . 102 ...... ments through string concatenation because they minimize the possibility of SQL- injection ...... [32] Neal Feinberg, Sonya E. Keene, Robert O. Mathews, and P. Tucker.
Presentation Techniques for more Expressive Programs
by Andrew David Eisenberg B.A., Rice University, 1998 M.Sc., The University of British Columbia, 2004
a thesis submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in
the faculty of graduate studies (Computer Science)
The University Of British Columbia (Vancouver) April 2008 c Andrew David Eisenberg
Abstract We introduce a class of program editors that present a program using a rich set of transformations; we call these kinds of editors composable presentation editors. Proper use of these kinds of editors appears to lead to more expressive programs—programs whose structure are aligned with the problem they are trying to solve. By default, the composable presentation editor presents program elements textually as concrete syntax and enables typical editor commands on the program. Metadata on program elements control how the transformations are applied. Customized metadata can re-order, pictorialize, collapse, duplicate, or expand the displayed form of program elements and can additionally alter the available editor commands. We have developed a set of presentation techniques to be used by presentation designers (i.e., the programmers who design how a program is presented in the editor). These techniques relate to well-understood programming language design, editor design, and programming best-practices techniques including scoping, higher order functions, refactoring, prettyprinting, naming conventions, syntax highlighting, and text hovers. We introduce two implementations of composable presentation editors and a number of examples showing how programs can be made more expressive when presentation techniques are properly used. The first implementation is the ETMOP, an open editor, where a metaobject protocol is provided that allows language and editor designers to customize the way program elements are displayed. These customizations are called presentation extensions and the corresponding presentation extension protocol acts in a way similar to the way that syntax macros extend the syntax of a language. The second implementation is Embedded CAL, a closed editor that uses these presentation techniques to embed one language (CAL) inside a host language (Java) through the use of presentation techniques, without changing the syntax or compiler of either language.
ii
Table of Contents Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Table of Contents
. . . . . . . . . . . . . . . . . . . . . . . . . .
ii iii
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii List of Figures
. . . . . . . . . . . . . . . . . . . . . . . . . . . .
Acknowledgements Dedication
ix
. . . . . . . . . . . . . . . . . . . . . . . . . xii
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Program as Presentation Vs. Program as Datastructure . . . 1.2 Motivating Examples . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Preview of presentation composition in an extensible editor . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Preview of presentation composition in a closed editor 1.2.3 Behind the scenes . . . . . . . . . . . . . . . . . . . . 1.3 Thesis Statement . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Claims and Contributions . . . . . . . . . . . . . . . . . . . . 2 Architecture . . . . . . . . . 2.1 Program . . . . . . . . . . 2.2 Program + Metadata . . 2.3 Logical Layout . . . . . . 2.3.1 Display protocol . 2.4 Physical Layout . . . . . . 2.4.1 Render protocol . 2.4.2 Example . . . . . . 2.5 Controllers & Commands
. . . . . . . . .
. . . . . . . . .
iii
. . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
1 3 3 3 5 7 7 7 9 10 10 11 11 14 15 15 17
2.5.1
Serialize protocol . . . . . . . . . . . . . . . . . . . . .
3 Presentation Techniques . . . . . . . . . . . . . . . . . . 3.1 A Definition of Expressiveness . . . . . . . . . . . . . . . . 3.2 Spatial Arrangement . . . . . . . . . . . . . . . . . . . . . 3.2.1 Similarities to other (non-presentation) techniques 3.2.2 Example . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Implementation . . . . . . . . . . . . . . . . . . . . 3.2.4 Using spatial arrangement . . . . . . . . . . . . . . 3.3 Textual Manipulation . . . . . . . . . . . . . . . . . . . . 3.3.1 Similarities to other (non-presentation) techniques 3.3.2 Example . . . . . . . . . . . . . . . . . . . . . . . . 3.3.3 Implementation . . . . . . . . . . . . . . . . . . . . 3.3.4 Using textual manipulation . . . . . . . . . . . . . 3.4 Temporal Referencing . . . . . . . . . . . . . . . . . . . . 3.4.1 Similarities to other (non-presentation) techniques 3.4.2 Example . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Implementation . . . . . . . . . . . . . . . . . . . . 3.4.4 Using temporal referencing . . . . . . . . . . . . . 3.5 Graphical Enhancements . . . . . . . . . . . . . . . . . . . 3.5.1 Similarities to other (non-presentation) techniques 3.5.2 Example . . . . . . . . . . . . . . . . . . . . . . . . 3.5.3 Implementation . . . . . . . . . . . . . . . . . . . . 3.5.4 Using graphical enhancements . . . . . . . . . . . 3.6 Constrained Editing . . . . . . . . . . . . . . . . . . . . . 3.6.1 Similarities to other (non-presentation) techniques 3.6.2 Example . . . . . . . . . . . . . . . . . . . . . . . . 3.6.3 Implementation . . . . . . . . . . . . . . . . . . . . 3.6.4 Using constrained editing . . . . . . . . . . . . . . 3.7 Expressiveness of Examples . . . . . . . . . . . . . . . . . 3.7.1 Summation . . . . . . . . . . . . . . . . . . . . . . 3.7.2 JDBC . . . . . . . . . . . . . . . . . . . . . . . . . 3.7.3 Parameter lists . . . . . . . . . . . . . . . . . . . . 3.7.4 Initializers . . . . . . . . . . . . . . . . . . . . . . . 3.7.5 Code style AspectJ . . . . . . . . . . . . . . . . . . 3.7.6 Embedded CAL . . . . . . . . . . . . . . . . . . . 3.7.7 Variations on error handling . . . . . . . . . . . . . 3.7.8 Webservices . . . . . . . . . . . . . . . . . . . . . . 3.7.9 Embedding UML diagrams . . . . . . . . . . . . . 3.7.10 Getter presentation extension . . . . . . . . . . . . iv
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18 20 21 22 22 24 28 28 30 30 33 34 34 35 35 37 40 42 42 42 45 49 49 50 51 53 55 55 55 55 56 56 56 56 57 57 57 58 58
3.8
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
59
4 The ETMOP and CTMOP . . . . . . . . . . . . . . 4.1 Implementation and Architecture . . . . . . . . . . . 4.1.1 ETMOP . . . . . . . . . . . . . . . . . . . . . 4.1.2 CTMOP Architecture . . . . . . . . . . . . . 4.2 Implementation Specifics . . . . . . . . . . . . . . . . 4.2.1 Persistent store . . . . . . . . . . . . . . . . . 4.2.2 Combining multiple metaobjects . . . . . . . 4.2.3 Hooks into Eclipse . . . . . . . . . . . . . . . 4.2.4 Other semantic processors . . . . . . . . . . . 4.2.5 Drawing framework . . . . . . . . . . . . . . 4.2.6 Extensions palette . . . . . . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
61 63 63 65 66 66 67 67 69 69 69
5 ETMOP Examples . . . . . . . . . . . . . . . . . . . 5.1 Setter . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Quasiquote . . . . . . . . . . . . . . . . . . . . . . . 5.3 Uses . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Equations . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Code Style vs. Annotation Style AspectJ . . . . . . 5.6 Stream Processing Automaton . . . . . . . . . . . . 5.7 Note Boxes . . . . . . . . . . . . . . . . . . . . . . . 5.8 Webservices . . . . . . . . . . . . . . . . . . . . . . . 5.9 JDBC . . . . . . . . . . . . . . . . . . . . . . . . . . 5.10 Life Cycles in jEdit . . . . . . . . . . . . . . . . . . . 5.11 Event handling in jEdit . . . . . . . . . . . . . . . . 5.12 Message Handling in jEdit . . . . . . . . . . . . . . . 5.13 Services in jEdit . . . . . . . . . . . . . . . . . . . . 5.14 XML . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.15 Optional Arguments with Default Values . . . . . . . 5.16 Slider, Toggle Button, and Drop Down . . . . . . . . 5.17 Sum Constraints on Constants . . . . . . . . . . . . 5.18 Documentation . . . . . . . . . . . . . . . . . . . . . 5.19 Architectural Constraints . . . . . . . . . . . . . . . 5.20 Hovers on Variables to Show References . . . . . . . 5.21 Alternate Method Names . . . . . . . . . . . . . . . 5.22 Formatting Floating Points . . . . . . . . . . . . . . 5.23 JUnit . . . . . . . . . . . . . . . . . . . . . . . . . . 5.24 Not-null . . . . . . . . . . . . . . . . . . . . . . . . . 5.25 Inlined Images . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . .
71 71 73 75 76 79 80 81 84 86 87 89 90 92 93 94 95 96 97 98 100 101 102 102 104 104
v
5.26 Image Embedding for Documentation . . . . . . . . . . . . . 104 6 Embedded CAL . . . . . . . . . . . . . . . . . . . . . . 6.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Architectural View of Embedded CAL . . . . . . . . . . 6.3.1 Program + metadata . . . . . . . . . . . . . . . 6.3.2 Logical layout and the display protocol . . . . . 6.3.3 Physical layout and the render protocol . . . . . 6.3.4 Controllers, commands and the serialize protocol 6.4 Embedded Editors . . . . . . . . . . . . . . . . . . . . . 6.4.1 Expression editor . . . . . . . . . . . . . . . . . . 6.4.2 Module editor . . . . . . . . . . . . . . . . . . . . 6.4.3 Embedded editors at compile and runtime . . . . 6.5 Second Example . . . . . . . . . . . . . . . . . . . . . . 6.6 Embedded CAL and Presentation Techniques . . . . . . 6.7 Summary and Conclusion . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
106 107 107 109 110 110 110 110 111 111 112 113 114 116 117
7 Related Work . . . . . . . . . . . . . . . . . . . . . . 7.1 Editor Architectures . . . . . . . . . . . . . . . . . . 7.1.1 Traditional editors . . . . . . . . . . . . . . . 7.1.2 Hard structure editors . . . . . . . . . . . . . 7.1.3 Soft-structure editors . . . . . . . . . . . . . . 7.1.4 Display-driven ASTs . . . . . . . . . . . . . . 7.2 Expressiveness Through Language Extension . . . . 7.2.1 Syntactic and semantic extension . . . . . . . 7.2.2 Non-meta libraries . . . . . . . . . . . . . . . 7.2.3 Macros . . . . . . . . . . . . . . . . . . . . . 7.2.4 Metaobject protocols (MOPs) . . . . . . . . . 7.2.5 Attribute grammars . . . . . . . . . . . . . . 7.2.6 Pretty-printing . . . . . . . . . . . . . . . . . 7.2.7 Syntactic stylesheets . . . . . . . . . . . . . . 7.3 Expressiveness Through Multi-Lingual Programming 7.3.1 Language embedding through syntax . . . . . 7.3.2 Common runtime environments . . . . . . . . 7.3.3 Pipelines . . . . . . . . . . . . . . . . . . . . 7.3.4 Service-oriented architecture . . . . . . . . . 7.4 Summary . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
119 119 120 121 124 126 130 130 130 131 132 133 133 134 134 135 137 137 138 138
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 vi
8.1 8.2 8.3
8.4
Summary . . . . . . . . . . . . . . . . Revisiting Claims and Contributions . Limitations . . . . . . . . . . . . . . . 8.3.1 Definition of expressiveness . . 8.3.2 User evaluations . . . . . . . . 8.3.3 Presentation design . . . . . . 8.3.4 Implementations and the use of Implications for Future Work . . . . .
Bibliography
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . internal APIs . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
141 142 143 143 143 144 144 144
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
vii
List of Tables 7.1
Summary of techniques for multiple language use. . . . . . . . 135
viii
List of Figures 1.1 1.2
2.1 2.2
2.3 3.1 3.2 3.3 3.4 3.5 3.6
3.7 3.8 3.9
Screenshot of a program presented using the ETMOP editor . Screenshot of a program that uses Embedded CAL to perform message processing. . . . . . . . . . . . . . . . . . . . . . . . .
4
Composable presentation editor architecture . . . . . . . . . . Some of the kinds of transformations capable using the transformable editor architecture when the program + metadata structure is hierarchical. The left side shows a small abstract syntax tree and the right side shows shows the display. The arrow is the application of the display protocol. The smaller boxes are metadata. This metadata drives the display protocol. Metadata in solid boxes apply some transformation that is not one to one. . . . . . . . . . . . . . . . . . . . . . . . . . Transforming getters on fields. . . . . . . . . . . . . . . . . .
10
Spatial arrangement to declare a summation . . . . . . . . . . Spatial arrangement to declare JDBC stored procedures . . . Spatial arrangement of formal and actual parameter lists (mockup) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Spatial arrangement of initializers (mock-up) . . . . . . . . . Screenshot of a C++ header file in Emacs. Syntax highlighting is determined by regular expressions. . . . . . . . . . . . . Syntax highlighting in Eclipse is based on a full or partial parse of the program. References to fields are in blue and references to static members are italicized. . . . . . . . . . . . Code folding can elide import statements. Import statements can be un-elided by clicking on the icon in the gutter bar. . . Use of textual manipulation to display @AspectJ annotations (below) as code style AspectJ (above). . . . . . . . . . . . . . Temporal referencing in embedded CAL. . . . . . . . . . . . .
ix
6
12 16 24 26 29 29 31
32 33 34 38
3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 3.19 3.20
4.1 4.2 4.3
5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 5.11 5.12 5.13 5.14 5.15 5.16
Four variations of error reporting. . . . . . . . . . . . . . . . . A use of graphical enhancements in DrScheme . . . . . . . . . JavaDoc for Eclipse’s IJavaElement . . . . . . . . . . . . . . Eclipse JDT Plug-in Developer Guide documentation for IJavaElement . . . . . . . . . . . . . . . . . . . . . . . . . . . Variations on delineating groups of field declarations . . . . . Embedding external graphical information in a program. This is a UML activity diagram. . . . . . . . . . . . . . . . . . . . Using the render protocol to draw graphics in the ETMOP. . Constrained editing in Microsoft’s Visual Basic 6.0. . . . . . . Constrained editing in Embedded CAL . . . . . . . . . . . . . Two variations of constraining input on getter declarations. The programmer has just typed ‘0’. . . . . . . . . . . . . . . . Two variations of constraining input on getter declarations. The programmer has just typed ‘0’. . . . . . . . . . . . . . . .
39 43 44
Screenshot of a program in the ETMOP editor . . ETMOP and CTMOP architecture . . . . . . . . . Palette from which extensions can be dragged and into appropriate locations. . . . . . . . . . . . . . .
62 62
. . . . . . . . . . . . dropped . . . . . .
Complete code for SetterEMO . . . . . . . . . . . . . . . . . Screenshot of the implementation of the Setter CDMO . . . . Screenshot of the uses extension . . . . . . . . . . . . . . . . Alternative presentation for the uses extension . . . . . . . . Screenshot of the equation extension . . . . . . . . . . . . . . Screenshot of the AspectJ extension . . . . . . . . . . . . . . Stream Processing Automaton that accepts c(ad)*r . . . . . Nesting of right note boxes: the presentation (a) and the persistent store (b) . . . . . . . . . . . . . . . . . . . . . . . . Combining top note boxes: the presentation (a) and the persistent store (b) . . . . . . . . . . . . . . . . . . . . . . . . . . Webservice presentation extension. . . . . . . . . . . . . . . . Positive affordance for adding a webservice component. . . . Negative affordance for adding a webservice component. . . . Specifying parameter names for webservices. . . . . . . . . . . Declaring a stored procedure on an interface method declaration Using a drop down menu to edit SQL query options . . . . . Life cycle defined by method ordering . . . . . . . . . . . . .
x
45 48 50 51 53 54 54 58
70 72 74 75 76 76 79 80 82 83 84 84 85 85 86 87 88
5.17 Expanded code for the presentation extension. An aspect is used to update a finite state machine that keeps track of the life cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.18 Event handler for text changes . . . . . . . . . . . . . . . . . 5.19 Message handling for jEdit components . . . . . . . . . . . . 5.20 Defining services inline using presentation extension in jEdit . 5.21 Embedding XML using presentation. . . . . . . . . . . . . . . 5.22 Optional arguments . . . . . . . . . . . . . . . . . . . . . . . 5.23 Using sliders, a checkbox, and a dropdown list in a program. 5.24 Sum of constants must equal 100. . . . . . . . . . . . . . . . . 5.25 Temporal referencing allows error feedback to appear while typing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.26 Inline HTML formatting of comments . . . . . . . . . . . . . 5.27 Architectural constraints . . . . . . . . . . . . . . . . . . . . . 5.28 JavaDoc formatting combined with architectural constraints . 5.29 Using mouse hovers to identify all references of an identifier. . 5.30 Alternate method names for a single method declaration . . . 5.31 Formatting numeric fields . . . . . . . . . . . . . . . . . . . . 5.32 JUnit 4.0 presentation extensions . . . . . . . . . . . . . . . . 5.33 Icon used to ensure argument to a method is not null . . . . . 5.34 Inline images. Double-clicking opens a file dialog. . . . . . . . 5.35 Image embedding. Double-clicking opens a file dialog. . . . . 6.1 6.2 6.3
6.4 6.5 6.6
7.1
Embedded CAL in a Java program. . . . . . . . . . . . . . . . Architectural view of Embedded CAL . . . . . . . . . . . . . Features in the expression editor: a) content assist, b) gutter annotation hovering and navigation, c) highlighting errors, d) resizing, and e) text hovers to show CALDoc. . . . . . . . Embedded CAL expression editor expanded view . . . . . . . Embedded CAL Module Editor . . . . . . . . . . . . . . . . . Function passing in Embedded CAL. Numbers 1–7 have been added as labels of interesting parts of the program. . . . . . .
89 90 91 93 94 94 96 97 97 98 99 101 101 102 103 103 104 104 105 108 109
113 114 115 116
Program editor architectures . . . . . . . . . . . . . . . . . . 120
xi
Acknowledgements Thank you to everyone who has helped me along the way to complete this dissertation. Gregor Kiczales has helped immeasurably. He made me focus and did not let me get side-tracked. My supervisory committee members Eric Wohlstadter and Shriram Krishnamurthi, have provided much needed feedback and insight. Shriram has been especially patient, being 3,000 miles away. The BORG team has been very supportive and open by giving me the freedom to use the work I developed at Business Objects as part of my dissertation. Other members of the Software Practices Lab have patiently read drafts of my papers and listened to my practice presentations over, and over, and over. Hermie Lam, our lab’s assistant, has helped me cut through academic red-tape, and has always had chocolate for me whenever I visited her office (and I visited quite often!). This work was supported by IBM Centers for Advanced Study and The Natural Sciences and Engineering Research Council of Canada.
xii
Dedication I dedicate this dissertation to my son, Lucas, who was born as I was nearing completion of this work; and to my loving wife, Stephenie, who supported me emotionally and financially all these years.
xiii
Chapter 1
Introduction Many factors contribute to making a program expressive. Good modularity decomposes the program into natural and clearly interconnected components [80, 72]. The use of elegant and standard style makes a program easier to read and maintain [65, Chaps 18, 19]. Tailorability of the programming language makes it possible to express general or domain-specific functionality using specialized constructs [22, Chap 5] [43, Chap 9]. Within this space of language tailorability, there have been a range of approaches, from making it easier to define domain-specific languages (DSLs) [45, 82, 83, 85] to making general purpose languages (GPLs) extensible, so that programmers can define new language constructs and semantics that better suit their needs [11, 13, 17, 56, 18]. Use of these extension mechanisms has itself ranged from defining fairly general-purpose constructs [31] to more domain-specific constructs [10, 12, 58]. An alternative approach to language extensibility is to use an extensible editor to present salient parts of the program more explicitly [16, 55, 74]. Language extensibility has a long history [19, 44, 66], but outside of Lisp-like languages, interest in extensibility waned in the mid-1970s [76]. This dissertation is part of a revival of interest in extensible languages and shows how tailoring languages through their presentation in the editor can be a factor in making programs more expressive. We say that the presentation of a program includes both how the program is displayed on the screen and the range of editor commands that can affect it. In an ordinary program editor such as Emacs or Eclipse, the transformation from the program source to its display is surjective—that is the linear order of program elements is preserved in the display and it is sometimes
1
possible to collapse some elements into others. Additionally, elements can be specially formatted, including font selection, pretty printing, colors, etc.1 We introduce a class of program editors that present a program using a rich set of transformations; we call these kinds of editors composable presentation editors. Proper use of these kinds of editors appears to lead to more expressive programs—programs whose structure is aligned with the problem they are trying to solve.2 By default, the composable presentation editor presents program elements textually as concrete syntax3 and enables typical editor commands on the program. Metadata on program elements control how the transformations are applied. Customized metadata can re-order, pictorialize, collapse, duplicate, or expand the displayed form of program elements and can additionally alter the available editor commands. We have developed a set of presentation techniques to be used by presentation designers (i.e., the programmers who design how a program is presented in the editor). These techniques relate to well-understood programming language design, editor design, and programming best-practices techniques including scoping, higher order functions, refactoring, prettyprinting, naming conventions, syntax highlighting, and text hovers. We introduce two implementations of composable presentation editors and a number of examples showing how programs can be made more expressive when presentation techniques are properly used. The first implementation is the ETMOP, an open editor, where a metaobject protocol is provided that allows language and editor designers to customize the way program elements are displayed. These customizations are called presentation extensions and the corresponding presentation extension protocol acts in a way similar to the way that syntax macros extend the syntax of a language. The second implementation is Embedded CAL, a closed editor that uses these presentation techniques to embed one language (CAL) inside a host language (Java) through the use of presentation techniques, without changing the syntax or compiler of either language. 1 Strictly speaking, with respect to whitespace characters, pretty-printing is not surjective. 2 Our full definition of expressiveness is given in Section 3.1. 3 In this dissertation, we use the term concrete syntax to mean the stored form of the program in text, but this is not necessarily what the programmer will see on the screen.
2
1.1
Program as Presentation Vs. Program as Datastructure
This dissertation inherently works with two different notions of what a program is: program as presentation and program as datastructure. We consider that from the end-programmer’s point of view (i.e., the programmer who uses the editor and language, but does not create them), the program is the presentation. What the programmer sees on the screen is the program, not a view, model, or other representation of the program. We call this program as presentation. On the other hand, presentation designers (i.e., those who design editors and the presentation of programs in them) must look inside the presentation, understand the intermediate data structures, and apply appropriate presentation techniques to the program. We say that the presentation designer views the program as datastructure. This dissertation discusses both points of view. At times context clearly dictates which view is being taken, such as when the architecture and implementation are described (program as datastructure), or when example uses are described (program as presentation). Some sections of the dissertation, most notably the presentation techniques (Chapter 3), bridge the two points of view and use both of them although not at the same time. When this happens, we will be clear as to which view is being taken.
1.2
Motivating Examples
In this section, we briefly preview the two composable presentation editors described in this dissertation in order to concretely illustrate the kinds of presentations that are possible.
1.2.1
Preview of presentation composition in an extensible editor
In Chapters 4 and 5, we present an approach to language extensibility that works using presentation extension. In presentation extension, the presentation designer defines a rule for transforming a part of the language or program using presentation techniques to compose the presentation in the editor. This presentation can be combined with additional semantics so that a new behavior can be associated with the presentation. The stored form of the program remains in a standard syntax—there is no change to the concrete syntax. Instead, the program editor incorporates an edit-time metaobject protocol [50] to make it possible for the programmer to customize the way the program is read, edited, or browsed. We have created the edit time metaobject protocol (ETMOP), an exten3
Figure 1.1: Screenshot of a program presented using the ETMOP editor sible program editor that allows programmers to use presentation extensions to present programs [28]. The compile time metaobject protocol (CTMOP) works synergistically with the ETMOP to provide semantic extensibility. Figure 1.1 is a screenshot of a short Java program presented using the ETMOP. Taking the program as presentation view, the programmer works with existing language primitives, such as class and method declarations, but also has access to an open-ended set of language extensions. The visual form of the program is simple and elegant in ways that are not possible with ordinary Java-like languages. In this example, five different extensions are used to allow the concise definition of getter and setter methods, field invariants, and pre- and postconditions. The program in the screenshot should be read as saying that the fields x and y, have getter and setter methods respectively named getX, setX, getY, and setY. Additionally, the field uid has a getter and a setter method 4
called uid and setUid, and an invariant, which says that the field must be non-negative (the $$ represents the new value to be set). Finally, the move method has both a pre- and a postcondition. The display part of presentation extensions makes the code appear more concise and also helps minimize scattering of different concerns in the code. In addition to the display customization, presentation extensions can also enhance the editing commands. The simplest effect is to make typing and cursor movement behave normally, even when text does not necessarily flow in straight lines and can be delineated by graphical elements such as vertical lines. Some editing is constrained. The words getter, setter, precondition, postcondition, and invar are read-only and are used as labels. The area to the right of the getter and setter labels is editable, but the programmer may only enter valid Java identifiers in the area, text that would make the area invalid is rejected. In Chapter 4 we introduce the implementation of the ETMOP and CTMOP, describing how they present using programs-as-datastructure. In Chapter 5 we return to this example and introduce others, showing how they take advantage of presentation techniques to produce more expressive programs.
1.2.2
Preview of presentation composition in a closed editor
Language-oriented programing [86] and meta-linguistic abstraction [1] are two terms used to describe software systems built using a collection of programming languages. Each language is used for a specific part of the program that the language is particularly adept at expressing. There is a range of mechanisms that enables this multi-lingual programming from serviceoriented architecture [71] to syntactic language embedding [12]. We use an editor-based approach that uses presentation techniques to embed the CAL programming language4 in Java. Embedded CAL5 is an editor-based technique for combining Java and CAL, so that both languages coexist in the same compilation unit. This tool allows programmers to enter CAL expressions directly into Java code, making it possible to write programs that use both CAL and Java in a single editor. Thus, viewing the program as presentation, this technique most resembles language-based techniques where there is little separation 4
CAL is a statically typed, non-strict, lazy functional language that was developed as an open source project by Business Objects. CAL has syntax similar to Haskell. The CAL compiler, libraries and tools are available for download http://openquark.org/. 5 Embedded CAL is also available for download at http://openquark.org/.
5
Figure 1.2: Screenshot of a program that uses Embedded CAL to perform message processing. between the languages.6 Figure 1.2 shows a Java program that uses Embedded CAL to perform message processing. The box on the screen forms an embedded editor that contains a CAL expression. The part of the program that is outside the embedded editor is standard Java. The CAL expression in the embedded editor can use any Java variables that are in scope. As expected, outside the boxes, editing feels like a standard Java editor, and inside the boxes editing feels like a standard CAL editor. As the programmer types, the embedded editor automatically resizes to fit all of the CAL code. Alternatively, the programmer can drag the lower right corner of the embedded editor to resize it manually. When program execution encounters one of these boxes, the expression is evaluated and its value is returned as a Java object. In this example, the CAL code is used to define the message processing logic, and the rest of the program—including I/O and event handling—are written in Java. The message processing logic in the CAL box takes a single message, finds its message processor, and either returns the processed message or the original message. The implementation and more examples are described in Chapter 6. 6
The line between extending a language (as in the ETMOP approach) and embedding one language in another (as in the Embedded CAL approach) is not always clear. Whether a presentation is an extension or an embedding seems to depend on the size and maturity of the presentation. We see Embedded CAL as an instance of language embedding, but agree that it could be considered an extension of Java.
6
1.2.3
Behind the scenes
The editor architecture is similar to a model-view-controller [57] with two models—the abstract syntax of the program, and an abstract description of the program’s layout. This provides an extra layer of indirection between the underlying program and the presentation, allowing a non-surjective relationship between the two. This is described in more detail in the next chapter.
1.3
Thesis Statement
Program editors that can present programs using a non-surjective relationship to the concrete syntax enable a set of presentation techniques that are natural alternatives to existing language design, editor design, and programming best-practices techniques. Careful use of these presentation techniques appears to lead to more expressive programs.
1.4
Claims and Contributions
This dissertation makes the following claims: 1. It is feasible for an editor to present (i.e., display and provide editing support for) programs using a non-surjective relationship to the concrete syntax. We support this claim by describing an architecture for a composable presentation editor (Chapter 2), and two implementations of this architecture: the ETMOP (Chapter 4) and Embedded CAL (Chapter 6). 2. Such an editor enables a set of simple and composable presentation techniques that can be used as strategies to present programs. This claim is supported by a description of the presentation techniques in Chapter 3 and how they arise from the architecture presented earlier. 3. Even though they operate in the domain of program presentation, these presentation techniques are natural extensions to existing language design, editor design, and programming best-practices techniques. An analysis of the presentation techniques to support this claim is provided in Chapter 3. 4. Careful use of the presentation techniques appears to lead to more expressive programs. 7
This claim is supported by our description of how the presentation techniques enable higher-level ideas about a program to be expressed, also described in Chapter 3. This claim is additionally supported by numerous examples that are described in Chapters 5 and 6. And the following contributions: 1. a composable presentation editor architecture, 2. a set of presentation techniques that are enabled by the use of this architecture, 3. a qualitative framework for characterizing tool and language effects on the expressiveness of programs, 4. an analysis of how these presentation techniques compare to existing program language design, editor design, and programming best practices techniques, 5. an open, extensible editor that enables presentation extensibility, 6. a set of presentation extensions that illustrate the use of the open editor, and 7. a closed editor that uses the presentation techniques to embed a lazy functional language into an object-oriented language.
The next chapter describes the architecture of the composable presentation editor. Chapter 3 introduces the presentation techniques that arise from the architecture and contains our argument for why these techniques can make programs more expressive. Chapters 4 and 5 describe the implementation of the ETMOP and some examples of its use. The implementation of Embedded CAL and an example of its use are described in Chapter 6. We discuss related work in Chapter 7 and we conclude in Chapter 8.
8
Chapter 2
Architecture An editor that can transform a program before it is presented involves indirection between the program text and what appears on screen. This chapter describes the architecture of this indirection by characterizing the structures produced during the different phases of creating the presentation; this architecture is shown in Figure 2.1. This chapter also describes the following three protocols in general terms with details deferred to Chapters 4 and 6: Display protocol converts an in-memory version of the program to logical layout, a structure that defines the display in abstract, relative spatial terms. Render protocol converts the logical layout to physical layout, a structure that concretely defines the display on the screen in absolute coordinates. Serialize protocol converts the logical layout back to a program structure. The composable presentation editor architecture is based on a modelview-controller architecture [57] where the model has two sub-structures. These sub-structures together provide the indirection between the program and its presentation, giving room to apply transformations before finally presenting the program. When the controller reacts to input to update the model, either of the model sub-structures may be updated. When relevant for each structure and protocol, we briefly describe their implementation in both the ETMOP and Embedded CAL. This chapter describes the program being displayed as a datastructure. As noted in Section 1.1, this is the point of view of the presentation designer, not the end programmer. 9
Controllers Event Handlers
Program
Program + Metadata (Model)
Logical Layout (Model)
Display Serialize
Render
Physical Layout (View)
Figure 2.1: Composable presentation editor architecture
2.1
Program
The editor begins with an in-memory version of the program. From the architectural point of view, the program consists of a set of program elements. This may be an abstract syntax tree, a set of non-overlapping regions of text, or some other representation of the program.
2.2
Program + Metadata
The next structure is the same program with metadata attached to it. Each program element has metadata, which can be shared among program elements. The metadata drives the display and serialize protocols for those program elements. This structure is part of the model, so that when the controller reacts to an event from the programmer, this structure may be updated. The architecture does not prescribe how metadata is defined and applied. The ETMOP uses this metadata to define metaobjects in a metaobject protocol [50]. Every AST node has a single metaobject and this metaobject can be shared between siblings or between parents and children. The set of metaobjects is open-ended and is intended to be extended by language or presentation designers so that custom displays can be created. In Embedded CAL, metadata describes a partitioning of the program text into non-overlapping regions based on the abstract syntax. The metadata on regions describes how the text will be presented. The kinds of metadata available are fixed and can either define a color and font style for the region, or define an embedded editor. 10
Both the ETMOP and Embedded CAL generate the metadata directly from the program itself based on the structure of the program. However, this is not a requirement, and metadata could, for example, be generated from an external properties file.
2.3
Logical Layout
The logical layout of a program forms the part of the model that defines a topological containment hierarchy of the relative layout of the final display. Display elements are the building blocks of this structure. Display elements abstractly define the contents of the screen. They describe the text, formatting, and graphical elements that are eventually rendered on the screen topologically, in terms of relative positioning. The ETMOP produces the logical layout as a hierarchy of boxes that determine the containment of display elements on the screen. Display elements in Embedded CAL are style regions and embedded editors. Style regions have a start, length, and a description of the font (color, bold, italics, etc.) for a block of text. Embedded editor regions have a start, length, and a description. This description contains the embedded editor’s contents, size, and other attributes about its static and dynamic behavior.
2.3.1
Display protocol
The display protocol transforms the program + metadata into logical layout. The abstract syntax tree is typically visited in a post-order traversal and each program element’s metadata generates the logical layout for that element and its children. The metadata has control over the visit—it can decide to perform an in-order or pre-order traversal, continue the visit to the children, visit only some of the children, or end the visit. If the metadata is shared, then the metadata can combine, re-order, or elide the display of some or all of the program elements to which it is attached. The display protocol visits program elements in order and applies transformations to convert program elements to display elements. These kinds of transformations are described below. By combining these transformations, it is possible to build up the more complex forms of presentation that we have shown in Section 1.2. Transformations for hierarchical program + metadata structures Figure 2.2 shows the kinds of transformations that are possible when the program + metadata structure is hierarchical. The ETMOP, because it uses a hierarchical program + metadata structure, generates the logical layout 11
Program Element
Metadata that displays program elements as standard text
Display Element
Metadata that transforms the program
1 2 3 4 5 6 7 1 2 3 4 5 6 7
Display
Display
1 2 3 4 5 6 7
Display
1 2 3 4 5 6 7
Display
A B C D E F G
One-to-one
A B C D E
Ellision
A B C E D G F
Rearrangement
A B C D E F G H
Insertion (b)
(a)
1
(c)
A
A
Display
B C D E H F G
B C D E G F
1 2 3 4 5 6 7
Display
X
1 2 3 4 5 6 7
Display
C A F G
2 3 4 5 6 7
B D E
A B C D E
Sharing
Substitution Not possible: Metadata cannot walk up the tree
Figure 2.2: Some of the kinds of transformations capable using the transformable editor architecture when the program + metadata structure is hierarchical. The left side shows a small abstract syntax tree and the right side shows shows the display. The arrow is the application of the display protocol. The smaller boxes are metadata. This metadata drives the display protocol. Metadata in solid boxes apply some transformation that is not one to one. 12
by running a code walker over the program + metadata structure and it can apply the following kinds of transformations: One-to-one Metadata can convert from program to logical layout on a one to one basis. The display elements will look similar to the program text. Although, this does not necessarily mean that the final presentation will be the original program text. The presentation may look like the text, but may be implemented by a composite of many display elements. Elision Metadata can elide program elements from the final display by ignoring the results of visiting a node or by not visiting it in the first place. Rearrangement Metadata can rearrange program elements in the display by altering the order in which child nodes are combined. Insertion Metadata can insert extra display elements. The inserted display elements can be graphical or textual. They may also contain callback handlers to the controllers (which is particularly useful for implementing temporal referencing, Section 3.4). Sharing Sharing of metadata between program elements provides alternatives for some of the other kinds of transformations. The sharing of metadata means that this metadata can combine the presentations of the program elements it is attached to (a), rearrange them (b), or elide them (c). Substitution The metadata can ignore its program element and all of its children and produce something independent of the underlying program. This kind of transformation can be used to replace a block of code with a box, graphics, or documentation. Not possible This last transformation is not possible under the protocol. Here, the metadata on program element 3 affects the presentation of its parent, program element 1. Metadata can only affect program elements that it is transitively attached to. This is not an exhaustive list of transformations that are possible when converting program elements to display elements. This is meant to be representative of the kinds of transformations that are possible. Proper placement of metadata combined with the composition of these acceptable transformations are sufficient to produce all of the examples shown in this dissertation. 13
Based on the way that metadata affects the presentation of program elements, it is natural, but not required, to place metadata as close as possible to the program elements that it will affect. Thus, the scope of metadata should be as limited as possible. As an analogy, variables in programs should have limited scope as well. Both global variables and metadata should be avoided as much as possible, rather, they should be declared as close as possible to where they are used. Transformations for flat program + metadata structures If the in-memory representation of the program is non-hierarchical, as it is for Embedded CAL, then there are fewer kinds of transformations available. A non-hierarchical program structure is a special case of the hierarchical program structure where the hierarchy is flat. The transformations are simpler, but for clarity, we provide a similar description of possible transformations as in the previous section. Most of the same kinds of transformations are possible for flat program + metadata structures, but they are restrained. Each program element’s metadata is visited sequentially to generate logical layout. One-to-one There is a one to one correspondence between program elements and display elements (same as hierarchical structure). Elision A program element is not transformed to a display element (same as hierarchical structure). Insertion Metadata adds new display elements before or after the display element corresponding to its program element (this is a restricted form of the hierarchical structure transformation). Shared When metadata is shared between program elements, the corresponding display elements maybe combined or rearranged (this is a restricted form of the hierarchical structure transformation). Not possible It is not possible for display elements to be rearranged unless they share metadata. Embedded CAL uses these kinds of transformations as described in Chapter 6.
2.4
Physical Layout
The physical layout is the third structure of the architecture and it forms a set of graphical objects that defines how the program is physically laid 14
out on the screen. This structure forms the view, it is comprised of render elements, and it is the structure that the programmer directly interacts with. Coordinates in the physical layout are in terms of locations on the screen. In both the ETMOP and Embedded CAL, the physical layout is created by external graphics libraries: GEF for the ETMOP and SWT for Embedded CAL. The relationship between display elements and render elements are m to n. A display element may be transformed into multiple render elements for graphics-intensive displays. Or, optimizations may be applied that combine multiple display elements into a single render element, for example pieces of text may be split across display elements, but if the font, size, and style are identical, then they may be combined. The logical and physical layouts are expected to be consistent. Changes to the logical layout are pushed to the physical layout as part of the render protocol, described next.
2.4.1
Render protocol
The render protocol applies a final set of transformations to perform the actual rendering on the screen, converting the logical layout to physical layout. The concrete rendering on the screen is in terms of the particular graphics library used by the editor. The render protocol walks the logical layout in order and produces render elements that precisely lay the program on the screen. The transformations perform the actual drawing of text and graphics. They determine the size, shape, and location of render elements. Textual transformations write the text and apply fonts and formatting to it. The size of textual render elements is determined implicitly based on the height and width of the font. The graphical transformations render the actual graphics on the screen and determine their physical coordinates and sizes. Layout information from the logical layout phase such as row, column, or centering can provide hints as to the precise positioning and size of render elements. The render protocol can be applied partially to render or update parts of the screen as the logical layout changes in response to commands executed by the controllers described below.
2.4.2
Example
Figure 2.3 shows a short example of how the two protocols work to produce the presentation of a program. This example shows the entire process of presenting a small snippet of a program, but focuses on the kinds of transformations required for the display and render protocols. 15
(a) Program Store
field x:
@Getter(“get”) private int x; @Previous private int y;
Getter Metadata: get
Parse
(b) Program + Metadata
@Getter(“get”)
private
int
x
private
int
y
field y: @Getter(“get”)
Display (c) Logical Layout row
column
(d) Physical Layout
row private
int
x;
row private
int
y;
L i n e
Getter:
get
Render
Figure 2.3: Transforming getters on fields. The example starts with some program text as in part (a). The program is parsed, converted to an AST and metadata is applied as in part b. Note that the getter is shared between the two field declarations. The child nodes of the field declarations all have default metadata, and for the sake of clarity they are not shown. The display protocol is applied to convert the program + metadata structure to the logical layout. The metadata on the two field declarations apply the following transformations to the AST nodes: 1. Most of the children of the x and y fields are displayed one-to-one, except for the annotation program elements. 2. The program elements corresponding to the @Getter and @Previous annotations are elided. This is controlled by the shared metadata on the field declarations. 3. The display elements including and to the right of the line are inserted. 16
4. The sharing of metadata additionally allows the line and subsequent display elements to span the display elements of the field declarations. In the logical layout, the containment hierarchy is described, but sizes are not. Text, graphics, and formatting are described, but not yet drawn. The render protocol performs another set of transformations, converting display elements into render elements. The display element hierarchy is walked and render elements are generated in the correct location and of the correct size. Colors, graphics and fonts are applied. Event handlers are added to the render objects to ensure proper error checking exists in the get box. From the programmer’s perspective, the field declarations private int x and private int y appear to be contiguous plain text. However, implementation details of the protocol will determine if this is true, or if there is a one-to-one correspondence between those display elements and the render elements that they beget. Thus, the presentation is generated, and programmers can now interact with the program as described below.
2.5
Controllers & Commands
Controllers define how the editor reacts to programmer input, such as key strokes and mouse movements. They maintain a correspondence between the logical and physical layouts, ensuring that the logical layout is never stale with respect to the physical layout for extended periods of time. Controllers form the fourth structure and there is typically a one to one correspondence between controllers and logical layout, so that controllers follow the same containment hierarchy. Each controller manages the region of the screen that is generated by the display element it is related to. Controllers have event handlers that react to user inputs by generating and executing commands. If a controller does not have an event handler defined for a particular event, then the controller hierarchy is walked up until an appropriate handler is found, or else the event is ignored. The execution of a command typically does one or more of the following: • Make a change to the display element it is attached to, which will in turn update the physical layout. For example, this includes editing text and graphics, or opening hovers and context menus that are implicitly rooted at the display element whose controller handled the event.
17
• Perform an external action that affects neither model sub-structures. This includes actions like starting the debugger or initiating a rebuild of the project. • Initiate the serialize protocol to regenerate the program (described next).
2.5.1
Serialize protocol
The serialize protocol is the reverse of the display protocol—it generates program elements from display elements. As described above, the serialize protocol is typically invoked by commands acting on the editor. Like program elements, each display element has metadata and when asked to serialize itself, defers the request to its metadata. The default metadata will visit all of the display element’s children in post-order, and combine it in the proper way. Alternatively, metadata on display elements may skip some of its children, alter the serialization results of its children, convert from post-order to pre-order or in-order transversal, or otherwise manipulate the visitation process. It is often useful to periodically run the serialize protocol because external tools typically require program text, or an abstract syntax tree of the program. Thereby, the discrepancy between what the programmer sees (i.e., the physical layout) and the in-memory program used by the editor and external tools is minimized. This is particularly useful for tools such as an eager parser or code completion. Commands may have additional control over the serialize protocol and alter some display or program elements as serialization is occurring. This additional control can affect any part of the display, thus giving commands back-door global access to the display. This power is provided as part of a layered protocol. Typically, commands may only change the display element that triggered the command (and its children). But occasionally, this is too restrictive and commands require access to other parts of the display. If the triggering command introduced any global changes, then after the serialize protocol is completed the display and render protocols are re-run to synchronize the display with these changes. The serialize protocol is therefore a powerful way to globally transform the program, but it should be used with care: 1. Commands should be as locally scoped as possible. Typically, commands only affect the display element whose controller executed it. However, using the serialize protocol is a back door, which can by-pass 18
this and produce global changes. Sometimes this is a requirement to execute commands that affect disparate parts of the program, but this can add complexity to the command. 2. The serialize protocol alters multiple structures when it is invoked and so it is potentially time consuming. Accordingly, there is a penalty for using this power. This kind of power is analogous to using reflection in Java to bypass privacy restrictions. Most of the time, programmers should follow public, private, protected keywords, but occasionally, it is required that a programmer be able to bypass it, for example when using a third-party library for a new kind of use. If required, then the programmer can explicitly bypass the security using reflection. This should be used with care because it is slow and it is error-prone. However, because the programmer explicitly invokes the reflective access, the programmer implicitly agrees to take responsibility for its danger. Thus, a design decision when implementing this architecture is to determine what kinds of commands will trigger the serialize protocol. Minimizing the use of the serialize protocol can make the editor more responsive. However, serialize can be useful to synchronize the program with the display in order to ensure that tools have access to the most up to date version of the program. And also, serialize allows the implementation of commands that have global affect. How this decision is addressed in the ETMOP and Embedded CAL will be explained in Chapters 4 and 6. At a minimum, the serialize protocol must be invoked when a save command is issued. In the next chapter, we describe various presentation techniques that arise from the transformations described in this chapter and how they can be applied to create more expressive programs.
19
Chapter 3
Presentation Techniques This chapter introduces five presentation techniques that provide guidance on how to design a program’s presentation. These techniques arise from a combination of the display and rendering protocols described in Chapter 2. The techniques are not orthogonal, rather their precise use and effect are often interconnected. We present a qualitative argument that the proper use of presentation techniques can make programs more expressive; this argument is based on comparisons to prior art in programming language design, editor construction, and programming best practices. Our argument is this: • The composable display architecture enables the presentation techniques. • These techniques establish various editor and language properties such as abstraction, referencing, and grouping. • Through establishing these properties, higher-level ideas are expressed about the program. • When the ideas that are expressed are aligned with a program’s intended meaning, then we say that the program is expressive. Later, in Chapters 5 and 6 we further explore this argument and show more examples of the use of these techniques. Because this chapter links the composable presentation editor architecture and its use, this chapter uses both the program as datastructure view and program as presentation view (Section 1.1). When it is not clear from context which view is being taken, we will explicitly state it. 20
We begin this chapter by introducing our definition of expressiveness, which provides a framework for understanding how various techniques can affect a program’s expressiveness. Then, we describe the five presentation techniques: spatial arrangement, textual manipulation, temporal referencing, graphical enhancement, and constrained editing. We describe each technique and how it is related to prior art, provide examples, show how this technique can be implemented in a composable presentation editor, and propose some informal principles on how these techniques should be used. We end this chapter by exploring how the expressiveness of a program is affected by the program’s presentation. We review all of the examples brought up earlier in the chapter and show how the ideas expressed in each program’s presentation emphasizes the program’s intended meaning—hence making the programs more expressive.
3.1
A Definition of Expressiveness
Our definition of expressiveness involves two core concepts: the intended meaning and the perceived meaning. The intended meaning of the program is the combination of the program’s semantics, the problem that the program addresses, a description of the solution, and how the program interacts with the external world (e.g., how it is used by people or other programs). The perceived meaning of the program is the ideas about the program that are elicited in a programmer’s head when the program is read or browsed. Some fragments of the perceived meaning may be inherent in the semantics of the program, whereas other fragments may be connoted through presentation or documentation of the program. An expressive program is one where the perceived meaning of a program reliably aligns with its intended meaning. Properties of a program that help expressiveness include properties of the language such as its abstraction mechanisms and its approach to scoping, properties of the editor such as its approach to error reporting and code formatting, and properties of the program itself such as modularity, naming conventions, and layout. If, however, a program’s intended meaning is obscured by the perceived meaning of the program, we say that the program is not expressive. Programs that are overly verbose, are improperly formatted, are too low level, use obscure identifier names, or use too many or the wrong kinds of abstractions often fall into this category. These issues tend to hinder debugging, extension, and general comprehension of a program. Expressive programs are not necessarily concise, but there is a tendency for shorter programs to be more expressive. Expressive programs are at a 21
level of abstraction that facilitates debugging, extension, and general comprehension of the program. The programming language (or languages) used to construct a program can contribute to expressiveness if the language is particularly suited to solve problems of the kind specified by the program’s intended meaning. Expressiveness (both the intended and perceived meaning) is partly subjective. A program’s intended meaning is subjective because different relative weightings can reasonably be applied to different fragments of a program’s intended meaning. For example, “a Java class that implements a web service” and “a web service implemented by a Java class” may be describing the same program but would have different intended meanings. The former emphasizes the Java semantics, and the latter emphasizes how the program interacts with the external world, i.e., as a web service. A program’s perceived meaning is also subjective because all programmers have different skill sets and backgrounds and will therefore read a program differently. An analogy to expressiveness can be made with design patterns. The definition of a design pattern is also partly subjective, but the use of design patterns has proven to be an important and enduring way to describe software systems [39]. This account of expressiveness is unrelated to Felleisen [33]. Felleisen formalizes a measure of expressiveness of languages, but does so in a way that is incompatible with our definition. For Felleisen, a language A is more expressive than a language B if transforming a program written in A to B requires a global rewrite of the program. We use our definition of expressiveness as a qualitative framework for characterizing the expressiveness of programs. Throughout this chapter, we apply this framework to various example programs—some that use presentation techniques and others that do not.
3.2
Spatial Arrangement
Spatial arrangement is a presentation technique that allows program elements to be displayed in a two-dimensional format on the screen, rather than linearly (with line breaks and whitespace) as with classically parsable concrete syntax.
3.2.1
Similarities to other (non-presentation) techniques
This section and all other sections with the same title serve to ground the discussion of presentation techniques by comparing them to other techniques that establish the same kinds of properties. This may be language design, editor design, or programming best practices techniques; or they may be 22
variants of these presentation techniques that are already in wide-spread use. Typically, concrete syntax uses lexemes, line breaks, and whitespace to establish layout, algebraic grouping, context, scoping, and documentation. These properties are well understood in textual programming language design, and we shall see how spatial arrangement establishes these same properties using two-dimensional layout. In the Java snippet below, we show manifestations of these properties: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
/* * * Defines a point relative to the origin */ class Point { private int x , y ; public int getX () { return x ; } public int getY () { return y ; } public Point ( int x , int y ) { this . x = x ; this . y = y ; } public double distanceFrom ( Point other ) { return c o mp ut eD i st an ce (x , y , other . getX () , other . getY ()); } }
Line 2 Documentation is written in text and delineated through the use of special characters (i.e., /* and */).1 Lines 6–8 Layout can be used to group related program elements. Such groupings implicitly state that the grouped elements are somehow related. Exactly how they relate is often unstated but can typically be determined by context. Here extra blank lines before and after the three declarations identify them as a group; the group contains the class’s fields and accessor methods. Lines 10 and 14 Context is delineated using various forms of brackets. The line breaks, whitespace, and indentation serves to emphasize the new context, but serves no semantic purpose.2 In this way, it is possible to rapidly distinguish between formal parameter declarations and method bodies. 1
This particular comment uses the /** and */) characters to define a JavaDoc comment whose contents have a semantics that can be used to generate documentation. 2 Other languages such as Python apply semantics to whitespace and indentation.
23
Figure 3.1: Spatial arrangement to declare a summation Lines 10 and 14 A scope is a special kind of context that defines the boundaries within which names are bound. In languages like Java where the bodies of methods are enclosed in the class declaration, there is a context in which fields are available as free variables in the body of the methods. This nesting sets up a kind of indexical reference where the use of a variable (e.g., x) is tantamount to this.x, which is also indexical but more explicitly so. Lines 15 Algebraic grouping combines program elements so that they can be used or viewed together. For example, here it is used when passing arguments to a method invocation. These properties in turn express higher level ideas of the program. For example, by grouping the fields with their accessor methods, there is the implication that these are all the accessor methods in the Point class. Hence, the fields are externally read-only. Furthermore, because these are the class’s only fields, Point objects are externally immutable. We argue later that when the ideas are aligned with the program’s intended meaning, the program is more expressive.
3.2.2
Example
The use of spatial arrangement in the following two examples of code presented in the ETMOP shows how the use of spatial arrangement can also establish these properties, but in a different way. The screenshot in Figure 3.1 shows code that includes an equation and uses spatial arrangement to establish three of the above properties: scopes, context, and algebraic grouping.
24
In this example, three different contexts are defined spatially in relation P P to the .3 As with standard mathematical notation, above the is the upper-bound of the loop, below is the iterator variable and the lower bound, P and to the right is the expression to be evaluated. The spans the expression x.get(i) and defines the scope for the variable i within it. The boundary of each of these contexts are defined spatially, with respect to the P . The horizontal bar is a form of algebraic grouping because it is standard mathematical notation for division.4 The above example uses standard mathematical notation to express the fact that it contains a summation. This is not possible using textual Java: public double mean ( List < Integer > x ) { int sum = 0; for ( int i : x ) { sum += i ; } return sum / x . size (); }
In this textual version, it is clear that both the semantics and the notation of Java are limiting. Because Java does not have first class procedures or macros, the code must be written with an explicit loop. Second, because the display is limited to text, standard mathematical notation cannot be used. This following version in Scheme is more concise than the Java version since it uses a first-class procedure and the loop is implicit in the foldl function: ( define mean ( x ) (/ ( foldl + 0 x ) ( length x )))
Alternatively, using Scheme’s macro system, it is possible to textually capture the domain properties of summation. The explicit loop is removed, which suggests that the summation may be done in any order or in parallel: ( define mean ( x ) (/ ( sum i from 0 to ( - ( length x ) 1) ( get x i )) ( length x )))
This final version is closer to standard mathematical notation, but without the use of two-dimensional layout, standard notation cannot be used, and the structure of the mathematics is obscured. 3
P
This is a clear example of the non-orthogonality of the techniques. The icon uses graphical enhancements to establish abstraction, while at the same time spatial arrangement establishes other properties. 4 Although the above example uses standard mathematical notation to present the summation, more work needs to be done to properly typeset the notation in the text. See Section 5.4.
25
Figure 3.2: Spatial arrangement to declare JDBC stored procedures This next example shows a presentation of code that uses JDBC 4.0 [2] that uses spatial arrangement to establish documentation, layout, and algebraic grouping.5 The code in Figure 3.2 defines an interface method that is associated with a SQL prepared statement.6 The compiler creates a class that implements this interface. At runtime, objects of this type are instantiated by the database manager and these objects execute the prepared statement when their methods are called. Using spatial locality, this presentation of the program expresses that the query definition applies to the method declaration getBigMammals, i.e., the query definition is the method body. Additionally, attributes of the stored procedure are defined in the rightmost column and apply to the query definition. Vertical bars are used as separators. This example uses an idiom employed in many ETMOP extensions, where two display elements are sideby-side with a vertical bar between them. This implies that the right-most display element applies to the left-most display element. The meaning of applies to depends on context. In the JDBC example above, spatial layout is used to shows that the query text applies to the method declaration (i.e., it is the method body). Hence, a new context is defined to the right of the first bar. To the right of that, the query options are defined, using a kind of algebraic grouping where each element of the group is laid out in a column. The far right column documents metadata about the query and describes how the query can be used in the program. Here, spatial arrangement is also used to save space. By placing the query definition to the right of the interface method, rather than above or below, vertical screen real estate is 5
Using JDBC 4.0 it is possible to create stored procedures by defining SQL queries as annotations on interface methods. 6 Prepared statements in SQL tend to be safer than constructing SQL statements through string concatenation because they minimize the possibility of SQLinjection attacks. Furthermore, since prepared statements are pre-compiled, they tend to be more efficient. See http://dev.mysql.com/tech-resources/articles/4.1/ prepared-statements.html for a discussion of this.
26
preserved.7 Compare this to a specially formatted textual presentation of the prepared SQL statement in JDBC 4.0: public interface Animals { @Select ( sql = " select name , description " + " from mammal where weight > ?1 " , readOnly = false , connected = false , a l l C o l u m n s M a p p ed = false ) DataSet < Mammal > getBigMammals ( int weight ); }
Here, the SQL statement is located above the method declaration. Quotation marks are used to delineate the boundary of the SQL statement’s context. The algebraic grouping of the query options are defined using commas, and emphasized through the use of whitespace. However, because the query and its attributes are not placed in separate columns, the applies to idiom is lost. The semantics are the same in both presentations, but in the textual presentation the query and its attributes are combined even though they are of different kinds. Furthermore, this presentation is dependent on a programmer actively maintaining proper whitespace. Alternatively, a pretty printer can be used to automatically format the query definition, but unless properly configured, the results will be less readable. For example, the code formatter that comes standard with Eclipse will format the above annotation as shown below, which further obscures the query and the metadata: public interface Animals { @Select ( sql = " select name , description from mammal " + " where weight > ?1 " , readOnly = false , connected = false , a l l C o l u m n s M a p p ed = false ) DataSet < Mammal > getBigMammals ( int weight ); }
An alternative to defining queries using metadata is to promote query declarations to first-class entities. This is the approach taken by Linq [68] in the .Net framework 3.0, where queries over relational data are specified as language constructs. This tactic addresses some of the syntactic issues that JDBC faces because editor support such as syntax highlighting and 7 An alternative approach to presenting the query metadata would be to use mouse hovers to display it and temporal referencing to make changes. The choice to display the metadata in a column was made because some of the metadata, such as read-only, is fundamental to how the query can be used that we believe it should be visible to programmers whenever they read the query. However, if the primary goal is a more concise program, then displaying the metadata in hovers is a reasonable choice.
27
formatting can be applied to the queries. We describe Linq and how it relates to our approach in more detail with the related work in Section 7.3.1.
3.2.3
Implementation
Taking the program as datastructure view (Section 1.1), the display protocol of a composable presentation editor specifies the layout of display elements relative to each other. This is the foundation of spatial arrangement. The simplest way to use spatial arrangement is to use the display protocol to compose display elements in nested rows and columns, thereby abstractly arranging display elements. The render protocol converts the abstract rows and columns into coordinates on the screen. An alternative approach to implementing spatial arrangement would have been to use layout managers, which is the standard approach used by user interface libraries such as Swing, SWT, and AWT. In these libraries, child graphical objects are laid out inside of parent graphical objects based on an algorithm contained in the layout manager of the parent. For example, layout algorithms can place children in rows or columns, in a table, or using absolute coordinates. We chose not to use layout managers in the ETMOP in order to maintain simplicity of the protocol.
3.2.4
Using spatial arrangement
Spatial arrangement offers alternatives for presenting both standard and non-standard programming language properties, but it is possible to overuse this technique. In general, beneficial uses of spatial arrangement should clearly and concisely express ideas in the program that are not easily exposed textually and this benefit should outweigh any additional clutter introduced in the display by spatial arrangement. This benefit must outweigh the cost of learning a new presentation. If the program contains many new ways of presenting code, this could confuse a programmer, obscuring the program’s meaning. One way to avoid this problem is to only use presentations with nearly obvious meanings, thus reducing the cost of learning new presentations. Consider two feasible presentations of Java programs in the ETMOP. First, Figure 3.3 shows what we consider to be an overuse of spatial arrangement. This shows a possible display of the GetterEMO implementation of the ETMOP. This code creates the Getter display in Figure 1.1. Here, spatial arrangement is used for algebraic grouping in two kinds of places: formal parameter lists, and actual parameter lists. This algebraic grouping of parameters is formatted nicely, but a similar effect can be done using pretty-printing. In fact, many standard Java pretty-printers can be 28
Figure 3.3: Spatial arrangement of formal and actual parameter lists (mock-up)
Figure 3.4: Spatial arrangement of initializers (mock-up) configured to display method parameters in this manner (the Eclipse prettyprinter is one of these), but without a vertical bar. Thus, these presentation extensions clutter the display without promoting new properties and may obscure the intended meaning of the program. The code snippet in Figure 3.4 uses spatial arrangement to present initializers in Java. This display is a departure from standard Java syntax. It provides visual separation of field declarations by surrounding them in a box. Compare this to the standard Java version: public class Rectangle { int x = 5 , y = 5, w = 5, h = 5; Color border = Color . BLACK , interior = Color . BLUE ; }
The frames add clutter to the display by over-emphasizing the initializers and do not expose new properties, but they do allow the program to be more concise. There are advantages for a program to be more concise, but in our 29
judgement, this presentation obscures the intended meaning of the program.
3.3
Textual Manipulation
Textual manipulation is a family of closely related techniques that all control the contents and style of textual display elements and consists of: Formatting (font, color, size, etc.) can be used to emphasize the syntactic structure or semantics of programs. Layout (whitespace, indenting, etc.) is also used for emphasis, but is helpful to demarcate blocks of code, contexts, and scope boundaries. Insertion (adding text) can be used to add documentation to the text, to label regions of text, to explain errors or warnings, or to provide an alternate concrete syntax for program elements. Elision (removing text) can be used as a way to de-clutter the display, and hide code that is irrelevant for the current task. Rearrangement can be used to re-order program elements. This is essentially the combination of elision and insertion of blocks of text. As we shall see below, textual manipulation can establish language extensions, as well as various editor properties such as code folding, syntax highlighting, and pretty-printing. These, in turn, express ideas about the structure of the program (syntax-highlighting), the relative importance of program elements (folding), and domain-specific entities (language extension).
3.3.1
Similarities to other (non-presentation) techniques
Textual manipulation is a presentation technique that is available in a limited form with many program editors in use today. Formatting and syntax highlighting Many program editors use formatting techniques to enhance the display of code. These formatting techniques establish syntax highlighting, which help to express the structure of a program: by highlighting the proper keywords, the start of a new control or datastructure can be clearly identified. Older program editors such as Emacs and Vi typically apply syntax highlighting based on matching regular expressions in source code, as shown in Figure 3.5. Keywords, comments, and variable references can therefore
30
Figure 3.5: Screenshot of a C++ header file in Emacs. Syntax highlighting is determined by regular expressions. be highlighted. In the figure, this highlighting emphasizes the start of the class declaration, as well as its public and private components. Eclipse performs syntax highlighting via a parse of the program. As shown in Figure 3.6, field references and static member references can be specially highlighted. This is not possible by regular expression alone; name binding is additionally required.8 Additionally, work in Literate Programming [53] and typesetting of programs [5] has emphasized reading programs on paper and has shown that proper formatting can make a program’s intended meaning closer to the perceived meaning and hence more expressive. 8 Actually, Eclipse uses a combination of highlighting by regular expression (for keywords and comments) and highlighting by parsing (for semantic highlighting on members).
31
Figure 3.6: Syntax highlighting in Eclipse is based on a full or partial parse of the program. References to fields are in blue and references to static members are italicized. Pretty-printing Pretty printing uses textual manipulation to configure the whitespace of programs. It uses a canonical layout to emphasize syntactic structures. Many program editors provide a pretty printer; Interlisp editors [14] and the Cornell Program Synthesizer [81] were some early editors to provide this functionality. Code folding Many modern program editors use code folding, a property established by elision, to hide parts of the program. Eclipse is one example of such an editor. As shown in Figure 3.7, Code can be folded (elided) or unfolded in the editor by clicking on the icon in the gutter. Foldable regions are based on syntactic structures.9 Language extensibility Most standard program editors do not allow textual manipulation in the form of insertion or rearrangement, but language extension mechanisms such as those established by macro systems allow the program to be manipulated before the program is sent to the compiler (i.e., program elements can be inserted, deleted, or rearranged). All macro systems facilitate the introduction of new language features into an existing language and help the 9
Emacs does not include code folding by default, but Emacs extensions for code folding in various languages do exist (http://www.emacswiki.org/cgi-bin/wiki/FoldingMode). Folding regions are defined by special characters, not syntax. Vim, on the other hand, offers more kinds of folding: by syntax, special characters, indentation, and keyboard commands (http://www.vim.org/htmldoc/usr_28.html).
32
Figure 3.7: Code folding can elide import statements. Import statements can be un-elided by clicking on the icon in the gutter bar. programmer express new ideas in a program through the addition of new syntactic structures. As we describe in Section 7.2.3, macro systems are typically associated with the compiler, not the editor. Some macro systems transform the lexical structure of the language [37] and others transform the syntactic structure [56]. The example in the next section shows how textual manipulation provides an alternative form of language extension.
3.3.2
Example
Textual manipulation can be used to mimic syntactic extensions of languages. For example, Figure 3.8 shows a snippet of AspectJ in code-style AspectJ, but the program itself is stored in @AspectJ style. By presenting aspect code in what appears to be language features, rather than embedded metadata annotations, the structure of the aspect is expressed directly. This example shows how multiple forms of textual manipulation can 33
public @After ( value = " changes ( p ) " ) void changesAdvice ( Point p ) { Screen . l o g D i s t a n c e F r o m O r i g i n ( p ); } @Pointcut ( " this ( p ) && execution ( void Point . set *( int )) " ) void changes ( Point p ){ }
Figure 3.8: Use of textual manipulation to display @AspectJ annotations (below) as code style AspectJ (above). be used together to create what appears to be a new concrete syntax for Java programs. Elision is used to remove the annotations and the method declarations. Formatting is used to highlight AspectJ keywords. Insertion is used to add AspectJ-specific syntax. Rearrangement is used to move pointcut designator strings to the correct locations. Semantically, all of this is supported by a weaver10 that coordinates the runtime so that aspect and non-aspect code can properly run together.
3.3.3
Implementation
Taking a program as datastructure view, to implement text formatting (e.g., font, color, and size) in a composable presentation editor, the formatting can be associated with a display element and stored as a style descriptor. When the render elements are created, the style descriptor generates text with the appropriate attributes. Elision, layout, and insertion can be performed by the display protocol (Section 2.3). The display protocol alters the visitation path of the program to skip over or rearrange some program elements— hence eliding or rearranging program elements as they are transformed into display elements.
3.3.4
Using textual manipulation
Above, we have shown three general uses for textual manipulation: • making the display of programs more concise, which can emphasize or de-emphasize code based on relative importance (e.g., code folding), • emphasizing syntactic or semantic constructs that are already present, 10 A weaver is the program that properly coordinates aspect and non-aspect code so that they can run together [51].
34
in the code (e.g., syntax highlighting and pretty printing), and • introducing new constructs into the language (e.g., as an alternative to macros). The first two uses for textual manipulation are already in use in many program editors, and we have shown an instance of the third use to establish language extensions.
3.4
Temporal Referencing
Temporal referencing is a presentation technique whereby an editor event handled by a display element initiates some action on some other element— either a display element, program element, or on an object external to the editor. Many modern program editors provide some form of temporal referencing to establish properties such as eager parsing, text hovers, and code folding. To the end programmer, who views the program as presentation, temporal referencing should feel like the program is reacting to programmer input; just like any other part of the development environment does. Temporal referencing establishes a relationship from the source display element to a target element; or it can show or hide additional information about a program. Temporal referencing momentarily emphasizes or de-emphasizes relationships that are typically already expressed through other kinds of referencing.11 The following examples provide a general notion of temporal referencing is and what is not temporal referencing. After that, a more precise definition of temporal referencing is given.
3.4.1
Similarities to other (non-presentation) techniques
As described above, the use of temporal referencing in many modern development environments establishes some properties of the program, which in turn express certain relationships about the code. Some examples are: eager parsing The recently typed text is parsed and any new errors are displayed, thus expressing the relationship between the newly typed text and the error. text hovers Hovering over a region of the screen will bring up a window with extra information (e.g., source code, documentation, etc.) about 11
Note that this is a different kind of temporal referencing than described in [64]. Lopes et al describe an indexical relation between a program element and a point in the execution of a program. AspectJ pointcuts are examples of this kind of temporal referencing.
35
the program elements in the hovered region, thus expressing a relationship between the hovered region and the extra information. code folding Clicking on a button just outside the editable region will show or hide a region of the program, thus making the program more concise. As a side effect, this expresses the relationship between the button and that region. This is an example of how textual manipulation and temporal referencing work together to seamlessly alter the display of the program. showing external information Visual Basic and other visual user interface creation tools typically display a context sensitive properties window when a programmer selects a widget. This expresses the relationship between the widget and its properties. automatic identifier highlighting Eclipse highlights all identifiers with the same name when one identifier is selected, thus the existing relationship between all the identifiers is emphasized. DrScheme uses arrows to accomplish the same: when DrScheme is in a special mode, hovering over an atom will draw an arrow from its definition to its use. inline renaming Using a special refactoring technique in Eclipse, it is possible to edit the name of all uses of the same reference at once. This emphasizes the relationship between all of the references in a way similar to automatic highlighting. This is also a form of constrained editing (as explained below). mouse actions and typing There are many editor interactions that fall under the definition of temporal referencing, but are uninteresting. These include typing (e.g., as the programmer types, characters appear on the screen), and mouse actions (e.g., a double click will select a word, etc.). Typically, these are all actions that are available in standard, simple text editors (e.g., Microsoft’s Notepad). Although these actions fall under the definition of temporal referencing, they are not specific to programming and so we disregard them from here on. All of these uses of temporal referencing are found in mainstream program editors. The underlying similarity between all of these uses is that the programmer triggers an event in the editor window that is handled by a display element. The display element, in turn, executes a command. The command may emphasize a region of the editor itself, may change part of
36
the program, or may display some information about the program externally. Temporal referencing is a factor in making editors more interactive and responsive to the programmer, and can add depth to the programming experience. As a side effect temporal referencing emphasizes or expresses a relationship between a display element and what it affects.
3.4.2
Example
We introduce two examples of temporal referencing. The first is a clear example of temporal referencing in Embedded CAL. The second is a characterization of the boundaries of what constitutes temporal referencing and what does not. This first example, shown in Figure 3.9, shows a use of temporal referencing in Embedded CAL. The behavior of this program is described in Chapter 6, but here we focus on temporal referencing only. The main text area presents the CAL expression and the list to the right presents its input parameters. As the the programmer types in the text area, the editor reparses the CAL code in the background and determines all name bindings. When an unbound identifier is discovered, this identifier is converted into an input parameter and added to the list of input parameters on the right. Thus, temporal referencing is used to establish a relationship between the text just entered and the new parameter in the parameter list. The editor is therefore expressing the idea that a new parameter has been created and it is unobtrusively notifying the programmer of this change. Temporal referencing tends to be combined with other techniques, and is often used to emphasize existing relationships. The second example illustrates this use of temporal referencing. Consider four variations of error reporting in an editor as in Figure 3.10. Two variations use spatial arrangement to report errors; and two use temporal referencing to do the same. Together the four variations are the crossproduct of using or not using the two techniques together: a Here, both techniques are used. An error mark in the gutter appears in the appropriate location (spatial arrangement) while typing (temporal referencing). b This variation shows errors in the gutter as above, but the errors are only updated when the programmer explicitly invokes the refresh button. Thus, spatial referencing is used, but no temporal referencing. Because the refresh button is not part of the editor, it is not a display element and therefore cannot trigger temporal referencing.
37
Programmer types
Editor responds by reparsing and adding new input parameter
Figure 3.9: Temporal referencing in embedded CAL.
38
b) Errors appear inline after refresh is clicked
a) Errors appear inline while typing
c) errors appear in log file while typing
d) Errors appear in log file when refresh is clicked
Figure 3.10: Four variations of error reporting.
39
c This variation does not show errors in the gutter, but rather appends them to a log file as the programmer types. Thus, temporal referencing is being used since an event on a display element triggers an action external to the editor to occur. But spatial arrangement is not used. d To complete the analogy, there is one more variation that logs errors to an external file only when the refresh button is invoked. Thus, here neither presentation technique is used. In practice, it is doubtful that the last two variations are particularly useful, but they serve to isolate the effects of temporal referencing on the editor. As this series of examples shows, temporal referencing can be coupled with spatial arrangement to express relationships between parts of the program.
3.4.3
Implementation
In this section, we first define more precisely what temporal referencing is, what it isn’t, and discuss why this definition is appropriate. Then we present some potential implementation strategies for temporal referencing. Definition Temporal referencing is a relationship that unfolds over time between a display element and another display element, program element, or some object external to the editor. This relationship becomes apparent to the programmer through some visible action taken by the editor in response to an event. The event that triggers the action is typically a programmer event such as a mouse click or a keypress, but non-human events—such as clock ticks, the completion of a compilation, or the return of a remote procedure call—are possible as long as the handler of the event is a display element (i.e., a part of the program that can be visualized). By restricting the event handler to being a display element, there is assurance that the event handler is associated with the program, not the development environment. Thus, it is the program that has control over how the editor responds to events, rather than the development environment. For example, the following is not temporal referencing. In the ETMOP, when the end-programmer clicks on the outline view in Eclipse, the caret will move to the expected editor location corresponding to that click. But because the outline view is external to the ETMOP, the presentation designer has no control over how the outline view affects the program. Therefore, this
40
event belongs to the development environment, not the program (the program itself has no control over how clicks on the outline view are handled) and is not temporal referencing. An alternative implementation of the ETMOP could conceivably allow the program to extend its presentation in the outline view. In this scenario, the above operation would be temporal referencing because the presentation designer now has control over how the event affects the program. However, this is not the implementation described in this dissertation. The outline view is external to the editor because the ETMOP as currently implemented does not provide protocols to extend it. In general, whether or not to categorize an event and its effect as temporal referencing depends on where the boundary between editor and development environment is drawn. This boundary is determined by the kinds of protocols available to the presentation designer and which development environment components are made extensible through these protocols. In the ETMOP, the boundary is the editor pane and all the code required to implement it. Implementation strategies In a composable presentation editor, temporal referencing can be implemented through the display protocol, which can add handlers and callbacks to display elements as they are being created. This allows the editor to react to programmer input based on the static semantics of the program being displayed. The ETMOP and Embedded CAL are built on top of an event-based graphics framework [69]. Event-based programming is often used in user interfaces to propagate data and control flow from the screen to a data model. Temporal referencing can leverage the existing event handling infrastructure to generate and handle events appropriately. As described in Chapter 2, the user-interface is the physical layout and the data model is the combination of the logical layout and the program+metadata. Reactive programming [21] is an alternative to event-based programming where the setting up of control flow occurs implicitly during object construction. With a reactive programming model, data flows continuously from the data source and changes are immediately reflected in the model. Although neither implementation of composable presentation editors use a reactive programming methodology, this is a feasible alternative to eventbased programming.
41
3.4.4
Using temporal referencing
All of the examples of temporal referencing above share three traits: they are predictable, restrained, and reversible. Predictability requires that the programmer can anticipate that triggering a particular editor event will cause a particular action, such as showing or hiding information on a mouse click. A violation of predictability would be a notification system that reminds the programmer to save after every 250 keystrokes. (This is a large enough number of keystrokes that a programmer would likely be surprised each time this occurs.) Notifying after every two keystrokes would be more predictable, but even less useful. Being restrained requires that the scope of the action taken is commensurate to the importance of the relationship being expressed. Pop-up windows that appear on the program have the potential to be unrestrained if they are large and block large parts of the program or if they appear at inopportune moments. If these windows are properly placed and do not hide the display element that they are expanding on, then their negative effects are likely to be minimized. Another example of violation of restraint would be an error handling system that causes the editor to flash red twenty times whenever a syntax error is entered. Reversibility requires that any change to the program should be undoable and any information shown (such as pop-up windows) should be easy to hide. Eclipse text hovers are good examples of this because as soon as the mouse is moved, the hover window disappears.
3.5
Graphical Enhancements
Adding graphics to a program’s display provides an alternative to a pure textual display. When used appropriately, graphics can concisely express ideas about a program that are verbose or awkward when using text only.
3.5.1
Similarities to other (non-presentation) techniques
Language-based approaches to abstraction Abstraction is a standard programming technique that reduces the information content of a program element so that only information relevant for a particular purpose is retained. The following short program illustrates several linguistic mechanisms can establish various forms of abstraction: functional abstraction (line 12), polymorphism (line 11), and sub-classing (line 5): 1 2 3
abstract class Shape { abstract int area (); }
42
Figure 3.11: A use of graphical enhancements in DrScheme
4 5 6 7 8 9 10 11 12 13
class Rectangle extends Shape { int h =1. w =3; int area () { return h * w ; } } ... Shape s = new Rectangle (); int area = s . area (); ...
These various incarnations of abstraction serve to clarify the object model and the appropriate operations on it. For example, sub-classing and polymorphism in the above code explain that programmers should think about this program in terms of Shapes and they also hint at possible ways that this program can be extended (e.g., sub-classing Shape to create Circle). Here, functional abstraction expresses the idea that area is an important operation that will be used in many places and that its implementation is less important than what it means. Graphics to emphasize existing relationships Graphical enhancements can be used to emphasize existing relationships in programs—a form of referencing. Figure 3.11 explores one way that DrScheme uses graphical enhancements in this manner. When the Check Syntax button is enabled, hovering over an identifier will cause an arrow to appear that associates an identifier with its declaration. This use of graphical enhancements is combined with temporal referencing so that existing relationships are emphasized in the presentation. Defining contexts Graphical enhancements enable context and scoping boundaries to be defined graphically. As we show in the examples below, graphical enhancements can be combined with spatial arrangement synergistically. Graphical 43
Common protocol for all elements provided by the Java model. Java model elements are exposed to clients as handles to the actual underlying element. The Java model may hand out any number of handles for each element. Handles that refer to the same element are guaranteed to be equal, but not necessarily identical. Methods annotated as “handle-only” do not require underlying elements to exist. Methods that require underlying elements to exist throw a JavaModelException when an underlying element is missing. JavaModelException.isDoesNotExist can be used to recognize this common special case. This interface is not intended to be implemented by clients. Figure 3.12: JavaDoc for Eclipse’s IJavaElement enhancements can further be used to associate these contexts with each other, which is a form of referencing. External information Not all information about a program is concisely or explicitly stated from within the code. Rather, information or clarifications about a program can exist outside of the program itself; examples are bug databases, UML diagrams, the results of regression testing, and various kinds of formal and informal specification. The following example illustrates this concretely. In the Eclipse API, JavaDoc comments are sometimes low-level and targeted towards programmers who are already familiar with the protocol and are just seeking to understand details. New developers must go to documentation included with Eclipse for sample code and a detailed description of how to use the APIs. In the Eclipse JDT12 the JavaDoc for the class IJavaElement in Eclipse is shown in Figure 3.12. The JavaDoc concisely and completely describes what an IJavaElement is, but may be confusing to programmers who are new to the Eclipse JDT API. Inside the Eclipse help system there is more information on IJavaElements. Figure 3.13 shows a documentation snippet taken from the Eclipse JDT Plug-in Developer Guide that helps to place various subclasses of IJavaElement into context in a way that is difficult using text only. As this example illustrates, low-level documentation is typically included with the program as comments, while supporting or explanatory documentation typically exists external to the program. The former is meant for 12
Eclipse v3.2
44
All Java elements support the IJavaElement interface. Some of the elements are shown in the Packages view. These elements implement the IOpenable interface, since they must be opened before they can be navigated. The figure below shows how these elements are represented in the Packages view.
Figure 3.13: Eclipse JDT Plug-in Developer Guide documentation for IJavaElement expert users and the latter is meant for beginners. Including the graphical enhancements in the presentation of the program may be particularly useful to beginners.
3.5.2
Example
There are four general ways that graphics can be used: graphical abstraction, association, delineation, and external information embedding. Graphical abstraction Graphics can be used as an abstraction for a set of program elements. This use of graphical enhancement serves to focus the programmer’s attention on a particular facet of the program, while momentarily hiding less important details. This usage may manifest itself as an icon, a graph, a chart, or some 45
other graphic that replaces chunks of text. Graphical abstraction is an approach to abstraction that is somewhat similar to the language-based abstraction mechanisms described above. Linguistic abstraction uses a semantic engine (e.g., a compiler, a MOP, or a pre-processor) to enable the semantics of the abstraction, while graphical abstraction is an editor technique and is wholly contained in the editor. Graphical abstraction can be attached to a semantic engine to provide many of the same kinds of language enhancements, but using graphics instead of or in addition to text. An example of graphical abstraction is the following class declaration, which substitutes graphics for a webservice definition using JSR-18113 [25]:
In this declaration, the parameterization of the webservice declaration has been abstracted and replaced with icons. The as being a webservice. The
icon defines the class
defines the method as being a web method
and therefore exposed by the service. defines the soap binding. These icons hide details like the XML namespace of the service. Graphical abstraction may be combined with some form of temporal referencing that can show concretizations of what the graphics abstracts. In the webservice example, by double-clicking on any of the icons, it is possible to view and edit the details for that part of the webservice. Here we show the effect of double-clicking on the echoAdd webmethod:
13 JSR-181 defines webservice information as annotations on Java declarations. A translator is used to convert the Java declarations into the appropriate WSDL descriptor.
46
The operationName field defines the name of the webmethod used by clients. By using graphical abstraction, several verbose metadata annotations are hidden. The graphical enhancements are expressing that some program elements contribute to the webservice that the Echo class defines, but without precisely detailing how each part of the webservice is parameterized. Association Graphics can be used to associate disparate parts of the program. Association describes indexical relationships from one display element to another and is often combined with spatial arrangement. Association can be read as display element X applies to display element Y, where applies to is dependent on context. The use of graphics for association is similar to some lexical techniques. For example, we can say that a method body applies to its method declaration, which is accomplished textually, not graphically. Graphical association may be manifested as arrows that connect display elements as in DrScheme, shown in Figure 3.11. Alternatively, the JDBC example, which we return to now, uses lines to associate display elements. Vertical lines are used to associate the method declaration with the SQL statement and the SQL statement with the SQL options. As mentioned in the spatial arrangement section, these graphics contribute to a common idiom used in many presentation extensions with the ETMOP:
In association, the relationship is between display elements, rather than between program elements. This distinction is important only to presentation designers who view the program as datastructure because leveraging the relationships between display and render elements enables the use of other presentation techniques such as temporal referencing. Here, the association expresses that the SQL statement is the method body of the interface method declaration. This is the same kind of association used when lexical techniques—line breaks, white-space, and brackets— associate a method declaration with its method body.
47
Figure 3.14: Variations on delineating groups of field declarations Delineation Graphics can be used to delineate disparate parts of the program—closely related to association; delineation and association are most often used together. As with association, delineation is often combined with some form of spatial rearrangement and may be manifested through lines or boxes to delineate regions of text. Delineation is a mechanism to define contexts and scopes that uses graphical boundaries to delineate the extent of syntactic structures, variables, behaviors, and actions. Delineation can be clearly seen in Figure 3.4 where the scope of the initializer is defined using a border. In this example, the delineation proclaims (perhaps a little too loudly) that variable declaration fragments contained in the box are all of the same type and are all related—they constitute the spatial and color characteristics of the class respectively. And also, returning to the JDBC example again, the vertical bars are used to delineate the boundaries of the three columns (this is in addition to their role of association). These vertical lines express the separation of the declaration from the method body and the method body from its attributes. One of the issues with delineation is how to subtly, but clearly define the boundary between parts of the program. We can see this issue arising in Figure 3.14 where we show two variants of delineation. Delineation is used in each variant in two ways: delineating field declarations from accessors and invariants, and also delineating one group of field declarations from another group. Delineation in Figure 3.14a is more subtle because it uses a vertical bar only. Whereas delineation in Figure 3.14b is more clear, but less subtle. Here, we believe that the subtlety concern overrides the clarity concern and opt for Figure 3.14a. Sometimes the use of subtlety is inappropriate. Mathematical notation tends to combine delineation with semantics. We can see this through the 48
use of symbols such as Σ that are used for delineation, but at the same time, they have a clear semantic purpose. Because these symbols serve multiple purposes, there is less need for subtlety. Compare the use of delineation in the field declarations of Figure 3.14 with that of the summation of Figure 3.1. The graphics used to delineate groups of fields are as subtle as possible, whereas the graphics used to delineate regions of the summation (i.e., the Σ) is both large and conveys computation. External information embedding Finally, graphics may be used to add external information into the presentation of the program. This allows non-textual forms of documentation, error reporting, and analysis to be included within the program. For example, using this technique, Figure 3.15 shows a UML activity diagram associated with a method. The method, calculateTotal, is shown in the diagram and is shaded in yellow. This diagram expresses the method’s relationship to control flow of the rest of the program. Note that this particular example shows an uninterpreted image attached to the method. However, it is possible for a more sophisticated implementation to construct this image directly from the program, or to ensure that method names in the image are synchronized with the method names in the text.
3.5.3
Implementation
Graphics are abstractly specified in the logical layout and concretely drawn in the physical layout. The ETMOP, for example, attaches drawing objects to display elements that are configured to draw the render elements during the render protocol, as shown in Figure 3.16.
3.5.4
Using graphical enhancements
In the above examples graphical enhancements have been used to enhance a program in the following ways: • by emphasizing or de-emphasizing some facet of the program (e.g., the details of the webservice are de-emphasized in order to emphasize the behavior of the program), • by defining the program structure and how this structure is internally related (e.g., for delineating and associating the columns of JDBC stored procedures), or • by showing information that is external to the program text and cannot easily be expressed in words alone (e.g., the graphic of the UML diagram).
49
Figure 3.15: Embedding external graphical information in a program. This is a UML activity diagram. It is possible to overuse graphical enhancements. As much as possible, graphics should be subtle and express concepts that cannot be clearly or concisely expressed through text alone.
3.6
Constrained Editing
Constrained editing is a technique whereby an editor can constrain the kinds of inputs that are acceptable in a particular region of the display. The following are some uses of constrained editing: • To make text read-only, useful for adding a label or documentation to the presentation. • To constrain possible choices of the text to a fixed set. For example, 50
Display Element Drawer
Render Protocol
Render Element Some Nifty graphics
draw
Figure 3.16: Using the render protocol to draw graphics in the ETMOP. this can be done through a drop-down menu and would be useful for choosing from a set of constants or enumerated data types. • To validate input to a text field by, for example, constraining the field to a numeric value, or by disallowing whitespace. • To use some other input mechanism such as a slider, checkboxes, or radio buttons. All of these uses of constrained editing establish some kind of safety or documentation mechanism that can express how a program is expected to be used or extended.
3.6.1
Similarities to other (non-presentation) techniques
There are three ways to think about constrained editing: as a kind of editor enforced type system, as a kind of structure editor, or as a way to demarcate syntactic structure in the program. Like a type system Both static type systems and constrained editing can provide editor feedback on the correctness of some kinds of expressions entered into the editor. A 51
type system defines how a programming language categorizes the values of expressions and how these expressions can interact. By defining which expressions evaluate to valid types, a type system also expresses constraints on programs. Static type systems enforce these constraints as the program is compiled, and many mainstream editors provide this feedback directly in the program editor using temporal referencing (as described above). It is this use of a type system that most closely resembles constrained editing. An example of this is shown below. Like a structure editor Structure editors inherently provide a kind of constrained editing. Hard structure editors such as those described in Section 7.1.2 assist programmers by constraining input to ensure that syntax errors can never occur— the programmer is prevented from entering text that would transition the display to an invalid state. Alternatively, soft structure editors provide more flexibility and allow the programmer to freely enter any text. Feedback is provided when input is erroneous. There is tension between soft and hard structure editors. Hard structure editors tend to be easier to implement, but are sometimes too restrictive in what they allow programmers to do [49, 87]. Because of their restrictiveness, hard structure editors are rarely used in modern program editors. However, aspects of hard structure editors can be useful to help programmers choose between a small number of valid values. Alternative editor mechanisms such as drop-down menus can be useful for this as we shall see in the next section. By establishing restrictiveness, both structure editors and constrained editing help to express what makes a potential program valid or invalid. VB6.0 uses a form of constrained editing in its GUI builder. Figure 3.17 shows a drop-down menu that assists a programmer in choosing between mouse pointers over the widget that is in focus. Because the set of possible values is small it is possible to include them all in the menu. Using this menu has two distinct advantages. First, it augments VB6.0’s relatively weak type system. Because VB6.0 does not have enumerated datatypes, all of the possible values of the MousePointer field are encoded as integer literals. The drop-down ensures correctness of the value by restricting possible choices to valid values. Second, the drop-down assists the programmer by including a human-readable description of the value. A way to define context Constrained editing can also help delimit code contexts. As described above in Section 3.3, lexical structures such as parentheses, brackets, and inden52
Figure 3.17: Constrained editing in Microsoft’s Visual Basic 6.0. tation are typically used to define different contexts in programs. For example, these contexts differentiate between method bodies and parameter lists. Additionally, some editors emphasize certain contexts by using syntax highlighting on keywords (e.g., for and while). However, when new kinds of contexts are created through language extensions, it is possible to label the context with some sort of description. This kind of labelling has parallels in programming languages. Many languages, such as OCaml, Smalltalk, and some Lisp dialects support labelled parameters, where function calls (or message passing) can specify actual parameters using labels. By using more explicit delimiting of contexts in the display, the structure of the program is more clearly expressed.
3.6.2
Example
We show two examples of constrained editing. The first is from Embedded CAL. The second compares two alternatives for constrained editing in the Getter presentation extension in the ETMOP. Figure 3.18 uses the same CAL expression shown previously. Recall that the table on the far right shows the input parameters. Since these input parameters come from Java variables of the same name, it is necessary to marshal the data from Java to CAL. The InputPolicy column controls the marshaling by associating a marshaling strategy with the input parameter. The programmer can choose from a set of pre-made InputPolicys or the 53
Figure 3.18: Constrained editing in Embedded CAL
Figure 3.19: Two variations of constraining input on getter declarations. The programmer has just typed ‘0’. programmer can override this and enter a custom one. Thus, this example is analogous to a soft-structure editor. By providing a set of standard InputPolicys, the program is expressing the kinds of marshaling that are typically used in Embedded CAL, and gently constrains the programmer to use those marshalings. Figure 3.19 shows two alternatives for constrained editing in the Getter presentation extension of the ETMOP. In Figure 3.19a, the programmer is prevented from entering any text into the getter name field that would cause the prefix to be invalid. In this figure, the programmer has just typed ‘0’, but the input was rejected because this would have caused the name to be an invalid Java identifier. This kind of constrained editing is analogous to that of a hard structure editor. In Figure 3.19b, the programmer can enter any text into the field. But, if the entered text causes an invalid name, then the getter name field is underlined in red. This is analogous to a soft structure editor. Both alternatives express the fact that the only appropriate value for the editable region is a valid Java identifier. This getter example also illustrates a third use for constrained editing: context definition. The getter region has Getter: as a read-only label to the left of it, which documents the context of the region it is adorning, expressing that the region defines a getter on the associated field declarations. Because the context of the getter extension is physically separated
54
from the rest of the program, the extent of the constrained region is clear. Embedding this constrained region inside other code without any apparent boundaries could be confusing for programmers.
3.6.3
Implementation
Constrained editing can be added to the program’s presentation during the logical layout phase. Because each display element of the logical layout describes how some region of the display reacts to user input, display elements can be configured to accept or reject different kinds of inputs.
3.6.4
Using constrained editing
Because structure editors can be overly restrictive, constrained editing should be used with caution. Before using constrained editing, the programmer should determine what constraint is being expressed and how this can assist in communicating to other programmers how the program works, how it can be altered, and how it can be extended. The examples above show that constrained editing seems to be most effective when used on small regions of the display, when the boundary of the region is clear, and when the set of valid inputs is small.
3.7
Expressiveness of Examples
In this section, we briefly revisit each example described previously and apply our definition of expressiveness to them. For each example, we ask: is its their intended and perceived meanings and to what extent are they aligned? For each of these examples, we describe some part of the intended meaning and describe the extent to which it aligns with the perceived meaning. Each subsection below refers back to a program or code snippet introduced earlier. Some of this section will be a repeat of arguments made previously. They are reiterated here and explicitly connected back to our definition of expressiveness.
3.7.1
Summation
The intended meaning of the summation program of Section 3.2.2 is that it should sum a list of numbers. The presentation of this program uses mathematical notation to connote a summation and therefore aligns the perceived meaning with the intended meaning. The Java version, however, with its explicit loop does not connote mathematical summation and is therefore less expressive. On the other hand, through their syntax, the two Scheme examples connote summation more strongly than the Java version,
55
but less so than the spatially arranged version and hence their expressiveness falls somewhere in between.
3.7.2
JDBC
The intended meaning of the JDBC example, shown in Figure 3.2, is that a SQL stored procedure is defined on an interface method and the stored procedure has some metadata. Because the query is both clearly separated from, but attached to the method declaration and its metadata, this relationship is connoted in the perceived meaning. The plain Java version, on the other hand, does not connote the query definition as strongly because the boundaries between declaration, query, and metadata are not as clearly defined.
3.7.3
Parameter lists
The intended meaning of the program in Figure 3.3 is the definition of the Getter EMO (i.e., this is the code that defines the Getter presentation extension of the ETMOP). The perceived meaning, however, emphasizes the formal and actual parameter lists at the expense of the rest of the program. The graphical enhancements and spatial arrangements therefore clutter the display without emphasizing anything interesting in the perceived meaning. Hence, the perceived and intended meaning are not closely aligned and this program is not expressive.
3.7.4
Initializers
The intended meaning of the program shown in Figure 3.4 program is to define a rectangle. The perceived meaning emphasizes the field declarations and initializers at the expense of the rest of the program. Since these fields and initializers are private, they neither describe the problem being solved by this program, nor how the program can be used. Therefore this emphasis of the perceived meaning is not aligned with the intended meaning and this program is not expressive. On the other hand, emphasizing private fields or methods does not necessarily obscure the meaning of the program. For larger programs, emphasizing proper parts of the internal structure of the program may be essential in understanding the program’s intended meaning.
3.7.5
Code style AspectJ
The intended meaning of the AspectJ snippet shown in Figure 3.8 is that the code records a Point’s distance from the origin every time the object moves. In the code style AspectJ, the perceived meaning captures the
56
aspect-oriented nature of the program because aspect-oriented language features are elevated to language features. For the annotation style AspectJ, the aspect-oriented nature of the program is obscured because of the use of metadata annotations. For this reason, we can say that the perceived meaning of the code style AspectJ is more closely aligned to the intended meaning than the perceived meaning of annotation style is, and code style is therefore more expressive.
3.7.6
Embedded CAL
We discussed two parts of Embedded CAL embedded editors in this chapter: the use of temporal referencing to update input parameters while typing (in Figure 3.9), and the use of constrained editing to help a programmer choose input policies (shown in Figure 3.18). The part of the intended meaning that is related to the temporal referencing is that the input parameters define the CAL expression’s interface to the Java code. The use of temporal referencing emphasizes changes to these parameters. The perceived meaning therefore states this change. When the interface changes, the programmer is required to re-learn the intended meaning in order keep the perceived meaning aligned with the intended meaning. This emphasis on the change assists with this task. Furthermore, because the temporal referencing is subtle, it is possible that the programmer can ignore the change if it is not interesting during the moment it changes. The constrained editing part of the intended meaning is that there are a small number of input policies that most likely to be used by the parameters of the embedded editor. This part of the intended meaning is related to how the CAL expression is used by external code. By listing these input policies in a drop-down menu, the perceived meaning is characterizing the suggested input policies and hence emphasizing the intended meaning.
3.7.7
Variations on error handling
As we show in Figure 3.10, we say that errors are part of a program’s intended meaning. Understanding these errors are essential for fixing the program. When error reporting is integrated into the editor, the program’s perceived meaning includes the location and kind of errors. Hence, integrated error reporting allows programs to be more expressive.
3.7.8
Webservices
The webservices example in Section 3.5.2 shows a trade-off between weights of various parts of the intended meaning. The intended meaning of the webservices example as implemented is: this is a Java class that implements a
57
Figure 3.20: Two variations of constraining input on getter declarations. The programmer has just typed ‘0’. webservice. Hence, hiding the plumbing details of the webservice emphasizes the Java part of the program, while still declaring that the program is a webservice. Thus, the perceived meaning emphasizes the part of the intended meaning that is most important. This presentation of webservices would be most valuable to those who intend to maintain or extend the Java implementation. An alternate version of the intended meaning is: this is a webservice implemented by a Java class. In this case, details about the webservice would be emphasized, while the Java code itself would be hidden, minimized, or obscured. This kind of presentation would align the perceived meaning with the intended meaning for programmers who are creating clients of the webservice, but are not interested in how the service is implemented.
3.7.9
Embedding UML diagrams
By attaching the UML diagram to the method declaration as we show in Figure 3.15, the perceived meaning of the method includes its relationship to other methods in the program. If the intended meaning includes such a detailed description of how this method is used in the context of the rest of the program, then embedding the UML diagram allows the perceived meaning to align with the intended meaning.
3.7.10
Getter presentation extension
We have already shown examples of getter and setter presentation extensions in several places in this dissertation. We describe how they can add to expressiveness here. To focus the discussion, we use Figure 3.20. This figure shows a simple Point class that uses a presentation extension to define accessor methods on fields. The intended meaning of this example includes several nuances. This is a class that defines a two dimensional point 58
that is immutable once it is created. There are only two integer fields of stored state and they are accessible to the outside world through accessor methods prefixed with get. The perceived meaning is partially constructed through the use of presentation extension. Because the convention is to put all Java field declarations at the top of the class, it is easy to see that the class has only two integer fields. It is also easy to see that they both have getter methods prefixed with get. Since the fields have getters, but no setters, it is a reasonable assumption that they are read-only from outside the class. Note that there is no guarantee that these constraints hold, but it is a reasonable assumption based on presentation. Constrained editing and temporal referencing is used to ensure that the getter prefix is a valid Java identifier, as shown in Figure 3.19. Thus, this presentation extension characterizes the acceptable names for accessor methods and consequently dictates how fields that it is attached to can interact with external code. Through all these connotations that are a part of the presentation extension, the perceived meaning is kept in line with the intended meaning.
3.8
Summary
In this chapter, we started with a theory of expressiveness, which states that programs are expressive to the extent their perceived meaning is aligned with their intended meaning. We then introduced five general techniques to compose the presentation of a program and argued that with proper use, they can make programs more expressive. For each technique, we started with an analogy to other, non-presentation techniques or to variants of the presentation technique in current use. We presented an example to show how this other technique can establish certain general properties in programs. Then, we showed how these general properties can be used to express specific ideas about that program. We then demonstrated how the presentation technique can establish the same or a similar property, but in a different manner. And we concluded the analogy by showing a specific example of the use of this property and how it can also express certain ideas about a program. After finishing the analogy, we described a general approach to how the presentation technique could be naturally implemented in a composable presentation editor. We ended each presentation technique description by describing some guidelines and possible pitfalls for using each technique. After describing the presentation techniques, we revisited each example and described how the use of presentation techniques can influence the perceived meaning of a program to align it with the intended meaning. 59
In the next chapter, we describe the implementation of the ETMOP and CTMOP, the extensible editor and compiler that uses presentation extensions to make a programming language extensible through the use of these presentation techniques.
60
Chapter 4
The ETMOP and CTMOP The edit-time metaobject protocol, or ETMOP, is an extensible editor that uses presentation extensions to display and edit programs. With presentation extensions, the stored form of the program remains in a standard syntax—there is no change to the parsable syntax.1 The program editor incorporates a metaobject protocol (MOP) [50] to make it possible for a presentation designer to customize the way the program is read, edited, and browsed. Presentation extensions rely on semantic extensions to deliver static and dynamic program semantics, which is provided by the compile-time metaobject protocol, or CTMOP. The CTMOP provides semantic extensions to Java programs and works in conjunction with the ETMOP. Other semantic processors can be used instead of or in addition to the CTMOP. In this chapter, we describe the implementations of the ETMOP and CTMOP. We show how the two MOPs work together to provide a particular kind of language extensibility—presentation extension coupled with semantic extension [28]. We show how the ETMOP uses the composable presentation editor architecture (Chapter 2) to construct the presentation of programs. The example from Section 1.2.1 is repeated below in the screenshot in Figure 4.1. This example is used to motivate the following description of the implementation.
61
Figure 4.1: Screenshot of a program in the ETMOP editor
Display
Render
Serialize
External Compiler
Figure 4.2: ETMOP and CTMOP architecture 62
4.1
Implementation and Architecture
The architecture consists of two MOPs that run at separate times, but work synergistically to enable implementation of language extensions, as illustrated in Figure 4.2. The ETMOP architecture uses the composable presentation editor architecture of Chapter 2, and the CTMOP mirrors many of the structures found in the ETMOP. The ETMOP is responsible for presentation extensibility and the implementation of the visual form of the program. The CTMOP is responsible for semantic extensibility and the implementation of traditional static and dynamic language semantics (warnings, errors and execution behavior). In Section 4.2.4 we discuss how other semantic processors can be used together with or instead of the CTMOP. This section describes the two MOPs by focusing on the data structures that are produced. Both MOPs start with a program AST annotated with metaobjects: edit metaobjects (EMOs) for the ETMOP, and compile metaobjects (CMOs) for the CTMOP.2 . These EMOs and CMOs are responsible for driving the display and semantic extensions. The relationship between AST nodes and metaobjects is n to 1. In a typical program, most AST nodes have default EMOs and CMOs. A smaller number of AST nodes have a custom EMO and/or CMO. This structure is the program + metadata structure of the architecture described in Section 2.2. To handle the common cases where we want a single AST node to have multiple metaobjects (e.g., both a Getter and a Setter), we provide a collection of composite metaobjects, which are discussed in more detail in Sections 4.2.2 and 5.1. As shown in Figure 4.1, the n to 1 sharing of EMOs allows the vertical bar to cover multiple fields. Composite EMOs allow, for example, having both a getter and setter on the same field declaration.
4.1.1
ETMOP
The implementation of the ETMOP editor is described below, which is described as a composable presentation editor. Model The model is made of two sub-structures: AST + EMOs, which form the program+metadata (Section 2.2) and are described above, and boxes, which are the display elements that form the logical layout (Section 2.3. 1
See Chapter 7 for a discussion of syntactic extension, an alternative to presentation extension. 2 We defer the discussion of how MOs are created and stored to Section 4.2.1.
63
The logical layout boxes are produced as a result of the ETMOP running a code walker over the AST + EMOs structure. The code walker asks each EMO to produce a box for the AST nodes to which it is attached. The default EMO, which is attached to the majority of AST nodes, makes a recursive call, and gathers the boxes of child nodes into the canonical layout of the program text. Specialized EMOs may change the canonical layout by using one or more presentation techniques. Different kinds of boxes produce different kinds of containment, which an EMO can use to assemble customized layout. Text boxes lay out a string of characters. Code boxes know how to lay out source code. Row and column boxes lay out structure horizontally and vertically. And graphics boxes display graphics, though the actual drawing of the graphics is configurable and deferred until the view is created. A simple example of a graphics box is the vertical line box, which produces the lines in Figure 4.1. Boxes are configured at construction time: setting their contents, how they respond to events, how they serialize themselves back to the persistent store, and, for graphics boxes, the graphics they produce. Their contents may change in response to user generated commands. View The view, or the physical layout (Section 2.4), is a hierarchical set of figures drawn on the screen. Figures are the render elements described in Section 2.4 of the physical layout. The figure hierarchy is almost identical to the box hierarchy with the exception that graphics boxes are able to draw any number or kind of render elements, which have no counterpart in the logical layout. Controllers Controllers react to user generated keyboard and mouse events by creating commands that affect the state of the editor. There is a 1 to 1 correspondence between boxes and controllers. Controllers are created by the boxes at the same time as the figures. The controller hierarchy is a standard part of the MVC architecture that handles updates to the model after a user interacts with the presentation. Commands created by the controllers affect editor state by changing some part of the model and triggering a refresh of the ETMOP view. As described in Section 2.5, there are three possibilities. A command can perform an external action, update the box structure only, or can trigger the serialize protocol and make global updates to the program. An event processing sub-protocol is available to assist the implementa64
tion of some kinds of temporal referencing (Section 3.4). Individual boxes can choose to emit typed events in response to user input. Other boxes can listen for specific events and react appropriately. In this way, it is possible to effect global changes to the program in a more principled way than by using the serialize protocol. For example, the event processing sub-protocol can be used to implement a status line that changes as the programmer types. Traceability The ETMOP maintains a simple correspondence between the boxes and the program text. Boxes know their offset into the program text, and there is a reversable mapping from any region of the display back to the text.
4.1.2
CTMOP Architecture
The CTMOP implements the static and dynamic semantics of the program through a process similar to expansion of syntactic macros [63]. It uses data structures and concepts similar to the ETMOP. A code walker traverses the AST, asking each AST node’s CMO to produce the expanded AST for each node it is attached to. The default CMO continues the walk recursively to follow a deep-copy like process. Customized CMOs can return an alternate expansion. This protocol is similar to both the mechanism and usage patterns of Common Lisp’s defmacro. Most customized CMOs do their work by picking out relatively high-level sub-parts of the AST nodes they expand, and then call the walk recursively to allow nested CMOs to do what they would like. But, a CMO can customize or entirely override the recursive walk if required. This allows a CMO to give context sensitive behavior to nested AST nodes with default or special CMOs. In contrast to typical syntactic macro systems the protocol allows CMOs to not only expand the current node, but also create additional top-level or sibling nodes if allowed by the abstract syntax. For example, this allows a CMO associated with a method declaration to not only customize that method, but also produce additional method declarations. Similarly, a CMO associated with a method parameter can add additional parameters. Once the AST is expanded, a standard compiler is called to complete the compilation. Traceability of warnings and errors To facilitate traceability of warning and error messages, the CTMOP provides several kinds of built-in behaviors. AST nodes that are simply copied or relocated in the expanded code retain their original source locators. Syn-
65
thetic AST nodes are assigned a source locator by the CMO that created them. By default, this is the source locator that is associated with the CMO itself when it is created, but the CMO can explicitly assign different source locators to synthetic nodes it creates when the default is not appropriate. After compilation, the CTMOP gathers all errors and warnings generated during expansion and compilation, and uses the source locators to translate an error in the expanded AST to an error in the original AST. Syntactic errors can also be detected at expansion time. For these kinds of errors, the CTMOP can produce messages meaningful to the programmer in the specific context of the extension involved. This is why in Figure 4.1 we see an error message that is specific to the precondition extension. Also, note that we do not see error messages for invocations of the getter methods (getX, etc.) even though they are not explicitly defined. Other kinds of errors including semantic errors like name binding and typing, are deferred to the Java compiler. For these kinds of errors, the messages produced by the compiler will be shown and will not be specific to the extension that caused the error. The traceability of source locators of the error allows the errors to be integrated into the extension as shown in Figure 4.1.
4.2
Implementation Specifics
This section describes some of the rationale behind the design choices made during the construction of the ETMOP and CTMOP. One of the key properties of our approach is its compatibility with existing source repositories, tools, and semantic extension mechanisms. Our implementation is Java based: Java is the source, target, and implementation language. We have implemented the two MOPs as Eclipse plugins. This section focuses on the ETMOP, and mentions the CTMOP only when needed for a comparison.
4.2.1
Persistent store
The architecture is neutral with respect to the form and concrete syntax of the persistent store. In our implementation, we use JSR 175 [9] annotated Java code. Metaobjects are serialized to annotations. The architecture does not prevent the possibility of storing this metadata in other ways, as XML, in comment strings, or as some kind of binary file, but we have chosen annotations because they are programmer, tool, and code-base compatible. In particular: • Annotations are already familiar among developers. • Annotations work with existing tools in modern IDEs. • They are valid Java syntax. 66
• They are attached to source code. • They can be edited by hand. • The Java compiler checks for syntax and some type correctness of annotations. To support extensions that want to cluster sibling AST nodes, we use two special annotations, @Previous, which designates that an AST node shares the metaobject of its previous sibling, and @Parent, which designates that an AST node shares the metaobject of its parent. By relying on JSR 175 annotations, we restrict ourselves somewhat: annotations can only be attached to declaration nodes of an AST, annotations are not subclassable, and declarations cannot have multiple annotations of the same type unless they are in an array. To partially address these limitations, metaobjects can also be created based on simple structural patterns in the AST. We show an example of this in Section 5.4. EMOs are registered with the ETMOP by extending an Eclipse extension3 point that associates an annotation class with a EMO class, and there is a similar mechanism for CMOs.
4.2.2
Combining multiple metaobjects
Although each AST node has exactly one EMO and one CMO, there are situations where the effect of multiple extensions should be combined on a single AST node or cluster of them. For example, the field declaration uid in Figure 4.1 has a getter, a setter, and a guard. We provide a generic suite of composite EMOs and CMOs that can combine the effects of multiple metaobjects onto a single or a group of AST nodes. Extension writers can also provide their own composite metaobjects. These composite EMOs and CMOs are created implicitly during the EMO and CMO creation process, and do not need to be stored as annotations. Which composite metaobject to create during the EMO and CMO creation process is defined during EMO/CMO registration. An example of this is given in Section 5.1.
4.2.3
Hooks into Eclipse
Each ETMOP editor is connected to a hidden JDT editor. Through our traceability mechanism described in Sections 4.1.1 and 4.1.2, there is a reversible mapping between JDT editor locations (lines and offsets), ETMOP 3
The use of the term Eclipse extension here is an unfortunate overloading of the term extension. When used in this context, we will use the full term Eclipse extension to differentiate it from our own language extensions.
67
editor locations (boxes), and the expanded AST. Building on this, the ETMOP editor hooks into existing Eclipse infrastructure in the following ways: • Syntax highlighting is applied to the ETMOP editor by listening for syntax highlighting events on the hidden JDT editor, and applying them to the correct boxes (or portions of boxes). • Similarly, clicks on the overview ruler, outline view, class hierarchy view, call hierarchy view, etc., will open an ETMOP editor with the expected text highlighted. We do this by having the ETMOP editor listen for the proper events. • Content Assist works in the ETMOP editor. The content assist proposals include code from the expanded source. For example, if a field, x, has a getter extension applied to it, the getX() method is returned as one of the proposals, if the request warrants it. • Errors and warnings on files persist between sessions by linking them to Eclipse resources using standard APIs. • Breakpoints can be applied in the ETMOP Editor. JSR 454 is used as the basis for this feature. Line mapping data is added to the class file after the class file is generated. This data maps from expanded line number to original line number. It is enough information for the debugger to properly display breakpoints in the original source code and pause program execution when the runtime hits the mapped line number in the expanded source code. In this way, breakpoints are applied at a particular display element, mapped to a program element, mapped to a line number in the original code, and then mapped to a line number in the expanded code. This expanded line number is used by the compiler and debugger. Our strategy to integrate new features into the ETMOP editor and CTMOP preprocessor leverages the traceability between a program’s persistent store, display, and expanded code. There are two general steps to integrating a new feature: 1) determine which Eclipse platform events to listen to and how they apply to the persistent store, and 2) determine how these events should affect the display or expanded source. Not all features from the JDT editor have been integrated into the ETMOP. Our goal with the ETMOP is to show that such integration is possible, even if the implementation is not complete. One such feature is text compare of the standard JDT editor. This feature compares two versions of the same file (or two different files) in a split screen, using graphics and colors 4 JSR 45 adds debugging support for other languages http://jcp.org/en/jsr/detail? id=45.
68
to emphasize differences. Such a feature would be possible to construct in the ETMOP, but there are three possibilities for its implementation. Should the comparison operate on text, abstract syntax, or display elements? This is an open question that we have not addressed in this work.
4.2.4
Other semantic processors
Using a different semantic processor is possible, either in conjunction with the CTMOP or separately. Our current implementation of the CTMOP is a source-to-source translation, so a requirement is that if the CTMOP is to be used, it must receive source code with the proper annotations applied to it. Taking into account this requirement, other processors may be used before or after the CTMOP is run. Section 5.5 shows an example where we use this approach. The default behavior of the CTMOP is for annotations to be carried over to the expanded code, where a built-in Java mechanism can control whether they are carried over to the class file and runtime [9].
4.2.5
Drawing framework
We use the Graphical Editing Framework (GEF)5 to build the figures (Section 4.1.1) and controllers (Section 4.1.1) of the editor. In addition to providing an editor framework, GEF also allows us to hook into built-in undo/redo, copy/paste, selection, highlighting, and cursor movement functionality.
4.2.6
Extensions palette
We provide a drag and drop palette from which a programmer can add instances of language extensions to a program, Figure 4.3. An extension can only be dropped onto sections of the program where it is syntactically valid to have this extension. Shriram et al [60] describe a common maintenance problem with designing tools that apply to recursive data definitions. The extensions palette side steps this issue. First we describe the problem and then we show how the extensions palette side steps it. The problem described in [60] occurs when there are recursively defined data definitions (such as the abstract syntax of a programming language) and a set of tools that operate on the data. If the system is designed using a standard object-oriented approach, it tends to be easy to add new data types, but hard to add new tools. If the system is designed using a functional approach or visitor pattern, then it tends to be easy to define new tools, but difficult to define new data types. 5
http://www.eclipse.org/gef/
69
Figure 4.3: Palette from which extensions can be dragged and dropped into appropriate locations. The extensions palette side steps this issue because the abstract syntax of the underlying language is fixed. Data type extensibility is therefore prevented by design. Because tools are defined using a visitor pattern, the extensions palette is operationally extensible. There may be other maintenance problems with extensions and the extensions palette that we have not discovered, but this issue is unlikely to be a problem.
70
Chapter 5
ETMOP Examples In this chapter, we walk through a number of examples that use the ETMOP and CTMOP to show that presentation extensibility can be used to make programs more expressive. These examples show the breadth of extensions that are possible using the ETMOP.
5.1
Setter
Getter and setter extensions have already been mentioned in several locations. These extensions use multiple presentation techniques to provide visual cues regarding the existence of accessor methods on fields. This extension uses the vertical bar as a presentation idiom to imply that the code to the right of the bar applies to the code to the left of the bar. We will see variations of this idiom below. Here, we describe the setter extension. The getter extension is analogous. The complete setter extension consists of four parts: 1 The @Setter annotation: public @interface Setter { String value (); }
2 The SetterEMO class (Figure 5.1) 3 The SetterCMO class (Section 5.2) 4 Registration of the EMO and CMO. This links the disparate parts of their definition to their annotation and describes how they can be used in the IDE. Additionally, the registration process defines how the extension can combine with others and what composite EMOs and CMOs are created to handle this composition. Complete code for the SetterEMO is in Figure 5.1. Following the line numbers in the Figure, we describe the code: 71
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
public class SetterEMO extends AbstractEMO implements I C o m b i n a b l e R i g h t { private final Annotation annotation ; public SetterEMO ( Annotation annotation , List < ASTNode > asts ) { this . annotation = annotation ; } private Expression prefix () { return annotation . getExpression (); } private IC onfigura tion p r e f i x C o n f i g u r a t i o n () { return new U p d a t e A n n o t a t i o n C o n f i g u r a t i o n ( annotation ); } private IC onfigura tion n o t e B o x C o n f i g u r a t i o n () { return new R i g h t N o t e B o x C o n f i g u r a t i o n (); } @Override public Box logicalLayout ( L o g i c a l L a y o u t V i s i t o r v , List < ASTNode > passedInASTs ) { return makeRow ( super . logicalLayout (v , passedInASTs ) , makeVLine () , l o g i c a l L a y o u t N o t e B o x (v , passedInASTs )); } public Box l o g i c a l L a y o u t N o t e B o x ( L o g i c a l L a y o u t V i s i t o r v , List < ASTNode > asts ) { return m ak e Ce nt e re dR ow ( n o t e B o x C o n f i g u r a t i o n () , emph asizeBla ck ( m a k e R e a d O n l y T e x t B o x ( " Setter : " )) , makePrefixBox (v , asts )); } private Box makePrefixBox ( L o g i c a l L a y o u t V i s i t o r v , List < ASTNode > asts ) { return emphasizeBlue ( makeJavaBox ( p r e f i x C o n f i g u r a t i o n () , prefix ())); } }
Figure 5.1: Complete code for SetterEMO Line 1 SetterEMO implements ICombinableRight. This interface states that an instance of this EMO can visually combine with other EMO instances that also implement this interface. Getter, Setter, and Invariant all implement ICombinableRight, which is what allows them to appear together at the right of a vertical line. Line 4 The EMO constructor protocol is that the constructor receives the annotation as well as the list of AST nodes that share the metadata. Some EMOs save both values, but in this case the setter EMO only needs to store the annotation, not the AST nodes. Line 12 The prefixConfiguration and noteBoxConfiguration methods 72
configure the boxes’ controllers. In particular, the ones used here are standard ETMOP library configurations that add extra commands to the controller. UpdateAnnotationConfiguration ensures that anything typed into the figure it is attached to automatically updates the annotation accordingly. RightNoteBoxConfiguration ensures that other ICombinableRight EMOs can be added and removed via the context menu. The UpdateAnnotationConfiguration class additionally ensures that the contents of the prefix box is always a valid Java identifier. This use of temporal referencing and constrained editing is illustrated in Section 3.6.2. Line 20 The logicalLayout method implements a simple layout in which the field declarations go on the left, then a vertical line, and then the note box for this EMO. This method is is part of the box construction protocol, and all EMOs must implement it to customize the layout. As the code walker constructs the boxes, it recursively calls the logicalLayout for every EMO that is directly attached to AST nodes. Line 27 The logicalLayoutNoteBox method does the part of the layout that corresponds to the getter/setter behavior of the EMO, which consists of the text “Setter:”, followed by the prefix in bold, blue letters. This particular method is part of the setter extension’s combining protocol. All EMOs whose extensions combine on the right side of a declaration (e.g., invariants, getters, and setters) must implement ICombinableRight and implement this method, which ensures such EMOs play nicely together. We have described how presentation extensibility is achieved for the setter extension using only a modest amount of code. The semantic extensibility for setters is described next.
5.2
Quasiquote
Recall that semantic extension is the ability to alter the computational meaning of programs through an open compiler or interpreter. Most semantic extension mechanisms have some sort of template-based generation facility such as source-level code generation in Dylan [3], mayans in Maya [6], and define-syntax in Scheme [56]. Our CTMOP has a similar facility:1 1
MOP annotations and inserted library code are fully qualified in the generated code by default. Typically, the programmer never sees the fully qualified names since ETMOP hides them. To make the underlying code easier to read in this dissertation, however, we
73
Figure 5.2: Screenshot of the implementation of the Setter CDMO
@Quasiquote () private M e t h o d D e c l a r a t i o n makeGetter ( String returnType , String varName ) { return makeMethod ( makeModifiers ( PU BLIC_KEY WORD ) , makeType ( returnType ) , makeName ( makeGet terName ( varName )) , EMPTY_PARAMETER_LIST , makeBlock ( m a k e R e t u r n S t a t e m e n t ( varName ))); }
The raw form of the code that generates the setter method is moderately expressive, but it still takes more work than we would like for the programmer to see just what will be generated. This is typical of the nonLisp languages which do not easily support an elegant quasiquote syntax. To improve expressiveness of CTMOP code, we have defined a presentation extension for our Quasiquote facility. With that extension in place the previous code is more compact and expressive as shown in Figure 5.2. The contents of the dashed box describe the generated code. The code in the inner solid boxes is quasiquoted (evaluated at code generation time), and the rest is generated as-is. Thus, the screenshot says that the makeSetter method returns a method declaration, using the variables typeName and varName, and by making a call to makeSetterName. Syntax highlighting is also provided by this extension. Programmers can type directly into the quasiquote box. On serialization, this box is parsed and converted to the raw form shown above. Unquote boxes can be added via the context menu. Only EMOs are necessary to implement this extension, no CMOs. The Java code in the persistent store has the full semantics. The EMO provides a library of abstract syntax generators through which code is generated at compile time. The EMO uses this same library to generate the display. The EMO converts library calls to create abstract syntax into the appropriate, visually simpler quasiquote display. have removed all name qualifications in this and the following examples.
74
Figure 5.3: Screenshot of the uses extension
5.3
Uses
Some objects, such as file readers, have complex initialization behavior involving construction inside a try/catch block, and corresponding cleanup. In Java syntax, this behavior requires a significant amount of code. Figure 5.3 shows the use of a simple extension designed to clean up this kind of code. The uses extension declares that the method uses an instance of a specified type—the extension has knowledge about the initialization and cleanup behavior for a set of types. This code says that the getFileContents method uses the InputStream called file. The stream is instantiated and closed, with simple error handling. The extension provides what looks like a new keyword, uses, but the underlying grammar does not have to change to accommodate this. The expanded code for this example is below: @Uses ( type = Buff eredRead er . class , name = " file " , init = " new B ufferedR eader ( new FileReader ( fName )) " ) public String g et Fi l eC on te n ts ( String fName ) { Buffe redRead er file = null ; try { file = new B ufferedR eader ( new FileReader ( fName )); { StringBuffer sb = new StringBuffer (); String l ; while (( l = file . readLine ()) != null ) { sb . append ( l + " \ n " ); } return sb . toString (); } } catch ( java . io . IOException e ) { throw new R u n t i m e E x c e p t i o n ( e ); } finally { try { file . close (); } catch ( Throwable t ) {
75
Figure 5.4: Alternative presentation for the uses extension
Figure 5.5: Screenshot of the equation extension
} } }
This kind of extension is application or domain specific in that the CMO only has knowledge of how to expand a limited set of types (although an extension implementor could certainly add to this set). The uses presentation extension was created early in the development of the ETMOP and CTMOP. Since its creation, our opinion of this extension has changed. This extension uses textual manipulation, combined with constrained editing (i.e., the label uses is read-only) to construct the presentation. However, there are no visual cues that part of the presentation is constrained. The programmer can only determine this through interaction. A more appropriate presentation for the uses extension would be use a top-note box as described below in Section 5.7. This alternative is shown in the screen-shot of Figure 5.4. Although less visually appealing, this version uses a standard ETMOP visual idiom. This helps to identify uses as a read-only label, rather than as a keyword like public or private.
5.4
Equations
We have already introduced equations in Chapter 3. Here we go into more detail on the implementation of this example. 76
Figure 5.5 shows the use of an extension that supports writing code that looks like the usual mathematical notation for the code. New equation parts can be added using the extensions palette (Section 4.2.6). And extensions can be configured to use integer or floating point arithmetic through context menus. The persistent store for this snippet of display is: public double mean ( final List < Integer > x ) { @Equation ( DOUBLE ) int MOP_FIELD$1 ; return Equ . div (1 , x . size ()) * new Summer () { public int loop ( int i ) { return x . get ( i ); } }. sum (0 , x . size () -1); }
The MOP FIELD$1 variable declaration is annotated with @Equation. This is field is a placeholder, and exists only so that an annotation can be attached to it. JSR 175 annotations may not be placed on arbitrary statements; this is a limitation of Java’s annotation system. We use this approach to avoid the limitation. The metaobject for this annotation is placed on the next statement. Through this technique, it is possible to store customized metadata on statements even though it is not syntactically possible to place annotations on Java statements. The @Equation annotation therefore signifies that there may be an equation in the statement succeeding the variable declaration that it is attached to. The DOUBLE means that the expanded code should use floating point arithmetic. During EMO creation, this annotation spawns an EquationEMO, which creates a new visitor that searches for code patterns matching parts of equations, and assigns EMOs to AST nodes based on the patterns. For example, the Equ.div(a, b) method invocation gets a DivisionEMO assigned to it, which then adds the horizontal bar in the appropriate location. The CTMOP creates CMOs similarly, assigning an EquationCMO based on the annotation and searching for patterns below the attached AST node. The CTMOP converts all invocations on methods in the Equ class to appropriately use integer or floating point arithmetic based on the specification in the annotation. Also, the expansion makes optimizing changes to the code by converting method calls into operators. The Equ.div(a,b) method invocation above will, as expected, be converted into a/b.2 In the expanded code for Figure 5.5 all locations where there may be integer references have 2 The use of Equ.div(a,b) distinguishes the use of division when the display should be altered from when it should not.
77
been explicitly cast to double: public double mean ( final List < Integer > x ) { @Equation ( DOUBLE ) int MOP_FIELD$1 ; return ( double ) (( double ) 1) / (( double ) x . size ()) * new SummerD () { public double loop ( int i ) { return x . get ( i ); } }. sum (0 , x . size () -1); }
This example of equation presentation is still rough. A full example would require a deeper study of the proper typesetting of equations. A possibly good typesetting style can be found in TEX and LATEX [61], where the above example would be formatted as follows: 1 return x.size()
x.size()−1
X
x.get(i) ;
i=0
There are several differences between the LATEX and ETMOP displays: Baseline LATEX ensures that each line of the equation is properly aligned around a baseline that is roughly the middle of the row (i.e., the vertical alignment is centered). This is not the case in the ETMOP version where vertical alignment is more ad hoc (e.g., the return keyword is vertically aligned with the denominator x.size(). This is a fault with the ETMOP version and would need to be addressed in a completed implementation. Font style LATEX uses an italicized, variable-width font, whereas the ETMOP uses a fixed width font. The former is aligned with standard mathematical texts, but the latter is more aligned to standard Java. Multiple fonts in the display (i.e., mathematical and standard Java fonts) might be more confusing compared to using a single, fixed width font for all. This is an open question. Font size LATEX uses variable sizes of text for different parts of the summation. We had experimented earlier with this in the ETMOP, but found that the smaller font sizes are less readable on the screen. Using a variable width font might mitigate this issue. Spacing The ETMOP version uses regular spacing (e.g., the space between the return keyword and the fraction is the same space as between the public and double keywords, which is the same space as between the * and surrounding equation parts). This is consistent with using a 78
Figure 5.6: Screenshot of the AspectJ extension fixed width font. LATEX, on the other hand uses variable spaces where the space between return and the fraction is different than the space between the fraction and the summation. The variable spacing provides a more visually appealing display. This is the same trade-off as with font style. Is it more effective to use a standard Java display or standard mathematical display? This is an open question as well. Multiplication LATEX uses an implicit multiplication by placing equation parts next to each other, whereas the ETMOP explicitly uses the Java multiplication symbol, “*”. This is yet another choice between remaining consistent with Java or becoming more consistent with standard mathematical notation. It is another open question. As the above shows, in order to properly implement equation presentation extensions, there is some engineering work to do (ensuring the use of proper baselines, etc.), and there is also more experimental work (consistency with a Java display vs. consistency with a mathematical display).
5.5
Code Style vs. Annotation Style AspectJ
Another use of our approach is to display annotation-style AspectJ as codestyle AspectJ. A programmer can view and edit standard code-style AspectJ as it appears in Figure 5.6, and this can be combined with other extensions, such as an equation. This code is stored as annotation style in a text file, as shown below: @After ( value = " changes ( p ) " ) @Equation ( FLOAT ) int MOP_FIELD$1 ; public void myAdvice ( Point p ){ Screen . l o g D i s t a n c e F r o m O r i g i n ( Equ . root ( Equ . power ( p . getX () , 2) + Equ . power ( p . getY () , 2) , 2)); } @Pointcut ( " this ( p ) && execution ( void Point . set *( int )) " ) void changes ( Point p ){}
We define one EMO metaclass for each annotation type used in annotationstyle AspectJ. These EMOs all work in a similar way: 79
Figure 5.7: Stream Processing Automaton that accepts c(ad)*r 1 During box creation, walk the annotation-style AST, creating boxes that look like code-style AspectJ. 2 Allow the editing of the code-style form as text. 3 When serializing back to text: • read the display by walking the box structure • create a code-style AspectJ AST • transform to an annotation-style AST, which also happens to be syntactically correct Java, which becomes the persistent store. Figure 3.8 shows how different semantic processors can be used together in our architecture. Here, the standard CTMOP executes, creating the expanded code for the myAdvice method, which includes expanding the contained equation. This code is then sent to the AspectJ compiler, which uses its own semantic processor.
5.6
Stream Processing Automaton
We have also implemented a stream processing automaton extension. This example shows how our tool can support more complex visualizations. Figure 5.7 shows a simple automaton that can accept or reject the regular 80
expression c(ad)*r. This example and its semantics are similar to that described in [59]. The presentation of the automaton is a simplified version of UML’s state chart.3 The automaton can be called in the following manner: try { // result is the list of characters accepted by // the input stream . List < Character > result = new A c c e p t C a d r S t r i n g (). accept ( inputStream ); } catch ( I n v a l i d T r a n s i t i o n E x c e p t i o n e ) { ... } catch ( U n e x p e c t e d S t r e a m E n d E x c e p t i o n e ) { ... }
Each element of the input stream transitions the machine to a new state. To determine which state to go to, all possible transitions are examined. Each transition has a label, which contains the transition event (an expression) and an optional guard (a method that returns a boolean) in brackets. The next value is taken from the stream and is compared to all transitions from the current state. If the value and the transition event are equal, then the guard method is executed and if this returns true, then the transition is taken, the current state is changed and the process continues. The automaton succeeds if the stream ends when the automaton is in an end state. The automaton fails and throws an exception if there are no transitions that can be taken or if the stream ends when the automaton is not in an end state. This is our most ambitious extension in terms of enhancements to presentation and compilation. Our intention with this example is to strike a balance between making the visual form powerful, but at the same time intuitive.
5.7
Note Boxes
A number of extensions previously introduced all use the same visual idiom where spatial arrangement and graphical enhancements are used to associate a presentation extension with a part of the program. We call this visual idiom a note box. The JDBC, getter, setter, and invariant extensions use right note boxes to present themselves; they appear to the right of a vertical bar, which is itself to the right of a set of field declarations. The precondition and postcondition extensions introduced in Chapter 1 use top note boxes to present themselves; they appear to the right of a vertical bar and on top of a method declaration. 3
The UML specification can be found at http://www.uml.org/.
81
(a)
(b) public class Ship { @Getter ( " get " ) @Setter ( " set " ) private int x ; @Previous () private int y ; @Previous () @Depth (1) @Invariant ( " $$ >= 0 " ) private int uid ; @Previous () @Depth (2) @Invariant ( " $$ < 100) private int groupid ; @Previous () private String name ;
Figure 5.8: Nesting of right note boxes: the presentation (a) and the persistent store (b) A field declaration typically consumes little screen real estate and this affords intricate sharing between extensions. Figure 5.8 shows an example of this; part (a) shows the presentation and part (b) shows the persistent store. Here, all of the field declarations have getters and setters with standard names. The uid and groupid fields must be non-negative, and the groupid field must additionally be less than one hundred. The persistent store uses the @Previous annotation to declare that a field shares the metaobject of its previous sibling. It uses the @Depth annotation to declare the nesting depth. Any additional extensions are stored as annotations after the @Depth. The following describes the algorithm used by the CTMOP to compile right note boxes: • Nested extensions are processed first. • Each extension at the same depth is processed in order. • Each extension is passed all of the declarations that it is attached to 82
(a)
(b) @Precondition ( " Ocean . inBounds (x , y ) " ) @Post conditi on ( " this . getX () == x && this . getY () == y " ) public void move ( int x , int y ) { this . x = x ; this . y = y ; }
Figure 5.9: Combining top note boxes: the presentation (a) and the persistent store (b) and may process them in any way it likes. • Subsequent extensions are passed the expanded form of previous extensions. Method declarations are typically both horizontally and vertically larger than field declarations, making this same kind of sharing and nesting of extensions on method declarations visually awkward. Extensions on methods may be combined as shown in Figure 5.9, but they may not be shared between method declarations (as right note boxes can for field declarations). Here the method has both a precondition and a postcondition. Figure 5.9a shows the presentation and how the two extensions are combined under the same presentation. Figure 5.9b shows the persistent store. When extensions are combined and shared, there is danger that their effects (both visual and semantic) conflict with each other. Although considerable leeway is given to each extension to process declarations in any way, there are two heuristics that can be applied to help ensure that extensions work together. 1 Extensions may add synthetic declarations (fields, methods, classes, or aspects) as siblings when processing, but these should not be accessible to subsequent extensions. 2 Expanded code should be strictly additive to ensure that the code needed by subsequent extensions still exists. Even if followed properly, these two heuristics are not sufficient to ensure that all note box extensions will work together in any order. However, they do provide some assurances and predictability of behavior. 83
Figure 5.10: Webservice presentation extension.
Figure 5.11: Positive affordance for adding a webservice component.
5.8
Webservices
The Webservices presentation extension has already been introduced in Chapter 3 and is revisited here with some new details. In particular, we describe how the extensions palette operates. The webservices presentation extension uses graphical abstraction to define a Java class that implements a webservice. This is shown in Figure 5.10. Webservices can be added using the extensions palette (described in Section 4.2.6). The programmer drags and drops the desired webservice component to the desired location in the editor. The editor provides feedback as to whether or not a particular location is 84
Figure 5.12: Negative affordance for adding a webservice component.
Figure 5.13: Specifying parameter names for webservices. capable of accepting the webservice component. For example, when a webmethod is to be added to the program and the mouse is moved over a class declaration, the editor shows negative affordance as shown in Figure 5.12. When the mouse is moved over a method declaration, positive affordance is shown, which implies that a mouse click will add the webmethod declaration to the Java method as shown in Figure 5.11. From the program as presentation view, the programmer can determine the acceptability of placing a webservice component at a particular location via feedback from the mouse. From the program as datastructure view, this is implemented by checking the AST associated with a display element as the mouse hovers. If that display element or any of its parents accept the component, then positive 85
Figure 5.14: Declaring a stored procedure on an interface method declaration affordance is shown. Otherwise, negative affordance is shown. Additionally, parameter names for webservices can be specified using spatial arrangement above the Java parameter declaration as shown in Figure 5.13. The underlying Java code for this is: @WebMethod ( operationName = " ech oSecureN ame " , action = " urn : Ech oSecureN ame " ) public String echoSec ureName ( @WebParam ( name = " greeting " ) String g , @WebParam ( name = " name " , header = true ) Name n ) { return g + " " + n . getFirst (). replaceAll ( " . " , " * " ) + " " + n . getLast (). replaceAll ( " . " , " * " ); }
Notice that there are three annotations: one for the webmethod declaration and one each for the two parameters. Also, notice that some of the webmethod annotation metadata is not in the presentation (operationName and operationValue). This information is abstracted away and hidden from view unless explicitly asked for.
5.9
JDBC
We have already discussed the JDBC example in detail in Chapter 3, but for completeness we revisit the example here. The JDBC presentation extension attaches a SQL prepared statement to an interface method declaration using JDBC 4.0. Figure 5.14 is the same screenshot as shown previously. To the right of the method declaration is the prepared statement and to the right of that are options and metadata on the prepared statement. Using constrained editing, the programmer can edit these options through a drop-down menu as shown in Figure 5.15. This extension uses the same visual metaphor as right note boxes. Display elements to the right of the vertical bar apply to display elements to the left of the bar.
86
Figure 5.15: Using a drop down menu to edit SQL query options
5.10
Life Cycles in jEdit
These next four examples are inspired by jEdit.4 Many frameworks require clients to call methods on its classes in a specific order. For example, Java Applets require the init method to be called before the start method, which must be called before the finish method, which must be called before the destroy method. This is called the applet life cycle.5 These kinds of application life cycles are standard in many frameworks that implement a plugin architecture, see Confluence,6 Eclipse [38], and Hibernate [8]. The life cycle that is addressed here is for the JEditPlugin class where users of this class must call the init, start, and finish methods in that order. This extension makes an implicit restriction on method invocation order explicit in both the presentation and in the execution. Figure 5.16 shows a simplified version of the JEditPlugin class that uses the life cycle presentation extension. the JEditPlugin class has three method stubs that must be called in the specified order. The base methods should not 4
jEdit is a cross platform programmer’s text editor written in Java that has a plugin architecture. http://jedit.org 5 http://java.sun.com/applets/ 6 http://confluence.atlassian.com/display/DOC/Lifecycle+Plugins
87
Figure 5.16: Life cycle defined by method ordering be called directly, but rather they should be overridden in sub-classes and those methods should be called. The presentation clearly shows the order in which these methods should be called. At runtime, if the methods are called out of order, an exception is thrown. The stored form of this extension uses a single @LifecycleOrder annotation to start the life cycle and @Previous annotations for subsequent steps in the life cycle, as shown below: @ Li fe cy c le Or de r int init () { ... } @Previous int start () { ... } @Previous int finish () { ... }
Accordingly, all methods in the life cycle share the same life cycle metaobject and the order is dependent on the order that the methods are declared. The methods must be neighbors. The benefit of this implementation is that it is simple to maintain consistency with the life cycle as the class evolves. The programmer is given immediate feedback if an edit causes the life cycle order to change. The CTMOP provides the semantics behind the life cycle presentation extension. The expanded code adds a simple finite state machine that moves from an initial state to subsequent states as the next method in the life cycle is executed. This finite state machine is implemented through an aspect as shown in Figure 5.17 that updates state using before advice that is triggered before a method is executed. The perthis clause ensures that each instance of a subclass of JEditPlugin has its own instance of the aspect and state 88
aspect Ordering perthis ( execution (* JEditPlugin +.*(..)) { enum State { ORDERING_INIT , init , start , finish }; State currState = ORDERING_INIT ; before () : execution ( JEditPlugin +. init ()) { if ( currState == State . ORDERING_INIT ) { currState = State . init ; } else { throw new R u n t i m e E x c e p t i o n ( " Life cycle is off . Must invoke init () " + " before any other life cycle method . " ); } } // ... Two more before advice as above , but different states .... }
Figure 5.17: Expanded code for the presentation extension. An aspect is used to update a finite state machine that keeps track of the life cycle. machine.
5.11
Event handling in jEdit
The jEdit editor uses an event handling idiom where each class that fires events has a single list of listeners of various types. The type of the listener determines the event that it responds to. The name of the event is used to determine both the event method and the listener type—e.g., if the event name is MouseHover, then the event method is fireMouseHover and the listener type is XListenerMouseHover. The event method iterates through a list of listeners and when a listener of the specified type is found, a block of code is executed. This idiom is not used uniformly throughout the jEdit codebase, but it is used often enough to imply that the idiom is representative of the code base and that outliers may be implementation errors. We provide a presentation extension that enforces this idiom and explicitly demarcates which events are handled by a particular class. Figure 5.18 shows an instance of this extension. In this figure, the TextChange event is handled. The fireTextChange method is used as a handler, which can be invoked explicitly anywhere in the class. And for each TextChangeListener in the list of listeners, the method handleChange is invoked on it. The Event: line in the top note specifies the event that is handled. This part of the presentation is editable. The following two lines specify the event 89
Figure 5.18: Event handler for text changes method and the handler type, and are read-only. We use simple affordance to emphasize this: the Event: field is highlighted in blue and the following two lines are not. The Method: and Type: lines use temporal referencing to update themselves as a programmer edits the Event: line. The body of the event is specified below. The CTMOP provides semantics for this extension as we show in the code snippet below: private void fireTex tChange () { Object [] listeners = listenerList . ge tL i st en er L is t (); for ( int i = listeners . length - 2; i >= 0; i - -) { if ( listeners [ i ] instanceof T e x t C h a n g e L i s t e n e r ) { try { T e x t C h a n g e L i s t e n e r l =( T e x t C h a n g e L i s t e n e r ) listeners [ i ]; { l . handleChange (); } } catch ( Throwable t ) { Log . log ( Log . ERROR , this , t ); } } } }
The CTMOP fills in the boilerplate code. Note that the for loop goes from listeners.length - 2 down to 0; this is an implementation detail of how listener lists are implemented in jEdit. Also, note the simple error handling. By using the CTMOP to fill in the boilerplate code, we ensure that the event handling idiom is implemented correctly and consistently whenever this extension is used.
5.12
Message Handling in jEdit
jEdit uses an idiom to pass messages to different components. Messages are subclasses of the EBMessage class. To pass a message the send(EBMessage) method on the singleton EditBus class is invoked. All jEdit components (e.g., plugins, panels, and windows) must register with the EditBus and 90
Figure 5.19: Message handling for jEdit components must implement the EBComponent interface in order to receive messages. The EditBus passes each message to each registered component via that component’s EBComponent.handleMessage method. Each component handles messages of only certain types. The idiom states that each component’s handleMessage method should perform instanceof testing to determine if the EBMessage is of a type that is handled by the component. This code is clunky and error-prone. For readers of the code, it can be difficult to quickly assess which kinds of messages are handled by the component. We show a presentation extension that makes message handling explicit in the class’s declaration and facilitates the adding and removing of message handlers. Figure 5.19 is an example of this. The class declaration summarizes the message types handled by this class and the methods that are used for handling. The handlesMessage keyword specifies methods that perform the message handling. The single parameter of each of these methods must have a type that is a sub-type of EBMessage. The stored form of this class uses the @HandlesMessages annotation. Methods that perform message handling use the @Parent annotation, which indicates that the method declaration shares its metaobject with its parent type declaration: @HandlesMessages public class HelpViewer { @Parent () void p r o p e r t i e s M e s s a g e ( P r o p e r t i e s C h a n g e d m ) { ... } @Parent () void updateMessage ( PluginUpdate m ) { ... } ... }
The handlesMessage keyword is not really a keyword, but is added 91
to the editor through textual manipulation. If a programmer types handlesMessage before a method declaration, then that declaration is given the handles method metaobject (it is an error if the argument is not of the proper type).7 handlesMessage feels like a true keyword to programmers (even though it is not) there is no constrained editing in the body of the class. But, the message handling summary at the top of the class is read only. Each message is handled by at most one handler; and handlers are tested in order. The order of handlers is therefore important and they should be ordered from most to least specific handler type. The CTMOP provides semantics for this extension by adding the synthetic handlesMessage method: public void handleMessage ( EBMessage message ) { if ( message instanceof P r o p e r t i e s C h a n g e d ) { p r o p e r t i e s M e s s a g e (( P r o p e r t i e s C h a n g e d ) message ); } else if ( message instanceof PluginUpdate ) { updateMessage (( PluginUpdate ) message ); } }
This method adds the instanceof testing that determines if a message is handled by this class. The benefit of this extension is that handled messages are explicit in the class declaration and the handlesMessage idiom is facilitated by the extension.
5.13
Services in jEdit
jEdit uses various XML files to provide a generic way for plugins to define API extensions. The services.xml is one such file that defines how clients can implement and extend core jEdit functionality such as virtual file systems, code folding, and text encodings. The services.xml file exists at the root of the jar file of a jEdit plugin. The service description contains three parts: the unique service name, the fully qualified class name of the service that is implemented (one of a small number of pre-defined services, or clients can add new ones), and a BeanShell script8 that defines how the service is implemented (this script can be any BeanShell expression, but in practice the script is a no-argument constructor call on the service class). 7
Note that here making the handlesMessage text behave like a keyword is appropriate, while it is not appropriate to do so for the uses extension described in Section 5.3. The reason is that when a programmer types handlesMessage in the appropriate location, the editor knows from context how to serialize handlesMessage appropriately. 8 BeanShell http://www.beanshell.org is a dynamically compiled scripting language that is compiled to Java byte code and can create and manipulate Java objects. It is used throughout the jEdit implementation to dynamically construct operations.
92
Figure 5.20: Defining services inline using presentation extension in jEdit Rather than force programmers to declare services in an XML file that is separate from their implementations, presentation extension can be used to declare services inline with their implementations. Note that this is similar to the Webservice example, where structure is defined inline with a Java class and a processor extracts the structure to convert it to XML. Figure 5.20 shows a screenshot of how the FavoritesVFS class defines its corresponding service. The name and class are specified explicitly and the BeanShell script is generated from the class name. The resulting XML is this: < SERVICES > < SERVICE CLASS = " org . gjt . sp . jedit . io . VFS " NAME = " favorites " > new FavoritesVFS ();
5.14
XML
The XML extension allows a programmer to generate XML documents in Java using a presentation that looks like XML. This extension is similar in implementation and presentation to the Java quasiquote extension described above in Section 5.2. This example shows that the utility of using a quasiquote mechanism to construct datastructures is not limited to Java ASTs. Any hierarchically defined data can be presented in a similar way. Note the difference between the quasiquote mechanism and Embedded CAL—the value of a quasiquote box is a datastructure that corresponds to the contents of the box, whereas the value of an embedded editor box is the result of evaluating the contents of that editor box. In Figure 5.21, the large, dotted box defines an XML element. The smaller, solid boxes contained inside of the XML element are escapes to Java. If the type of the object contained in the escape box is an XML object, then that object is spliced in directly when generating the XML. Otherwise, the toString method is called on the object and it is spliced in as text. Note that DrScheme uses a similar approach to present X-expressions 93
Figure 5.21: Embedding XML using presentation.
Figure 5.22: Optional arguments (a representation of XML data using s-expressions) in an XML-like manner. DrScheme provides the ability to use XML boxes as a way to construct XML expressions. Similarly, Scheme boxes can be used to escape out of an XML context and enter Scheme code. XML and Scheme boxes are evaluated to x- and s-expressions by the DrScheme interpreter. There is a difference from our approach, however. XML and Scheme boxes are stored in a nonscheme format (ASCII-encoded binary). Thus, only interpreters that are DrScheme-aware are capable of running code that uses these boxes.
5.15
Optional Arguments with Default Values
Many common programming languages such as Lisp, OCaml, and Python allow a programmer to define optional arguments with default values to functions or methods. This is not a feature of the Java language, but using a form of presentation extension, it is possible to add the feature through textual manipulation. Here is a method with optional arguments presented in the ETMOP editor: In the persistent store, the default values are stored as annotations: @OptionalArguments public int dialogBox ( String message , @DefaultValue ( " null " ) Shell parent , @DefaultValue ( " false " ) boolean modal ) { return new MessageBox ( parent , ( modal ? SWT . A P P L I C A T I O N _ M O D A L : SWT . NONE ) | SWT . OK ). open (); }
94
Note the use of the @OptionalArguments annotation. This annotation is the stored form of an OptionalArgumentsEMO, signifying that the method may have one or more optional arguments. This metaobject exists to signify to the serialize protocol to parse the default value expression properly. Thus, a serializer that is specific to optional arguments is substituted for the default serializer that serializes text in the display as program text. Without this customized metaobject being present, the default serializer would parse the optional values as a syntax error. The CTMOP will process the annotations in the following way to produce overloaded methods with the arguments filled in (comments have been added for clarity): // Original @OptionalArguments public int dialogBox ( String message , @DefaultValue ( " null " ) Shell parent , @DefaultValue ( " false " ) boolean modal ) { return new MessageBox ( parent , ( modal ? SWT . A P P L I C A T I O N _ M O D A L : SWT . NONE ) | SWT . OK ). open (); } // Synthetic public int dialogBox ( String message ) { dialogBox ( message , null , false ); } // Synthetic public int dialogBox ( String message , Shell parent ) { dialogBox ( message , parent , false ); }
Note that the first synthetic method uses all the default values and each successive synthetic method adds one more optional argument. Default values are stored as strings. This allows any expression to be entered as a default value and accommodates syntactically invalid default arguments, which are caught by the Java compiler. There are two limitations with this extension. First, optional arguments must come at the end of the parameter list. Second, optional arguments must be filled in from left to right. For example, if arguments two and three are optional, then if argument three is specified, then argument two must be as well.
5.16
Slider, Toggle Button, and Drop Down
It is possible to add some standard widgets such as sliders, check boxes, and dropdown menus to a program that control the values of constants. This class of presentation extensions might be useful for testing, when a 95
Figure 5.23: Using sliders, a checkbox, and a dropdown list in a program. programmer may be tweaking the inputs of test cases. For example, the following defines a simple robot that has specified initial values. When the slider moves, the value of the integer field changes through the use of temporal referencing. Similarly, clicking the check box or selecting from the dropdown will alter the values of the fields they are attached to; this is an instance of constrained editing. The stored form of this program is: public class SimpleRobot { @IntSlider ( min =0 , max =100) int power = 34; @IntSlider ( min =0 , max =100) int speed = 33; @IntSlider ( min =0 , max =100) int shield = 33; @ B o o l e a n C h e c k b o x () boolean isSelfAware = false ; @ St ri ng D ro pD ow n ({ " Sad " ," Happy " ," Confused " , " Downright Pissed " ," Ecstatic " }) String mood = " Happy " ; }
The CTMOP is not required for these kinds of presentation extensions.
5.17
Sum Constraints on Constants
This presentation defines a constraint on a series of integer constant declarations. The constraint states that the sum of the constants must equal 100. 96
Figure 5.24: Sum of constants must equal 100.
Figure 5.25: Temporal referencing allows error feedback to appear while typing. When the constraint holds, there is no feedback, as shown in Figure 5.24. Temporal referencing is used to notify the programmer while typing if the constraint is broken, as shown in Figure 5.25. The constraint is defined using the following snippet: @SumC onstrai nt (100 , public final static @Previous public final static @Previous public final static
" power , speed , and shield must sum to 100 " ) int power = 90; int speed = 9; int shield = 1;
This constraint is specific to numeric constants, but alternative versions could be more general. This implementation of constraints works at edit time. However, it would be possible to implement constraints so that they can be checked both at runtime and edit time. This would eliminate the need for constraints to be placed on constants only.
5.18
Documentation
An external tool is typically used to render JavaDoc comments as HTML. The inlined version of Doc comments, however, are plain text and are often obscured by markup. Using presentation extensions, it is possible to present comments inline in a way that is similar to rendered JavaDoc comments. Figure 5.26 shows a Java class with its documentation rendered inline. This presentation extension is similar to JavaDoc and allows links and bold, 97
Figure 5.26: Inline HTML formatting of comments italics, and monospace font. Documentation text is shown using a variable width font because it is easier to read prose using variable width as opposed to fixed width [70]. Additionally, variable width makes the documentation feel more like reading a webpage. A WYSIWYG editor for creating this documentation could be implemented using the ETMOP, but we have not done so. Instead the text must be entered using some simple markup. The preceding documentation is stored using the following annotation: @Docu mentati on { doc = " [ GenericOrder ] is the abstract " + " base class for [ Orders ]. Clients should not subclass " + " this class . Instead , use one of the supplied " + " [ Order ] subclasses : [ OnlineOrder ] , [ MailOrder ] , or " + " [ InstoreOrder ].\\ n * Note *: clients should / not / " + " = execute ()= an order unless it has already been " + " = validated ()=. " , see ={ " Order " , " OnlineOrder " , " MailOrder " , InstoreOrder " }}
5.19
Architectural Constraints
Frameworks such as Eclipse place architectural constraints on how classes within the framework can be used. For example, the phrases “This class is not intended to be implemented by clients” and “Clients may implement this interface” or variations of them appear frequently in the JavaDoc of Eclipse framework classes. For example, IJavaElement and its sub-interfaces use the former statement in their JavaDoc comments, and the JavaDoc for IAdaptable uses the latter. 98
Figure 5.27: Architectural constraints Rather than enter this text by hand, an architectural constraint presentation extension can be used. Figure 5.27 shows a snippet of a class declaration that has three constraints placed on it. The exact wording of the constraint is defined in the constraint definition and the constraint itself is applied to the class. This presentation extension has the following benefits: • All text is uniform and can be edited in one place, with the effect that all instances of its use are changed consistently. This consistency can make documentation more readable and ensure that deprecated wordings of the constraint are not used. • These architectural constraints are easier to search for because they are marked by an annotation. • It is possible to apply static or dynamic analysis to ensure conformance to the constraint. • It is possible to categorize classes and interfaces by the constraints that they use. This would help to ensure consistency over the program and can help find classes and interfaces with missing or invalid constraints. 99
Architectural constraints are defined using the following annotation: public @interface A r c h i t e c t u r a l C o n s t r a i n t { Const raintTy pe [] value (); public static enum Cons traintT ype { N O _ I M P L E M E N T _ O U T S I D E _ F R A M E W O R K ( " This interface is not intended " + " to be implemented by clients . " ) , N O _ S U B C L A S S _ O U T S I D E _ F R A M E W O R K ( " This class is not intended to " + " be subclassed by clients . " ) , N O _ U S E _ O U T S I D E _ F R A M E W O R K ( " This class is not intended to be " + " used outside of the framework . " ) , A L L _ C O N C R E T E _ M E T H O D S _ A R E _ F I N A L ( " The non - final concrete methods " + " of this class are not meant to " + " be overrided outside of the " + " framework " ) , F I E L D S _ A C C E S S E D _ D I R E C T L Y ( " Fields may be accessed directly " + " without accessor methods " ) , IMMUTABLE ( " Once instantiated , objects of this class are " + " immutable . " );
Cons traintTy pe ( String javaDocText ) { this . javaDocText = javaDocText ; } private String javaDocText ; public String getJav aDocText () { return javaDocText ; } } }
Although the presentation in Figure 5.27 for architectural constraints is clear, it is also awkward. The constraints and the JavaDoc comment do not mix well visually since they are different fonts and use different visual idioms—the constraints use the vertical bar and the JavaDoc uses a * at the start of each line. An alternative approach is to combine the two into a single presentation extension as we do in Figure 5.28. The two metaobjects can combine and be displayed together.
5.20
Hovers on Variables to Show References
As shown earlier in Figure 3.11, DrScheme supports a check syntax mode, where hovering over an identifier will show all references of that identifier in the current file. It is possible to mimic this behavior using presentation extensions. A screenshot of this functionality is in Figure 5.29. This functionality is an alternative to Eclipse’s approach to highlighting all identifiers of the same name as the insertion caret is placed on an identifier. We implemented this example to show that such behavior is possible using the ETMOP. In most cases, however, we believe that Eclipse’s stan100
Figure 5.28: JavaDoc formatting combined with architectural constraints
Figure 5.29: Using mouse hovers to identify all references of an identifier. dard behavior is more useful because it is a subtler approach to showing all references.
5.21
Alternate Method Names
In Scheme, the functions first and car are synonyms; they both retrieve the first element of a pair. We can implement a succinct way of defining such synonyms using presentation extension in the ETMOP. Figure 5.30 defines a single method that retrieves the first element of a linked list of strings. 101
Figure 5.30: Alternate method names for a single method declaration It also defines several names for this function: first, head, top, and car. The stored form of the code defines a single method whose name is the first of the group and the rest are stored as part of an annotation: @Synonym ({ " first " , " top " , " car " }) public String head () { return store [0]; }
The CTMOP provides the semantics for this extension. For each synonym, the CTMOP creates a new method with the same formal parameters that calls the original method.
5.22
Formatting Floating Points
This presentation extension enables a numeric field to be formatted into strings in a specified way. Figure 5.31 shows how the growthRate and sharesOwned fields are formatted into strings. This extension uses the CTMOP to provide semantics. For each of the formatted fields, one method is added: formatGrowthRate and formatSharesOwned respectively. These methods return a properly formatted string representation of the numeric value of the field.
5.23
JUnit
The JUnit extensions are part of the top note box class of extensions. This extension allows programmers to add JUnit 4.0 test methods into classes as shown in Figure 5.32. An icon is used so that programmers can quickly determine which methods are test methods. Constrained editing is used to add or remove options for the tests. Using the JUnit icon, which is meaningful to regular Eclipse users, it is possible to clearly and succinctly characterize how JUnit tests are used in a class.
102
Figure 5.31: Formatting numeric fields
Figure 5.32: JUnit 4.0 presentation extensions
103
Figure 5.33: Icon used to ensure argument to a method is not null
Figure 5.34: Inline images. Double-clicking opens a file dialog.
5.24
Not-null
Using the not null extension, it is possible to assert that a parameter to a method is non-null, as shown in Figure 5.33. Graphical enhancements in the form of an icon is used to convey this information. The expanded version of the code contains an assertion statement as the first statement in the method. Note that the metaobject is on the parameter itself, rather than the method.
5.25
Inlined Images
When an image is defined, it is possible to show the image inline as we do in Figure 5.34. Double clicking on the image will open a file dialog box so that the image can be chosen in a standard way. Note that DrScheme also has a similar capability to embed images inline.
5.26
Image Embedding for Documentation
We first introduced this example in Chapter 3 and we show it again here for completeness. For documentation purposes, images can succinctly convey information that is difficult to using text only as shown in Figure 5.35. This presentation is similar to inlined graphics, but is used for documentation only. In this chapter, we have explored a number of domain specific and general purpose language extensions made possible through the ETMOP and CTMOP. They represent a non-exhaustive characterization of the kinds of extensions possible with the ETMOP.
104
Figure 5.35: Image embedding. Double-clicking opens a file dialog.
105
Chapter 6
Embedded CAL Programs that use multiple languages to solve different kinds of problems can be more expressive than programs that use only a single language [24, 42]. Language-oriented programming [86], meta-linguistic abstraction [1], modeldriven architecture [36], software factories [43], and intentional programming [74] are related techniques that all advocate problem solving through the use of little languages tailored to the particular problem domain. Recall the definition of expressiveness from Section 3.1: a program is expressive if its perceived meaning aligns with its intended meaning. Due to syntactic and semantic differences between languages, the language or languages chosen to construct a program can have an impact on the program’s expressiveness. This chapter introduces Embedded CAL1 , an editor-based mechanism for combining languages, where multiple languages exist in the same editor, but the stored form of the program is in the concrete syntax of a single, base language. From the program as presentation viewpoint, there is little separation between the languages, control flow between languages is explicit, and most glue code is hidden from the programmer. Embedded CAL applies presentation techniques (Chapter 3) and through the use of a composable presentation editor architecture (Chapter 2), allows a programmer to edit CAL2 code that is embedded in Java programs. Embedded CAL is a second implementation of a composable presentation 1
Embedded CAL is available for download: http://openquark.org CAL is a non-strict, statically typed, lazy functional language produced as an open source project by Business Objects. CAL has syntax similar to Haskell http://www. haskell.org/. The CAL compiler, libraries and tools are also available for download: http://openquark.org. Because the syntax and semantics of CAL are similar to Haskell, there is no discussion of how to use the language. 2
106
editor architecture. It is smaller in scope than the ETMOP. Embedded CAL is a closed editor and was created for a specific purpose as described next.
6.1
Background
CAL is a general purpose language, and was originally designed by the Business Objects Research Group (BORG) as a data modelling and manipulation language.3 One of the core functionalities of Business Objects’ software is to enable uniform access and manipulation of large amounts of data from heterogeneous sources. CAL was designed to make this kind of data modelling simple, efficient, and type safe. Since much of Business Objects’ software is written in Java, a requirement of CAL is that its byte code can run on a Java virtual machine. Typically, CAL is used to define a data universe4 and Java is used to handle state, UI, event handling, and communication with other components. Because of this style of programming, many programs written by BORG contain both CAL and Java. However, because the semantics between the two languages are so different, communication across the language boundary is complicated and can be error prone. Informal discussions with developers at BORG suggest that a significant portion of development time is spent writing and debugging glue code between Java and CAL. Furthermore, once the glue code has been written it can be difficult for new developers to determine the boundary between the CAL and Java languages. There are two parts to this problem: making the boundary between the languages explicit and easy to read, and writing and maintaining the glue code. Embedded CAL primarily addresses the first problem, by making the use of CAL explicit in Java code. Embedded CAL also touches on the second problem by providing a protocol that simplifies the execution of CAL expressions in Java. To further address the creation and maintenance of glue code other members of the BORG team have developed static code generation tools, which are orthogonal to Embedded CAL, and provide alternate mechanisms to access CAL from Java (as well as a ripe area for future work in presentation techniques).
6.2
Example
Figure 6.1 is a screenshot of a small Java program that uses Embedded CAL. It will be used as an example for much of the rest of the chapter. The 3
As of October 2007, CAL is no longer actively developed at Business Objects (recently purchased by SAP). It is now hosted on an independent website and is being maintained by current and former Business Objects employees. 4 A data universe is a distributed, persistent data model with object wrappers.
107
Figure 6.1: Embedded CAL in a Java program.
108
e) Managers (Controller)
a) Program Text
b) Program Text + Partitioning (Model) Embedded Editor Metadata
c) Descriptors+ Styles (Model)
d) Widgets (View)
Red
Editor Descriptor Blue
Figure 6.2: Architectural view of Embedded CAL rectangular regions in the Java editor define embedded editors that contain CAL expressions, which are evaluated when a Java thread reaches the editor in its control flow. This example defines two TransitTrips using CAL and Java. The first trip, NorthVancouverToUniversityOfBC, is defined entirely in CAL and then passed back to Java as a Java object. The second trip, nanaimoToBroadway, is constructed in different pieces. The bus Java object is constructed from a Bus CAL value, a To CAL value, and a Minutes Java object. Similarly, the skytrain and broadwayStation Java objects are constructed through CAL values. These Java objects are then used to create a CAL TransitTrip, which is converted to a Java object. Notice that results of CAL expressions are passed into Java. Also, notice how Java values can be used in the embedded editor. They are in boldface in embedded editors. This example shows how Java objects and CAL values can be constructed piecemeal from values and objects in either language.
6.3
Architectural View of Embedded CAL
Embedded CAL uses the composable presentation editor architecture described in Chapter 2. Figure 6.2 shows a description of the architecture and how it relates to the composable presentation editor architecture of Figure 2.1. This section describes Embedded CAL from a program as datastructure point of view.
109
6.3.1
Program + metadata
The implementation starts with the program as text (part a). Metadata is applied to the text in part b by analyzing the token stream and partitioning the text into different regions based on keywords and other tokens. When the tokens /* */ are discovered, the surrounded part of the program produces an embedded editor. Consequently, additional metadata is applied to the region by parsing the method declaration contained within the tags. This metadata is the raw contents of the embedded editor in AST form.
6.3.2
Logical layout and the display protocol
The next stage is the logical layout. The logical layout is a contiguous, nonoverlapping list of display elements where each element covers a region of text and has additional information attached to it. Textual display elements include a style, which defines the font and color of the text. Embedded editor display elements use the metadata defined previously to construct the characteristics of the editor, including the size, CAL contents, input policies, and output policy. The display protocol operates by walking the list of program elements sequentially and converting each program element into a display element. Program elements that have embedded editor metadata are converted into embedded editor display elements.
6.3.3
Physical layout and the render protocol
Finally, the physical layout is created by converting the display elements into render elements. The physical layout uses SWT5 as the library to construct render elements. Each textual display element is converted to a region of text in a StyledText widget6 . Embedded editor display elements are converted into render elements that comprise the different parts of the embedded editor.
6.3.4
Controllers, commands and the serialize protocol
Each embedded editor display element has an editor manager associated with it. The editor manager ensures consistency between the display element and the render elements of the embedded editor. Whenever the programmer performs an action on the embedded editor that mutates state, the editor manager executes a command that updates the display element, which in 5
http://eclipse.org/swt A StyledText widget is a graphical object that can contain both formatted text and images. 6
110
turn updates the state of the render elements. Thus, consistency is ensured between the logical and physical layouts. The underlying program is only updated when the serialize protocol is invoked. This occurs after a save command is issued or an embedded editor loses focus. The serialize protocol converts all dirty embedded editor display elements into text, which replace the relevant partitions of the program.
6.4
Embedded Editors
This section describes how Embedded CAL works from the end programmer’s point of view (i.e., program as presentation). It describes embedded editors and their features. There are two kinds of embedded editors: expression embedded editors, which contain a single CAL expression; and module embedded editors, which contain a module declaration, import statements, and top-level declarations. A class may have no more than one module embedded editor, but any number of expression embedded editors. The module editor defines the module that all expression editors execute in. Embedded editors have functionality similar to standard CAL editors that are part of the CAL-Eclipse plugin7 . For example, auto-indent, syntax highlighting, code completion, text hovers, eager parsing, and navigation are all available from within embedded editors. Both kinds of embedded editors are valid wherever a Java expression is valid. At runtime, they are treated as Java expressions. The return value of expression embedded editors is the evaluated expression. Module embedded editors have no return value, i.e., they return void.
6.4.1
Expression editor
Expression editors contain a single CAL expression. Figure 6.1 contains six embedded editors. The Java values of each of these editors is determined by the use of CAL output policies that marshal a CAL value into a Java value. The bottom-most embedded editor contains three identifiers in bold face. These are Java variables that have been captured and used in the CAL context. CAL input policies are used to marshal from Java values to CAL values. Any Java variable that is in scope can be used in the embedded editor. Figure 6.3 is a composite of several screenshots that illustrate some of the features of expression editors. Content assist (a) is available and includes Java local variable proposals. Hovering over gutter annotations (b) on the left and right side of the editor brings up a summary of the embedded editor 7 The CAL-Eclipse plugin provides Eclipse support for CAL and is available for download at http://openquark.org.
111
it is associated with. Additionally, clicking on an annotation will navigate the cursor to that embedded editor. When an invalid identifier is used in the editor, an error appears as the programmer types (c). The expression editor can be resized by dragging on the lower right corner (d), and is automatically resized as the programmer types. Hovering (e) over a CAL identifier will bring open a text hover with appropriate CALDoc documentation, and using keyboard commands or context menus, it is possible to follow a hyperlink to the declaration of the identifier. They can be used for navigation to that editor. Expanded mode Expression editors can be in one of two modes: standard mode, or expanded mode. Standard mode, described above and shown in Figure 6.1 shows only the CAL expression contained in the editor. Expanded mode, shown in Figure 6.4 displays detailed information about the embedded editor. The programmer can flip between expanded and standard modes by issuing a command from the context menu. Output policy The output policy of the expression can be selected in the drop-down. If the drop-down is blank, then the output policy is inferred at runtime with a small runtime cost. Alternatively, an output policy can be specified—a standard one from the drop-down, or a custom one by typing it in. Input policy On the right side of the editor, there is a list of the captured Java variables. Here, it is possible to control their order and specify their input policies. Standard input policies can be selected from a drop-down menu, or a custom one can be entered by hand. Error handling By default, all Java or CAL exceptions are caught by the Embedded CAL runtime library. However, this can be overridden. If the Explicit exception handling check box is selected, then exceptions must be explicitly handled by the caller.
6.4.2
Module editor
All expression editors run in the context of a module. Modules specify the namespace and declare top-level functions. Using a module editor in a Java class is a way to specify the context of expression editors. If no module editor exists, then expression editors run in a default module that imports a minimal number of identifiers. Figure 6.5 shows a module editor. They are distinguished from expression editors by being highlighted in green. Each Java compilation unit may have no more than 1 module editor. Module editors are somewhat simpler than expression editors and have no expanded 112
Figure 6.3: Features in the expression editor: a) content assist, b) gutter annotation hovering and navigation, c) highlighting errors, d) resizing, and e) text hovers to show CALDoc. mode, but in other ways, they have many of the same features including hovers, content assist, and navigation.
6.4.3
Embedded editors at compile and runtime
Embedded editors are stored as Java method invocations. They contain enough information both to present the embedded editors and to compile them. Because embedded editors are stored as Java, only a standard Java compiler and CAL binaries are required. At runtime, module editors are evaluated before the invocation of any 113
Figure 6.4: Embedded CAL expression editor expanded view expression editors. Typically, module editors are placed in a static initializer. The CAL expression of expression editors are dynamically compiled into a CAL function the first time the editor is invoked. This function is added to the module defined by the module editor (or a default module if no module is defined). Java variables used in the editor become arguments to the function. After compilation, the function is cached and reused upon each entrance.
6.5
Second Example
This section walks through a second example of Embedded CAL’s capabilities. Figure 6.6 shows a fragment of code that creates an anonymous function in CAL and passes it around as a Java object for later use. The numbers below correspond to the labels in figure: 1 This embedded editor contains a let expression. The let expression defines the tripLength function that determines the length of a transit trip in minutes. It takes a TransitTrip data type as input and returns a Minutes data type. A TransitTrip data type contains a list of trip Segments that describe a piece of a transit trip and its length in minutes. 2 The tripLength function uses collapseLengths as a helper function that takes a list of trip lengths and sums them. 3 getLengths is another helper that extracts the length from the Segment 114
Figure 6.5: Embedded CAL Module Editor
4 5 6
7
of a TransitTrip. The tripLength function is returned as an unapplied function. The result of the embedded editor (i.e., the unapplied tripLength function) is stored in the Java variable of the same name. tripLength is used in another embedded editor. It is passed the northVancouverToUniversityOfBC8 so that the tripLength function is applied. The result is a Minutes data type in CAL, which is output to Java as a Java Minutes object. And finally, tripLength is applied to nanaimoToBroadwayStation in the same way.
8
northVancouverToUniversityOfBC and nanaimoToBroadway are defined earlier in Figure 6.1.
115
Figure 6.6: Function passing in Embedded CAL. Numbers 1–7 have been added as labels of interesting parts of the program. In this example, data processing is performed in CAL, and I/O is performed in Java. Thus, language appropriate abstraction techniques and idioms are available from either language. At the same time, the control flow between Java and CAL is clear.
6.6
Embedded CAL and Presentation Techniques
Embedded CAL editors are constructed using the presentation techniques introduced in Chapter 3. We return to Figures 6.3 and 6.4 to describe how Embedded CAL uses these techniques to construct its presentation. The CAL contents of embedded editors are created using textual manipulation. The embedded editor converts Java Strings into CAL source code and formats it appropriately. Figure 6.3 parts a, b, c, and e all show various forms of temporal referencing. All of these uses of temporal referencing are built on top of standard Eclipse tools, such as hovering and content assist. Additionally, this temporal referencing provides feedback to the programmer that combines information from both languages. 116
Figure 6.3 part d combines temporal referencing with graphics. The graphics signifies that lower-right corner of embedded editors is able to be dragged, while temporal referencing provides the means of affecting change to the editor. As shown in Figure 6.4, spatial arrangement is used to layout the embedded editor. The main editing box defines the scope of the CAL language. Inside this scope, Java variable names are available for use and additionally, CAL names defined in the module editor are available. To the right of the main editing box is the input policy box, which defines a context for defining the input policies for the Java variables. These Java variables are arranged using algebraic grouping. Temporal referencing is used to relate the contents of the input policy box with the variables being used in the main editing box. As the programmer adds or removes Java variables to the main editing box, the input policy box is updated with references to these Java variables. Graphical enhancements are used to display buttons above and below the input policy box. Clicking on these buttons will change the order of the Java variables. Constrained editing is used in several places. The Explicit Exception Handling checkbox constrains this option to be a boolean. Below that is the output policy box, which uses a softer form of constrained editing. The programmer can choose from the suggested output policies or can override them with a custom one. Input policies can be changed in exactly the same manner.
6.7
Summary and Conclusion
We have described Embedded CAL and how it is implemented using a composable presentation architecture. We described its two kinds of embedded editors: expression and module editors. We showed how Embedded CAL can be used to make certain kinds of programs more expressive—programs that use and manipulate a data universe. In these kinds of programs, the universe is defined and accessed through CAL, and manipulated through Java. Although Embedded CAL is restricted to embedding CAL in Java, a similar implementation would enable other kinds of embedding as long as the following two requirements must hold: 1 The base and embedded languages must be able to run on the same platform. 2 The embedded language must be able to be compiled dynamically. Al117
ternatively, with a proper preprocessor and runtime, the embedded language can be extracted, compiled, and executed in conjunction with the base language.
118
Chapter 7
Related Work This chapter describes the work related to presentation techniques and presentation extensions. We begin by comparing our composable presentation editor architecture to other program editor architectures. We show how other editor architectures are different in one of two ways: either they trade simplicity for expressive power of the display, or they rely on a non-standard concrete syntax. The subsequent two sections introduce work that explores how the design of programming languages can add to the expressiveness of programs written in those languages. First, we describe language extensibility techniques and show how none provide the same kind of presentation extensions as the ETMOP does. In the last section, we describe multi-lingual programming techniques—that is tools and architectures that allow programmers to use the most appropriate language to solve a given problem and we show how this is related to our work in Embedded CAL. The distinction between language extension and multi-lingual programming is not crisp. With language extension, a single base language is extended with new constructs to solve problems in different domains. With multi-lingual programming, the languages are fixed, but multiple languages are able to communicate with each other. We use this distinction to illustrate that there are differences between an open and a closed approach to language extension.
7.1
Editor Architectures
We start with a tour of several historical, mainstream, professional, and research program editors and describe their architecture. Figure 7.1 shows various program editor architectures, which we describe in order and com-
119
Figure 7.1: Program editor architectures pare to our composable presentation editor architecture (Figure 7.1e). Not all the editors fit precisely in a single group, but Figure 7.1 provides a simple way to deconstruct the various editors and illustrate their similarities and differences.
7.1.1
Traditional editors
Traditional editors, such as Emacs and Vi do a single pass through the program text to present it. These traditional editors can present programs relatively quickly and use few resources compared to other editors described later. 120
Highlighting keywords notwithstanding, these editors typically do not parse the program into a language-specific representation. Therefore, the presentation is based on the lexical structure of the program and all manipulation occurs within the context of the text.1 Metadata, such as ctags can be applied to the program, but these tags are not applied to program structure. Rather, the metadata is applied to regions of text. Because presentation takes one pass to complete, nothing in the text can affect how something that precedes it appears. Consequently, the kind of manipulation of the program is limited to elision (code folding, etc.) and highlighting of keywords (including colors, fonts, size, etc.). The choice of what to elide or highlight can only be performed on the token stream. Higher-level manipulation, such as at the abstract syntax level, is not possible without performing a second pass or back-tracking through the program.
7.1.2
Hard structure editors
Structure editors (also syntax-directed editors) recognize the abstract syntax of the program being edited and use this knowledge to assist editing accordingly. Early structure editors such as the Cornell Program Synthesizer [81] and Interlisp-D’s DEdit [7] were hard structure editors that had strict constraints on how programmers could edit code, but this gave way to softer structure editors that we describe later. Typically, some variant of an attribute grammar [47] is used to aid structure editors with parsing and displaying code. This section describes the architecture required for hard structure editors. Many of the more modern editors use a kind of soft-structure editing, but use a different kind of architecture to achieve this. The program is often stored in a format that is not meant to be accessible outside of the editor.2 Before displaying, this store is converted to an inmemory abstract syntax tree. Note that the store is often optimized to make this conversion process trivial. The store of some editors, such as Barista (described below), also contains metadata to describe how individual parts of the program are displayed. After creating the AST, the program can be displayed. All edits to the screen are directly reflected back to the AST. This is typified by InterlispD’s DEdit editor [7] (described in greater detail below). In DEdit, the AST, 1 Some emacs commands are syntax based, but these commands must create and manage their own syntax tree. There is no model of the abstract syntax that is shared between commands and the editor. 2 Interlisp-D is a notable exception to this where the stored form of a program is prettyprinted lisp concrete syntax.
121
a Lisp S-expression, is walked and pretty-printed to the screen. To edit, a programmer would select an inner s-expression. That expression would pop up in a separate editing buffer, where it could be edited and replaced by another syntactically correct s-expression. In this manner, a programmer is prevented from creating a syntactically incorrect s-expression. Some structure editors use techniques like syntactic stylesheets (described below) to present regions of the AST in non-standard ways, but despite this, there is always a 1 : [0, 1] correspondence between AST elements and display elements. AST elements can be folded into their parent or displayed. This restriction helps ensure that the AST and the presentation are synchronized. Another benefit of hard structure editors is tooling support. Because the AST is always accessible, it is relatively easy to make tools for hard structure editors. However, hard structure editors can be restrictive and difficult for programmers to use [49, 87]. To address this problem, soft structure editors were developed, which have many of the benefits of both standard text editors and hard structure editors. When typed into, the soft structure editors behave largely like a standard text editor, but they also aware of some amount of program structure. Typically, soft structure editors, such as the Eclipse JDT editor, and the Squeak implementation of Smalltalk recompute structure as typing occurs. However, in general they are more difficult to create and make tools for; and they require more processing power and memory than hard structure editors. What follows is more detail on specific hard structure editors. Cornel Program Synthesizer An early structure editor was included with the Cornell Program Synthesizer (CPS) [81], which was one of the first development environments to integrate a debugger, compiler, and an editor for multiple languages. The environment supported incremental compilation, thereby greatly increasing the speed at which the program can be executed after editing. Versions of CPS were created to target different languages, including PL/I, Pascal, and Interlisp. One of the goals of CPS was to preclude the creation of syntactically incorrect programs to let “the programmer focus on the intellectually challenging aspects of programming”, but this was found to be overly restrictive for most programmers.
122
Interlisp-D Interlisp-D [7]3 is a programming and runtime environment for Interlisp developed for special-purpose lisp machines4 at Xerox PARC in the late 1970s. Among other tools, it provides a structure editor for lisp. The original form of the structure editor, called DEdit, was fairly constrained in the kinds of editing possible: only one s-expression could be edited at a time, and cut/copy actions could only contain complete s-expressions. The later version of the editor, called SEdit, had fewer constraints. The lisp code could be edited almost as if it were text and parentheses were automatically matched. The move from DEdit to SEdit showed that programmers strongly preferred the less restrictive nature of soft structure editors. Interlisp-D also gave the programmer explicit control over the AST through metaprogramming, making the editor customizable. The programmer could make some function calls in the interpreter that could (for example) delete a declaration or control how the program is pretty-printed. These functions could then be stored as editor commands, which could be triggered on certain events, thus allowing the editor to be tailorable for a specific programmer’s or program’s needs. Barista Barista [55] is a modern and experimental structure editor that can enhance the way code looks by using specialized typeface, images, and syntactic stylesheets. Editor enhancements are defined using a separate toolkit. Code is parsed and unparsed as the programmer types. Thereby, the editor is able to dynamically reformat its contents. The protocol to produce editor extensions is in the Citrus language and toolkit [54], developed to create domain specific structure editors. A programmer can enter most constructs through simple typing. Like Literate Programming’s WEB editor, the focus is on creating code that looks good. Unlike WEB, however, the enhanced view is directly editable. Syntactic stylesheets are available to, for example, display arithmetic Java statements in a mathematical form. Although the target language of Barista is Java, the code and its metadata are stored as XML. Code created in Barista cannot be easily edited 3
Interlisp, along with MacLisp, was one of the two main flavors of lisp throughout the 1970s, until Common Lisp was developed in 1980. See Steele and Gabriel’s Evolution of Lisp [78] for an in-depth discussion of this. 4 These are D machines; the most common of which are the Dolphin, the Dorado, and the Dandelion [78]. A version of Interlisp, called Interlisp-10, also ran on PDP-10 machines. It had an even more restrictive editor.
123
in other Java editors, nor is the reverse true. Since Barista does not store code in standard Java, tools such as debuggers would have to be specifically created to work in Barista. Boxer The boxer programming environment and language [23] uses a structure editor to allow programmers to define domain specific presentations of data structures. Programmers describe computation declaratively in terms of a hierarchical set of boxes. Each box is an object that can reference and manipulate other boxes using a form of temporal referencing (Section 3.4); it can also contain code, comments, or graphics. There is some amount of extensibility of the language in the form of defining new kinds of boxes, defining new interactions with them (e.g., creating widgets that respond to mouse events), and defining how internal data is laid out within the box. This is done graphically. A limitation of Boxer is that, because the environment is meant for students, it lacks some of the niceties of professional object-oriented languages such as dynamic memory allocation and inheritance. Also, boxes are strictly hierarchical, and usually have dynamic scoping. Lexical scoping can be achieved through ports where one box contains a reference to another. Boxer is meant to be a teaching language and environment, and as such compatibility with external languages and tools is not attempted. Boxer programs cannot be edited outside of the environment.
7.1.3
Soft-structure editors
Many of the integrated development environments (IDEs) used in the industry today employ some kind of soft-structure editor so that programs can be edited directly as text, but the editor still has some knowledge of the abstract, language-specific structure of the program. For many of these IDEs (Smalltalk IDEs are an exception), the program is stored as text and can freely be edited using other tools. Like traditional editors, the program text is typically displayed in the editor directly. Additionally, however, the program is parsed into an AST and is used to create an overlay of fonts and colors that can enhance the textual display. Typically, this is implemented in a background thread, so this extra work is not noticed by the programmer with a fast computer. This AST overlay provides additional, syntax-based entry points for tools that can apply helpful techniques such as refactoring and debugging. The AST overlay is light-weight and flexible, but typically this means that the presentation is limited to the text with formatting and colors. As 124
with traditional program editors, parts of the program text can be elided, but program elements cannot be rearranged or otherwise manipulated. What follows are two canonical soft-structure editors that the general form that these kinds of editors can take. Smalltalk Smalltalk is an object-oriented language and environment built with an explicitly object oriented meta-level (in other words, a MOP [50], although it was not called as such at the time it was created) [41]. It is not possible to describe the Smalltalk editor without describing some of the language capabilities. The programmer has access to the MOP and all the meta and non-meta libraries that implement the system. Thereby, language features can be redefined and new ones can be added. Smalltalk environments were built using what are now well-recognized object-oriented framework design principles, which rely on having libraries and well-documented protocols to use them. All classes have special methods that control the way they are prettyprinted in code browsers. In addition, some Smalltalk implementations such as Squeak,5 and those that use the refactoring browser have multiple syntactic stylesheets available that make small changes to the way source code is displayed, such as replacing certain sets of ASCII characters with Unicode, or displaying declarations of graphical elements (menus, buttons, etc.) as either text or graphics. However, there are no protocols that are specifically designed for creating new syntactic stylesheets or extending the prettyprinting. Even though the programmer has access to the Smalltalk compiler, there are no protocols exposed to help extend the language syntax or add new syntactic stylesheets. Code in a Smalltalk environment is locked in the Smalltalk image. There is no file system and code cannot be accessed outside of the environment unless it is specifically exported. This is by design since a Smalltalk image is meant to be its own unit, isolated from the rest of the world. All tools for the image must be written in the environment and are accessible to all other parts of the image. The Eclipse Platform Instead of describing all of the popular Java and .Net IDEs, we will focus on the Eclipse platform for two reasons. First, it is a good proxy for the other platforms, and second both the ETMOP and Embedded CAL are built on top of Eclipse. 5
http://www.squeak.org
125
Eclipse’s Java editor (the JDT editor) is a standard soft-structure editor. The editor displays the text of a Java compilation unit. Text may be elided from the view through the use of code folding. Additionally, a background thread parses the program and applies highlighting based on the semantics of the program. For example, static variables are typically displayed in italics. Additionally, tools such as content assist use this AST to provide context sensitive completion proposals at a particular text location. The Eclipse IDE borrows many ideas from Smalltalk and Interlisp environments in that it emphasizes extensibility, but it avoids some of their pitfalls by using the file system as the code repository, thereby making code accessible outside of the environment. The environment itself is open and extensible, with all of its code exposed to programmers through protocols. Through this, developers can leverage much of Eclipse to create their own tools while still programming with standard Java (the biggest exception to this is the JDT editor itself). The editor is a soft structure editor and thus text can be entered freely, while still maintaining a parse tree of the program through a background thread. The editor is somewhat aware of the semantics of the program being edited. Thus, tools such as refactoring and hyperlinking can be applied easily. However, the JDT editor itself is essentially closed and there is no exposed protocol to customize the presentation of programs. Aside from annotation processing, Eclipse offers no protocol for syntactic or semantic extension of Java.
7.1.4
Display-driven ASTs
There is a kind of editor architecture where the AST is generated from the display, not the other way around as in structure editors. The persistent store is a serialization of the display and is typically not in a form meant to be directly readable by programmers. The benefit of this kind of AST created from the display is that more radical forms of display are possible. However, because the AST is dependent on the particular display infrastructure of the system, there is less flexibility to plug in external tools or edit using other editors. With the exception of DrScheme, these editors tend to be more experimental. Subtext Subtext [26] is a “paper-free medium of programming, one designed for usability”. The program is a graph-like data structure that the development environment interprets to display on the screen using syntactic stylesheets. All edit operations operate on the graph directly. References in the language 126
are through direct links in the data structure, not through names as in traditional languages. These links can be labelled with a name, which serve only as a comment and have no semantics. The semantics of the language is lazy-functional and the code is always live, so any change in the code is directly reflected in the output. One of the goals of the project is “language extension through presentation”. Although this is a similar goal to that of the ETMOP, Edwards means that concrete syntax in Subtext should be separate from its abstract syntax, being defined only through syntactic stylesheets. This is a difference from our presentation extensions, that extend the presentation of a program, rather than define it. The program itself is not meant to be viewed, edited, or executed outside of Subtext. Hence, it is unlikely to be compatible with external tools and languages. Consequently, it is unclear how a system like Subtext would scale to larger examples, and how to apply its concepts to existing, popular general purpose languages. More recent work in Subtext [27] has shown how providing an alternate representation of conditional logic can lead to programs that are more declarative, easier to read, and easier to edit. The conditionals in Subtext, called schematic tables, have a structure similar to standard truth tables, but also have semantics associated with it. Schematic tables are laid out in a canonical form, thus freeing the programmer from making layout decisions. Since the code is always live, the programmer can manipulate the inputs and immediately see the display change. Domain Workbench The Domain Workbench [74] is an instance of intentional programming (IP) [73]. Similar to MPS (described next), the Domain Workbench assists in the development of domain specific languages that interact with each other. Code is stored in a proprietary data structure called the intentional tree. Unlike MPS, the Domain Workbench enables the creation of graphical languages and multiple views of single programs through using syntactic stylesheets to present the intentional tree. Nodes in the intentional tree are called intentions which are instances of DCLs (intention definitions). To obtain an executable program, the tree is walked and transformed into a more basic, reduced tree, which is then compiled into machine code. The DCL of an intention defines xmethods that define behavior on intentions, for example how they are input, displayed, browsed, versioned, debugged, etc. Hence, DCLs define an attribute grammar—intentions are the productions and the xmethods are their at127
tributes. The xmethods themselves are similar to rewrite rules of syntactic macros. However, intentions seem not to be lexically scoped. Any intention seems able to affect any part of the intentional tree, which may lead to problems with interactions between intentions and analysis of programs. Our discussion is speculative because there is no publicly available version the Domain Workbench or of IP. The Domain Workbench has a similar compatibility issue to Subtext. Although an existing program in any recognized language can be imported into the Domain Workbench, once it is there, it is stored in a proprietary format and is inaccessible to other tools and editors. There are many open questions with IP. How exactly is the code executed and how can do domains intermingle? What happens if two DCLs contradict each other? How are DCLs scoped? How difficult is it to define a new projection or domain? How does it scale? Because there are no publicly available versions of IP or the domain workbench, the answers to these questions must wait. Work in IP has existed for over a decade, first in Microsoft and now as a private company. Metaprogramming System (MPS) MPS6 is a development environment that is intended to enable the easy definition of new programming languages that can easily interact with other languages. In addition to the languages themselves, MPS aids the creation of editors and tool support such as code completion and debuggers. Essentially, each new language defines an extension of some existing language, with the exception of the base language, which has syntax and semantics similar to Java. Transformation translates an extended language into a simpler language. There are several steps to defining a new language. A language designer must create the structure (abstract syntax), the editor (how the language is displayed, edited, keyboard shortcuts and commands), and the generator (how to transform the language into a lower-level language). Each of these steps is accomplished by using a language defined in MPS: the structure language, the editor language, and the transformation language respectively. To transform programs, the transformation language makes three strategies available to programs: templates (similar to XSLT), pattern matching (a regular expression search and replace over the AST), and iteration (walk6
MPS is a commercial product currently in beta and is available through JetBrains at http://jetbrains.com/mps.
128
ing the AST to produce an expanded AST, non-hygienically7 ). The editors are hard structure editors, restricting the kinds of interactions that the programmer can have with the editor. MPS is often described in terms of language oriented programming (LOP) [24]. LOP [86] is similar to metalinguistic abstraction (described below in the macros section), but with a difference in focus. LOP tends to be more focused on tool support, emphasizing custom editors for the new languages. MPS also incorporates much of the architecture of Intentional Programming. Compatibility in MPS is not as much of an issue as in the Domain Workbench. The persistent store is in XML, and its format is documented, though not intended to be viewed by programmers. It would therefore be possible to build tools external to MPS that work with code-bases written in it, but there is little support for this as of yet. DrScheme DrScheme8 is a programming environment for PLT Scheme that provides semantic and syntactic extensibility through the use of Scheme’s hygienic macro system and provides presentation extensibility through the ability to embed boxes into programs. Boxes contain textual or graphical objects that can be evaluated by the Scheme interpreter. For example, DrScheme has box syntax for fractions, comments, images as values, XML with quasiquotelike escape, and test case boxes. More recently, DrScheme has included picture boxes for WYSIWYG layout of slide content that is otherwise created programmatically [34]. There is no documented protocol for creating new kinds of boxes, but their implementation is fairly simple [20]. Two pieces need to be defined: the visual form and a translation to an s-expression. The visual form of the box is defined using DrScheme tools and graphics interfaces. Translation from a box to an s-expression occurs when the box is evaluated. Both steps have full access to scheme’s hygienic macro system. Optionally, it is possible to write extensions to the stepper (the debugger) so that boxes can be stepped through in a natural way during evaluation. DrScheme is largely not a structure editor, except that boxes themselves are separate graphical objects and focus cannot be gained through standard keyboard movement. Thus, interacting with boxes can be awkward. The persistent store for any DrScheme program that contains these boxes is ASCII-encoded binary. The semantics of the boxes is largely dependent on 7
Hygienic expansions ensure that program transformations will not cause unintended name capture [56]. 8 http://www.drscheme.org
129
the display itself, rather than coming from the persistent store. Therefore, in order to compile, interpret, or edit the program, a box-aware editor is required. This affects compatibility since any box-enhanced scheme program must be edited and executed from within the DrScheme environment (or a large subset of the environment).
7.2
Expressiveness Through Language Extension
The kind of expressiveness that we have explored in the ETMOP is expressiveness through language extension. This section describes language extension mechanisms that can be used to create more expressive programs. We describe how they work, why they are important, their strengths and weaknesses, and the type of extensibility they provide: syntactic, presentation, or semantic.
7.2.1
Syntactic and semantic extension
Most approaches to language extensibility have worked by pairing syntactic extension with semantic extension. Under syntactic extension, the language designer defines a new syntactic rule for the language, and under semantic extension, the language designer defines some new semantics for a part of the language. These two kinds of extensions may be orthogonal, but typically they are related. Syntactic extension can be used to bind semantic extension to a particular location in the code. The Common Lisp defmacro facility is a typical example of this approach [77]. This combination of syntactic and semantic extension has worked most easily in the Lisp languages, where the syntactic constraints are simple [3]. In the larger set of languages with Algol-like syntax (including C, Java, etc.) enabling syntactic extension is more complex, although systems like Dylan, JSE, and Maya have shown that it is workable [3, 4, 6]. The following few sections describe variations of both semantic and syntactic extensions.
7.2.2
Non-meta libraries
Non-meta level libraries9 of various kinds—procedures, classes and datastructures—are simple mechanisms for language extensibility in that they add vocabulary to an existing language, adapting it to a new domain. They are a form of semantic extension to a language. They enable programmers to create language extensions without the need for meta-programming of any kind [75]. However, these libraries cannot always sufficiently capture domain 9 We distinguish these from meta-level libraries which are written to support meta-level mechanisms such as MOPs that we describe later.
130
expertise because expressing concepts specific to a domain is restricted by the core syntax and semantics of the language [79]. Libraries only extend the language by adding to its vocabulary, thereby increasing its range, but they cannot make any changes to the core language itself, such as controlling order of evaluation, changing syntax, adding laziness, changing method dispatch, etc. To provide these kinds of extensibility, some form of meta-programming is required, as in the following mechanisms.
7.2.3
Macros
Macros are a form of meta-programming in which programmer defined code is allowed to make local transformations to the program at the point of the macro’s invocation. The term macro is a shortened form of macroinstruction, and was originally used to mean a single machine instruction that stood for several (micro-)instructions [44]. The term has evolved to mean any word or phrase in a programming language that stands for another (usually a larger and more complicated phrase) [32, Chap 21]. Next, we describe different kinds of macros and some of their uses. Syntax macros receive an AST as input and produce an expanded AST as output [63]. These kinds of macros have gained the most traction in languages in the Lisp family of languages that have S-expression based syntax [3]. Scheme uses a variant of syntax macros, hygienic macros [56], which ensures that macro expansion will not cause unintended name capture. Due to the syntactic complexity of Algol-like languages such as C, C++, and Java, creating and using syntax macros in these languages is more complicated. For this reason, they have not been as widely adopted as they have been in Lisp [3, 88]. Despite this, there have been several syntactic macro mechanisms developed for these languages. All of these mechanisms provide a way to specifically extend the grammar of the language so that macros can be attached to syntax. This puts a burden on macro developers to not only understand language design and parsing technology, but also the specific grammar of the target language. Brabrand and Schwartzbach [11] describe , a high-level language for developing interactive web services. Weise and Crew [88] describe MS2 , a programmable macro language for C. Bachrach describes the Dylan language macro system [3] and an implementation of Dylan style macros in Java, called the Java Syntax Extender [4]. Maya [6] is a hygienic macro system for Java. Lexical macros operate on the token stream that a lexer produces as it reads a program. At the point of the macro invocation, these macros produce a new, expanded token stream from the original. CPP, the C preprocessor, is a lexical macro system that is used in many large programs 131
[37].10 It is efficient and flexible, but being token-based it introduces a number of problems—an improper use of a macro can make the source program unparsable, and syntactic correctness of the expanded program is not guaranteed. Decades of experience with macros show that the proper use of macros can make programs more expressive because they make languages more responsive to change [62], they are a powerful tool for the extension of language syntax [1], and they can be used to add syntactic sugar and little languages to a host language [52]. Metalinguistic Abstraction (MLA) [1] is a programming paradigm in which macros (or other language features) are used to solve complex problems by creating a new language and vocabulary to better understand the problem space. This is a major theme of Abelson and Sussman’s seminal Structure and Interpretation of Programming Languages [1]. However, creating structurally complex macros can be difficult even with a powerful macro system [18]. One way to address this problem is to use a more structured approach to local semantic extension such as a metaobject protocol or an attribute grammar. Depending on definitions, these two technologies are either alternatives to macros, or are mechanisms to implement macros. Regardless of which definition is used, MOPs and attribute grammars provide a more structured way to extend a language compared to standard macro approaches.
7.2.4
Metaobject protocols (MOPs)
MOPs [50] provide a protocol to extend and customize a compiler or interpreter in an object oriented manner. They have been used in the past to make extensible interpreters [50] and extensible compilers [17]. MOPs are a powerful mechanism to extend language semantics, but most MOPs have not provided much syntactic extensibility. For example, OpenC++ [17] has a fixed set of extensions to the syntax (such as new modifiers and for-style statements), and is therefore not as syntactically flexible as many of the macro systems described previously. Similarly, the CLOS MOP [50] does not by itself extend syntax, but rather relies on Common Lisp’s macro system to do that. Our own work in the ETMOP makes use of MOPs to provide both semantic and presentation extensibility. 10 More specifically, CPP reads in the program as text and does some minimal text processing (e.g., line splicing of escaped newlines) before converting the text into a token stream and expanding that. Also, pre-ANSI versions of CPP did work on the character stream of a program [48].
132
7.2.5
Attribute grammars
Attribute grammars are another mechanism that can attach semantics to syntax. As such, they are sometimes used to assist with code transformation [85]. Kennedy and Warren succinctly describe what attribute grammars are [47]: An attribute grammar is an ordinary context free grammar extended to specify the semantics of each string in the language. Each grammar symbol has an associated set of attributes, and each production rule is provided with corresponding semantic rules expressing the relationships between the attributes of the symbols in the production. To find the meaning of a string, the parse tree is first found and then the values of all the attributes of symbols in the tree can be found. One example of extensibility through attribute grammars is the JastAdd extensible Java compiler [29]. Its attribute grammar can be modularly extended to add new forms of static checking or language mechanisms. It remains modular through the use of intertype declarations. The attribute grammar is specified using a DSL that uses intertype declarations that enable the modularization of both new syntactic elements and operations on those elements. Another notable example of extensibility through attribute grammars is work by Cardelli et al [15]. Here, the authors show an example of how an extensible attribute grammar can be used to extend a simple typed lambda calculus into a SQL-like query language. The grammar is expanded incrementally with new keywords and constructs that reduce to existing terms in the unexpanded language. Throughout this expansion, hygiene is preserved because the names of free variables are replaced with unique names at each stage of the expansion. This kind of grammar extensibility is more general than macro expansion because the language can be both extended and contracted. For example, keywords can be added to or removed from the language. Thus, this work shows a hygienic alternative to macro expansion that uses parser technology to provide syntactic extensibility.
7.2.6
Pretty-printing
Pretty-printing, also called code formatting, is the process of laying out source code, altering whitespace, and changing fonts, color, and font style to enhance readability and emphasize program structure. Pretty-printing customizes the way programs are presented without affecting syntax or semantics, and works by parsing a program into an AST and then laying out 133
the AST according to rules defined by the pretty-printer. The output is typically rich text in the concrete syntax of the language. Most hard structure editors have some sort of built-in pretty-printer. A pretty-printer can be implemented by attaching an attribute grammar to a structure editor. An example of this is Interlisp-D [7]. The attributes on each production in the grammar describe how the production is displayed. In Interlisp-D, the programmer can control the way macros and other code structures are pretty-printed and displayed in the debugger. Pretty-printing alone is typically restricted to altering fonts, colors, and whitespace; it therefore only has minimal control over how the language is presented, but from here it is a small step to using syntactic stylesheets as described next.
7.2.7
Syntactic stylesheets
Using a syntactic stylesheet [26] is a way to provide a new concrete syntax for an existing language’s abstract syntax. Syntactic stylesheets take prettyprinting one step further and provide the ability to rearrange text or insert graphical notation into the display. Stylesheets can be used, for example, to expand or combine declarations, and to add or remove explicit typing information in the display. There are many ways that syntactic stylesheets can be implemented. For example, Eclipse (Section 7.1.3) implements them by using document projections, Subtext (Section 7.1.4) and the Domain Workbench (Section 7.1.4) implement them through attribute grammars. Also, the ETMOP can be used to implement syntactic stylesheets. As with pretty-printing, the semantics of the language are not affected, but syntactic stylesheets have considerable power to make concrete syntax malleable.
7.3
Expressiveness Through Multi-Lingual Programming
Programming languages are typically tuned to provided certain kinds of services and therefore make solving certain kinds of problems easier. Key examples of this are domain specific languages (DSLs) [82, 83, 84], which are little languages designed to solve problems in a particular domain. Even general purpose programming languages (GPLs) tend to be tuned to solving specific kinds of problems at the expense of other kinds. For example, GPLs such as the LISP family of languages tend to be most effective for symbolic processing, JavaScript provides powerful capabilities to manipulate web-
134
Language embedding through syntax Common runtime environments Linking Pipeline SOA
Separate parsers. parser.
Communication through
Separate compilers. virtual machine.
Communication through
Separate compilation steps. Communication in memory mapped address space. Separate processes. Communication through input and output Separate machine. Communication through message passing.
Table 7.1: Summary of techniques for multiple language use. pages on the client side, and Fortran is tuned for numeric computation and scientific computing. The technique of using multiple languages to construct programs is an alternative to the previous section where we describe techniques to extend a single language for the same purpose. However, the line between language extension and multi-lingual programming is blurry as we illustrate below. Martin Fowler distinguishes internal from external DSLs [35]. An internal DSL is integrated into a host language and is often syntactic sugar for an application library. An external DSL is not derived from a host language and typically runs separately. While this distinction is useful, it is not specific enough for our purposes. We would like to be more specific about how the languages interact and what effect this has on the expressiveness of the program. This section explores various techniques that have been used to allow multiple languages to communicate in one program. The techniques are ordered from the strongest connection between languages to the weakest. The work is summarized in Table 7.1.
7.3.1
Language embedding through syntax
Combining languages through their syntax is a powerful technique to program using multiple languages. One language can be embedded in another through the use of a parser or a set of parsers that can parse individual, but connected snippets of the program. This is a fine-grained connection
135
between languages, and therefore it is possible for each language to do small pieces of work before handing the computation off to another language. However as we show below, there are limitations with the kinds of languages that can be embedded since this technique is typically implemented through a pre-processor or a compiler front-end. An example of this is writing server-side scripts to generate web-pages. For JSP, HTML is used to describe the static structure of the page, and Java is used to generate parts of the page that are constructed dynamically. PHP uses a similar technique. For these kinds of language embeddings, one language is considered the base language (Java, PHP) and the second language (HTML) is transformed to statements in the first. In this way, embedding is accomplished through the use of a pre-processor. The final compiler sees only a single language. This kind of embedding is possible only because the HTML is largely declarative and can easily be rewritten in the base language using only calls to a stream writer. Bravenboer et al [12] have used attribute grammars to embed domain specific languages (DSLs) inside of general purpose languages (GPLs), using a process they call MetaBorg. To compile the program the entire program is first parsed, producing an AST that uses syntax elements of both the GPL and the embedded DSL. Next, the entire, multi-lingual AST is type checked. The third step is to assimilate, which transforms the multi-lingual AST into a new AST that uses only the base language. This MetaBorg approach is used to promote libraries in the base language into DSLs. The restriction is that any DSL written using MetaBorg is largely syntactic sugar over the base language. The DSL must borrow much of its semantics from the GPL. Design decisions like the type system or laziness are not extensible using their process. A slightly different approach is taken with Microsoft’s LINQ [68] (Language INtegrated Query) introduced in the .Net framework 3.0. LINQ provides querying capabilities for general datasources such as relational, XML, or in-memory datastructures in languages that compile to the .Net framework’s CLR. LINQ is a collection of new language semantics (such as anonymous lambda expressions, extension methods (similar to intertype declarations), simple type inference, and syntactic sugar to make the use of these extensions more SQL-like. LINQ provides hooks for programmers to write their own drivers to customize the kinds of datasources that are accessible. Although LINQ is not a general purpose extension to .NET, it provides some standard functional programming techniques that are not typically available in mainstream object-oriented languages. LINQ differs from our approach in Embedded CAL because it is language-based not editor-based. Also, 136
LINQ is a query language only and is therefore a domain specific approach. Embedded CAL is general purpose, even though the language is tuned for accessing and manipulating data universes.
7.3.2
Common runtime environments
An increasingly common way to communicate across languages is to use a common virtual machine and compile all languages to byte code for that machine. Compared to language embedding, there is more freedom in the kinds of languages that can be used together. But with common runtime environments, the connection between languages is weaker than it is language embedding because languages are not mixed within a compilation unit. The .Net framework uses the CLR or common language runtime [67]. All languages written in the .Net framework compile into CLR byte code and hence all these languages are interoperable to an extent. It is possible for languages with a dynamic type system, VB.Net11 to interoperate with functional languages with a static type system, F#,12 and to additionally interact with object oriented languages, C# 13 . On the Java side, there is a growing number of languages including Jython, JRuby, Scala, and CAL that all compile to Java byte code. These languages range from dynamic and procedural: Jython14 , JRuby15 to objectoriented and static: Java, to functional and lazy with type inference: CAL.16 . This technique allows a diverse range of languages to interoperate, but the restriction is that they must run in the same process sharing an address space on the same virtual machine.
7.3.3
Pipelines
A pipeline architecture [40] arranges a series of processing elements so that the output of one element becomes the input of the next. Pipelines enable multi-language use in a way that provides slightly more flexibility from common runtime environments. Each processing element is typically a distinct program and exchanges with the other processing elements through streams. The canonical example 11
http://msdn2.microsoft.com/en-us/vbasic http://research.microsoft.com/fsharp/manual 13 http://msdn2.microsoft.com/en-us/vcsharp 14 http://www.jython.org 15 http://jruby.codehaus.org Actually, Jython and JRuby are multi-paradigm and can be written using a functional, object-oriented, or procedural approach, but the original implementations of the languages were both procedural. 16 The Java-CAL connection is discussed in greater detail in Chapter 6. CAL can be downloaded at http://labs.businessobjects.com/cal/ 12
137
of this kind of architecture is the Unix command line. Each processing element is agnostic to the language used to create other processing elements. A processing element can be a compiled component in machine code, or it can be a dynamically interpreted script. The restriction is that all processing elements must run on the same machine and therefore must use compatible machine code.
7.3.4
Service-oriented architecture
Service-oriented architecture (SOA) [71] is a style of programming that enables a program to perform computations on remote machines. This style is the most decoupled form of multi-lingual programming that we discuss. Under SOA, the basic unit of communication is the message and target systems are agnostic with respect to the architecture and language used to produce the message. Thus, components can be written in any language and compiled to any architecture. The price of this flexibility is difficulty in communication between components. Message passing is slow because components typically reside on separate machines. Additionally, it is important to ensure that the message contents are not tied to a particular language or machine type. The set of valid messages are constrained by an interface through which the components interact. Using an interface is typically less flexible than using a more tightly integrated approach to multi-lingual programming. There are many SOA implementations. The most common ones are SOAP17 , which wraps remote invocations in XML to send data in a platform independent way, and CORBA, which encodes remote invocations in a binary format specific to CORBA. In general, CORBA implementations are faster [30] in part due to the verbosity of SOAP’s XML messages. SOAP’s primary advantage is that it runs on top of HTTP [46] and therefore fits nicely with existing protocols.
7.4
Summary
In this chapter, we have described a number of editor architectures and shown how this architecture enables or limits the presentation of the programs they display. We have shown how our composable presentation editor architecture provides fewer restrictions on the kinds presentations that are possible, while not prescribing the form of the abstract syntax of the program presented. Next we showed a number of language extension mechanisms that are 17
http://www.w3.org/TR/soap
138
used to create more expressive programs, and we described how none of them provide the same kind of flexibility in presentation as found in composable presentation editors. We ended with a description of the range of multi-lingual programming techniques. The Embedded CAL approach is similar to language embedding through syntax (Section 7.3.1) in that the programmer can view and interact with both languages in a single editor, but because in Embedded CAL, the embedding is editor-based, not syntax-based, there is more flexibility in the kinds of languages that can be embedded.
139
Chapter 8
Conclusion Having presented the technical material of our work, we summarize our results, review the claims and contributions, discuss some of its limitations, and close with implications for future work. Although clear in retrospect, the most surprising part of this work has been the analogies between presentation techniques and other more traditional techniques. For example, it was not originally evident that spatial arrangement can provide scoping for the language, or that graphical enhancement is yet another technique for abstraction. From the beginning, we were fairly certain of the feasibility of building something like a composable presentation editor. We figured early on that using a MOP to provide editing semantics for a program would be the most appropriate way for the program to gain control over its own presentation. Other pieces of the implementation followed from that. The model-viewcontroller style of architecture arose later as the pieces of the metaobject protocol were being laid down. However, determining the precise characterization of the protocol, metaobjects, and other datastructures required significant tinkering throughout the early and middle part of our work. From the beginning, our philosophy on the use of presentation techniques was that they should be used with a light touch—that is, the presentation should be unobtrusive, emphasize the meaning of the program, and its semantics should be self-evident. The end result of this thinking was that few extensions used graphics. Where graphics were applied, this was typically as accents to associate or delineate parts of the program. Perhaps na¨ıvely, we had initially assumed that graphical enhancements would play a larger role in the uses of the techniques.
140
8.1
Summary
In this dissertation, we explored how program presentation can affect the expressiveness of a program through the use of editor technology. Our definition of expressiveness states that a program is expressive when it has properties that reliably align the program’s perceived meaning to the program’s intended meaning. We proposed an editor architecture that can compose the presentation of a program non-surjectively—that is, the transformation from program elements to display elements can insert, delete, or re-order. The architecture is based on a model-view-controller style architecture. The model is a combination of a program with metadata and the logical layout, an abstract representation of the display. The view corresponds to the physical layout and concretely describes the display. The controllers are a set of handlers that react to user events and can affect change on the program. This architecture enables various presentation techniques: spatial arrangement, textual manipulation, temporal referencing, graphical enhancement, and constrained editing. These techniques, in turn, establish editor and language properties, such as abstraction, referencing, and algebraic grouping; and these properties express higher-level ideas about the program. By providing the programmer with flexibility in the way these ideas are expressed, a composable presentation editor can more easily allow the program’s perceived meaning to align with its intended meaning. We tested these ideas by implementing two instances of a composable presentation editor: an open and extensible editor—the ETMOP, and a closed, domainspecific editor—Embedded CAL. To show the breadth and flexibility of the use of presentation techniques, we implemented the ETMOP, which provides presentation extension. Presentation designers can combine the ETMOP with a semantic processor, such as the CTMOP, to enable presentations extensions to have semantic effect. Both the ETMOP and CTMOP are implemented through the use of a metaobject protocol and provide extensibility in a way similar to syntax macros. We described numerous extensions that can be created by the ETMOP. Some extensions require the CTMOP, others require external semantic processors (such as an aspect weaver), and still other extensions require no semantic processing at all. The breadth of extensions we show characterizes the kinds of language enhancements that are possible when using presentation extension. Embedded CAL shows the depth of presentation techniques. When restricting the flexibility of presentation techniques, we can achieve greater 141
compatibility with the development environment. We show how Embedded CAL embeds one language within another through the use of editor technology. The two languages are visually connected, and each language can be used for the most suitable purposes. We showed how embedding CAL inside of Java programs can coherently combine the two languages in a way that supports the strengths of each language. We discussed how our work in presentation techniques fits in with other forms of language extensibility and multi-lingual programming. We have shown how other editor architectures differ from ours. Some do not have the same flexibility between stored form and presentation, and others are are display-oriented, where the stored form is based on the presentation, thus requiring the compiler to rely on the display infrastructure.
8.2
Revisiting Claims and Contributions
We revisit the claims and contributions, describing how they have been achieved. The claims (from Chapter 1): 1 It is feasible for an editor to present (i.e., display and provide editing support for) programs using a non-surjective relationship to the concrete syntax. The composable presentation editor architecture provides a template for how these kinds of editors can be constructed; and the ETMOP and Embedded CAL show that it is possible to implement. 2 Such an editor enables a set of simple and composable presentation techniques that can be used as strategies to present programs. The five presentation techniques introduced in Chapter 3 arise from the architecture by taking advantage of indirection between program and presentation: the program + metadata is transformed into the logical layout, which is then presented as physical layout. 3 Even though they operate in the domain of program presentation, these presentation techniques are natural extensions to existing language design, editor design, and programming best-practices techniques. Language design, editor, and programming best practices techniques enable properties of a program such as scoping, referencing, and prettyprinting. We have shown how presentation techniques can enable the same kinds of properties, but differently, through the manipulation of presentation. 4 Careful use of the presentation techniques appears to lead to more expressive programs. Program properties such as scoping, grouping, and formatting can ex142
press higher level ideas about a program when used appropriately. When these higher level ideas align with the intended meaning of the program, that program is more expressive than when they do not. This dissertation makes the following contributions. We describe a composable presentation editor architecture and two implementations of this editor: an open implementation and a closed implementation. We propose a qualitative framework for characterizing program expressiveness. We describe a set of presentation techniques enabled by this architecture and an analysis of how the techniques compare to existing program language design, editor design, and programming best practices techniques. We have shown how these presentation techniques help to enable programs to be more expressive.
8.3
Limitations
There are a number of limitations with this work, which we enumerate below.
8.3.1
Definition of expressiveness
Our definition of expressiveness is qualitative and not precise. It cannot be used for making absolute comparisons of expressiveness between programs. Rather, the definition can be used descriptively and so it is sufficient for our purposes.
8.3.2
User evaluations
The ETMOP and Embedded CAL are largely proofs of concept—i.e., they show that implementing a composable presentation editor architecture is feasible. However, we have not evaluated either implementation through formal user studies. There are two kinds of questions that such studies could help answer. First, are editors that use presentation techniques beneficial to programmers? Second, how effective are the particular presentations that we have chosen? These kinds of user evaluations are not currently a focus of our research. To date, we have instead focused on expressiveness. The implementations are currently not mature enough to place in front of users and the engineering effort required to make them so would come at the expense of our current research goals. Some informal usability studies were performed on Embedded CAL. The editor was used by some programmers at Business Objects to develop a small, sample application. The anecdotal feedback on usability and API guided our efforts and led to improvements in the implementation.
143
8.3.3
Presentation design
Some of the ETMOP extensions, most notably the equations and the automaton, could be helped by a closer study of typesetting and graphic design. A full analysis of proper fonts, spacing, and colors could make these extensions easier to read and hence further contribute to more expressive programs. This is the kind of work performed by Baecker et al [5] where they explored the most expressive way to format the paper printing of C programs.
8.3.4
Implementations and the use of internal APIs
Both implementations are built as plugins to the Eclipse development environment. Eclipse has proven to be a good foundation to explore these ideas because it is extensible, it is well documented, and there is a community of researchers who are also using Eclipse as a research platform. Despite this, however, there are limitations to having chosen Eclipse and Java as the target platform. Eclipse is a large and complicated development environment. It is meant for development work in industry and therefore much of its infrastructure exists to ensure proper response time and ease of use. This adds complexity and size to many Eclipse’s extensions. This is especially true for extensions such as the EMTOP and Embedded CAL that depart from the development environment’s original intensions. The ETMOP and (less so) Embedded CAL rely on using and extending internal APIs that are not exposed to client applications. To use these APIs, the implementation uses a combination of subclassing, intertype declarations, and reflection. Future versions of Eclipse may change or remove any of these APIs without warning. This is already a problem that we have encountered. The ETMOP works for Eclipse version 3.2 only. Later versions of Eclipse are incompatible with the ETMOP.
8.4
Implications for Future Work
Our work with composable presentation editors contributes to a body of work with editors that help make programs more expressive. This contribution is an analysis of presentation techniques and presentation extension. We hope that there will be further exploration of these ideas. Through our experiments with presentation techniques, they appear to be a factor in making programs more expressive. Further exploration in this area may be fruitful as there are many open questions. Can we characterize presentation idioms and patterns? Will they show trends of the kinds of
144
presentations that are effective? Are there certain problem domains that are naturally more amenable to presentation techniques? Is it possible or useful to construct a more quantitative framework for describing expressiveness? As we describe above, usability and usefulness of presentation techniques are not the focus of our current work. Exploring these ideas will be essential for a full evaluation of presentation techniques and presentation extension. The most appropriate means to achieve this is through a user study. As noted above, the two implementations would require some work before they could be the foundation of a user study. But, there are a number of alternatives that can help minimize the programming effort while still being able to answer research questions. For example, Eclipse 3.4 will provide rich hovers, which are pop-up windows that can be edited and can contain links, buttons, and graphics. This will be a rich area for the exploration of a lightweight incorporation of presentation techniques into mainstream editors without the need to use or extend internal APIs. Alternatively, Visual Studio 2008 will provide DSL authoring tools and ways to use multiple languages together in the same editor. A third option is to use an environment like DrScheme that can provide a rich playground for incorporating presentation techniques in Scheme. Program presentation is an important, but as yet under-studied component of expressiveness. This dissertation explores how presentation can affect expressiveness by applying a novel editor architecture to present programs.
145
Bibliography [1] Harold Abelson, Gerald J. Sussman, and Julie Sussman. Structure and Interpretation of Computer Programs. MIT Electrical Engineering and Computer Science. The MIT Press, second edition, July 1996. [2] Lance Andersen. JSR-221: JDBCTM 4.0 API Specification. Technical Report Public Review, Sun Microsystems Inc., December 2005. [3] Jonthan Bachrach and Keith Playford. D-expressions: Lisp power, Dylan style. Technical report, Massachussetts Institute of Technology, 1999. http://www.ai.mit.edu/people/jrb/Projects/dexprs.pdf. [4] Jonthan Bachrach and Keith Playford. The Java syntactic extender (JSE). In Proceedings of the 16th ACM SIGPLAN conference on Object oriented programming, systems, languages, and applications, pages 31–42. ACM Press, 2001. [5] Ronald M. Baecker and Aaron Marcus. Human factors and typography for more readable programs. ACM, New York, NY, USA, 1989. [6] Jason Baker and Wilson C. Hsieh. Maya: multiple-dispatch syntax extension in java. In PLDI ’02: Proceedings of the ACM SIGPLAN 2002 Conference on Programming language design and implementation, pages 270–281, New York, NY, USA, 2002. ACM Press. [7] David R. Barstow. Overview of a display-oriented editor for Interlisp. In Proceedings of the 7th International Joint Conference on Artificial Intelligence, pages 927–929, 1981. [8] Christian Bauer and Gavin King. Java Persistence with Hibernate. Manning Publications, revised ed edition, November 2006. [9] Joshua Bloch. JSR-175: A Metadata Facility for the JavaTM Programming Language. Technical Report Final Release, Sun Microsystems Inc., September 2004. [10] Claus Brabrand, Anders Moller, and Michael I. Schwartzbach. The project. ACM Trans. Inter. Tech., 2(2):79–114, 2002. 146
[11] Claus Brabrand and Michael I. Schwartzbach. Growing languages with metamorphic syntax macros. In PEPM ’02: Proceedings of the 2002 ACM SIGPLAN workshop on Partial evaluation and semantics-based program manipulation, pages 31–40, New York, NY, USA, 2002. ACM Press. [12] Martin Bravenboer and Eelco Visser. Concrete syntax for objects. Domain-specific language embedding and assimilation without restrictions. In OOPSLA ’04: Proceedings of the 19th ACM SIGPLAN conference on Object-Oriented Programing, Systems, Languages, and Applications, Vancouver, Canada, October 2004. ACM SIGPLAN. [13] Avi Bryant, Andrew Catton, Kris De Volder, and Gail C. Murphy. Explicit programming. In AOSD.02: Proceedings of the 1st international conference on Aspect-oriented software development, pages 10–18. ACM Press, 2002. [14] Richard R. Burton, Ronald M. Kaplan, Larry M. Masinter, B.A. Sheil, Alan Bell, Daniel G. Bobrow, L. Peter Deutsch, and Willie Sue Haugeland. Papers on Interlisp-D. Technical report, Palo Alto Research Center, Xerox Corporation, September 1980. [15] Luca Cardelli, Florian Matthes, and Mart´ın Abadi. Extensible syntax with lexical scoping. Research Report 121, Digital SRC, 1994. [16] Philippe Charles, Julian Dolby, Robert M. Fuhrer, Jr. Stanley M. Sutton, and Mandana Vaziri. Safari: a meta-tooling framework for generating language-specific ide’s. In OOPSLA ’06: Companion to the 21st ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications, pages 722–723, New York, NY, USA, 2006. ACM. [17] Shigeru Chiba. A metaobject protocol for C++. In OOPSLA ’95: Proceedings of the tenth annual conference on Object-oriented programming systems, languages, and applications, pages 285–299, New York, NY, USA, 1995. ACM Press. [18] Shigeru Chiba, Michiaki Tatsubori, Marc-Olivier Killijian, and Kozo Itano. OpenJava: A Class-based Macro System for Java. In Walter Cazzola, Robert J. Stroud, and Francesco Tisato, editors, Reflection and Software Engineering, Lecture Notes in Computer Science 1826, pages 119–135. Springer-Verlag, Heidelberg, Germany, June 2000. [19] Carlos Christensen and Christopher J. Shaw, editors. Proceedings of the Extensible Languages Symposium. Association for Computing Machinery, 1969. Appeared as SIGPLAN Notices, 4(8):1–62, August 1969. 147
[20] John Clements, Matthew Flatt Matthias Felleisen, Robert Bruce Findler, and Shriram Krishnamurthi. Fostering little languages. Dr. Dobb’s Journal, March 2004. [21] Gregory H. Cooper and Shriram Krishnamurthi. Embedding dynamic dataflow in a call-by-value language. In European Symposium on Programming, pages 294–308, March 2006. [22] Krzysztof Czarnecki and Ulrich W. Eisenecker. Generative programming: methods, tools, and applications. ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 2000. [23] A. A diSessa and H. Abelson. Boxer: a reconstructible computational medium. Commun. ACM, 29(9):859–868, 1986. [24] Sergey Dmitriev. Language oriented programming: The next programming paradigm. JetBrains onBoard, 1(2), February 2005. [25] Stuart Edmondston and Brian Zotter. JSR-181: Web Services Metadata for the JavaTM Platform. Technical Report Proposed Final Draft, BEA Systems, April 2005. [26] Jonathan Edwards. Subtext: uncovering the simplicity of programming. In OOPSLA ’05: Proceedings of the 20th annual ACM SIGPLAN conference on Object oriented programming, systems, languages, and applications, pages 505–518, New York, NY, USA, 2005. ACM Press. [27] Jonathan Edwards. No ifs, ands, or buts: Uncovering the simplicity of conditionals. In OOPSLA ’07: Proceedings of the 22th annual ACM SIGPLAN conference on Object oriented programming, systems, languages, and applications, 2007. [28] Andrew D. Eisenberg and Gregor Kiczales. Expressive programs through presentation extension. In AOSD.07: Proceedings of the 6th international conference on Aspect-oriented software development. ACM Press, 2007. [29] Torbj¨ orn Ekman and G¨orel Hedin. The jastadd extensible java compiler. SIGPLAN Not., 42(10):1–18, 2007. [30] Robert Elfwing, Ulf Paulsson, and Lars Lundberg. Performance of soap in web service environment compared to corba. apsec, 00:84, 2002. [31] Jean-Charles Fabre and Shigeru Chiba, editors. Proceedings of Workshop on Reflective Programming in C++ and Java, October 1998. http://www.csg.is.titech.ac.jp/~chiba/oopsla98ws.html. [32] Neal Feinberg, Sonya E. Keene, Robert O. Mathews, and P. Tucker Withington. Dylan programming: an object-oriented and dynamic 148
[33]
[34] [35] [36]
[37] [38]
[39]
[40]
[41]
[42]
[43]
[44]
language. Addison Wesley Longman Publishing Co., Inc., Redwood City, CA, USA, 1997. Matthias Felleisen. On the expressive power of programming languages. In ESOP ’90: Selected papers from the symposium on 3rd European symposium on programming, pages 35–75, Amsterdam, The Netherlands, The Netherlands, 1991. Elsevier North-Holland, Inc. Robert Bruce Findler and Matthew Flatt. Slideshow: functional presentations. J. Funct. Program., 16(4-5):583–619, 2006. Martin Fowler. Meta Programming System. http: //www.martinfowler.com/articles/languageWorkbench.html. David Frankel. Model Driven Architecture: Applying MDA to Enterprise Computing. John Wiley & Sons, Inc., New York, NY, USA, 2002. Free Software Foundation. The c preprocessor. http://gcc.gnu.org/onlinedocs/cpp/. David Gallardo, Ed Burnette, and Robert McGovern. Eclipse in Action: A Guide for the Java Developer. Manning Publications, seventh ed. edition edition, May 2003. Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. Design patterns: elements of reusable object-oriented software. Addison-Wesley Professional, 1995. David Garlan and Mary Shaw. An introduction to software architecture. In V. Ambriola and G. Tortora, editors, Advances in Software Engineering and Knowledge Engineering, pages 1–39, Singapore, 1993. World Scientific Publishing Company. Adele Goldberg and David Robson. Smalltalk-80: The Language and its Implementation. Addison Wesley, Xerox Palo Alto Research Center, 1983. Jeff Gray and G´ abor Karsai. An examination of dsls for concisely representing model traversals and transformations. In HICSS ’03: Proceedings of the 36th Annual Hawaii International Conference on System Sciences (HICSS’03) - Track 9, page 325.1, Washington, DC, USA, 2003. IEEE Computer Society. Jack Greenfield, Keith Short, Steve Cook, and Stuart Kent. Software Factories: Assembling Applications with Patterns, Models, Frameworks, and Tools. Wiley, August 2004. Irwin D. Greenwald. A technique for handling macro instructions. Commun. ACM, 2(11):21–22, 1959. 149
[45] JetBrains. Meta Programming System. http://www.jetbrains.com/mps/. [46] I. Jong. Web services/soap and corba. http://www.omg.org/news/whitepapers/CORBA_vs_SOAP1.pdf, 2002. [47] Ken Kennedy and Scott K. Warren. Automatic generation of efficient evaluators for attribute grammars. In POPL ’76: Proceedings of the 3rd ACM SIGACT-SIGPLAN symposium on Principles on programming languages, pages 32–49, New York, NY, USA, 1976. ACM Press. [48] B. W. Kernighan and D. M. Ritchie. The C Programming Language. Prentice-Hall, Englewood Cliffs, New Jersey, 1978. [49] Amir Ali Khwaja and Joseph E. Urban. Syntax-directed editing environments: issues and features. In SAC ’93: Proceedings of the 1993 ACM/SIGAPP symposium on Applied computing, pages 230–237, New York, NY, USA, 1993. ACM Press. [50] Gregor Kiczales, Jim des Rivi`eres, and Daniel G. Bobrow. The Art of the Metaobject Protocol. MIT Press, Cambridge, MA, USA, 1991. [51] Gregor Kiczales, Erik Hilsdale, Jim Hugunin, Mik Kersten, Jeffrey Palm, and William G. Griswold. An overview of aspectj. In ECOOP ’01: Proceedings of the 15th European Conference on Object-Oriented Programming, pages 327–353, London, UK, 2001. Springer-Verlag. [52] Oleg Kiselyov. Macros that compose: Systematic macro programming. In Generative Programming and Component Engineering (GPCE’02), October 2002. [53] Donald E. Knuth. Literate programming. The Computer Journal, 27(2):97–111, May 1984. [54] Andrew J. Ko and Brad A. Myers. Citrus: a language and toolkit for simplifying the creation of structured editors for code and data. In UIST ’05: Proceedings of the 18th annual ACM symposium on User interface software and technology, pages 3–12, New York, NY, USA, 2005. ACM Press. [55] Andrew J. Ko and Brad A. Myers. Barista: An implementation framework for enabling new tools, interaction techniques and views in code editors. In CHI ’06: Proceedings of the SIGCHI conference on Human Factors in computing systems, pages 387–396, New York, NY, USA, 2006. ACM Press. [56] Eugene Kohlbecker, Daniel P. Friedman, Matthias Felleisen, and Bruce Duba. Hygienic macro expansion. In LFP ’86: Proceedings of 150
[57]
[58] [59] [60]
[61] [62] [63] [64]
[65] [66] [67] [68]
[69]
the 1986 ACM conference on LISP and functional programming, pages 151–161, New York, NY, USA, 1986. ACM Press. G. Krasner and S. Pope. A description of the model-view-controller user interface paradigm in the smalltalk-80 system. Journal of Object Oriented Programming, 1(3):26–49, 1988. Shriram Krishnamurthi. Automata as macros. Journal of Functional Programming, 2005. Shriram Krishnamurthi. Automata as macros. Journal of Functional Programming, 2005. Shriram Krishnamurthi, Matthias Felleisen, and Daniel P. Friedman. Synthesizing object-oriented and functional design to promote re-use. In ECCOP ’98: Proceedings of the 12th European Conference on Object-Oriented Programming, pages 91–113, London, UK, 1998. Springer-Verlag. Leslie Lamport. LaTeX: A Document Preparation System. Addison-Wesley Professional, 2nd edition edition, 1994. Paul J. Layzell. The history of macro processors in programming language extensibility. Comput. J., 28(1):29–33, 1985. B. M. Leavenworth. Syntax macros and extended translation. Communications of the ACM, 9(11):790–793, 1966. Cristina Videira Lopes, Paul Dourish, David H. Lorenz, and Karl J. Lieberherr. Beyond aop: toward naturalistic programming. In OOPSLA Companion, pages 198–207, 2003. Steve McConnell. Code complete: a practical handbook of software construction. Microsoft Press, Bellevue, WA, USA, 1993. M. Douglas McIlroy. Macro instruction extensions of compiler languages. Commun. ACM, 3(4):214–220, 1960. E. Meijer and J. Gough. Technical overview of the common language runtime, 2000. Erik Meijer, Brian Beckman, and Gavin Bierman. Linq: reconciling object, relations and xml in the .net framework. In SIGMOD ’06: Proceedings of the 2006 ACM SIGMOD international conference on Management of data, pages 706–706, New York, NY, USA, 2006. ACM. Bertrand Meyer. The power of abstraction, reuse, and simplicity: An object-oriented library for event-driven design. In Olaf Owe, Stein Krogdahl, and Tom Lyche, editors, Essays in Memory of Ole-Johan Dahl, volume 2635 of Lecture Notes in Computer Science, pages 236–271. Springer, 2004. 151
[70] Carol Bergfeld Mills and Linda J. Weldon. Reading text from computer screens. ACM Comput. Surv., 19(4):329–357, 1987. [71] Eric Newcomer and Greg Lomow. Understanding SOA with Web Services (Independent Technology Guides). Addison-Wesley Professional, December 2004. ISBN: 0321180860. [72] D. L. Parnas. On the criteria to be used in decomposing systems into modules. Commun. ACM, 15(12):1053–1058, 1972. [73] C. Simonyi. The death of computer languages, the birth of intentional programming. Technical Report MSR-TR-95-52, Microsoft Corporation, September 1995. [74] Charles Simonyi, Magnus Christerson, and Shane Clifford. Intentional software. In OOPSLA ’06: Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming languages, systems, and applications, pages 451–464, New York, NY, USA, 2006. ACM Press. [75] Richard M. Stallman. Emacs the extensible, customizable self-documenting display editor. In Proceedings of the ACM SIGPLAN SIGOA symposium on Text manipulation, pages 147–156, New York, NY, USA, 1981. ACM Press. [76] Thomas A. Standish. Extensibility in programming language design. SIGPLAN Not., 10(7):18–21, 1975. [77] Guy L. Steele. Common Lisp the Language, 2nd edition. Digital Press, 1990. [78] Guy L. Steele and Richard P. Gabriel. The evolution of lisp. In History of programming languages—II, pages 233–330. ACM Press, New York, NY, USA, 1996. [79] Bjarne Stroustrup. A rationale for semantically enhanced library languages. In In proceedings of LCSD’05 workshop, October 2005. [80] Peri Tarr, Harold Ossher, William Harrison, and Jr. Stanley M. Sutton. N degrees of separation: multi-dimensional separation of concerns. In ICSE ’99: Proceedings of the 21st international conference on Software engineering, pages 107–119, Los Alamitos, CA, USA, 1999. IEEE Computer Society Press. [81] Tim Teitelbaum and Thomas Reps. The cornell program synthesizer: a syntax-directed programming environment. Commun. ACM, 24(9):563–573, 1981. [82] USENIX, editor. Proceedings of the Conference on Domain-Specific Languages, New York, NY, USA, 1997. ACM Press. 152
[83] USENIX, editor. Proceedings of the 2nd conference on Domain-specific languages, New York, NY, USA, 1999. ACM Press. [84] Arie van Deursen, Paul Klint, and Joost Visser. Domain-specific languages: an annotated bibliography. SIGPLAN Not., 35(6):26–36, 2000. [85] Eelco Visser. Program transformation with Stratego/XT: Rules, strategies, tools, and systems in StrategoXT-0.9. In C. Lengauer et al., editors, Domain-Specific Program Generation, volume 3016 of Lecture Notes in Computer Science, pages 216–238. Spinger-Verlag, June 2004. [86] M. Ward. Language oriented programming. Software—Concepts and Tools, 15(4):147–161, 1994. [87] Richard C. Waters. Program editors should not abandon text oriented commands. SIGPLAN Not., 17(7):39–46, 1982. [88] Daniel Weise and Roger Crew. Programmable syntax macros. In PLDI ’93: Proceedings of the ACM SIGPLAN 1993 conference on Programming language design and implementation, pages 156–165, New York, NY, USA, 1993. ACM Press.
153
