MANIPULATION

Many developers have yet to grasp what makes direct manipulation interfaces successful

•

lrect

MANIPULATION

nte aces ser interface technology has made considerable progress during the past few years, particularly in the area of direct manipulation. User interfaces based on direct manipulation are characterized by their support for interaction organized around the use of a pointing device, such as a mouse or touchscreen. Other features commonly associated with this style of interface include graphics, windows, and advanced menu techniques. Direct manipulation hal' been successfully used in a wide range of applications, including spreadsheets, desktop publishing, CAD / CAM and public information systems, software development environments, and expert systems.

U

28

LIMITING FACTORS Despite its success, acceptance of direct manipulation has been less than universal for a number of reasons. First and perhaps foremost, direct manipulation interfaces involve considerably more overhead than teletypebased interfaces. Therefore, the machines associated with them have, until recently, been expensive. Second, risk is incurred in software development; design and implementation of applications using direct manipulation often involve unconventional languages, unorthodox design techniques, and unfledged development tools. No consensus has been reached as to the best way to design soft-

ware for direct manipulation interfaces. This lack of consensus is reflected in the tools emerging on the markeL'·2 Third, many software developers have yet to fully grasp what makes these interfaces successful. This issue is the focus of this article-in particular, the elements of direct manipulation and how they may be applied to knowledge-intensive interactive environments such as expert systems. Prospective developers of direct manipulation interfaces are "all dressed up with nowhere to go." The technical capabilities needed to develop effective user interfaces exist, but often no one knows what to do with them. The process of creating an effective user interface remains very much an art, yet artistic inclinations among software developers remain an undervalued commodity. Support for developers in the form of interface guidelines tends to be geared to yesterday's technology, viewing the world as if it were a series of full-screen displays (if not 80-column punch cards). This is quite different from direct manipulation environments, which compress considerable power into a single display through the use of windows, pop-ups, icons, graphics, and other gadgetry. The art of creating a direct manipulation interface involves more than piling windows knee-deep on the display, anticipating that the world will rejoice at the prospect of getting lost among them. Designs based on naive implementation of advanced techniques

AI EXPERT. OCTOBER 1988

ARTWORK: DON CARROLl/IMAGE BANK

tend to result in inept, garish, and irrelevant interfaces causing user overload, disorientation, and frustration. 3. 5 Development should be undertaken within a framework of relevant user interface concepts. An effective user interface encourages its users to focus their energies on the substance of their work rather than on extraneous requirements imposed by the computer. For expert systems, this can mean the difference between success and failure. This is not simply a matter of user acceptance. Even if people can be induced to use .a system and the results are generally accepted as epistemologically sound, success may remain elusive. Ensuring that the problems posed by the user and solved by the system are one and the same requires close cooperation between user and system. Furthermore, if the user cannot be presumed capable of properly posing a question, the system must ascertain not only the facts concerning the case at hand but the questions worth asking. Otherwise, effectiveness is compromised, and this may go undetected by the unsuspecting user. If expert systems come into common usage (and become as common as the shells we use to develop them), it will be in part because obstacles associated with the user interface have been surmounted. Direct manipulation represents the best solution available using today's technology. The success of direct manipulation interfaces can be attributed not to any single technique but to a convergence of interrelated elements, of which direct manipulation is but one. Other elements include symbolic representation, choice constructs, and concurrent contexts. Each of these elements is essential to the interface as a whole: symbolic representation is important in modeling user interfaces on their problem domains, choice constructs guide and inform user decisions, and concurrent contexts let the user integrate the benefits of several interactive systems at the same time. For its own part, direct manipulation is the mechanism that permits users · to navigate among and interact with these elements. Because of their significance to the effectiveness of direct manipulation, these elements will be discussed in some detail.

SYMBOLIC REPRESENTATION Symbolic representation is a key element in modeling user interfaces on their domains. To the extent a user interface successfully AI EXPERT. OCTOBER 1988

I.

29

30

models the system's domain, the process of interacting with the computer begins to assume the character of directly manipulating domain objects. This lets the user think about the problem being solved rather than the way it is being solved. Symbolic representation is the relationship between symbols and domain objects. This relationship is one of association, resemblance, or convention, and is accomplished using pictorial, metaphoric, arbitrary, and composite depictions of the objects and relations comprising the problem domain. In direct manipulation interfaces, symbols are interactive. As output entities, they depict the problem domain in its current state. As input entities, they are the handles, knobs, and other gismos whereby the user manipulates the problem domain. Any changes effected in the domain are reflected symbolically. The symbols populating an interface are not isolated entities. The interface itself consists of a collection of symbols, their spatial and temporal relations, and their dynamics. Spatial relations define the behavior of symbols sharing the display at the same time; temporal relations define the interactive scenarios possible in the system. These relations may be effective, effected, concomitant, or inconsequent; that is, they may affect other objects, be affected by other objects, belong to groups of mutually affected objects, or contrive to have no relationship whatsoever. The dynamics of a symbol define the set of behaviors possible in the symbol's life in the system. Viewed from a rudimentary perspective, the process of designing a user interface consists in defining the collection of symbols, their relations, and their dynamics. Selecting and integrating symbols into a system requires a knack for graphically visualizing complex information, an intuitive grasp of user behavior, and sufficient domain knowledge for delineating the significant problems. The dimension of domain dependency suggests that, for expert systems, interface design is analogous to knowledge acquisition. 4.6 Following this analogy, the human expert's role in interface design is comparable to his or her role in knowledge acquisition. In expert systems, effective use of symbolic representation can help contain complexity and make the system intuitive and credible. Because expert systems are knowledge intensive, this is not merely a matter of easing the mechanics of interaction or minimizing the memorization of commands, but also of exploiting the user's expectations as to how ideas are organized and expressed within the system's target domain. A number of factors contribute to accomplishing this.

Symbols and their relations. Symbols and their relations must be defined in terms of how knowledge is represented across the user interface. This includes · the terminology, rhetoric, notations, depictions, and styles of interaction associated with the problem domain. Some aspects of the interface, such as diagramming techniques, are closely associated with the problem domain. Others, such as the presentation of general reasoning, tend to be shared across multiple domains or common to the user's culture. Effective use of symbols and their relations requires understanding both the domain and the user. Data flow diagrams, for example, are appropriate to a system supporting software design, but are unlikely to inspire enthusiasm among many users of a guide to Chinese cooking, regardless of the quality of the recipes. Such an approach, however, might be useful if the users were software engineers with no prior experience in the kitchen. Reasoning. Reasoning, as represented iIi relations among symbols, must be expressed in human rather than machine terms. Human reasoning is highly compressed, leaving many premises and intermediary inferences implicit. It is unnecessary and tedious to present all relevant premises and conclusions with mathematical rigor. Consider this example:

Any horse can outrun any dog; some greyhounds can outrun any rabbit; therefore, any horse can outrun any rabbit. If we disregard the possibility that some horses are quite slow while some dogs are pretty fast, we can readily observe that two additional premises are needed to render the argument valid:

If Xcan outrun Yand Ycan outrun Z, then Xcan outrun Z All greyhounds are dogs Even so, the argument is still not complete. To demonstrate from these premises that any horse can outrun any rabbit requires 27 steps of formal logic. 7 Such reasoning is not human reasoning; it is the reasoning of computers (and logicians). In the user interface, the degree of compression advisable depends on the domain and the user's familiarity with the domain. 3 Premises presumed by an. expert may seem less than obvious to a nOViCe.

Explanations. Explanations should be explanatory rather than tracebacks of activated rules. The purpose of explanation is to support user orientation and system credibility. Explanations should provide the user with a view of overall strategy, the factors


r governing the strategy, and the user's options with respect to controlling the strategy. While rule-based representation is conducive to system manageability from the developer's perspective,S its modularity and simplicity tend to subvert the user's ability to understand the overall structure of the problem domain. 9 Explanations to expert systems are so important that they cannot afford to rely solelyon textual discourse for their symbolic representation. Pictorial and' metaphorical representations can usually be more easily and quickly grasped, especially when implemented as interactive entities capable of sustaining user exploration and manipulation. Explanations should, from the user's perspective, be presented as an integral part of the system, not a subsystem developed as an afterthought. Questioning. For interactive expert systems to do their work, they must solicit sufficient information from the user to form a basis for their recommendations. Consequently, it is not unusual for expert systems to subject users to lengthy interrogations. However, users quickly tire of an expert system that only wants to play Twenty Questions. Other means of soliciting user input should be considered. Alternatives to the interrogative approach include model building and mixedinitiative interaction. Both of these are wellsuited to direct manipulation. With the model-building approach, the system provides the user with a set of symbolic tools to depict the target problem graphically. This depiction may be carried out cooperatively, with the user and system prompting one another as necessary until the problem is refined to their mutual satisfaction. Mixed-initiative interaction 3 is a variation on the interrogative approach, except the user is able to interrupt the interrogation at any time to offer as-yet-unsolicited information. To the extent that the interrogative approach is necessary, it should be progressive rather than arbitrary; that is, sequences of questions should be intuitively organized rather than sequenced according to the firing of a randomly arranged rule base. While the analogy between interface design and knowledge acquisition illuminates the path toward improved expert system user interfaces, it also suggests that the knowledge acquisition bottleneck may be tighter than we realize. If expert resources are required, not only to construct the knowledge base but to design the interface, development time and expense will increase accordingly. Before sounding the alarm, though, we may wish to consider the possibility that we are taking the analogy a bit too far. The demands for domain expertise in interface de-

sign, while very real, may not be as stringent as in knowledge acquisition. It is at least conceivable that in domains characterized by a considerable body of publically accessible knowledge, developers experienced in user interface design could make considerable headway in prototyping the interface with only a modicum of expert assistance.

CHOICE CONSTRUCTS Choice constructs constrain, guide, and inform user decisions. Menus are the most familiar type of choice construct; other types include icons, buttons, and text prompts. Menus support user decisions by presenting useful sets of alternatives. Icons and buttons support tightly constrained decisions, while text prompts, being essentially open-ended, usually must be augmented with error handling. Serialized text prompts are command languages. Menus cover the territory between buttons and text prompts. The combination of direct manipulation with bit-mapped display technology has fostered the maturity of menus. Using bitmapped displays, a host of menu types has arisen: pop-ups, multiple-item menus, and menu bars. Direct manipulation provides the mechanism for deftly summoning and dispatching these menus. For example, in Figure 1, the use of windows and menu bars lets the user select from a variety of contexts. Pop-up or momentary menus are handy for transient, single-item decisions. Multiple-item menus express slightly more complex notions, such as selecting values for groups of parameters. Scrollable list boxes are provided for managing' dynamically sized lists of selectable items, such as a list of files. Menus can be combined with other choice constructs to form dialogue boxes. Menu bars are exemplary choice constructs. They consist of a row-of mouse-sensitive symbols, .each of which, when selected, presents a momentary menu (in this context, momentary menus are usually called pull-down menus). To enter a command, the user opens a pull-down, selects an item, and the pull-down disappears. Menu bars support continuous access to a repertoire of commands within a shallow hierarchy, fully accessible within a single screen. Consequently, they can be browsed rapidly without loss of orientation. The importance of this kind of choice construct is not that it momentarily overlaps a region of the display, but that the region overlapped is small and the overlap is momentary. Choice constructs are essentially structures of related interactive symbols, grouped and displayed to assist the user in conveying decisions to the system. The symbols comprising a choice construct may be pictorial, arbitrary, or composite. The pal-


Users quickly tire of an expert system that only wants to play Twenty Questions

I·

31

MENU BAR

-----iiiliiiliiiiiiiliiiiiiiiiiiiil

~

ALERT INDICATORS -

~it~~~ Current

~

(Amps)

~~[225 .

,.~] Bead profile Is offset to right

To maintain constant level fill r "'l.rsll"ale,>v. torch has b een moved eloser to

left sidewall

EXPLANATION SUBSYSTEM, SHOWING GRAPHIC AND TEXTUAL ELEMENTS

~-;~i ~~~~t@ Travel

Speed

(ipm)a15.~,:

"~j

60 55 50 45

40 35 30

25

20 Actual Planned Pass : . .

15 10 05 00

WINDOWS

FIGURE 1. Operator interface display for an expert welding control system.

32

GRAPHIC REPRESENTATION OF CURRENT SYSTEM STATUS

ette of drawing tools used in the Apple's chanics of interaction and thus reduce the MacDraw is a metaphoric menu. mental load required to use the system. For expert systems, the importance of choice constructs is their approach to the lo- CONCURRENT CONTEXTS cus of interactive control. In recognizing Concurrent contexts let users interact with this, it is helpful to note the fundamental several systems, processes, domains, or virdifferences between systems based on ad- tual worlds at the same time. This is comvanced choice constructs and conventional monly realized using windows, where each menu-driven systems. window behaves as a virtual terminal, and Conventional menu-driven systems guide the user can select a window for interaction users through a series of decisions, travers- using direct manipulation techniques. ing from one decision point to the next. BeWindows provide a fix for what is othercause the locus of control resides with the wise a shortcoming in user interface envisystem, the user has no alternative but to re- ronments. This shortcoming is the narrowspond to the current menu. Such systems ness of domain of computer applications, can be very easy to use on a step-by-step ba- sometimes referred to as the tool metaphor sis, but for complex tasks, the associated because of the tendency for programs to menu trees tend to become prolix, disor- handle individual, specialized tasks. 1o Typiienting, and confining. cal tools in a user support environment Advanced choice constructs, however, are might be a word processor, drawing prouser-driven. They are passive until the user gram, and spreadsheet. Without windows, chooses to act upon them. By letting the these programs or tools operate for the user take the initiative, the system gives the most part as discrete, sequential entities. user flexibility in deciding what to do and Any attempt to force them to cooperate in a when to do it. Because the realm of possible joint interactive venture is cumbersome at actions is structured and bounded by best. menus, the user is largely relieved of reThis difficulty can be vexing when atsponsibility for remembering what com- tempting to integrate emerging technolmands exist and in what contexts they make ogies into an ongoing environment. For exsense. Choice constructs simplify the me- ample, using today's technology, it is AI EXPERT. OCTOBER 1988

frequently feasible to develop expert systems to assist operators of monitoring and control applications in diagnosing problems. However, if the expert system executes independently and takes exclusive control over the user interface, forcing the operator to abandon the monitoring and control application, considerable inconvenience will be involved in consulting the expert system. This considerably diminishes the probability that the system will be consulted at all. The windows solution is to let the user run and interact with more than one program at a time , with each program running independently in its own window. Several additional services are usually associated with window systems. These include commonality across domains, dynamic display control, freedom of navigation, and data transfer. Commonality across domains means similar functions are implemented similarly from one application to the next; for instance, the procedure for deletion will be the same regardless of whether the user is deleting a paragraph from a document or erasing a rectangle from a drawing. Applications can exploit and reinforce the user's experience with other applications in rendering the interface easy to use. With dynamic display control, the user can orchestrate windows as desired. As the needs of the user shift among the domains occupying the display, the arrangement of windows can readily be altered. The ability to set aside a window (and in doing so, collapse it into its iconographic representation) without destroying it lets the user put it on hold and return to it later as needed by reopening it. In this fashion, the user can set aside collections of iconographic domains, thus tailoring a metaphorical menu of systems. Freedom of navigation lets the user roam at large among domains. For example, a programmer might have a text editor containing source code in one.window, a display of detailed design documentation in another, and a command processor in a third. The programmer might scroll through the design documentation in search of a bit of pseudocode and, upon finding it, shift attention to the text editor to key in an implementation of the pseudocode using a highlevel programming language, all the while keeping the pseudocode visible on the display. This being accomplished, the programmer might instruct the editor to save the source code to disk, and then focus on the command processor, entering commands appropriate to compiling and linking the source code. If the compile results in error messages, the programmer can correct the source code without losing sight of the rpessages.

Data transfer is the ability to move data from one window to another. For example, a user composing a document can insert drawings from a draw program or tables from a spreadsheet directly into the text. In the previous example, the programmer might prefer to insert a copy of the designlevel pseudocode directly into the text-editor, where it could be revised to conform to the syntax of the high-level language. While windowing is clearly a key element in direct manipulation interfaces, it is perhaps a concept whose full potential has yet to be realized. Several deficiencies are associated with windows: • If poorly managed, windows can contribute to user disorientation, which is only rarely productive in the workplace. Windows containing vital information may appear and vanish under program control, giving the bemused user no clue as to whence they came nor whither they have gone. • To the extent that windows represent isolated islands of activity, they can hamper the user's efforts. It is up to the user to create and maintain whatever organization and relationships might exist among the windows. Thus the desktop metaphor found in some systems can be realized with a disturbing degree of accuracy. Too many windows may be required on display, forcing the user to thrash through mounds of clutter whenever some bit of information is desired. • The current state of the user's focus of attention may not be obvious. For example, users may think they are entering text into an editor window, only to discover that the system has been directing the input to a command processor. Problems due to poor management of windows can be corrected in a straightforward manner. Rather than presume that the cheap thrill of a windowing environment will pass for user friendliness, developers must use windows as a means to better designs. This includes observing, within the context of direct manipulation, many conventional tenets of good user interface design, such as providing the user with a stable work area, continuous access to status information, and requiring confirmation for important actions. II To some extent, the problems with windows result from the tool metaphor. That windows operate primarily as independent entities or "alienated windows," as R. Reichman 12 calls them, is contrary to the way users use them. Windows are not arbitrarily selected for display and interaction. In current systems, the structure and relation of a given set of windows is entirely subjective to the user. There is little attempt on the part of the system to reinforce this structure. Particularly in window systems based on existing operating environments,


Windows may appear and vanish under program control, giving the bemused user no clue as to whence they came nor whither they have gone

33

I ,

applications originally developed for tele- cal representation can be found may lend type interaction are forced to fit into the themselves readily to direct manipulation window scheme of things. As more applica- designs. Tasks intrinsically bound up in systions are developed to take advantage of tems of arbitrary symbols may be more suitwindows (as well as of other window applica- ed to high-level languages. To suppose that the issue is this simple, tions), we may expect to see some improvement in integration among windowed however, is to sell direct manipulation short. It is the nature of direct manipulation to abprocesses. Another possible basis for improvement is sorb other user interface techniques, as evihypertext. Using concepts arising out of hy- . denced in the assimilation of Small talk, pertext, clusters of applications can be dyna- LISP, and some implementations of the mically linked together at access points UNIX shell into the direct manipulation specified by the user. In any case, for the scheme of things. In each of these environpresent, the isolation of individual processes ments, the user interacts with the system uswithin windows is not nearly so complete as ing a language based on arbitrary symbols. that of processes running in a conventional But this is accomplished in a manner inteteletype environment. grated into and augmented by direct manipulation. Apparently the ability to select and EXPERT SYSTEM USER INTERFACE manipulate symbols, combined with features E.L. Hutchins et al. 13 note that for some such as commonality across domains, dytasks, high-level languages may be superior namic display control, freedom of navigato direct manipulation. If so, interface de- tion, and data transfer, is a paradigm that is signers need to be able to tell which type of sufficiently powerful to encompass the capainterface they should use for a given bilities of earlier, more conventional technologies. application. Because selecting the symbols appropriate Expert systems help people solve comto a system is a domain-dependent activity, plex, demanding problems. These problems the problem may not be susceptible to an a may entail a higher level of accountability priori solution. Instead, decisions as to in- than conventional interactive software systerface type must be made as part of an iter- tems. Consequently, it is important that usative design and prototype process. Tasks ers be able to focus their energies on the for which a suitable pictorial or metaphori- substance of the problem domain rather

INDUSTRIAL STRENGTII OOps. You have three options in today's world; lead, followor get out of the way. You've already taken a leadership position in hardware with the latest 286 or 386 system. Now you can use that triple-digit architecture to blast ahead of the pack with the most powerful new Object Oriented Programming (ooPs) software on the market: Smalltalk/V286. Smalltalk!Y, the original oOPS tool for the PC, gave scientists, engineers, programmers and educators a brand new way to solve .problems. And soon they were developing exciting new applications in everything from economics to medicine to space. Now Smalltalk/V286 gives you true work station performance with industrial strength capabilities like: push-button debugging; multi-processing; portability

between DOS, OS/2 and Presentation Manager operating environments; integrated color graphics; a rich class library; and access to 16 MB of protected mode memory, even under DOS. The new Smalltalk/V286, which is e~n easier to learn and use than Smalltalk!Y, retails for just $199.95. Or you can buy Smalltalk!Y, still the world's best selling OOPS, for only $99.95. And both come with our 60 day moneyback guarantee. Check out the new Smalltalk/V286 at your dealer. If he doesn't have it, order toll free, 1-800-922-8255. Or write to: Digitalk, Inc., 9841 Airport Blvd., Los Angeles, CA 90045. And let us put you ahead of the power curve.

SmalItalk/V 286

CIRCLE 14 ON READER SERVICE CARD

than on extraneous requirements imposed 1987, pp. 327-331. Sponsored by NASA Marshall Space Flight Center a nd the University of Alabama in by the computer. The elements of direct manipulation Huntsville. 5. Shneiderman, B. Designing the User Interface: Stratecombine to form an environment hospitable gies Jar Effective Human-Computer Interaction. Reading, to the interactive demands of expert sys- Mass.: Addison-Wesley, 1987. 6. Baroff, J., R. Simon, F. Gi lman, and B. Shneidertems. Symbolic representation provides the basis for externally representing knowledge man. " Direct Manipulation User Interfaces for Expert Systems," in Hendler, J.A. (ed.), Expert Systems: The User using artifacts intrinsic to the problem do- Interface. Norwood, N .J .: Ablex, 1988 , pp. 99-1 25 . main. Advanced choice constructs support 7. Copi , I.M. Symbolic Logic, third ed. London , U.K.: user-driven interaction control strategies MacMillan , 1967. 8. Barr, A., and E.A. Feigenbaum (eds.). The Handwhile explicitly limiting the range of possioj Artificial Intelligence, vol. II. Los Altos , Calif.: ble initiatives. Concurrent contexts enable book William Kaufman , 1982. users to consult expert systems without hav9. Kidd, A.L, and M.B. Cooper. "Man-Machine Ining to do so in a vacuum; instead , they can terface Issues in the Construction of an Expert Sysbe run alongside other systems, expert or tem. " International J ournal of Man-Machine Studies 22: otherwise, and consulted as needed. Direct 91-102,1985 . 10. Laurel, B.K. " Interface as Mimesis," in Norman, manipulation provides the mechanism for. D.A. , and S.W. Draper (eds.), User-Centered S),stem Denavigating among symbols, choice con- sign: Nrw Perspectives on Human-Computer Interaction. structs, and systems in a manner that is intu- Hillsdale, N .J.: Lawrence Erlbaum Associates, 1986. OJ pp. 67-85. . itive, transparent, and productive. REFERENCES 1. Potter, A. "Soft ware Developme nt Under Windows," COMPUTER LANGUAGE 5(1): 36-44, J an . 1988. 2. Stern, H.L "Comparison of Window Systems," BYTE, No v. 1987, pp. 265-272. 3. Berry, D.C. , and D.E. Broadbent. " Expert Systems and the Man-M achine Inte rface," Expe rt Systems 4(1), Feb. 1987. 4. Potter, A. " Interfacing th e Expert: Characteristics and Req uireme nts for the User Interface in Expert Systems ," in Prorfl'dillgs oj the Third Anllual Conference all Artificial IlI telligPl/rP Jar S/Ja(e AP/J/icatiolls, Nov. 2-3,

11. Rubinstein, R., and H .M. Hersh. The Human Factor. Burlington , Mass. : Digital Press, 1984. 12. Reichman , R. " Communication Paradigms for a Window System," in Norman, D.A., and S.W. Draper (eds.), User-Centered System Design: New Perspectives on Hu man-Computer Interaction. Hillsdale, N.J.: Lawrence Erlbaum Associates, 1986 , pp. 285-3 13. 13. Hutchins, E.L, J.D. Hollan , and D.A. Norman. " Direct Manipulation Interfaces," in Norman , D.A., and S.W. Draper (eds.), User-Centered S),stem Design: New Perspectil.es on Human-Computer Interaction. Hillsdale, N.J .: Lawrence Erlbaum Associates, 1986, pp. 87-124. Andrew Potter is a senior software engineer with General Digital Industries Inc., Huntsville, Ala.