A Mobile User Interface for Threading, Marking, and Previewing Email Li-Te Cheng, Daniel Gruen Collaborative User Experience Group IBM T.J. Watson Research Center Cambridge, MA 02142 {
[email protected],
[email protected] } Abstract Email has become a vital center for managing work, and has a growing role on mobile devices. However, the challenge of creating user interfaces to support the complex ways people manage their email are exacerbated by the limitations of mobile devices, notably in screen size and input mechanisms. We address this challenge by enhancing a PDA’s inbox with user interface tools for viewing and navigating threaded email discussions, marking messages with gestural pop-up menus, and previewing content with one-line “ticker-tape” displays. Our paper describes these user interface contributions, the underlying implementation of our prototype, and reflects on reactions from prospective users. Introduction Email has become a “home” for many people: a center for organizing, remembering, and conducting work. Electronic conversations are being used as alerts, invitations to events, background information, contact lists, records of past activity, pieces of business workflow, and components of electronic commerce [3]. As the role of email has grown over the years, the need for people to access their email while away from their desktop computers has grown as well. This can be seen in the growth of tools such as the RIM Blackberry, Handspring Treo, Symbian-based devices, Palm, Danger Hiptop, and connected Pocket PC devices that enable people to be informed of new mail as it arrives regardless of where they are, and lets them exchange messages with each other. While useful for monitoring and replying to incoming mail, most traditional mobile email clients are limited, scaled-down versions of desktop email applications, and do not address the complex ways people manage email on desktop computers [3][9][7]. For example, people frequently read through a number of related messages before replying. They consult one message – often after a search of some sort – for information or content to be included in one they are writing. They copy from one, or from several messages, into a new one. People also use their mail as a dynamic contact list, searching for previous messages as a way to locate a person’s name, phone number, or email address. And people use their mail as a personal knowledge base, for example by searching through old mail to find a resume someone sent them or to remind themselves when (and why) a decision was made. Mobile devices exacerbate the email management challenge in a number of ways. Limited resolution and small screens are significant obstacles. They make it difficult to keep several messages open at once, making traditional windowing schemes unsuitable. Navigation among items scattered around the inbox often results in laborious and excessive scrolling and tapping. Input mechanisms using styli, onscreen keyboards, or small thumb keypads make text entry slower and more errorprone than traditional keyboards. Processors, storage, and network capabilities are generally less capable, often resulting in a less responsive, lower fidelity experience than on the desktop computer. No longer stuck at a desk, the mobile user is usually “on the go”, with only seconds, not hours, to spend dealing with email per session. While mobile device technology, multimedia, form-factor design, networking infrastructure, and middleware services continue to advance, we focus on improving the software user interface of the mobile email client itself. With time being a scarce resource for mobile users, we explored the user interface possibilities around the mobile inbox, the likeliest place where the most messages can be dealt with at a glance. In this paper, we describe related work, introduce our PDA mobile email prototype, MoMail, and describe its implementation. We then present MoMail’s user interface contributions around managing messages in the email inbox and conclude with reactions from prospective users and directions for future work. Related Work PDAs on the market have available email clients, as do many smartphone devices, and some mobile devices are devoted to email. They all offer messages in an inbox, with some functionality to manage messages therein, usually through traditional context menus. While messages can come in many flavors beyond plain text (e.g. spreadsheets, voice, photo, video), the inbox is often presented as a list of subject lines, senders, and dates. Users can act upon messages, such as delete and mark as important, through context menus or buttons.
The mobile inbox tries to present a large number of messages in a limited amount of space. Besides information content, the inbox must show user interface widgets. A notable example is Kamba et al’s work in semi-transparent widgets to provide a reasonably sized user interface on top of information content. Although such widgets provide a large area for user interaction without obscuring too much content on the screen, there are some issues on how to select content behind the widget [5]. With so many messages to present, a map may be an interesting alternative to the inbox list, such as Dunlop and Davidson’s use a Starfield visualization to map information as icons on an x-y grid [4]. However, the salient features of email messages are text-based and difficult to map to a discrete set of icon types and an x-y coordinate scale. Email visualization on the desktop often takes the form of conversation threads. Email messages are often related by being part of the same discussion thread or activity, yet they often appear scattered as separate items in a traditional inbox list view. This can make it difficult for people to locate related items, to notice if an issue raised in a message has already been handled by someone else, or to follow a discussion through to get up to speed. This can be particularly difficult when using a portable device with a small screen as there is less opportunity to see related items in a large on-screen list. Furthermore, due to the effort required to enter text on many such devices, the cost of responding to a message that has already been handled by someone else is greater. An email thread helps group many messages into a single conversational context, and thus is a meaningful yet compact construct worth using in a mobile email inbox. Newsgroups and applications like Microsoft Outlook provide inbox views that present hierarchical lists of threads organized by topic. Research such as Bellotti et al’s “thrasks” group related messages, files, and tasks into threads [1]. Rohall and others have a variety of thread map visualizations to depict all the messages in a thread and its entire hierarchical structure [8]. One technique is the use of color highlighting, where a selected message is shown in one color, and the other messages in the thread are shown in a secondary color. Such a technique can enhance the traditional appearance of an inbox list view, and the graphical resources can be easily met by any current generation PDA. With regards to input interfaces requiring little screen real estate, PDAs and desktops offer a variety of methods such as voice commands, pen-stroke recognition, and stylized small widgets. Another example is the pie menu, a circular menu arrangement of icons, featuring a compact screen footprint, and using spatial icon placement to support memorization of directional strokes [6][2]. We consider a variation of pie menus in our user interface later on in this paper. An Overview of MoMail Our chief goal in developing the MoMail prototype is to provide a user interface that supports email inbox management, where actions must occur within the short time frame needed by a mobile user. This is accomplished in a number of ways. We minimize the amount of time-consuming screen switching by instrumenting the one screen with many pieces of functionality. We use the notion of email threads to help organize the mobile inbox, and as a means of navigation. We also leverage a variant of pie menus to mark and act directly on messages in the inbox. We do support email features beyond the inbox (e.g. composition, full-screen reading) but this is a topic for another paper. Like most email applications, MoMail’s default screen is the “Inbox screen” (see Figure 1). This screen contains the traditional inbox list of email messages sent to the user, with each message being identified by the sender, the date, and the subject. But, this screen is also enhanced with a number of features to examine message relationships and manage them, without leaving this screen. Thus, the inbox becomes more than a browsing widget; it also becomes the mobile user’s workspace to triage email messages. More time is focused on marking, deferring, replying, and deleting many messages in the queue at once, and less time is spent reading entire messages one by one. Implementation Our prototype was created using Macromedia Flash and C++ on a Pocket PC 2002 PDA. Flash was selected for its relative ease of use in creating compelling prototype user interfaces, while C++ was used to provide direct access to supporting infrastructure. Figure 2 summarizes MoMail’s implementation. The user interface was built in Flash, and communicates with a C++ backend piece. The C++ backend receives requests from the Flash user interface, retrieves email message data from a DB2 Everyplace database and leverages addressbook data from the PocketPC’s Pocket Outlook data store. All of these pieces operate on the Pocket PC PDA, and are packaged to appear as a single standalone program rather than a browser-based application. The total footprint is about 450k, with around 100k devoted to the Flash user interface, 100k for the C++ backend, and 250k for supporting libraries.
Figure 1: The inbox with thread color highlighting and mini-map (a) A single tap on Scott’s email reveals two replies (b) Navigating to the first reply (c) Navigating to the second reply (d) An example of how a reply in a complex email thread gets highlighted
Figure 2: MoMail’s Architecture
The Flash user interface and the C++ backend communicate via a custom XML messaging protocol. During development, this allowed us to experiment and add new commands between the UI and backend quickly. Our command set has a close mapping to user interface functionality, including commands for retrieving new email, for marking messages as urgent and for deleting, creating and sending new messages. While simple and readable, an XML protocol does require processing resources spent on text parsing, and a production system should definitely consider a binary format. The C++ backend uses direct API calls to store and retrieve data from the DB2 and Pocket Outlook stores. The DB2 Everyplace database provides a first-class relational database engine to help organize email messages efficiently. The Pocket Outlook data store enables access to existing information on the PDA itself. MoMail can operate without network access, by presenting only the data copied into the local database. The C++ backend also talks with an email agent that acts as an intermediary with the mail server. Again, we used an XML protocol (similar to the one between the UI and backend) to query and retrieve messages from the agent. The main purpose of the agent is to compute email thread information, so that the PDA only needs to copy the data over and render it, and to select a manageable subset of messages from the mail server for replication onto the PDA. The supporting C++ infrastructure was built in parallel with the user interface. Developing the XML protocol was the only major place to coordinate efforts between the backend and the user interface. Since our focus was on exploring user interface ideas, our choice of Flash allowed us to iterate through interface ideas quickly, given its relative ease of creating compelling interfaces. In the following sections, we present MoMail’s four main user interface contributions: email thread highlighting with color, a thread mini-map, one-liner previews, and gestural pop-up menus. Color Highlighting Email Threads MoMail retains the list-like view of a standard inbox, as opposed to the newsgroup-like notion of collapsing messages around discussion topics. Email threads can have a strong sense of temporal and personal urgency (e.g. responding to a high priority question from a manager before the end of the day), and thus a list layout helps present some of this urgency using timebased ordering. However, it does scatter messages from a thread around the inbox, so to show thread relationships, we use color highlighting. Color highlighting helps identify entire collections of conversational context, allowing users to skip over entire groups of irrelevant messages, and focus on the important discussions. Tapping any message item in the inbox will “select it” (whereas double-tap “opens” it, as discussed later), which highlights the message blue. All the message’s replies (direct replies and replies to other replies) and parents (direct parents and indirectly related ones) are highlighted in orange. For example, in Figure 1 (a), the user selects Scott’s email by tapping on it once with the stylus, which colors his message blue. The two replies to Scott’s email (from Vladimir and Linda) immediately get highlighted in orange. This shows that Scott’s email is at the beginning of the thread. Tapping on Vladimir’s message selects his message in blue and highlights the preceding and following message in orange, as shown in Figure 1 (b). This shows that Vladimir’s email is in the middle of the thread. Tapping on Linda’s message selects her message in blue, and highlights the preceding two messages in orange, as shown in Figure 1 (c). This shows that Linda’s email is at the end of the thread. A Mini-map to Navigate Related Messages While color highlighting shows a group of messages, it does not reveal the real hierarchical structure of responses to a message. Instead it simplifies a thread as a linear sequence of messages sorted by date. We exploit this simplification by presenting a compact “mini-map” interface to show the existence of other messages in the thread before or after the current one, and to allow the user to step backwards or forwards through the thread. The mini-map also gives a general sense of how many items precede or follow the current message (none, one, two, or many). Color highlighting is also limited to coloring items currently in view, whereas the mini-map shows items off-view and lets the user jump immediately to threaded messages that would normally take much scrolling and searching. The mini-map is best illustrated by reusing the same example about Scott, Vladimir, and Linda’s email thread. Figure 1(a) shows a selected message from Scott and two replies from Vladimir and Linda. The mini-map is shown in the bottom right
corner of the image, depicting the current position in this thread as a circle (the currently selected message) connected to two circles stacked on top of each other (the two replies). Tapping on the stack of two circles with the stylus advances to the next message in the thread (Vladimir’s), as shown in Figure 1 (b). The mini-map now shows a middle circle (representing Vladimir’s message) preceded by a circle (Scott’s original email) and followed by another circle (Linda’s reply). Tapping on the left circle would navigate back to Scott’s email (the email prior to Vladimir’s). Tapping on the circle on the right selects the next message in the thread, Linda’s email, as shown in Figure 1 (c). The minimap now shows the end of the email thread, with the last message (Linda’s) represented by a circle preceded by a stack of two circles (Scott and Vladimir’s). The mini-map is only able to depict email threads with none, one, two, or “many” items. Figure 1 (d) shows a long complex thread with many messages. Jorge’s email is currently selected (in blue), and the mini-map shows his message preceded by one message (Dan’s) and followed by three or more messages. The one preceding message is shown by a circle, while the following messages are represented by a single “clump” of three circles. One-Liner Preview Thread highlighting and mini-maps provide inbox cues for the user to identify relevant and irrelevant discussions. A one-line preview of the beginning of a message, combined with sender, date, and subject fields already in the inbox view, allows users to identify relevant and irrelevant messages. By embedding the preview in the inbox screen, the user does not have to waste time switching contexts just to take a glance at the contents of a message. Instead of taking up screen real-estate for a typical preview pane, we designed a control that allows the subject areas in the list to be turned into a one-line horizontal scrolling field in which the body text of the message can be displayed and scrolled, in a manner similar to an on-screen ticker tape. The user can control the scrolling of the list in various ways, such as by clicking or pressing with a stylus on arrows at the side of the scroll area or "grabbing" and dragging within the scroll area. The user can also dismiss the scroll area, returning the item to the way it appears in the normal list. For example, consider the snapshot of the inbox screen in Figure 3 (a), where Scott’s email is selected in blue. Tapping the “eye” icon by the subject line (“Thoughts on the UIM Proposal?”) replaces the subject with the one-liner preview, as shown in Figure 3 (b). The preview shows the beginning of the "body" of the message (“Please send me your thoughts…”). The user can scroll the text by pressing on the arrows at either side of the scroll area, or by grasping and dragging the text within the preview area, as shown in Figure 3 (c). This way of controlling text in scrolling "ticker tape" style windows exists in other applications as well. Our contribution is having such a ticker area appear in-place among other items in a list, without affecting the other items. It is possible to have such scrolling text areas appear for multiple items in the list at the same time, as desired by the user, as demonstrated in Figure 3 (d). Each one-liner preview is invoked and controlled separately. Clicking on the eye icon again would dismiss the scroll area, and display the subject information as it initially appeared in the list. Gestural Pop-Up Menus In addition to browsing the inbox for messages, users typically perform actions on email in the inbox, such as marking a message as urgent, marking for deletion, forwarding messages, and replying to them. We did not want to take up space with a toolbar, nor did we want to make users to tap and wait for a context menu to pop-up. Instead we preferred mapping stylus strokes with menu actions (like quickly marking a message for deletion). On the other hand, we want to provide a visual guide to help users memorize the strokes. This suggested the use of pie menus. We designed a “gestural pop-up menu” interface component allowing users to mark an item for deletion through a quick dragging gesture (using the mouse or a stylus) to the edge of the display (corresponding to the notion of "throwing the item out") or mark it as urgent by dragging upward (corresponding to the notion of "elevating it" to "higher" importance). Additional possible actions are displayed and available as well. The component was designed so options appear to the left and above the initial stylus click, where they would be less likely to be blocked by a hand holding a stylus than traditional desktop-style pop-up menus which appear below and to the right of the mouse click.
Figure 3: Inline preview in the inbox (a) A single tap selects Scott’s message (b) A single tap on the “eye” icon left of the subject line, “Thoughts on the…”, replaces it with a preview of the body of message (c) Tapping the triangles beside the preview scrolls the message (d) Several message bodies can be previewed at once in this manner While inspired from pie menus, our gestural pop-up menus are non-circular and more like a hybrid of pie menus and a toolbar: “memorizable” elements are closer to the pen-tip to the immediate left and top (“delete” and “urgent”), while the remainder of the strip is a horizontal layout of icons much like a toolbar. Another distinction is that our gestural pop-up menus, which consist of six menu items, are carefully laid out to minimize any obscuring of the inbox content, whereas traditional pull-down context menus and circular pie menus with the same number of items would block more inbox content. Figures 4 (a)-(d) show an example of marking a message for deletion. Figure 4 (a) shows the “Hot Deals” message being selected by a single tap of the stylus. In Figure 4 (b), the gestural pop-up menu is invoked by pressing the stylus down on the dot (the menu disappears as soon as the user stops pressing). In Figure 4 (c), dragging the stylus to the left onto the “X” icon, marks the item for deletion. Selecting the “X” icon also puts up the “Delete” tooltip over the pop-up menu. After dragging to the left, the stylus is lifted and the item is then shown to be marked for deletion as shown in Figure 4 (d). Similarly, in Figure 4 (e), pressing on the dot and dragging the stylus upwards to the “!” icon marks the message as urgent. Figure 4 (f) shows the item after it is marked as urgent.
Figure 4: Using gestural pop-up menus (a) A single tap selects the message from “Hot Deals” (b) Holding the stylus down on the grey dot by the message brings up the pop-up menu (c) Sliding the stylus to the left selects the “Delete” choice (d) The message is now marked for deletion (e) Similarly, sliding the stylus up selects the “Urgent” choice (f) The message is now marked as urgent
Discussion and Future Directions While a formal user evaluation is planned for the future, we did collect informal reactions from a large group of prospective users. The prototype was shown to more than 50 people through demos during three consecutive days in a public event, as well as during visits to our research lab over the course of several months. Feedback was positive, and respondents felt MoMail’s user interface ideas were usable and desirable in commercial offerings. Many people saw the need to directly address email user interface issues on mobile devices, and that it would directly influence whether users will bother using mobile email on a regular basis. People saw the value of using threading to manage mobile email. There was a great appreciation in the aesthetics of the user interface design, and the integration of threading, navigation, gestural menus, and preview in the same inbox screen. Choosing to use Flash as a user interface and C++ as a backend allowed rapid development and a clear division of labor. Although we found overall performance tolerable, we believe developing everything in C++, and with a binary protocol rather than XML, would yield better user interface responsiveness and scalability. Our separation of a user interface front end from an infrastructure-oriented backend, suggests a web portal implementation, where the browser runs the user interface, and talks with a server hosting the remaining pieces. Some of our respondents concurred on this and pointed out that web portal components are often displayed in a small region on a larger screen. Thus, MoMail’s compact inbox interface might be a good fit in such a situation. Lists of items, not necessarily just messages, could be enhanced with a gestural-pop menu to allow users to quickly mark or act on items. There were suggestions for improvements. Mini-map navigation could be more fine-grained than its simple one-step forward and backward approach. The one-liner preview should provide automatic scrolling in addition to manual scrolling. Folder management should be supported. We need to address how to clearly identify read and unread messages. A number of people would like to see MoMail to step beyond email and consider integration with other common functionality found on PDAs, such as calendaring, to-dos, and addressbook management. The gestural pop-up menu is oriented for right-handed people, and we should consider an alternate version for left-handed users. There are other mobile device form factors than just PDAs, such as smartphones. A smartphone design would have to consider how to present different types of messages in the inbox than just email, such as voicemail and “multimedia-mail” offered by some 3G phone services. Many smartphones still do not have touchscreens, and thus gestural pop-up menus and mini-map navigation would have to be mapped against physical buttons. In conclusion, as mobile email use grows, the mobile inbox’s user interface needs to support more sophisticated ways to manage messages than just reading and replying to an endless list of items. Limitations such as screen space and input mechanisms, combined with the short attention afforded by mobile users make designing such an interface quite challenging. Our approach is to consolidate functionality into one inbox screen. We use email threads to group messages into conversational contexts through color highlighting, and derive a mini-map to navigate among related messages. Inline previews add another cue for users to judge messages without leaving the inbox. Gestural pop-up menus allow one-stroke actions on messages in the inbox without a learning curve. There remains more work be done. This includes validating the user interface with a formal user study, supporting more ways how people cope with email, and addressing non-PDA devices. Acknowledgements We would like to give special thanks to our interns, Devon Rueckner and Jorge Ortiz, who contributed greatly to the creation of MoMail. References [1] V. Bellotti, N. Ducheneaut, M. Howard, and I. Smith, “Taking Email to Task: The Design and Evaluation of a Task Management Centered Email tool,” Proc. ACM CHI ’03, Ft. Lauderdale, FL, April 5-10, 2003, pp. 345-352. [2] J. Callahan, D. Hopkins, M. Weiser, and B. Shneiderman, “An empirical comparison of pie vs. linear menus”, Proc. ACM CHI ’88, Washington, D.C., 1988, pp. 95-100. [3] N. Ducheneaut and V. Bellotti, “E-mail as Habitat: An Exploration of Embedded Personal Information Management,” ACM interactions, vol. 9, no. 5, 2001, pp. 30-38.
[4] Dunlop, M., and Davidson, N., “Visual information seeking on palmtop devices,” Proc. HCI ‘00, vol. 2, September, 2000, pp. 19-20. [5] Kamba, T., Elson, S., Harpold, T., Stamper, T., Sukaviriya, P. “Using small screen space more efficiently,” Proc. ACM CHI ‘96, Vancouver, Canada, 1996, pp. 383-390. [6] Kurtenbach, G. and Buxton, W. “User learning and performance with marking menus”, Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, Boston, MA, 1994, pp. 258-264. [7] W. Mackay, “More Than Just a Communication System: Diversity in the Use of Electronic Mail,” Proc. ACM CSCW ’88, Portland, Oregon, 1988, pp. 344-353. [8] S. Rohall, “Redesigning Email for the 21st Century Workshop Position Paper,” IBM Technical Report 02-17 http://domino.research.ibm.com/cambridge/research.nsf/pages/papers.html, (Current 2003). [9] S. Whittaker, and C. Sidner, “Email Overload: Exploring Personal Information Management of Email,” Proc. ACM CHI ’96, Vancouver, Canada, 1996, pp. 276-283.
