Multimedia Client Implementation on Personal Digital Assistants Markus Lauff and Hans-Werner Gellersen TecO - Telecooperation Office - University of Karlsruhe Vincenz-Prießnitz-Str. 1 - 76131 Karlsruhe - GERMANY Phone +49 (721) 6902-69, Fax +49 (721) 6902-16 e-mail: {markus, hwg}@teco.uni-karlsruhe.de Abstract. Their small size and ease of use makes Personal Digital Assistants (PDAs) very attractive for inclusion as clients in mobile multimedia applications. Alas, the cost for integration of PDA in distributed multimedia systems is high, as the resource limitations on PDA require client adaptation, and as software development on PDA is comparatively expensive. We present PocketWeb, a tool facilitating PDA integration in multimedia applications based on the web protocols while freeing application developers from PDA specific software development. We have applied PocketWeb for integration of PDA as multimedia clients in MILLION, a distributed multimedia system developed in an ESPRIT project. Keywords. Multimedia Client, Personal Digital Assistant (PDA), Mobile Computing, Mobile Multimedia Applications, Multimedia Software Development.
1
Introduction
The new generation of small computing devices such as the Apple Newton MessagePad, the US Robotics Pilot or the Psion Series 3c are currently primarily used in personal stand-alone applications, for example as electronic filofaxes. Now, their small size and ease of use also renders them attractive and potentially useful as clients in networked applications in mobile environments. In fact, a range of applications have already been reported, demonstrating the useful integration of Personal Digital Assistants (PDA) in networked applications, for instance in the health care domain or in field engineering [1,2,3]. The integration of PDA in networked applications, though, remains difficult and expensive for three reasons: • Resource Limitations. PDAs impose limitations on client functionality because of comparatively limited user interface, memory and computing resources. These limitations imply a high adaptation cost for adapting a client to the available resources, for example for development of user interface components that require less screen real estate. • Network-level integration. The facilities for connecting to networks vary with available PDAs, but in general are rather low-level, thus implying additional cost for realizing the connectivity required for a PDA multimedia client.
• Expensive PDA Software Development. PDA software development is comparatively expensive for a number of reasons. PDAs in general come with their own development environments [e.g. 4,5,6]. These environments have not yet matured to the efficiency and reliability available on standard platforms. Further, there is a lack of experience in PDA software development in general, and PDA software developers are hard to find. Last not least, software components developed for standard platforms are difficult to reuse or even to port to PDA, and reusable PDA software components are not yet available to any noticeable degree. In this paper, we present the tool PocketWeb as a generic solution for integration of PDAs as multimedia clients in mobile multimedia applications. Pocket Web facilitates integration of PDA in applications based on the World-Wide Web protocols. From an end user perspective, PocketWeb is a web browser on Apple Newton MessagePad. From an application developers perspective, though, PocketWeb enables PDA integration in networked applications without requiring software development on PDA, thus reducing the cost of PDA integration effectively. Regarding client adaptation to limited resources on PDA, PocketWeb embodies a number of built-in adaptation strategies based on a thorough analysis of the design space for a multimedia client on PDA. In the following section we will discuss resource limitations on PDA, and the options for handling them. This discussion leads to consideration of pre-processing of multimedia information for a PDA client. The discussion of the design space for a PDA multimedia client extends to consideration of network-level integration in section 3. In section 4, we will describe the PocketWeb tool and briefly compare it to related work. Finally, in section 5 we will describe the application of PocketWeb to integration of PDA in MILLION, a large distributed multimedia system for the tourism sector currently being developed in an ESPRIT co-funded project [7]. The paper concludes with a brief summary and outlook on further development.
2
Resource Limitations and Multimedia Preprocessing
Today's information system clients such as WWW browsers have lots of built-in modules to display the different types of multimedia information. Apart from the weak computing power, discussed later in this document, PDAs have several other restrictions that makes it difficult to present multimedia information. These are: • small display • limited colour depth • small memory • weak computing power • poor digital audio interface 2.1 Restrictions concerning the display of PDAs The usual size of displays known from Desktop computers is between 14" to 21" and the size of the display from Laptop computers is about 12". Compared to theses values the size of PDA displays, from about 3" to 6", is very small.
Most of the information available on information systems are thought to be displayed on a display with at most 80 columns. It is no problem to reformat plain text to a smaller display, for example to 50 columns but this is not as easy for tables or multimedia information as pictures or video. 2.1.1 Tables: Tables need a lot of space because they use additional characters to separate the single columns. This leads in almost every case to a larger width than the available 50 columns on a PDA. There are several strategies to display a table on the small display: 1. two scrollbars Usually the user has the possibility to navigate up and down through the document. In addition to this vertical scrolling a horizontal scrolling is used to view the table.
This approach is very inconvenient for the user because it is necessary to scroll for every row from the left to the right margin to view the table. Also it is difficult to overlook the whole table and to remember the position within the table. If the thirst column can be locked, that means the first column is not scrolled and always shown, it is easier to remember the position within the document. The disadvantage is that the space is always used for the first column. 2. reformat the single columns This approach is used in common HTML viewers. The available space is divided into the number of used columns and the text is reformatted to fit into the column.
This above table is divided in 5 columns of almost the same width. The single columns are separated with one character as delimiter. So the average size of a single
column is about 9 characters. This means that usually only a maximum of two words fit on a single line within the column. A lot of space is lost if only one word fits on the line because the second word is too long. In this case we can force a word break to use the remaining characters. This is shown in the fifth column. Without a forced word break, that makes the document harder to read, the length of the document growth very fast. 3. Don't use a table. Reformat the table to multiple sections. The idea that leads to this approach is, that a table contains in every row the information that belongs under a given topic to a given point. This is not always applicable but will do for the most cases. The following example shows the original table
and the decomposition to multiple sections
This approach is useful for tables where it is not necessary to compare the different rows.
4. Use floating columns This approach uses the basic characteristics of a table. A table is a formation with a separation into columns and rows. It is not essential that the columns always start at the same horizontal position. It only must be recognisable for the user to which column the current text belongs. If the display supports colours a better separation of the columns can be reached by the use of different background colours. This approach results in a very uncommon structure and is therefore only acceptable on very small displays. 5. Ignore the table At first sight this approach may look mad or trivial but on a closer look it is at least reasonable to ignore the table construct in some cases. This approach can only be used in addition to another approach to display tables where it is not possible to ignore the table construct.
This leads to the question which table constructs can be ignored? In many cases tables are used to have a nicer presentation of the information. But on a small display this brings the contrary result. Deciding where it is possible to ignore the table or not can only be done by an heuristic algorithm.
This example shows a typical case where tables are not necessary. The table is only used to separate headers in a pretty way. In addition to these approaches it is always possible to reduce the size of the font to increase the number of characters per line but this isn't very useful on a PDA because PDAs usually still use small fonts. 2.1.2 Pictures: The problem with the pictures is, that the difference between the PDA and a PC is even more worse than with the before mentioned tables. The common resolution of a PC display is 800 x 600 pixel or more with at least 256 colours. For example the Apple Newton MessagePad only has a monochrome display with a resolution of 320 x 240 pixel. Nevertheless it is necessary to be able to display as many multimedia information as possible in any way because they may contain inalienable information. On the other hand there are a lot of useless pictures in documents for example pictures displaying bullets or delimiters. Displaying images can fail because of several limitations (computing power, memory, size of the display) this sections describes how to solve the problems with the small display the other problems are described later in this document. In every case where the picture has a higher colour depth than the display the image must be dithered before it can displayed. Similar to the procedure with the tables there are two main strategies to display images. 1. Use two scrollbars The easiest way to display a large picture is to use a horizontal and a vertical scrollbar to view the picture. There are two problems occurring with this solution: • it can be difficult to imagine the complete picture • the PDA must have enough memory to store the whole picture and enough computing power to scroll through the picture 2. Scaling the image An obvious solution is to scale the image to fit on the display of the PDA. But depending on the original size and the content of the image the scaled image can be useless and for that reason a fallback to the previous approach should be possible. 3. Ignore the image This approach is only applicable if the image contains no information, for example bullets or delimiters. A special algorithm has to decide if an image is worth to be displayed or not.
2.2 Restrictions because of the small memory and the low computing power The common PDA does not have a secondary memory for the storage of programs and data. Therefore all programs and data must be stored in the primary memory. With a usual size of about 2 - 4 MByte the memory of a PDA is a very restricting resource. 2.2.1 Processing and displaying images For image processing on the PDA there are three things to do: 1. load the image 2. decode the image 3. convert the image 1. load the image As a matter of course we need the memory to store the image if we want to load it. But this is not always possible. Large images or truecolour images may need to much memory so they can not be stored in the original format. If this happens it is necessary to process the image on-the-fly. Processing on-the-fly means that the small parts of the image are decoded and converted immediately when they arrive on the PDA and stored in a new format. 2. / 3. decode and convert the image Decoding a picture from the GIF or the JPG format to a raw pixel-format is a hard job for a PDA. Particularly decoding a JPG image is currently not done in a few seconds. In addition to the required computing power the image grows in size while it is decoded and more memory is necessary to store the decoded picture. Therefore the image is dithered (reduce the colourdepth) and scaled before it is stored. Conclusion loading, decoding and converting images In many cases it is not possible for the PDA to load, decode or convert a image because of the limited resources. Why should not another machine process the image for the PDA? The PDA is connected to a network with many powerful machines, why he should not use the computing power and memory from another machine to preprocess the information? The following section describes several possibilities of preprocessing information for the PDA. 2.3 Preprocess information to eliminate limitations A PDA is one of the smallest machines in the world of information. It is very comfortable to use a PDA to view as many information as possible. But that is the problem: „as many information as possible„. If the PDA is responsible for decoding, formatting and processing the requested information the limits are reached very soon. Now it is conceivable that the PDA is able to give some jobs to another machine on the existing information network. We can now build several models depending on what is done by the PDA and what is preprocessed from another machine. The machine that is responsible for preprocessing is viewed as a proxy.
2.3.1 The PDA only displays prebuild screens The PDA send the request to a special PDA-Proxy. In addition to the request the PDA sends some information describing his possibilities, for example • size of the display (320 x 240 pixel) • resolution (50 dpi) • number of colours (monochrome) • digital audio capabilities (no) • memory available (50 kByte) The proxy now computes the page, depending on the possibilities of the PDA and sends the result as a bitmap to the PDA. If the page contains large elements (tables, pictures) the proxy generates a preview and integrates this preview into the page. The user than is able to request the complete objects as an independent page. Advantages: • There are no more restrictions concerning the original format of the information. The limits are defined by the memory, the type of the display and the digital audio possibilities of the PDA. The computing power of the PDA is not the restricting factor. Disadvantages: • The PDA is not able to request new pages without a proxy • A page is stored as bitmap and may need a lot of memory depending on the resolution and number of colours from the PDA Operational area: • PDA with low computing power • low resolution and a small number of colours, because the size of the computed bitmap 2.3.2 The PDA can submit supported features with the request In addition to the request the browser sends his capabilities concerning the different HTML tags to the proxy. The proxy examines the HTML document and converts from the browser unsupported elements to supported elements. Example 1: If the browser is not able to format tables the proxy can restructure the table to a preformatted textarea using monospaced fonts. Example 2: If the browser can not display inline graphics the proxy can replace pictures that are used as bullets in front of a list with a usual bullet symbol. Advantages: • The proxy can replace some elements that are not supported by the browser with an at least readable alternative. Disadvantages: • The proxy must parse and modify the HTML document • If the browser uses heuristical algorithm to remove unsupported elements the information can be corrupted
Operational area: • The browser on the PDA is able to format standard items and the proxy is especially written to enhance the functionality of the browser 2.3.3 The PDA can submit supported formats with the request With the request the PDA sends the supported formats or desired properties (size, number of colours) to the proxy. The proxy examines the requested data and tries to convert the information to one of the supported formats. If the proxy is able to recognise the identification of the browser sent within the header of the request, there is modification of the HTTP necessary. In this case the PDA needs not to sent the desired properties or formats, because the proxy knows these facts from the identification of the browser. For example: If the PDA only supports GIF images at a given maximum size and requests a page containing a bigger JPG-coded image, the proxy converts the JPG image to the GIF format and scales the image to the maximum size. Advantages: • The PDA is also able to work if the proxy is not available • The proxy can use standard tools for image conversions the HTML document is not modified • No modification of HTTP is necessary Disadvantages: • The proxy only enhances the functionality of the browser that are originally limited by the number of supported formats • There is no support for preview items as described above Operational area: • The browser on the PDA is able to format standard items and the functionality should be transparent enlarged.
3
Network Connectivity for PDA
To retrieve any information it is necessary that the client is connected to a source where the information is stored, usually this is an information server. In many cases the information is spread over a lot of servers that are connected through a network. There are several possibilities to connect a PDA to a network. Almost every PDA has other facilities to connect to a network. For example the Apple Newton MessagePad can connect directly to an AppleTalk network. To use the Newton with an TCP/IP network a serial line to a PPP1-server is necessary. A US Robotics Pilot has no possibility to connect to a network, the Pilot only has a serial connector that can be used to connect to another machine. 1
Point to Point Protocol
A Psion Serie 3c also has only a serial connector with an overlying TCP/IP protocol to connect to a PPP-server or an IrDA infrared channel to connect to another Psion Serie 3c At the moment there is no PDA that is able to support a direct TCP/IP connection to a network and so in almost every case a serial line is used to connect to an network provider. With a PDA the usual mobile environment uses a modem to connect through the telephone line to a network provider. If we use a mobile phone we can speak from a real mobile environment because there are no restrictions we have to make against the environment. Using a mobile phone is surely the most convenient solution but it is also the most expensive one. A connection is either online or offline. Online means that we have a physical connection and so we can retrieve new information from our information provider. The opposite to this is offline meaning that there is no possibility to retrieve new information. A online connection always means maintaining a connection to an service provider and therefore producing costs. But an access to an information system is only meaningful if the costs are proportionate to the outcome. Because of this we have developed another interesting connection type besides online and offline that tries to use a few new elements to optimise the ratio between costs and outcome. We name this approach „semi-online connection„ meaning that we try to present the user the information like having a real online connection. 3.1
„Semi-Online„ Connection
If the standard request is used every information is requested with an own request statement. If the user wants to load the following information he has to open a new connection to do the request. In many cases it is possible to guess the information the user wants to have and to request the information in advance within the current connection. Whenever it is possible to modify the original information, it is suitable to add extra information to group the information to blocks which can loaded altogether to reduce the number of necessary connections. For this we have introduced a new HTML tag „REQUIRE“. The value of the tag is a page/picture that is required to view/follow the current page. With the help of this tag we can build a tree of information. The spanning tree is called a HTML package and is requested with a single request statement.
4
PocketWeb: A Generic Multimedia Client
As a result of our analysis we have developed „PocketWeb„ as an generic multimedia client on an Apple Newton MessagePad. PocketWeb is based on HTML/HTTP and FTP and extensions developed through our before mentioned studies. The current version 2.1 is available from the TecO. Features of PocketWeb: • support of most HTML 2.0 elements • pictures • forms • flexible cache management • supports download of complete HTMLpackages PocketWeb uses optimized methods for requesting, displaying and storing multimedia objects on the Apple MessagePad. Some examples on how objects are requested or displayed have been explained in section 2,3. Now we will show how the cache is optimized for performance and an optimum memory usage. Optimized cache architecture Instead of storing the information as they arrive the information is preprocessed and than stored in an object oriented cache architecture. This means, that the information within the HTML page is bind to the procedure that is responsible to display it. This cache architecture allows the information to be displayed very fast without wasting unnecessary memory. A comparison of the memory usage of cached pages between PocketWeb and the commercial product Nethopper from Allpen has shown, that Allpen uses 2-7 times more memory for caching pages than PocketWeb depending on the number of used HTML elements within the page. Optimized requests In the current version PocketWeb works with a special subclient. This subclient is responsible for translating serial requests to TCP/IP requests and for preprocessing information as explained in section 2. At the moment we are working on a new version of the subclient that acts as an usual proxy on the net and is able to preprocess the information as requested from the client. In addition to the explained optimizations we are thinking about adding a quality of service module that has been developed at the TecO for mobile clients to access the net.
5
Application of PocketWeb
5.1 Use of PocketWeb Clients in MILLION We have refined and applied the PocketWeb tool in the ESPRIT project MILLION. The objective of MILLION is to develop a coherent set of tools and services for municipal tourism, including tools for commercial transactions. In the cities of Venice, Bologna and Khania/Crete MILLION system pilots are being installed currently. In these pilots, PDA are integrated for mobile multimedia information access. In the pilot evaluation phase, PDAs will be handed out by tourism agencies to tourists for the duration of their stay. Tourists can download multimedia information according to their personal preferences and the use the PDA in an offline mode as travel guide. This part of the MILLION project is called VIP and covers all areas of mobile access and transactions within MILLION using PDAs. With the help of PocketWeb and the ideas we describe in this document we have reached the goal to use a PDA as a multimedia client without adapting and developing new specialized software for every new area. 5.2 Other Parts of MILLION MILLION BOOK MILLION BOOK is the fundamental part of MILLION and covers the integration and development of multimedia databases. A main goal of MILLION is the integration of existing databases of any type in a for the user transparent fashion. MILLION GOLD The Dutch company Digicash integrate their transaction and electronic cash payment system ecash in the MILLION project to allow secure transactions and payments on the net. MILLION VOICE This part of the projects studies the possibilities to add voice services to the MILLION system. For example one area for voice integration in MILLION is the access to the MILLION system through usual telephone using text to speech and voice recognition. 5.3 Further Applications of PocketWeb Through the universal architecture of PocketWeb it can be reused in many areas to access multimedia information without developing new expensive software for PDAs. A very early version of PocketWeb [8] has been successful tested within the ViaPerfecta project to access a route optimisation system through a PDA.
6
Conclusion
PDAs are in principal very attractive as clients in mobile multimedia applications with their major advantages being small size, ease of use, and comparatively low cost. Integration of PDAs in distributed multimedia systems, though, implies a range of problems: handling of multimedia in the context of limited resources, connecting
PDA at network-level, and, in order to solve these issues, expensive software development on PDA. We have presented a generic solution to these problems based on the tool PocketWeb. While PocketWeb is primarily perceived as web browser for end users, it actually facilitates integration of PDA in intranet/internet applications based on the web protocols. PocketWeb embodies strategies for handling multimedia on PDA, thus minimising if not eliminating the cost for adaptation of user interfaces to PDA resources. Further PocketWeb uses the http and html protocols, abstracting from low-level communication and granting wide-spread acceptance and applicability. PocketWeb is applied within the ESPRIT project MILLION for integration of PDAs as clients in a multimedia system for the tourism sector. As part of the MILLION system, the PocketWeb-based multimedia clients will be evaluated in pilot installations in Venice, Bologna and Khania/Crete.
Acknowledgements This work was partially supported by the European Union’s ESPRIT programme in Project 20772 MILLION, whose partners are Omega Generation, Gruppo Formula, Digicash, Venis, Intracom, MUSIC and the municipalities of Venice, Bologna and Khania.
References 1. Brian Grimm, Education Research Laboratories Inc., The Williams Pocket Clinical Consultant, Resident's Medical Reference Library, Contact:
[email protected] 2. PDA for Field Engineers in Railway, Information Systems Group, Railway Technical Research Institute, 2-8-38, Hikari-cho, Kokubunji-shi, Tokyo 185, Japan. 3. FieldWorker, Cindy Park, 551 Millwood, Toronto, Canada, http://www.fieldworker.com/ 4. Newton Developer Kit, Newton Script Programming Environment, Apple Computer Inc., http://www.newton.apple.com/ 5. Psion EPOC/16, SIBO hardware and software architecture, Organiser Programming Language, http://www.psion.com/developerreference/develbackindex.html 6. U.S. Robotics Palm OS Software Development Kits, http://www.usr.com/palm/5034.html 7. Multimedia Interactive Leading Life-giving Initiative On Net, Esprit Project MILLION, Omega Generation, http://www.omega.it/million/ 8. Gessler, S., Kotulla, A., PDAs as mobile WWW browsers. Proc. of 2nd International WWW Conference, Chicago, Oct. 1994.
