A Multimedia display extension to SQL: Language Design and ...

A Multimedia display extension to SQL: Language Design and Architecture Chitta Baral, Graciela Gonzalez and Tran Son Department of Computer Science University of Texas at El Paso El Paso, Texas 79968 U.S.A. fchitta,chelis,[email protected]

March 8, 1997

Abstract We propose here a loosely-bound SQL extension that will allow users to include high-level display speci cations with an SQL query, particularly when dealing with multimedia databases. This way, the user can specify both the data that needs to be extracted and how it should be displayed. We also describe an architecture that will allow a relatively simple implementation of dynamic query browsers using the proposed query language on stand-alone applications or Web pages. We have implemented a major subset of our proposed extension and are working on a complete prototype.

1 Introduction At the time when the concept of a \database" came into being, nobody imagined the extent and variety of data that will eventually be stored in such structures. To display data as columns and rows of text is no longer sucient these days. Multimedia databases, with their rich content of images, audio, video and animations, demand much more. Multimedia information is not necessarily stored using a dierent data model. Relational databases are often used to store pointers ( lenames) of the multimedia data. However, if the user tries to use a traditional query interface or a query language such as SQL, answers to queries made to multimedia databases are displayed in the same \spreadsheet" style as normal data queries, or, at best, as a list of navigatable links to the real elements that make up the answer. For anything beyond that, the user has to rely on a programmer to write a graphical interface in C, Power Builder, Visual Basic, C++ or any other programming language. The design of special graphical interfaces becomes more dicult if we wish to access databases and dynamically display data over the Web. This has been a subject of research for some time now [NS96, Ra94]. Accessing multimedia databases over the web stresses the challenge: now dynamic web pages are a must. For example, say a user poses a query to the database of a certain museum, and the answer to his query consists of a set of color images of some paintings, plus some textual information about 1

each. The user should be able to decide how to display the answer: one painting at a time or \thumbnails" of all paintings, with the related textual information together with the images or only when requested. Things get more complicated if the answer also includes some other related information, say a voice recording of the director of the museum explaining the meaning or history of the piece of art. When and how is each piece of data going to be shown? How do we incorporate new kinds of data elements as years go by? The above example suggests that in order to better answer queries that include multimedia objects as part of the requested information, a query language needs to include the following: 1. A way to specify, for each kind of data element, some display mechanism other than the column-row \spreadsheet" style. 2. A way to specify the general layout of the data on the screen. 3. A way to navigate through the answer set. 4. A way to control and specify the length of time the objects will be displayed. 5. A way to set presentation order, like if two images related to the same record should be shown in sequence, or some audio played at the same time as an animation is played. 6. A way to explore the data: that is, trigger queries from the query so that the user can go into as much detail as desired. 7. A way to incorporate display speci cations of new and proprietary data elements. To address these challenges, we propose here a loosely-bound SQL extension, SQL+D , that will allow users to display answers to queries posed to Multimedia Databases (and even \regular" databases) in a dynamic fashion.1 The idea is to include high-level display speci cations with the SQL query. Many pre-de ned multimedia display mechanisms are provided, and the user can add proprietary display elements to the language by adding classes to the language. Also, we propose an architecture that will allow a relatively simple implementation of dynamic query browsers using SQL+D on stand alone applications or Web pages. Our approach diers from other querying mechanisms in that it is speci cally designed for querying multimedia and other databases containing complex information, and emphasizes exible, in-the-query, by-the-user speci cation of the display of the output data, rather than visual speci cation of the query, for which many visual query languages and applications have been proposed [Zl77, ACS90, Ai95]. Other eorts have emphasized data browsing and exploration [WK82, Ca96, Ko93, MS96] through the use of xed display paradigms. With SQL+D , it is possible to build applications to allow visual speci cation of queries or data browsers (and we have done that), but these applications are just a couple of the possible uses of the language. Section 2 discusses some of the assumptions we made about representing multimedia data. Sections 3 and 4 present SQL+D through the use of some extended examples. The formal syntax of SQL+D is presented in Section 5. The extensibility of SQL+D , its data exploring capabilities, and how 1

Visit our web site at http://www.utep.edu/chitta/display.

2

it can interact with external programs are also explained in that section. Section 6 presents the overall Architecture of the full implementation of the language, and describes our current advances. Section 7 presents an overview of related work and how SQL+D compares to other query display mechanisms.

2 Representing Multimedia Data Throughout this paper, we want to concentrate on displaying multimedia data. However, some words are necessary as to which data model we are using and which assumptions we are making. There have been multimedia data models proposed, or rather data models that consider nontraditional or user-de ned data types, such as the Object Oriented and Object Relational data models [St96]. These models incorporate some knowledge about the nature of the element being stored into the database schema. We do agree that such a data model will be the optimal implementation of a multimedia database. In reality, however, switching from one data model to another is very costly. The relational model can be used, with certain limitations, if multimedia elements are stored as lenames that actually contain the data. Even though SQL+D is presented here as an extension to SQL, it is not intended to be used solely with relational databases. The current draft of the ANSI/ISO revision to SQL (known as SQL3) incorporates object extensions, user de ned abstract data types, methods, object identi ers, among other features that facilitate multimedia data storage in an object oriented fashion [SQL3, Be93, BC95]. SQL+D can be used both if the \pure" relational model is in use, and the schema says nothing of the multimedia nature of the attributes, or if the Object Oriented or Object Relational data model is in use, and the schema provides multimedia information. In the rst case, we leave it up to the user to specify how attributes are to be displayed: as image, video, audio, etc., assuming that the value of the attribute is the name of a le that should be loaded for display as image, video, audio, or whatever the user indicated. On the other hand, if the Object Oriented or the Object Relational data model is in use, however, the user might leave it to the system to gure out the nature of the attributes, and only indicate where the attributes should be displayed on the screen. Also, and if the pre-de ned display speci cations concept is used to its full extent, default display speci cations could be part of the schema in an Object Relational data model or of the class speci cation in an Object Oriented data model. In any case, SQL+D needs only information about the attributes that need to be displayed. How the values are obtained or in which model they were stored is not important for the understanding of SQL+D . In this paper, we present the features of SQL+D in general.

3 SQL+D by example The philosophy behind the design of SQL+D is to balance power and simplicity. We want to provide a simple and elegant, yet powerful, high-level 4th generation language to let the user specify the way data obtained from a query is to be displayed. How much power we build into the 3

language is always a question. We don't want to end up with a new C++, Java, or Visual Basic programming language, yet we would like to get acceptable multimedia graphical user interfaces from the display speci cations. We are not proposing another query language, but rather a display speci cation language extension to SQL. Let us present a few queries to a couple databases to introduce how we can specify the display of the extracted data. In SQL+D . queries are posed using standard SQL with an additional DISPLAY clause following it, which constitutes the extension. Even though some users might enter queries directly in SQL+D , it is more reasonable to assume not all users are entirely familiar with SQL, let alone with SQL+D . That is why in our implementation, there is a graphical query constructor that hides SQL+D syntax from the user, letting them layout the main pieces of the panel graphically, which then produces SQL+D queries. However, de ning it as a language rather than an interface, gives users the possibility of embedding display speci cations in an SQL query and not have to write a single line of code to construct a screen2 .

Example 1 Consider a database for a video rental store containing movie titles and other general information of the movies, plus a movie clip and a picture of the promotional poster. Also available is a list of the actors in a movie, and other information about the actors, including their picture. The schema looks as follows: MOVIE (available, title, director, producer, date, classi cation, rating, movie clip, poster) MOVIE ACTORS (title, name, role) ACTORS (name, dob, biography, picture) where all the attributes of MOVIE ACTORS and all the attributes of MOVIE are character strings, with the exception of movie clip, which is a video (mpeg or avi), and poster, which is an image (in tif, gif, jpeg or any other standard image format). The attributes name, dob and biography in ACTORS are also character strings, biography being a long (memo) eld, and picture is an image. We would like to pose the following queries: 1. List all actors in \Gone With the Wind" with their pictures and biographies. 2. Display all action movies and their posters, allowing to play the video clip if desired. 3. Browse through the action movies, showing just their poster and title. The rst query would look as follows after it is translated to SQL+D : 2

Of course, assuming the SQL interpreter being used when the program is run is SQL+D -enabled

4

SELECT

MOVIE ACTORS.name, ACTORS.biography, ACTORS.picture FROM MOVIE ACTORS, ACTORS WHERE MOVIE ACTORS.title = \Gone With the Wind" AND ACTORS.name = MOVIE ACTORS.name DISPLAY PANEL main, PANEL info ON main(east) WITH MOVIE ACTORS.name AS list ON main(west), ACTORS.picture AS image ON info(north), ACTORS.biography AS text ON info(south). Query 1. Querying the actors in \Gone With the Wind"

This is a standard SQL query up to the WHERE clause, thereafter the DISPLAY clause is used to specify where data is to be placed on the screen. Since the DISPLAY clause operates on the data extracted from the query, only the attribute names included in the SELECT clause can be used inside the DISPLAY clause. The list of panels following the DISPLAY keyword constitute the layout statement, and determine the overall layout of the screen: the panels listed here will contain dierent data display elements, speci ed in the WITH statement. The WITH statement contains a list of display speci cations, indicating the kind of display element desired for each attribute in the SELECT clause and their relative location within the panels previously de ned in the layout statement. By default,if the user does not specify how to display all the attributes, the values of the attributes that have no display speci cation are grouped and displayed as a table in the south-most and east-most panel (lower right). This query will display, as shown in Figure 1, a scrollable list of the actors on the west half of the screen, and a panel on the east half containing a picture at the top and the actor's biography at the bottom. The picture and biography displayed correspond to the currently selected tuple from the answer set. The \currently selected tuple", is in a sense a pointer, which at rst points at the rst tuple in the answer set. Thereafter, the user can move through the answer set by using one of the four master control buttons displayed on the main panel: one each to move the pointer to the previous tuple and the next tuple, plus two that allow jumping to the rst or last tuple in the answer set. The tuples are sorted by the attribute indicated by the user in the (regular) SQL \ORDER BY" and/or \GROUP BY" clauses. In addition, there are ways to control navigation via the displayed data directly, as is the case of a list or through the use of the keywords ALL or ALL DISTINCT, explained later. In a list, the value of all the tuples for a particular attribute are listed (the names of the actors, in the above example), and the user can select any tuple simply by clicking on a particular item on the list. If this is done, the pointer in the answer set is moved to the tuple that contains the selected attribute. The user has visual cues in all the displayed elements that indicate which is the selected tuple. For example, in a list, the current tuple is highlighted in reverse video. It is a good practice to display the key attributes of the relation as a list, whenever possible, to make navigation easier. That way, the user can see some portion of the data in advance and go 5

Figure 1: Display generated by Query 1 directly to the tuple of interest, instead of blindly going tuple by tuple until the correct one is found. The second query, to display all action movies and their posters, allowing to play the video clip if desired, will be written as follows in SQL+D : SELECT FROM WHERE DISPLAY WITH

title, poster, video clip MOVIE classi cation = \Action" PANEL main, PANEL b ON main(east) poster AS image ON main(west), title AS list ON b(north), \Play" AS button ON b(south) TRIGGER `mpegplay('+ video clip +`)' Query 2. Query to see the poster of action movies.

The display corresponding to this query is shown in Figure 2. The north, south, east, and west locations are sized automatically to t the data displayed on them. Since the button labeled \Play" has a TRIGGER associated to it, double clicking on the button causes an associated command to be executed. In SQL+D , the associated command is indicated in the TRIGGER portion of the display speci cation. A TRIGGER associates either a query or an executable program to a particular attribute or display element. The associated query or application is invoked when the element that contains 6

Figure 2: Display generated by Query 2 the TRIGGER is double clicked. The TRIGGER keyword follows an attribute or xed label speci cation and is followed by the name of a stored query or an executable (\mpegplay", in the above example) and a list of parameters to be passed to it (the value of the attribute video clip, above). For example, if the user were to double click on the \Play" button while \Jumanji" was highlighted on the list, the following command will be triggered: mpegplay(jumanji.mpeg) assuming the value of the video clip attribute of the tuple that contains \Jumanji" is \jumanji.mpeg". When database attributes are listed as parameters to queries or executables, the value of the attribute(s) when the TRIGGER is invoked is(are) passed to the associated query or executable. The value of the attribute is determined by the current tuple selected in the display. The video clip could be played directly upon selection of the tuple, by substituting the last attribute speci cation in the WITH statement by the following: video clip AS video ON b(south) In this case, the user would be presented with a video display element in the bottom half of panel b, with the standard command buttons \Stop", \Play", \Pause", \FF", \RW" found in any video player. However, since the query intended to let the user decide when to load the video rather than load it automatically for display, we must use the button as a display element and trigger the video player only when the user clicks on the button. This feature allows fast retrieval of tuples for navigation, loading \heavy" 3 attributes only when the user wants to see them. The same technique could be used for attributes that contain images, audio, postscript documents, or any other element which takes longer to load than text. Finally, lets take a look at the third query, browse through the action movies, showing just their poster and title, as it is expressed in SQL+D : By \heavy" we mean attribute values that are of a signi cantly larger size than the rest of the attributes of the tuple 3

7

SELECT FROM WHERE ORDER BY DISPLAY WITH

title, poster MOVIE classi cation = \Action" title PANEL main title AS text ON main(north), poster AS image ON main(center) AUTOSKIP 2 Query 3. Query to see the posters of Action movies as a \slide show".

This query speci es a display similar to the ones shown above. However, by using the AUTOSKIP statement, the display will automatically move through the answer set showing each tuple for 2 seconds. Also, a fth master control button appears in the main panel, to allow the user to pause the automatic skipping at any point. All the above example screens were generated by our current implementation of the Display Controller, explained in section 6.3. We will now introduce a more extended example application, which we are currently implementing. Example 2 The database contains the campus map, plus maps of each oor in the dierent buildings on the UTEP campus. For each building, some pictures are available, together with a list of rooms, their descriptions, and their corresponding coordinates in the oor map. There is also some additional information and=or the http address of an associated home page (if available) for some of the data items in the database. This information is in free-text format, and includes historic highlights or detailed directions to reach the buildings, and the home pages correspond to the people, organizations, oces or departments that occupy the rooms listed. The database schema is as follows: CAMPUS (campus area, campus area map) BUILDING (building name, building picture, campus area, coordinates) FLOOR (building name, oor, oor map) ROOM (building name, oor, room name, description, home page, detail info, coordinates)

To this schema, a user can pose queries that will let them nd a particular place on campus, but also there are many (common) queries pre-de ned and available. For example, the following pre-de ned query allows to view the main campus with a list of all of the buildings, as shown in Figure 3: SELECT

CAMPUS.campus area map, BUILDING.building name FROM CAMPUS, BUILDING WHERE CAMPUS.campus area = \Main Campus" DISPLAY PANEL main WITH BUILDING.building name AS list ON main(east), CAMPUS.campus area map AS image ON main(west) Query 4. Query to display area map and list of buildings in the Main Campus. 8

Figure 3: Display generated by Query 4 In addition, if we want the buildings to be outlined on the map (polygons drawn over the building in green), we can use the coordinates attribute (adding it to the SELECT clause). It contains a series of x,y coordinates for each vertex of the polygon that outlines the building (or room, in the ROOM relation), and substitute the DISPLAY clause in the previous query by the next clause: DISPLAY PANEL main WITH building name AS list ON main(east), CAMPUS.campus area map AS image ON main(west) ALL BUILDING.coordinates AS polygon ON main(west) The keyword ALL indicates that all the attribute values of \BUILDING.coordinates" in the answer set are to be shown at the same time. As the user navigates through the answer set, the polygon corresponding to the selected tuple (building), will be lled in red (or reverse video). The rest of the polygons (non-selected) will remain unchanged, to give a visual cue as to which tuple (building) is currently selected. If the user clicks on one of the polygons, the corresponding building will be highlighted on the list. Thus, the ALL keyword turns any display element into a virtual \list" in the sense the user can navigate through the answer set via the displayed elements, just as if they were displayed in a list. Specifying the same location for both the coordinates and the area map indicates that the elements should be overlaid. Overlaying order follows the order in which the elements are speci ed in the DISPLAY clause. The idea of overlaying can be extended and applied for such things as overlaying subtitles in English over a movie with audio in Spanish. This is more complex, however, as it would require matching subtitles to the corresponding frames in the movie. But it might be practical for short video clips in a language course, for example. Now, if we want to trigger a separate query to explore a building, we change the display speci cation for building name to the following: 9

building name AS list ON main(east) TRIGGER `building maps(' + building name + `)' where the symbol `+' indicates concatenation with the actual value of the attribute building name at run time, when the user clicks on the corresponding element on the list. The query to start exploration of a building is stored as building maps, receives as parameter the name of a building, and is written in SQL+D as follows: BUILDING.building name, BUILDING.building picture, FLOOR. oor FROM BUILDING, FLOOR WHERE BUILDING.building name = + input + AND FLOOR.building name = BUILDING.building name DISPLAY PANEL main, PANEL b ON main(south), WITH building picture AS image ON main(north), building name AS title ON main(north), ALL oor AS button on b(east) TRIGGER ` oor buildings(' + building name + `,' + oor + `)' \Select Floor to Visit" on b(west) Query 5. Query to display the building picture and buttons to go to dierent oors. SELECT

Here, \input" is the name of an input variable that the query receives when it is executed. In this case, the building name. It results in the screen shown in Figure 4. The triggered query, oor buildings, looks as follows: SELECT

FLOOR. oor map, ROOM. oor, ROOM.room name, ROOM.detail info, ROOM.home page, ROOM.coordinates FROM FLOOR, ROOM WHERE FLOOR.building name = +input 1+ AND FLOOR. oor = +input 2+ AND ROOM. oor = FLOOR. oor DISPLAY PANEL main, PANEL b on main(west), PANEL c on main(east), PANEL d on b(south) WITH

oor map AS image ON c(north), ALL coordinates AS polygon ON c(north), room name AS list ON b(north), detail info AS text on b(center), ALL DISTINCT oor AS button on c(south) TRIGGER ` oor buildings(' + building name + `,' + oor + `)' home page AS button on b(south) Query 6. Query to display the oor map and a list of rooms, plus extra information. It receives two input parameters: the building name and the oor. This is a more elaborated query, since now the trigger makes a recursive call. In SQL+D , recursively calling a query is possible 10

Figure 4: Display generated by Query 5 since now the same query is invoked, but with new parameters, and the display changes to show the changing data. The generated display is shown in Figure 5.

4 Graphs and Charts in SQL+D

There is a special display class included in SQL+D that deserves separate treatment: graphs and charts. These are dierent than display elements movie, audio, or image in that the data by itself is not of a complex type, but the information it represents is. Some information is better conveyed by the use of bar, line or pie charts, like the results of surveys, the month-by-month sales of a particular retail store, or the oscillations of stock market prices. The information contained in such charts, if it were presented in a \spreadsheet" fashion, will not show the relationship among the dierent pieces of information as clearly as the chart itself. In this sense, charts are similar to multimedia information, since the information they need to convey goes beyond its actual representation in the database. Similarly, a company's organization or the intermingled air routes of an airline's scheduled ights will be dull and very dicult to understand if not presented as a graph. SQL+D allows charts and graphs to be de ned as part of the display speci cations. Even though some systems have been proposed [Cr92, AS94] that allow users to design screens that display query results as plots, graphs, and charts, it is dicult or impossible in them to specify directly the display of results of online and user de ned queries. Moreover, unlike the other approaches, 11

Figure 5: Display generated by Query 6 we allow speci cation of triggers as part of the display. For example, a node or an edge label in a graph can be speci ed to trigger a dierent display ( say, the home page associated with that node or label) when double clicked.

Example 3 Consider a database for a University Undergraduate Studies plan, containing class ids

and prerequisites for each career, and other general information about the curriculum required for students to earn dierent majors. Part of the schema looks as follows: CLASS PREREQ (classid, prereq) CLASSES (classid, name, description, mingrade) REQUIREMENTS (major, classid) where all the attributes of all the relations are character strings.

We are interested in displaying a directed graph of the classes needed to complete a BS in Computer Science. The query would look as follows in SQL+D :

12

SELECT

REQUIREMENTS.classid, CLASS PREREQ.prereq FROM CLASS PREREQ, REQUIREMENTS WHERE REQUIREMENTS.major = \BSCS" AND REQUIREMENTS.classid = CLASS PREREQ.classid DISPLAY PANEL main WITH ALL DISTINCT nodes(classid, prereq) AS boxedtext ON main, edge(prereq TO classid) ON main, Query 7. Query to display a directed graph representation of the CS degree plan. Say that the query results in the following relation: CLASSID PREREQ CS4101 MATH4110 CS3202 CS4101 CS4201 CS4101 CS3320 CS4332 CS3330 CS3202 CS3335 CS3202 CS3350 CS3202 CS3410 CS3350 CS3411 CS3410 CS4332 CS3202 Relation resulting from SQL portion of query 7. This data will be shown as follows, following the speci cation in the display clause of query 7:

Figure 6: Display generated by Query 7 We can see in the display that, since the nodes are formed by the values of the two attributes in the relation and the ALL DISTINCT option is indicated, only a single instance of each value 13

is shown as a node, and there is one directed edge per tuple in the relation. It is obvious that the display is much easier to understand than the resulting relation itself. Note that we can easily modify the display speci cation of the query such that each node can trigger the display of the home page corresponding to that classid.

5 Syntax

5.1 Queries A query in SQL+D is simply a standard SQL query, or an SQL query plus some DISPLAY speci cations. Semantically, the only attributes that might be used inside the display speci cations are restricted to the attributes listed in the SELECT clause of the SQL query. hqueryi ! hSQL Queryi j hSQL QueryiDISPLAY hdisplay specsi

5.2 Presentation and layout Basically, the overall layout of the display is speci ed by rst positioning some \container" panels, then lling those panels with data elements (database attributes or constants) speci ed within the WITH clause. hdisplay specsi ! hdisplay panels i WITH hattr spec listi [SEQUENCE hpresentation seqi] [AUTOSKIP hinti] Each panel is assigned a speci c panel id, which is used to place panels on other panels and data elements on the dierent panels. To arrange panels within panels and the dierent data elements on the panels, a relative location might be speci ed, using the keywords north, south, east, west and central. hdisplay panelsi ! panel hidi j panel hidi, hdisplay panel listi hdisplay panel listi ! panel hidiON hidihlocationi j panel hidiON hidihlocationi, hdisplay panel listi hidi ! halphai j halphaihalphanumerici hlocationi !

(north) j (south) j (east) j (west) j (central) By nesting panels and placing them on relative positions one within the other, the user can place an element virtually anywhere on the screen. The presentation order of the dierent attributes is speci ed using either the SEQUENCE or the AUTOSKIP clauses, or a combination of both. The SEQUENCE clause indicates for all or some of the attributes displayed, the order in which they should be presented: serially or in parallel. The default is in parallel (all at the same time). The AUTOSKIP clause, if included, causes the display to move through the records in the answer set one by one, showing each for the speci ed number of seconds. 14

The presentation sequence of two attributes is serial if the attributes are separated by semicolon, or parallel if separated by \k". hpresentation seqi ! Attr j Attr hpresentation orderihpresentation seqi j (hpresentation seq i) hpresentation orderi ! ; j k

5.3 Data display speci cations For each attribute that needs to be displayed, the user speci es how and where it should be displayed. The how is speci ed by a display element, and the where follows the same logic described before. hattr spec listi ! hattr speci j hattr speci, hattr spec listi hattr speci ! [ALL j ALL DISTINCT] Attr AS hdisplay elementiON hidi[hlocationi] [htriggeri] ALL and ALL DISTINCT let the user display more than one attribute value at a time, using any of the display elements. ALL displays all attribute values, while ALL DISTINCT causes only non-repeated values to be displayed. For example, if the user has a database of pictures of the city, and wants to show all of them at the same time (in scrollable groups), he might use a speci cation like: ALL city pictures AS image ON main(center). By default, the SQL+D Display Controller will show as many as possible on the screen and let the user \page" through or \scroll" through the thumbnails. In addition, the user might display constant elements on the screen (character strings, logos, instructions to the user, and other elements that are not in the database), as follows: hattr speci ! hconstant elementiAS hdisplay elementiON hidi[hlocationi] [htriggeri] j There are some display elements built into SQL+D that need special treatment: graphs and charts. The display speci cation is a bit dierent: hattr speci ! [ALL j ALL DISTINCT] nodes (hattr list i) AS hdisplay elementiON hidi[htriggeri] j edge (Attr1 TO Attr2 ) [LABEL char string j LABEL Attrn ]4 ON hidi[htriggeri] j edge (Attr1, Attr2 ) [LABEL char string j LABEL Attrn ] ON hidi[htriggeri] j (AttrX , AttrY ) AS hchart elementiON hidi Graphs are de ned by their nodes and edges. The attributes listed in the nodes statement become the valid arguments in either of the edge statements. There are two kinds of edge statements: the rst one, with the (Attr1 TO Attr2 ) argument, speci es a directed edge from node Attr1 to node Attr2. The second one speci es an undirected edge between the two nodes. hattr listi ! Attr name j Attr name, hattr list i Only here we need to let the user specify a label, since no attribute name can be used by default, like in other display elements 4

15

5.4 Display elements and extensibility Many pre-de ned multimedia display mechanisms are provided, and the user can add proprietary display elements to the language by adding classes to the language. hdisplay elementi ! audio j image j video j oval j polygon j button j checklist j list j choice j text j boxedtext j document j user de ned data type hchart elementi ! linechart j barchart j piechart hconstant elementi ! lename j \char string"

5.5 Data exploration and external interfaces As explained before, dynamic data exploration is possible by creating queries from queries. Nested queries need to be stored in advance, but by passing parameters to them instead of xing them in the query de nition they are exible and allow the user to really explore the record(s) desired. The nested query has some other DISPLAY speci cations, or it can even be an SQL data modi cation query. Also, SQL+D allows the user to interact with any other program by allowing a trigger to invoke an external application. Any executable might be speci ed, optionally receiving constant values or attributes as parameters. htriggeri ! TRIGGER query [ ( hparameter listi) ] j TRIGGER executable [ ( hparameter listi)

6 SQL+D Architecture SQL+D is a loosely-bound extension to SQL that allows users to specify, at the query level, the layout of the data extracted. By loosely-bound we mean that SQL+D does not demand changes to the current syntax and semantics of SQL, except for the addition of the DISPLAY clause at the end of an SQL query. If we remove the DISPLAY clause, we are left with a standard SQL query. The only bound between the SQL query portion and the DISPLAY clause is the data extracted

by the query. The bound between SQL and SQL+D is one-way: the DISPLAY clause needs information from the SQL query clause, but not the other way around. Any SQL query results in a new relation which schema is speci ed by the attributes listed in the SELECT statement. It is on this relation that the DISPLAY clause operates. By relying solely on existing features of SQL for the query speci cation, we simplify the implementation and facilitate the adoption of SQL+D by any existing database access software. By keeping the simple SQL-like statements, users familiar with SQL can easily learn to use the display speci cations. As far as the implementation architecture, we are following a client/server, object-oriented design, which allows SQL+D to be extensible through the use of new classes. Figure 6 shows the three main components of the system: a Query Constructor, an Interpreter, and a Display Controller, plus the general data ow through the system. The Query Constructor 16

and the Interpreter run on the server, while the Display Constructor runs on the client. We include in the following subsections a brief description of these components.

Query Interface

Query Constructor

-

6

SQL+D Query

?

Syntax Checking

Error

SQL+D Query ? SQL query Interpreter  Data

DISPLAY Clause & Data

 ? ?

Trigger?

?

Display Controler

? ?

?

6

User Event

?

User Display

    



A A A

-

Database Interface 6

Call

A A

A AA U

?

DB

External Queries and Applications

Figure 7: SQL+D Components We do not consider the Database Interface a direct component of database access is transparent to SQL+D .

SQL+D , since the actual

6.1 Query Constructor It is important to understand that SQL+D is not a browser. It is a display speci cation language. Thus, how the query is entered into SQL+D is not as important as to what is entered and how it is interpreted. However, as mentioned before, one cannot reasonably expect all users to be familiar with SQL+D . We have implemented a graphical query constructor that helps the user visualize the data display and construct an SQL+D query with graphical aids. The rst two screens of our Graphical Query Constructor let the user open a database and then, with online help and prompts, construct an SQL query. After the SQL query has been validated, the user can go to the Display Speci cation Screen, shown in gure 8. Here, the user can select attributes and then specify how they are to be displayed. 17

As the user speci es the panels and data display elements, they are shown on the right as they would appear when the query is executed. In the example shown in gure 8, the user has created a container panel called ContA and has chosen an option from the menu to create a data element component: a button that will be located on the south portion of ContA that is going to show the value of the attribute \Homepage URL".

Figure 8: Graphical Query Constructor Display Speci cation Screen A more extensive explanation on our graphical query interface can be found at

http://cs.utep.edu/csdept/students/ssahu/SqlD

In general, the architectural component we call the Query Constructor (graphical or not), receives some user input that can be interpreted as an SQL+D query and analyzes this input for syntactic errors, accepting or rejecting the query at that point.

6.2 Interpreter A syntactically correct query is passed to an Interpreter, which serves as a link between SQL+D and the external data components needed to answer a query. These external components include the following: 

A Database Interface that receives an SQL query, accesses the database, and returns the extracted data directly or allows access to the extracted data. 18

  

Queries used in TRIGGER statements within the query being interpreted. External applications referenced in a TRIGGER statement. User-de ned display elements.

To make SQL+D portable, the Interpreter requests access to a database indirectly through the Database Interface. If the connection is rejected, then no data is displayed. Basically, the Interpreter just submits an SQL query to the Database Interface and waits for the data. If no data is returned, then no data is shown. The Interpreter takes care of accessing the data and making it available to the Display Constructor in a standard way. Schema information is also extracted this way. The Interpreter also handles TRIGGERS. When the user invokes a TRIGGER, the Display Controller makes a call to the Interpreter, which in turn calls the external elements. This is done to encapsulate system-dependent knowledge within the Interpreter and facilitate portability. Before passing the display speci cations to the Display Constructor, the Interpreter looks for the queries and external applications used in TRIGGER statements. If some query or external application is not found, then it substitutes the user speci cation by a call to a system-de ned DISPLAY that shows the user an appropriate error when the TRIGGER is invoked. Otherwise, it keeps a reference to the speci ed query or application, and information about the attributes that need to be passed to it. If there is a user-de ned display element speci ed, the Interpreter checks for its existence before passing it to the Display Controller. If it cannot be found, or it does not meet the interface requirements (it does not have all the methods required by the Display Controller) it substitutes it for the default table display element.

6.3 Display Controller The Display Controller receives a DISPLAY clause and the data to be displayed, and based on that information constructs the display and handles user requests (screen events). The Display Controller consists of a group of classes aimed at displaying and managing the data. All displays are constructed using instances of the display element classes (like list, table, checklist, image, video, audio), arranged and managed by an instance of a \master panel" class. The master panel class consists of two panels: one, the user query panel, is empty and becomes the rst panel speci ed by the user. The other one contains the master control buttons and all methods to handle navigation through the answer set. In addition, the master panel class has logic built in to adequate the panel to better t the data: for example, based on how many records and data elements need to be displayed, it might decide how many items are shown in a list at the same time or how many of a group of objects speci ed as ALL DISTINCT to show at a time, or whether scroll bars need to be displayed. User requests are expressed through screen events, and are basically of two kinds: (a) Move through the answer set or (b) trigger a query or application. The Display Controller handles all requests to move through the current answer set, and passes the trigger events to the Interpreter. We wanted to rst concentrate on specifying queries and displaying their answers, so we developed a prototype of the Display Controller, having all but a few of the display elements implemented. 19

A sample (downloadable) application where the user is given a database schema and can actually type a query using a subset of the display elements in SQL+D is available at http://www.utep.edu/chitta/display, and a more extensive explanation of the Display Controller can be found at http://mail.utep.edu/ reddy/paper.html.

7 Related Work Information can be extracted from databases either by browsing, i.e. going from a high level view through progressive re nement until nding the desired data, or by querying, i.e. specifying some criteria (conditions) that describe properties of the data desired. The rst method is better when the user does not know what exactly he/she is looking for, or does not know the database schema and \travels" or \explores" the data until he/she sees some relevant information. In addition, browsing is the only means to extract data when there is no query language or schema associated to the data, as is the case of the web. The second method addresses the needs of users that are looking for speci c information, which means they have to use expressions to describe data properties. Dierent research eorts have sought to address the challenges and exploit the bene ts of multimedia and object oriented databases through graphical browsing or querying interfaces. The schema of the database is, in most cases, the \highway" through which browsing ows. Such is the case of PESTO [Ca96] (Portable Explorer of STructured Objects), an integrated user interface that supports browsing and querying of object databases. Like other object browsers, PESTO allows users to navigate in a hypertext-like fashion, following the relationships among objects (schema). In addition, PESTO allows users to formulate queries online by allowing users to specify conditions to the object properties as they browse. It is meant to be extensible, providing \hooks that allow specialized predicate formation and display tools to be added to the system for important new data types". However, in [Ca96], the authors deal only with displaying images, and the architecture of the \hooks" is not discussed. Unlike PESTO, SQL+D is not an application (query browser), but a 4th generation display language that can be incorporated (as an extension to SQL) into any relational database application, and, through SQL3, into Object-Relational database applications, including those that run over the Web. As explained in section 6, a query browser is one of the applications that can be developed using SQL+D , and we are currently developing one that includes the functionality of PESTO. Querying a database involves de ning the domain of the search (where should data be looked for), expressing the selection conditions, and how the data should be organized. However, visual query systems in general emphasize the use of graphical interfaces to express the query itself, including pictorial representation of the queries and a variety of buttons and menus to convey the selection criteria. They rely on the selection of a visual metaphor [SM97] and means of interaction. As pointed out by [SM97], even though forms, icons and diagram paradigms are well-known, much work has still to be done ... most of the work has overlooked instance visualization, i.e., the way output data is presented. To allow exible (or user-de ned) instances visualization, one has to describe within the query, what visual representation should be employed. One of the pioneering works to address this issue is DOODLE [Cr92, Cr93]. DOODLE is a visual language to query object-oriented databases, allowing the user to specify how to construct 20

\pictures" from data. It supports the speci cation of graphs, bar charts, pie charts, plot charts. SQL+D includes a similar feature to display graphs and charts (see section 5.3). SQL+D goes far beyond DOODLE in allowing the user to specify within the query the visual representation and layout of the data, as presented throughout this paper. Another well-known project that addressed the issue of integrating database management and database visualization is the Sequoia 2000 project, TIOGA [St93], and more recently TIOGA-2 [Ai96], also known as Tioga DataSplash. The main feature of these works is that the visualization technique to be used for displaying a relation stored in the database is also stored in the database. In the Visualization Executive [Ko93], the name of the program to be executed and the parameters to be passed to it in order to visualize some data set were stored with the data set in the database. Tioga [St93] uses the boxes-and-arrows paradigm to provide for browsing the database and displaying objects. In Tioga, every box is a user-de ned function which has been registered in the POSTGRES database management system. Every Tioga program contains a viewer which displays the program's output. An expert programmer writes the viewer, which encodes information about how objects will be displayed. The default viewer provides pan and zoom features with which the end user can y over the canvas viewing areas of interest. SQL+D allows displays to be speci ed on the y by the user, rather than relying on an expert programmer. But, an expert programmer can also construct a viewer in advance, to allow exploration of a speci c database, as shown in our Campus Navigation example. In brief, our approach diers from other querying mechanisms in that it is speci cally designed for querying multimedia and other databases containing complex information, and emphasizes

exible, in-the-query, by-the-user speci cation of the display of the output data, rather than visual speci cation of the query or browsing. However, with SQL+D , it is possible to build applications to allow visual speci cation of queries or data browsers (and we have done that), but these applications are just a couple of the possible uses of the language. The expressiveness of SQL+D is dictated by SQL, and its syntax and semantics are based on it, giving SQL+D a solid formal foundation as a query language.

8 Conclusion We have presented the syntax of display speci cation language, SQL+D , designed as an extension to SQL. It is intended to be simple enough that even without the help of a sophisticated graphical user interface, any user that is familiar with SQL can pose a query using the DISPLAY extension to layout the data. We have implemented a graphical query constructor that aids users in constructing SQL+D query that incorporates most of the features of SQL+D . We are currently extending this interface to allow the speci cation of graphs and TRIGGER statements. Also, we are gathering user feedback to make it more user friendly. We expect a full beta version of the graphical query constructor to be ready by the end of April. We have completed the second phase of development of the Display Controller (see section 6.3). As it stands, the controller receives as input two text les: one containing a DISPLAY clause, and the other one containing some (formatted) query results in the form of a table where the rst row contains the selected attributes separated by a tab and the following rows contain the corresponding attribute values, one row per tuple, with the attribute values separated by a tab. With this, it 21

constructs a fully operational display window. The only features of SQL+D not yet implemented are the TRIGGER statement, and the graph and chart speci cations. We expect a full beta version of the display controller to be ready by the end of April. In the future, we will expand on other uses of the language, including dynamic querying of multimedia databases over the web, querying multidimensional data and scienti c data, among others.

References [ACS90] Angelaccio, M., Catarci, T., and Santucci, G. \QDB*: A Graphical Query Language with Recursion," IEEE Transaction on Software Engineering, pp. 1150-1163. October 1990. [Ai95] A. Aiken, J. Chen, M. Lin, M. Spalding, M. Stonebraker, and A. Woodru. \The Tioga-2 Database Visualization Environment," Proceedings of the IEEE Visualization '95 Workshop, Atlanta, GA, USA, October 1995. [Ai96] Aiken, A., Chen, J., Stonebraker, M., and Woodru, A. \Tioga-2: A Direct Manipulation Database Visualization Environment, " Proceedings of the 12th International Conference on Data Engineering, New Orleans, LA, February 1996. [AS94] C. Ahlberg and B. Shneiderman, \Visual Information Seeking: Tight Coupling of Dynamic Query Filters with Star eld Displays," Proceedings of ACM Conference on Human Factors in Computing Systems CHI '94, Boston, USA, pp 313-317. [Ba91] Batini, C., Catarci, T., Costabile, M.F., and Levialdi, S., \Visual Query Systems: A Taxonomy," Proceedings of the 2nd IFIP WG2.6 Working Conference on Visual Databases, 1996. [Ba93] Batini, C., Catarci, T., Costabile, M.F., and Levialdi, S., \On Visual Representations for Database Query Systems," Proceedings of the Interface to Real and Virtual Worlds Conference, Montpellier, France, March 22-26, 1993. pp. 273-283. [BC95] F. Banchilhon and A. Carlson, \Providing rich query functionality," SQL3 discussion paper, ISO/IEC JTC1/SC 21/WG3 DBL LHR, ASC X3h2-95-462, Dec 1995. [Be93] D. Beech, \Collections of objects in SQL3," Proceedings of the 19th VLDB Conference, August 1993. [Ca96] M. Carey, et al, \PESTO: An Integrated Query/Browser for Object Databases," Proceedings of the 22nd VLDB Conference, August 1996. [CCM92] Consens, M.P., Cruz, I. and Mendelzon, A.O., \Visualizing Queries and Querying Visualizations". SIGMOD Record, 21(1):39-46, March 1992. [CCS94] Catarci, T., Chang, S.K., and Santucci, G., \Query Representation and Management in a Multiparadigmatic Visual Query Environment," Journal of Intelligent Information Systems, vol. 2, 1994. [Cr92] Isabel F. Cruz, \DOODLE: A Visual Language for Object-Oriented Databases," Proceedings of SIGMOD, June 1992. [Cr93] Isabel F. Cruz, \User-De ned Visual Languages for Querying Data," Technical Report, Department of Computer Science, Brown University, Providence, Rhode Island, December 1993. 22

[Ko93] Kochevar, P., Ahmed, Z., Shade, J., and Sharp, C., \Bridging the Gap Between Visualization and Data Management: A Simple Visualization Management System,"Proceedings of IEEE Visualization, 1993. [MS92] J. Melton and A. Simon, Understanding the New SQL: a Complete Guide, Morgan Kaufmann Publishers, Inc., San Francisco, California, 1992. [MS96] A. Massari and L. Saladini, \Virgilio: A VR-Based System for Database Visualization," Proceedings of the International Workshop on Advanced Visual Interfaces, AVI '96, ACM Press, New York. [NS96] T. Nguyen, V. Srinivasan, \Accessing Relational Databases from the World Wide Web," Proceedings of SIGMOD, June 1996. [Ra94] B. Rasmussen, \WDB- A Web Interface to SQL Databases," European Southern Observatory, http://arch-http.hq.eso.org/bfrasmus/wdb/wdb/html, 1994. [SM97] S. Sachs, V. Markowitz \Electronic Notebooks: Project Report" Lawrence Berkeley National Laboratory, http://www-itg.lbl.gov/ ssachs/notebook/report.html [SQL3] F. Manola (Editor) \SQL3," X3H7 Object Model Features Matrix, Document Number X3H7-93-007v10, http://info.gte.com/ftp/doc/activities/z3h7/by model/SQL3.html. [St91] Stonebraker, M. and Kemnitz, G., \The POSTGRES Next-Generation Database Management System," Communications of the ACM, 4:10, pp. 78-92. [St93] Stonebraker, M., Chen, J., Nathan, N., Paxson, C., Su, A., and Wu, J. 1993. \Tioga: A Database Visualization Tool," Proceedings of IEEE Conference on Visualization, 1993, p. 8693. [St96] M. Stonebraker, Object Relational DBMSs: The Next Great Wave, Morgan Kaufmann Publishers, Inc., San Francisco, California, 1996. [Va94] B. T. Ander Vanden, \Optimizing Toolkit-Generated Graphical Interfaces," Proceedings of the ACM Symposium on User Interface Software and Technology, pp 147-166, Marina del Rey, 1994. [WK82] Wong, H.K.T. and Kou,I., \GUIDE: Graphical user Interface for Database Exploration," Proceedings of the International Conference on Very Large Data Bases, pp. 22-32, 1982. [WS97] A. Woodru, M. Stonebraker. \Supporting Fine-Grained Data Lineage in a Database Visualization Environment". To appear in Proceedings of the 13th International Conference on Data Engineering, Birmingham, England, April 1997. [Zl77] Zloof, M.M. \Query-by-Example: A Database Language," IBM Systems Journal, 16(4):324343, 1977.

23