PDA-based Visualization Control and Annotation ...

PDA-based Visualization Control and Annotation Interface for Virtual Environment Nobuyuki KUKIMOTO, Yoshiko FURUSHO Department of Multimedia Tohwa University 1-1-1 Chikushigaoka, Minami-ku, Fukuoka, Japan {kukimoto, yoshiko}@tohwa-u.ac.jp

Abstract Virtual Reality (VR) system can be a useful tool to support visualization tasks. However, the current available input devices for VR limit the control and navigation within the Virtual Environment (VE). To overcome this problem, we designed a user interface by using Personal Digital Assistants (PDA) for scientific visualization purposes. Through this system, it is possible to change visualization parameters, put marks, or write a memorandum within the VE. We can use the VE for recording memoranda by using the PDA as a writing utensil or by drawing on the PDA touch screen for after-placement into the VE. The main advantage of this method is that it supports the use of any language, because the characters are drawn avoiding the necessity of typing. We evaluated both kind of drawing tasks and verified that the first method is suitable for drawing figures and the second one is useful for drawing characters within a VE. Keyword: Visualization Control, Virtual Environment, PDA, Drawing 1. Introduction Many scientific visualization systems have utilized 3D data. When we display this 3D data onto a 2D display, such as a CRT or LCD monitor, it is difficult to see an overview of the entire data because of the influence of occlusion and the difficulty in obtaining the precise depth perception. As a result, a facility to make possible easy and fast viewpoint changing through the input devices, such as the traditional keyboard or mouse, becomes crucial. Obviously, it is better to visualize 3D data in a 3D displaying system, such as stereoscopic display. For instance, visualization of 3D vector fields becomes easier and, as a result, the interactive

Jorji NONAKA,Koji KOYAMADA, Masanori KANAZAWA Department of Systems Science Graduate School of Informatics Kyoto University Yoshida-Honmachi, Sakyo-ku, Kyoto, Japan {jorji, koyamada, bwv147}@mbox.kudpc.kyoto-u.ac.jp observation becomes more useful. The application of VR in the scientific visualization field has been considered very useful. However, fast and easy parameter changing and data handling without the need of stopping the continuous VR operation would be useful. 2. Visualization Task 2.1 User Interface for Visualization Several methods have been proposed for visualization purposes. Applications in this field require some features such as facility for changing visualization parameters, switching analysis methods and changing object viewpoints. The application of VR in scientific visualization requires easy viewpoint changing. However traditional input devices for VR, such as “ Data Glove” or “Wanda” (Ascension Technology), limit the user interaction for parameter changing or function selection because of the difficult manipulation and few available functions, as a result, an alternative easyto-use input device becomes crucial. 2.2 Annotation Overtop a Visualization Result When we study using a textbook, writing a memorandum in the unfilled space or underlining an important part can become a key feature for subsequent works or for an easy review. For instance, physician could search for a diseased part on a volume rendered image of CT scan data and, put marks and write annotations on it. The annotated documents could be easily shared with other staff within the VE. Taking these into account, we can consider that the annotation feature is very useful and important in VE in order to promote better and easy understanding. An annotation onto the result of scientific visualization in VE could be effective for a collaborative and creative work.

2.3 PDA within VE We designed a user interface using a PDA for scientific visualization within the VE. PDAs have become a popular personal device for managing several kinds of data. In addition, they have become enhanced wireless network devices. The reasons of this popularity can be considered as its easy handling, light weight and easiness to connect with traditional desktop or notebook computers. PDA touch screen can be used as an input device for dynamic interaction that enhances VR ability and usability. At the same time, a PDA screen can transform in an extended VR output display. As a result, it can mediate between the virtual and real world. 3. Related Works Kageyama et al. [1] proposed the VFIVE (Vector Field Interactive Visualization Environment) system composed of

Figure 1. Overview of visualization task using a PDA

3D pop-up menus inside VE for visualization control and viewing field modifier. The purpose of VFIVE is the analysis of complex 3D vector fields as they are. In addition, it generates curved surfaces or objects automatically from a dataset. This is useful to display an object calculated beforehand by using some previous visualization techniques. However, small 3D pop-up menus are not useful due to the difficulty in handling. On the other hand, large menus can occlude the displayed objects difficulting the interaction. Our approach focused on applying PDA into VE. However, some researches have used PDA to provide collaboration with PCs. For instance, Ayatsuka et al. [2] proposed a hybrid computing environment for small computing devices, such as PDA, called “HyperPalette”. They used an electromagnetic tracking sensor attached to the PDA in order to detect the PDA’s position and movement. This PDA is used as an interaction device for a “computerized table” on

Graphical User Interfaces(GUI) are used everywhere in the personal computing and, as a result, ordinary users have no difficulty using GUI. However, GUI in VR is still rare. Hartling et al. [4] proposed “Tweek”, a Java-based GUI for PDA that enables the control of a VR application using the PDA. It uses CORBA (Common Object Request Broker Architecture) to communicate between the Java GUI and the C++ VR applications. The current development status of Tweek does not make possible the mapping of 2D GUI into the 3D virtual environment. In addition, Tweek focuses mainly in 2D navigation, diverging from our approach. In the Augmented Reality (AR) field, an annotation overlay on live video frames is considered an essential feature to fuse the real world with cyberspace. Kourogi et al. [5] proposed a panorama-based annotation system to overlay the annotations according to the image alignment between the input frames and the panoramic images. In our approach, we focused only on 3D volumetric datasets, and not on video images.

which a computer screen is projected by an LCD projector. This table works as an extended PDA screen and the PDA can be worked as a kind of pointing device, and a user can interact with the table using grab-and-move or scoop-andspread interaction. Myers [3] has investigated the use of PDA as a remote control for PC applications. His project called “Pebbles” focuses on how handheld computers, such as PDA, and a PC can work together when both are available. A PDA can connect with a PC by serial cable, infrared or a wireless network. Through the PDA, a user can perform some specific actions on applications on the PC side, as well as, general actions on Windows environment, such as scrolling and selecting.

4. Implementation of User Interface Figure 1 shows an example of a visualization task using PDA in a VE. The PDA is used for visualization parameter changing and for annotating free-hand memoranda in the VE through two different drawing methods. One of them is to move the PDA for drawing something directly in the VE while pressing a specific button, and the other is to first draw something on the PDA touch screen, and then transfer it into the VE. This implemented PDA application offers a simple GUI in order to make possible fast visualization control and easy annotation without visualization task interruption.

npmep_kk_`jc`srrml3

D?QRP?Ipcacgtcp

RAN-GN

TRI MnclEJNcpdmpkcp

Gk_ec

RmsafQapccl

A?TCJg`p_pw npmep_kk_`jc`srrml1 npmep_kk_`jc`srrml2

Qf_pcbkckmpw RAN-GN D?QRP?IQCS aspqmpicw

rp_aib

Gkkcpq_Bcqi

PQ+010a Fmqrk_afglc

Figure 2. System Configuration 4.1 System Configuration We used VTK (The Visualization Toolkit) for a supporting tool of visualization, OpenGL Performer for realtime graphics, QUANTA [6] for networking, and CAVE Library for VR. Figure 2 shows our proposed system configuration. ImmersaDesk is a display system for VR having a drafting table format. It possesses a 175x133cm rear-projected screen inclined at 45-degree. By using shutter glasses, users can see high-resolution, stereoscopic, head-tracked images. The host computer is a dual Pentium III (1GHz) with an ATI FireGL4-based video card. It uses RedHat Linux 7.1 as OS. ImmersaDesk applications can be executed in CAVE systems requiring only small changing in some configuration files. CAVE Library utilizes a tracking daemon called trackd for tracking measurement and data control. This daemon stores data into a memory space shared with the CAVE Library. PDA transmits control data, generated by cursor keys or buttons, such as front programmable or side ones, to the trackd daemon, which has extended TCP/ IP communication functionality. Control data and tracking data are separated from PDA application, so it is possible to control other CAVE applications. The PDA used in our experiment was a TOSHIBA Genio e550GX (PocketPC 2002) with wireless LAN adapter used for communication with host computer through the trackd daemon. An electromagnetic motion-tracking receiver (Polhemus FASTRAK with Long Ranger) was attached to the PDA case. This PDA case was modified by enlarging it 10 cm in order to attach the motion-tracking receiver far enough from the PDA to diminish the electromagnetic in-

Figure 3. The PDA with the reformed case. Left picture is the adjusted FASTRAK receiver on the back face of the case. fluence. This receiver was attached by using a piece of Velcro as we can see in Figure 3. Tracking data were sent to trackd daemon, running on the host computer, through the RS-232C serial interface. As a result, we had three communication channels in the PDA: serial interface for tracking data, wireless communication for hardware buttons and currsor data with the host computer, and GUI on the PDA screen through the implemented software buttons, sliders and annotation interface. 4.2. Visualization Application We implemented a simple VTK-based visualization application, by which we were able to modify and visualize isosurfaces from a given volume data. VTK does not sup-

(4-1) Button Interface

(4-2) Slider Interface

(4-3) Drawing Interface

Figure 4. Implemented tab-control interfaces for visualization parameter changing and free-hand memorandum making tomed to this interface without difficulty. port multiple viewport as CAVE Library does. Thus, we used OpenGL Performer in order to make possible the use of 4.3.2 Making Annotations CAVE Library with VTK. OpenGL Performer is a widely To draw directly in a VE by using the PDA as a writused as real-time 3D rendering library that runs on IRIX, ing utensil, move the PDA by pressing the programmable Linux and Windows platforms. In addition, it provides other button “3”. To erase a line, press the programmable button features not supported by VTK, such as scene-graphs, inter“4”. We can draw indirectly in the VE by using the PDA section testing, visibility culling and multi-viewport with touch screen and the implemented drawing interface. Inside multi-processing. the VE, the drawing data are changed to polygon data. This We used a function called vtkActorToPF [7,8] in orabove-mentioned drawing interface is composed of four der to convert data generated by VTK into data recognized buttons: the “PostIt” button is used to put the drawn data by Open GL Performer. This function translates the geominto the VE; the “OK” button is used to confirm the placeetry, color information, and properties from a vtkActor, in ment of this drawn data; the “Cancel” button is used to erase the VTK side, and creates or modifies a corresponding the current drawn data; the “Clear” button is used to repfGeode, which is a node within the OpenGL Performer move the last placed drawn data. scene-graph. The required conversion time for the utilized After drawing something on the PDA, press the CT scan data, obtained from a VTK sample dataset, was ap“PostIt” button in order to place it in a VE. At this time, it is proximately 0.05sec. By using vtkActorToPF, a developer possible to move freely and change the placement position. can build a pipeline between VTK and a OpenGL Performer The drawn data is displayed, by default, at 15cm deeper and scene-graph. Whenever the vtkActor changes, the pfGeode 7cm above the virtual pen position. Maximum drawing data is automatically re-translated. size inside the VE was set as 32x37cm. After the necessary 4.3. PDA Application 4.3.1 Visualization Parameter Changing Figure 4 shows the user interfaces of our proposed PDA application. Three distinct interfaces were designed in order to make possible the interaction with VE: a set of buttons, a set of sliders and a drawing interface. The generated data from manipulating this interface are packed, with the corresponding item ID and value, and transmitted to the host computer. The functionality of each button or slider is defined in the host application. Users who are familiar with PC graphical user interface are expected to become accus-

movement, such as translating or rotating, user can paste it by pressing the “Ok” button. 4.3.3 Software Archtecture The PDA application was developed by using Microsoft Embedded VC++3.0. Figure 5 shows the configuration of PDA application and its interface with visualization application.The PDA's menu items can be selected by using TabControl item. When an annotation is performed by drawing directly on the PDA screen, the Windows Message Handler captures the stylus pen positioning data, and counts the number of

NB? ?nnjga_rgml R?@Amlrpmj

@srrmlq

GB)T_jsc

TRINgncjglc

QrwjsqA_nrspc

Tgqs_jgx_rgml ?nnjga_rgml NcpdmpkcpQaclc+Ep_nf

B_r_Qmspac

ndQaclc

Dgjrcp K_nncp tri?armp

Bp_u

Qjgbcpq

tri?arRmND

ndBAQ

ndBAQ

ndEcmbc

ndEcmbc

(6-1)Drawing the evaluator’s family name within a 40cm wire framed cube displayed in VE.

Figure 6. Evaluation of drawing

Table 1 Evaluation of character drawing

Figure 5. Software Archtecture lines. The drawn data and the number of lines are sent to the visualization application. This data is converted to pfGeode for subsequent utilization by pfDCS. The functions of the programmable buttons and cursor key are separate from the PDA application. These functions are based on GAPI (Game API), which is a Windows CE specific series of gaming technology. This API allows for controlling the hardware buttons and makes the use of the full-screen on the PDA possible. By using this API, we implemented similar functions to the default CAVE controller, which is called Wand. 5. Evaluation We evaluated both drawing methods. One by drawing directly in the VE by moving the PDA pressing a specific button (Method 1), and the other by drawing on the PDA touch screen for after transferring into the VE (Method 2). Evaluation 1 corresponds with the task of drawing Chinese characters (family name of the evaluator) in a 40cm wire framed cube in the VE. Each evaluator drew his or her own family name as we can verify in the Figure 6-1. The reason for choosing the family name was because of familiarity. Evaluation 2 corresponds to the task of drawing three different circles surrounding a 45cm solid cube as we can verify in Figure 6-2. The set of evaluators was composed of 22 undergraduate students who had only one lesson about the use of this drawing system before this evaluation. The evaluation items were composed by: usability of each method scored from 1 through 5 (5 corresponds to satisfactory), the

(6-2) Surrounding a 45cm solid cube with circles.

Subjective Value Subjective Value Method1 Method2

Drawing time Method 1

Drawing time Method 2

Average value

2.4

3.9

19.3 (sec)

15.1 (sec)

Median Value

2

4

14.5 (sec)

16.5 (sec)

Table 2 Evaluation of circle drawing Subjective Value Subjective Value Method1 Method2

Drawing time Method 1

Drawing time Method 2

Average value

3.7

2.8

16.2 (sec)

27.7 (sec)

Median Value

4

3

12.5 (sec)

34.5(sec)

explanation of the scoring and the time required for drawing. In Method 1, the drawing time was measured from the moment of pressing the corresponding button for drawing until the moment of completing the drawing process. In Method 2, it was measured from when the stylus pen touched the touch screen until the confirmation of placement through the “Ok” button. Table1 shows the average and median values of evaluated times and scores for evaluation 1. We are not able to identify significant drawing time difference between Method 1 and 2. However, through the subjective evaluation, we were able to verify that Method 2 was considered easier for drawing Chinese characters than Method 1. The results of evaluation 2 are shown in Table 2. From the point of view of the subjective evaluation of usability, we cannot verify any significant difference between either methods. However, from the average drawing time, we can conclude that the

Table 3. Average drawing time for accustomed and unaccustomed user

Drawing Average Time

Accustomed user

Unaccustomed user

5.61sec.

15.75sec.

Figure 7 Drawing "TOHWA" in Chinese characters evaluators found Method 1 easier for drawing figures than Method 2. Through this experiment, we can conclude Method1 is useful for drawing and erasing, however it is not suitable for drawing small figures or characters. On the other hand, Method 2 is suitable for drawing characters and, in addition, it can place characters at any position in the VE. The drawing times of both methods show that they are not interactive. We presume that this is caused by the inexperience of the evaluators in using PDA to draw in the VE. Table 3 shows the average drawing time for both accustomed and unaccustomed users of PDA. We performed an evaluation, similar to Method 1, by drawing “TOHWA” in Chinese characters on the PDA screen and putting it into the VE (Figure 7). The group of evaluators was composed of four students accustomed, and four students unaccustomed with this interface. Accustomed users took approximately 5.61sec., while unaccustomed ones took 15.75sec. It is necessary to investigate the required manipulating times for a beginner to become familiar with a PDA. 6. Conclusion and Future Work This paper describes a PDA application for visualization control and annotation interface in a VE. By a GUI, this application gives buttons and sliders interfaces for changing visualization parameters. Two different annotation interfaces were implemented. The first one uses the PDA as a direct writing utensil in the VE and, the other one uses the

PDA touch screen for pre-drawing in the VE. The results of the first method were not satisfactory for drawing characers, especially when the hand stroke become short. However, this method was suitable for drawing large figures. In contrast, the use of PDA touch screen was suitable for writing detailed characters. In both methods, it was difficult to place the drawing data wherever evaluators wanted. This was caused because the depth perception was not set up correctly. In this experiment, we did not evaluate the visualization parameter changing. For future work, we expect to compare this PDA GUI with other methods, such as 3D pop-up menu, in order to verify our proposed method’s effectiveness. In addition, we expect to extend the use of this visualization control and annotation interface in a CVE (Collaborative Virtual Environment). References [1] Kageyama,A., Tamura, Y., and Sato,T., Visualization of Vector Field by Virtual Reality, Progress of Theoretical Physics Supplement, No.138,2000,pp.665-673. [2] Ayatsuka,Y.,Matsushita,N., and Rekimoto,J., HyperPalette: a hybrid computing environment for small computing devices, In CHI 2000 Extended Abstracts, 2000,pp.133-134. [3] Myers, B.A.,Using Handhelds and PCs Together, Communications of the ACM, Volume 44 ,pp.34-41,2000. [4] Hartling,P., Bierbaum,A., Cruz-Neira,C.,Virtual Reality Interfaces Using Tweek,ACM SIGGRAPH 2002, San Antonio, Texas, USA, 2002. [5] Kourogi,M., Kurataka,T., Sakaue,K., Muraoka,Y.,A Paranoma-Based Annotation on Video Frames and Its Real-TIme Implementation” The transactions of the institute of electronics information and communication engineers D-II Vol.J84-D-II No.10,2001, pp2293-2301. [6] Leigh, J., Yu, O., Schonfeld, D., Ansari, R., et al., Adaptive Networking for Tele-Immersion, Proc. Immersive Projection Technology/Eurographics Virtual Environments Workshop (IPT/EGVE), Stuttgart, Germany, 2001. [7] Rajlich,P., An Object Oriented Approach to Developing Visualization Tools Portable Across Desktop and Virtual Environments, MS THESIS,1998 [8] Leigh, J., Rajlich, P., Stein, R., Johnson, A. E., DeFanti T. A., LIMBO/VTK: A Tool for Rapid Tele-Immersive Vi s u a l i z a t i o n , C D R O M p r o c e e d i n g s o f I E E E Visualizaton’98, North Carolina,USA,1998.