Generic Visualization and Manipulation Framework for ... - IEEE Xplore

Generic Visualization and Manipulation Framework for Three-Dimensional Medical Environments Brian S. Tani, Tiago Nobrega, Thiago R. dos Santos, Aldo von Wangenheim The Cyclops P r o j e c t C o m p u t e r Graphics and Image Processing Lab Federal University of Santa Catarina {bstani,tigarmo,thiago,awangenh}@inf.ufsc.br

Abstract W e present a visualization and manipulation framework for three-dimensional medical environments. The M V C architecture of the framework allows the use of different API technologies such as OpenGL and Direct3D, different model representations, and controllers. This facilitates the implementation of medical applications offering resources commonly needed by such systems. Two case stud2es are shown as examples of such applications. Key-words: Model View Controller Framework, Generic Manipulation, Generic Visualization

1. Introduction Medical applications present a distinct set of characteristics when it comes to visualization and manipulation of visual data [5, 111. Visualization of medical data must be precise enough to avert ambiguities in the information perceived and must be intuitive when manipulating such visual data [Ill. These characteristics have been addressed with various application specific techniques. The Analyze [8] software package's hierarchical arrangement of visualization, manipulation and measurement tools consists of a set of separate, conceptually independent programs that share memory space and communicate amongst themselves. In a MVC paradigm [14,6], processes/programs that deal directly with the user display and visualization can be encapsulated into the View portion, while processes that deal with manipulation of volumes can be seen Controllers. Processes that make calculations regarding tissue elasticity and diameter of arteries are surely Models.

In the 3DVIEWNM [13] system, the Data interface that deals with handling and storing of data corresponds to the Model entity. The Graphics interface shares the View's role of handling the display device and visualization. The Process interface, much like the Controller, handles manipulation and interaction. Medical volume visualization applications or threedimensional medical environments can be expressed as a MVC model. We present a generic visualization and manipulation framework using a MVC architecture that can be used in three-dimensional medical environments. The purpose of this framework is to provide solutions to some of the characteristics shared by medical applications, in terms of manipulation and visualization of data and their integration. Such framework would allow faster software development. Another work in this area is the MEDIFRAME framework[9],which also provides an extensible framework for medical applications. They use a different adaptation of the MVC paradigm to achieve this.

2. MVC Architecture The MVC Engine framework is based on the MVC architecture, where the Model represents the data, the View represents the visualization of the Model and the Controller works as a bridge between the two, catching events created by user on the view and altering the model accordingly and as needed. The Model can be, for instance, any geometric representation of data generated from medical images [12, 7, 21, or acquired by other means. In the framework proposed this can be achieved by specializing the engine model class. High level integration doesn't have to be reimplemented. Figure 1 shows the MVC diagram. The same can be done t o the View and Controller.

Proceedings of the 19th IEEE Symposium on Computer-Based Medical Systems (CBMS'06) 0-7695-2517-1/06 $20.00 © 2006 IEEE

all the Views which one is activated when it receives an event. This way the Controller can issue redraw commands only to the Views affected by the change. Specific needs on what type of control we want are implemented by specializing the Engine Controller class. Controls for input devices such as mouse, keyboard, DataGlove, or any other device can be created without the need to change the View or Model classes.

2.3. View

Figure 1. The MVC architecture used in this framework.

The View can be specialized to use any API technology, such as Direct3D or OpenGL, or different display types (Computer Monitor, HMD, etc.). The basic interface remains the same, whether the View is an OpenGL specialization for HMD with stereoscopic display or a SDL specialization in a common P C monitor. Controller specialization also maintain the same high level integration.

2.1. Model The Engine Model class contains interface functions to assign transformations to the entire model and a prototype of the data request function. This function ha5 a different implementation depending on the model specialization chosen, but always compares the state of the View requesting the Model data, and the inner state of the Model itself. Different behaviors of the Model, such as subdivision of solids, have to be implemented accordingly. This also affects the Controller which has to know which features the Model supports.

2.2. Controller The Engine Controller class has a list of all models and all views associated with it. In order to know which model to send commands to, the Controller class catches events originated from the active View and translates them to commands to the Model associated to the respective View, or the View itself. To know which View is activated at the time, the Controller asks

The Engine View class creates an association to its controller which is used to send commands through the callback functions registered in the View. The events generated in the View are sent to the engine view callback functions and from that to the appointed control instance. The interface of the Engine View class is based on a simplified OpenGL interface and is not actually implemented. The specialized class translates the interface to specific API functions or output device requirements.

2.4. State Management Common implementations of the MVC Engine framework will support efficient storage and handling of large data such as a volumetric representation of a series of medical images - by making use of the graphics card's dedicated memory, for instance. The Views are the entities in charge of lower level handling of data, specific to the system, and the Model is only aware of a higher level abstraction. As a result, the data representation of the View and of the Model must be synchronized to avoid inconsistencies. As a practical example, say the system is currently handling a graphical representation of a heart, with four Views, one Model and one Controller. In this specific implementation, the user can cut through the virtual heart with a scalpel tool. He chooses the View that suits him the most, and clicks his mouse pointer, dragging a short line representing the scalpel's path. The Controller notices the action, and sends the relevant information to the Model. This Model knows the kind of tissue it is dealing with, and knows in what the action will result: an eye-shaped cut across that section of the heart. Because these specific Views have been implemented with speed and efficiency in mind, they had stored their representation of the virtual heart (now outdated) in a portion of the graphics card dedicated memory. As a result, if nothing else is done, each time the Controller asks a View to redraw itself, it will simply go through

Proceedings of the 19th IEEE Symposium on Computer-Based Medical Systems (CBMS'06) 0-7695-2517-1/06 $20.00 © 2006 IEEE

the data structure kept in that special memory and draw the heart as it previously was.

3.1. Trackball Engine

The solution chosen for this problem is this: Each time the Model is changed, it updates its inner state. All Views have a state attribute related to the Model assigned to it. Then, when a View is asked to be redrawn, a message is sent to the Model with the state information stored in the View. If this information is different from the Model's current state, the View receives the new updated data overwriting the preexisting representation, along with the current Model state. Otherwise, if the states value are the same, nothing changes. Finally, the View draws its current representation.

In this simple application the user rotates, translates and scales the model instead of the camera. The rotation uses a trackball metaphor [3]. Much like the actual trackball mouse, consisted of a base and half a sphere on top, an imaginary sphere (virtual trackball) is used and moving the mouse on top of it has the desired effect. Mouse coordinates are projected onto the sphere and the quaternion representing the rotation from the initial point to the current point is calculated. Because of the extreme changes when the mouse approaches the edge of the virtual trackball, a hyperboloid is used around the edged of the sphere. This is converted to a rotation matrix and applied onto the model by means of a transformation matrix, which is the compound transformation of the translation, scaling and rotation done within the frame to the model. The trackball engine can have many Views assigned to it, as well as many Models. Transformations applied onto a Model are updated on the Views associated to that model. This application lets the user to manipulate the visual data using the mouse in a seemingly intuitive manner. Different applications that use this framework can use the same controller with little programming effort.

model flushDeta(v1ew.stat41

Figure 2. The controller sends the draw command to the view which asks the model to flush its data, but also sends the current state it has stored. The model compares the states, and if it is different it flushes its data and sends the new state back to the view. If the states are the same, nothing is done.

In this solution the View will always ask the Model for the representation data, and will always draw the representation it has afterwards. It is up to the Model to compare states and update the View accordingly.

3. Case Study: Application using The MVC Engine Framework, OpenGL and Doubly Linked Face Lists We have developed two applications with the MVC Engine framework which we believe will help exemplify the use of the framework. In both cases, the Views make use of the Open Graphics Library (OpenGL) for graphics calculations, and the Model stores its data in Doubly Linked Face Lists (DLFLs) [I]. Using OpenGL allows faster graphics calculations and rendering (if the system's hardware supports it), and DLFLs are efficient data structures for the representation of solids.

3.2. Third Person Engine This application features four (or more) views: One is an instantiation of a specialization of the ViewGL class, called ThirdPersonView, and the remaining are standard ViewGL objects. There is also a DLFLModel object (although the application could be used with other Models with little effort), and a new type of Controller called ThirdPersonController. Each View object represents a "camera" in the universe [lo]: It has a position and a direction it is facing. With each View we associate a Viewpoint object, which in essence has a center (a point in the universe representing the position of the camera) and two vectors, a view plane normal (representing the direction the camera is facing) and an up vector. By manipulating these Viewpoint objects, we can change the Views' positions. This is possible through the two new MVC specialization classes for this application - the ThirdPersonView and the ThirdPersoncontroller. The ThirdPersonView adds the other Views' representations (the Viewpoints) to the graphical visualization of the Model, as shown in figure 3. The ThirdPersonController is the key to manipulating the Viewpoints. Specifically, it allows the user to travel

Proceedings of the 19th IEEE Symposium on Computer-Based Medical Systems (CBMS'06) 0-7695-2517-1/06 $20.00 © 2006 IEEE

in the virtual universe as if they were flying in space - it is possible t o go forward, backwards, t o the sides, up, and down. In addition, the user has free control of the direction the camera is facing (by manipulating the view plane normal vector) with the mouse (a "free look" effect).

Figure 3. Shows a reconstruction of a series of DlCOM images in which a contrast agent has been used to highlight the blood vessels. The left image is the normal view, and the right image is the ThirdPersonView, or Overview Window, showing the viewpoint of the left window as well as the viewing volume.

With each keystroke, the ThirdPersonController calculates and updates the active View's Viewpoint location. Afterwards, it notifies the active View of the change. There is no need to notify the ThirdpersonView of the change to the Viewpoint - it will query the Viewpoint itself when it is asked to be redisplayed. With this application, the user can examine the Model's structure with a great degree of freedom. It is possible to "walk" around it t o look at its other side, from above, bellow, and in all angles. The user can place each "camera" (the Views) anywhere he wants and see the structure from many viewpoints at the same time, all with keyboard strokes and mouse movements. The ThirdPersorlViewls window displays the current location of each ViewGL object. An example is shown in figure 4. Event compression [4] also takes place to enhance performance. Many consecutive keystrokes will result in only one final position of the Viewpoint, and by asking the hardware to display only the final scene, unnecessary graphics computing is avoided.

Figure 4. Shows many different views of the same series of images. The upper right picture shows the Overview Window.

4. Conclusions The use of a generic visualization and manipulation model allows the implementation of this framework in any application, not limited to the medical field. Being able t o reuse the code with little implementation effort, or apply similar control mechanisms on different application models is an important aspect of this work and should facilitate further development of medical applications. The limitations of the project in this early stage, however, prohibit the immediate applicability of the applications in the case study. These applications only serve as examples of the capabilities of a generic model. Using virtual reality equiparnent to improve the way t o manipulate the model and visualize it would also improve the usability of the application. Faster rendering method for highly complex models, allowing rendering of highly detailed surfaces can be implemented in a generic way, and would greatly improve this frarnework.

References [I] E. Akleman, J. Chen, and V. Srinivasan. A new

paradigm for changing topology during subdivision

Proceedings of the 19th IEEE Symposium on Computer-Based Medical Systems (CBMS'06) 0-7695-2517-1/06 $20.00 © 2006 IEEE

modeling. Pacific Conference on Computer Graphics and Applications, pages 192-201, 2000. [2] N. Hata, T. Wada, T. Chiba, Y. 'Butsumi, Y. Okada, and T. Dohi. Volume-rendered 3d display of mr angiograms in the diagnosis of cerebral arteriovenous malformations. Academic Radiology, 10(3):309-312, March 2003. [3] K. Henriksen, J. Sporring, and K. Hornbsk. Virtual trackballs revisited. IEEE Transactions on Visualization and Computer Graphics, 10(2):206--216, March 2004. [4] M. J. Kilgard. OpenGL Programming for the X Window System. Addison-Wesley, 1st edition, 1996. [5] J. Kniss, J. P. Schulze, U. Wossner, P. Winkler, U. Lang, and C. Hansen. Medical applications of multifield volume rendering and vr techniques. IEEE T C V G Symposium on Visualization, 2004. [6] G. E. Krasner and S. T. Pope. A description of the model-view-controller user interface paradigm in the smalltalk-80 system. Journal of Object Oriented Programming, 1988. [7] M. Nishihara and T. Norihiko. Usefulness of volume rendered three-dimensional computed tomographic angiography for surgical planning in treating unruptured paraclinoid internal carotid artery aneurysms. Kobe Journal of Medical Science, 10:206-216, October 2001. [8] R. Robb and D. Hanson. Analyze: a software system for biomedical image analysis. Proceedings of the First Conference on Visualization i n Biomedical Computing, 1:507-518, May 1990. [9] S. Seifert, 0 . Burgert, and R. Dillmann. Mediframe an extendable software framework for medical applications. In Proceedings of Surgetica Conference, pages 224-231, France, September 2002. [lo] D. Shreiner, M. Woo, J. Neider, and T. Davis. OpenGL Programming Guide. Addison-Wesley, 4th edition, 2004. [Ill G. Szttkely and R. M. Satava. Virtual reality in medicine. British Medical Journal, 319(7220):1305, November 1999. [12] K. Tsuchiya, S. Katase, J. Hachiya, and Y. Shiokawa. Volume-rendered 3d display of mr angiograms in the diagnosis of cerebral arteriovenous malformations. Acta Radiologica, 5(6):675-679, November 2003. [13] J. Udupa, D. Odhner, H. Hung, R. Goncalves, and S. Samarasekera. 3dviewnix: A machine-independent software system for the visualization and analysis of multidimensional biomedical images. Proceedings of the Annual International Conference of the I E E E Engineering i n Medicine and Biology Society, 5:20822083, November 1992. [14] W. L. J. Wittel and T. G. Lewies. Integrating the mvc paradigm into an object-oriented framework to accelerate gui application development. Technical report, Oregon State University, 1991.

Proceedings of the 19th IEEE Symposium on Computer-Based Medical Systems (CBMS'06) 0-7695-2517-1/06 $20.00 © 2006 IEEE