Virtual and Augmented Reality Applied to Educational and ... - CiteSeerX

Virtual and Augmented Reality Applied to Educational and Cultural Heritage Domains N. Mourkoussis, F. Liarokapis, J. Darcy, M. Pettersson, P. Petridis, P. F. Lister, M. White The Centre for VLSI and Computer Graphics School of Engineering and Information Technology The University of Sussex BN1 9QT, United Kingdom {n.mourkoussis, f.liarokapis, j.darcy, m.pettersson, p.petridis, p.f.lister, m.white}@sussex.ac.uk Abstract This paper discusses four research projects currently running in the Centre for VLSI and Computer Graphics. These projects exploit a variety of state-of-the-art technologies, such as photogrammetry, 3D modeling, augmented and virtual reality, and multimedia applied to the educational and cultural heritage domains, while ensuring interoperability through the use of XML. Furthermore, our user-driven prototype approach facilitates the development of valid and exploitable virtual and augmented reality systems through the use of user trials and assessment and evaluation, thus closing the quality assurance loop.

Representation of Cultural Objects (ARCO); all of which address the aforementioned shortcoming. These four projects are related in several ways, for example, some share the same base technologies, such as XML [W3C 2000] and database technologies; some target similar application fields, e.g., learning environments; while some require similar visualisation techniques. Figure 1 illustrates the relationships between the various application domains and the technologies used. Technologies XML/Databases

Photogrammetry

1.

Introduction

The Centre for VLSI and Computer Graphics embraces a number of research areas, including but not limited to multimedia, augmented and virtual reality, the generation of virtual environments applied to virtual prototyping, cultural heritage visualisation systems and teaching and learning environments. Of particular interest are the cultural heritage and educational sectors, because although computer graphics in general and—more specifically—virtual reality have been successfully applied to a great many areas, there still seems to be a lack of innovation within these particular fields. In this paper we will discuss four interrelated projects: Lambert Watercolours Visualisation (LWV); Virtual Interactive Teaching Environment (VITE); Augmented Reality in Cultural Heritage (ARICH); and Augmented

Application Domains

Visualisation

LWV Cultural Heritage Education

VITE

Web-based Rendering

Education

ARICH

Manual 3D Modelling

Cultural Heritage

AR Hardware/ Software

Cultural Heritage

Photo-realistic Rendering

ARCO

Figure 1. Technologies, application domains and visualisation functionality provided by the four projects outlined in this paper A key socio-economic research area of the work is the identification and subsequent involvement of real users, e.g., museum pilot sites as in the ARCO project (see section 5), who provide valuable feedback, which we gather through the development of prototype systems. These prototype systems are then assessed and evaluated at museum user trials. Hence, we ensure that final systems

Witold Abramowicz (ed.), Business Information Systems, Proceedings of BIS 2002, Poznan, Poland

368

BUSINESS INFORMATION SYSTEMS – BIS 2002

fully conform to actual user needs, and that future exploitation efforts have a greater chance of success.

2.

The Lambert Watercolour Visualisation Project (LWV)

The LWV project outlines a method of viewing the collection of watercolours—painted by the artist George W. Lambert—through a typical Internet browser. This involves the configuration of a database and the subsequent development of two different interfaces to the database: a web-based visualisation of the watercolours and associated metadata, and a user-oriented administration interface. The web interface allows the public user to query the database and visualise the watercolours in a multimedia environment, enriched by 3D models and interactive real-time rendering, which produce a more interactive and attractive user experience. The user- orientated administration interface is developed to operate on the database, providing the user of the system with an easy-to-use graphical user interface for manipulating server-side data. The architecture of our system is illustrated in Figure 2. Public User Web Interface

XML formatted SQL queries dfdgdfdg dfddg dfddg dfdgdfdg dfddg dfddg

Query results - - - --- -- - -- - - -- - - - -- --- - - - -- - - ---- ----- ----- --- ---- -- -- - ---

Oracle 8i

XSQL Servlet

JDBC

Administrator Admin Interface

2.2

Web Interface

One of the objectives of this project is to provide the general public with access to the watercolours and associated multimedia metadata over the Internet in a learning environment. Therefore, it is necessary to design a gateway to the database that is compatible with web browsers, such as Microsoft’s Internet Explorer or Netscape. This is achieved through the use of web pages, designed using HTML, Macromedia Flash, etc. The chosen database, i.e., Oracle, supplies a Java-based XSQL servlet, which we use to access the watercolour collection information stored in the database. Figure 3 presents a step-by-step diagram of the XSQL servlet, which illustrates how data flows from a client (i.e., web browser), to the server, and back to the client. During this process, the user accesses the project’s web site and enters search criteria using the provided search engine (step 1). These parameters are then taken from the web page and incorporated into a dynamically constructed XSQL file. The servlet sends this file (step 2) to the ‘XML parser for Java’, which parses the document and allows the contents of the document to be accessed. The ‘XSQL Page Processor’ then passes the XML parameters and SQL statements to the ‘XML SQL Utility’ and the XSL processing statements to the ‘XSLT Processor’ (step 3). Then the ‘XML SQL Utility’ queries the database and the results are returned back to the ‘XML SQL Utility’ (step 4), which passes the results to the ‘XSLT Processor’ (step 5). Finally, the ‘XSLT Processor’ transforms the results with the use of a style sheet (step 6), and the final document is passed back to the web browser (step 7). This process is described in more detail in [Wait 2001]. This basic process is adopted for both the LWVR and the VITE architectures, see Figure 3.

Figure 2. The overall architecture of the LWV system

2.1

Watercolour Database

The Lambert Watercolour collection is a physical medium, and, as such, it needs to be transferred to a digital one through a photographic digitisation process. The resulting digital images are then stored in the database, and are transferred to and from it using the processes described in sections 2.3 and 2.4. For the implementation of the database, an Oracle 8.1.7 server is used. The reasons for choosing this Database Management System (DBMS) are its extensive support for Java and its smooth integration with XML technologies.

Figure 3. Technical illustration of the XSQL servlet, see [Wait 2001] for further information


VIRTUAL AND AUGMENTED REALITY APPLIED TO EDUCATIONAL AND CULTURAL HERITAGE DOMAINS

2.3

Administrative Interface

The administrative interface is implemented in Java and designed to provide end-users (museum staff) of the Oracle database with a simple, user-friendly tool to administer the database of Lambert watercolours. The current version of the administrative interface is functionally dependent on the cultural metadata; however, future implementations will be dynamic and change with the metadata. Another option in the administrative interface enables the user to insert new entries into the database by importing an XML file. Finally, the administrative interface provides the functionality to export a database entry to an XML file. The administrative front-end is connected to the database through a JDBC driver. The JDBC API provides our application with easy access to our database, and allows us to use any JDBC compliant SQL database.

2.4

369

Figure 5. Editing a database entry

First Prototype

We have implemented a first prototype system, which enables the user to administer the Orcale 8i database without any knowledge of SQL. The user can create an XML output file, which exposes the metadata to other applications. In addition, the user can import any metadata from existing XML files as long as they use the same Document Type Definition (DTD). Presentation is enhanced through supplementary virtual reality content, e.g., VRML. Figure 4 to 6 illustrate the LWV prototype interfaces.

Figure 6. XML-based file I/O

3.

Augmented Reality in Cultural Heritage (ARICH)

Our current research efforts have been focused on the design, development and implementation of an indoor Augmented Reality (AR) system for visualisation of archaeological artefacts. The system, presented in Figure 7, consists of three major parts: 3D modelling, photogrammetry and mixed or augmented reality. Two of these are within the scope of this project, as listed below, while photogrammetry is part of the ARCO project, see section 5: • Automated modelling from architectural plans • Efficient and realistic AR rendering Figure 4. Querying the database from both the web and administrative interfaces


370


software/hardware and the existing electronic design automation tools that operate with VITE, see Figure 8 below.

Figure 7. A general overview of our system The input to our system is 2D architectural plans provided by the archaeologist. Then, using 3ds max from Discreet, semi-automated modelling takes place based on scripts, which are algorithms written in MaxScript (the 3ds scripting language), developed as part of this project. Furthermore, manual modelling, such as geometrical reconstruction of missing parts, will be utilised to refine the model. Or, as in ARCO, photogrammetry techniques will be used to digitise 3D models. Finally, the 3D model (geometry, normals, material colour, texture, and lighting information) is output, ready for interactive rendering or visualisation. Our chosen 3D modelling package (3ds max 4) has a utility called ‘camera match’ that allows us to mix the video input with the computer-generated model. However, we plan to use the ARToolKit [Billinghurst] to integrate the video and 3D models in an augmented reality environment.

4.

Virtual Interactive Teaching Environment (VITE)

We are currently developing a new approach to the teaching of top-down design of VHDL using a novel Virtual Interactive Teaching Environment [White 2001]. This environment enables students to learn more effectively using virtual multimedia content, while exploiting XML and augmented reality. VITE is an AR teaching system that will provide teachers or trainers with 3D visualisations of teaching materials in an augmented reality environment, e.g., sample designs, tutorials, 3D objects, reference manuals, etc. VITE has to support communication and interaction between users, i.e., the students and the teacher, within the virtual environment. The Virtual Multimedia Context, or VMC, which is stored in a database and visualised on an appropriate medium, is the information interface between the AR technology and the users. The technology encompasses both the AR

Figure 8. The VITE system architecture The effectiveness of VITE is based on many parameters concerning technological issues such as: choice of display systems; haptic and kinaesthetic devices; object and user tracking systems; and auditory perception devices. Other factors concern virtual multimedia issues such as: generation of augmented reality software, creation of a large database and threedimensional modelling. Effective implementation of these systems in VITE will allow real-time interaction between users and the system [White 2001]. As we can see from Figure 8 we use the same database technology to drive the VITE system as we do for the LWV project

5.

Augmented Representation of Cultural Objects (ARCO)

There are a multitude of small and large museums across Europe that hold countless archives of varioussized artefacts, which they currently cannot exhibit in a convenient, low-cost manner. There are several reasons for this: many museums do not have the space required to exhibit all their archives to the public; and many cultural artefacts are simply too fragile. Hence, the current cost of exhibiting these archives is prohibitive. Furthermore, digitisation of these archives using traditional 3D modelling techniques requires great expertise, something that museum staff generally does not possess. Museums need a cost-effective way of creating virtual representations of their archives, suitable for presentation to the public and cultural researcher alike. The methodology used has to be suitable for existing museum staff with little expertise. Additionally, it has to adopt, or even drive, international standardisation efforts to facilitate exploitation in the museum and public domain.


VIRTUAL AND AUGMENTED REALITY APPLIED TO EDUCATIONAL AND CULTURAL HERITAGE DOMAINS

5.1

The ARCO System

The system proposed in the ARCO project is based around innovative and robust digital capture and presentation techniques, allowing curatorial staff to efficiently solve the aforementioned problems. We will analyse and provide innovative yet simple-to-use technical solutions for: • Automated creation of virtual cultural objects using Object Modelling through photogrammetry. • Refinement of these objects—including combination with environmental data as well as other supplementary multimedia data—through Interactive Model Refinement and Rendering. • Management of all these data through an XMLdriven Cultural Object Relational Database Management System. • Reconstruction and presentation of the collections and their environments, based on an Augmented Reality Interface or a web browser. Modules for the above solutions will be developed as part this project. Furthermore, XML will be used as the interfacing technology between all system components, fully exploiting its interoperability aspects. During the project lifetime we will track current trends in VR languages, and may incorporate those that seem likely candidates for emerging standards. As an example, the dynamic and database-connectivity capabilities of XVRML make this new meta-language look particularly promising. X-VRML [Walczak 2002a, Walczak 2002b] is being developed by PUE, one of the ARCO project’s consortium partners. Automatically Acquired 3D Cultural Objects

Digital Acquisition Of the Cultural Object (Digital Video)

(The XML Object Description)

Application of camera based modelling algorithms (XML Polygon Mesh)

Interactive Model Refinement and Rendering

The Cultural Object Archives

Cultural Objects selected by the museums

Museums’ provide the XML attributes that describe the cultural object

Museum Curatorial Steering Group (Museum Input)

(3D Studio Max plug-in based on Max Script, Enhanced XML Object Description)

Registration

Museum evaluation Augmented Virtual Representation

The Cultural Object ORDBMS Augmented Reality Interface

Figure 9. The overall architecture of ARCO The project will evolve iteratively through a series of prototype systems, enabling feedback from the users to continuously influence the development work, and

371

eventually leading to a final, fully integrated system. This system will closely conform to actual user needs, and will be evaluated both technically and socio-economically. We offer a fast, simple and flexible method of capturing, refining, storing and presenting archives, while allowing for advanced searching, retrieval and browsing of archives—both locally and via the web—subject to security that protects museum Intellectual Property Rights (IPR). The system will also increase the curators' efficiency in identifying the origin of new artefacts, by offering efficient analysis using the ARCO system, and facilitate standardised cataloguing through the use of XML-based archives. 3D cultural objects could be enhanced with interpretations of what the complete cultural object might have looked like. For example, a fragment of pottery may be augmented by rendering the missing parts. Other types of refinement might include for example the application of textures, colour, shadows, lighting, plus the addition of multimedia information, such as text and audio—all tied together through XML descriptions.

5.2

Exploitation Opportunities

Cultural archives—vast in number, located all over Europe, and many never even displayed to the public— will be made available to a large audience through the deployment of ARCO systems. In theory, this will create new business and employment opportunities—a key objective of ARCO. Museums will be in a better position to exploit their assets in a more effective way, as they will be in full control of the new technologies. ARCO also aims to help remove the economic barriers and technological complexities that currently prevent the widespread use of 3D capturing technology in the cultural sector. The methods used to manually create 3D models, and to render, animate and visualise them are enhanced and augmented with ARCO. The notion of converging media is also a key aspect to the ARCO system. Through the use of innovative Information Communication Technology ARCO brings together the media of 3D, 2D, text, audio and augmented reality, providing an important cultural resource that can be disseminated throughout Europe. Such a resource may provide exploitable business opportunities not only for the ARCO system but also for any future systems developed in this area.

6.

Conclusion

We have described several projects currently underway in the Centre for VLSI and Computer Graphics. The largest is the ARCO project, a major European Union IST Framework 5 project involving a total of seven European partners. The LWV project is an ongoing research project


372


that was initiated as part of a fourth-year master level group project. All projects include the application of augmented or virtual reality, and share some base technologies, such as the database system and various visualisation techniques.

7.

Acknowledgements

Part of this work has been funded by the EU IST Framework 5 Programme under Key Action III Multimedia Content and Tools, Action Line III.1.6 Virtual Representations of Cultural and Scientific Objects.

8.

References

[Billinghurst] M. Billinghurst, “ARToolKit”, Human Interface Technology Laboratory and Center for Environmental Visualization, University of Washington. http://www.cev.washington.edu/artoolkit/

[W3C 2000] T. Bray, J. Paoli, C. M. Sperberg-McQueen, and E. Maler (ed:s): “Extensible Markup Language (XML) 1.0” 2nd Ed., W3C, October 2000. http://www.w3.org/TR/2000/REC-xml-20001006 [Wait 2001] B. Wait, R. Hall: “Using XML in Oracle Database Applications” (updated), Oracle Corporation, June 2001. http://technet.oracle.com/tech/xml/info/htdocs/otnwp/about_or acle_xml_products.htm [Walczak 2002a] K. Walczak, W. Cellary: “X-VRML – XML Based Modeling of Virtual Reality”, Proc. Int’l. Symposium on Applications and the Internet (SAINT-2002), January 2002. [Walczak 2002b] K. Walczak, W. Cellary: “Building Database Applications of Virtual Reality with X-VRML”, 7th Int’l. Conf. on 3D Web Technology (Web3D 2002), February 2002. [White 2001] M. White, E. Jay, F. Liarokapis, C. Kostakis, P. Lister: “A Virtual Interactive Teaching Environment (VITE) Using XML and Augmented Reality”, International Journal of Electrical Engineering Education, vol. 38, no. 4, pp. 316– 329, October 2001.
