Inductive Design and Testing of a Performance Ontology for Mobile ...

Chapter 29 INDUCTIVE DESIGN AND TESTING OF A PERFORMANCE ONTOLOGY FOR MOBILE EMERGENCY MEDICAL SERVICES Thomas Horan, Ugur Kaplancali, Richard Burkhard, and Benjamin Schooley Claremont Graduate University

Abstract:

Ontology provides an overarching framework and vocabulary for describing system components and relationships. As such, they represent a means to devise, analyze and compare information systems. This research investigates the development of a software-based ontology within the context of a rural wireless emergency medical (EMS) services. Wireless EMS has developed in response to the unprecedented growth of wireless as a means to communicate in emergency situations. Using an inductive, field-based approach, this study devises and tests a new ontology-based framework for wireless emergency response in rural Minnesota. The ontology is developed by integrating concepts and findings from in-depth field reviews in Minnesota into an ontological software originating out of bioinformatics. This software, Protégé 2000, is an open source ontological software system developed by Stanford University’s Medical Informatics group. Using Protégé 2000, the authors developed a wireless EMS ontological framework populated by the real data gathered from field interviews and related data collection. This EMS framework distinguishes between classes of systems, instances within the classes, and the relationships among classes and instances. The next step in the research involved conducting a simulation of performance using a sample of case study data and demonstrated important linkages among system classes. It is expected that use of such performance ontology will assist researchers and program managers with identifying basic problems in terms of technical and non-technical rural EMS issues, as well as possible patterns of inconsistency or discrepancies across EMS deployments.

Key words:

Web-based Ontology Development; Emergent Systems; Wireless Emergency Medical Services; Rural Mayday; Enhanced 911; Simulation

824

1.

Raj Sharman, Rajiv Kishore and Ram Ramesh

INTRODUCTION

Ontology is of increasing interest to Information Science researchers and professionals (McGuinness, 2002). This interest stems from both their conceptual use of organizing information and their practical use in communicating about system characteristics (Jurisica et al., 1999). Many ontological frameworks have already been developed by academic disciplines such as computer science and bio-informatics and applied to broad variety of businesses from high-tech industries to agricultural sectors (Noy et al., 2000). Within the field of IS, attention to “ontology driven information systems” is now on the rise because the concept of ontology promises a framework for communicating among architectures and domain areas (Smith, 2003). In general, ontology refers to explicit specification of a conceptualization (Gruber, 1993). Ontology development and use of supporting tools offer an opportunity to utilize a unifying framework that embodies objects and concepts, their definitions and relationships between them. Ontology also makes representative content available for knowledge sharing by providing a set of “consistent vocabularies and world representations necessary for clear communication within knowledge domains” (Leroy et al., 1999). Three main uses of ontology are for communication, for computational inference and reuse of knowledge (Gruninger and Lee, 2002). This study is motivated by all three uses of ontology and specifically develops a software-based ontology-driven system to tackle the complexity of wireless EMS and its end-to-end performance. This research takes an inductive approach to ontology system development and applies it within a framework to clarify the domain’s (wireless EMS) structure of knowledge.

2.

EMERGENCY MEDICAL SERVICES

There is increasing pressure to use mobile wireless communications for medical emergencies, yet little is known about its functionality and performance dimensions. Conditions driving the problem include rapid growth of cellular phone use for mayday, strong policy interest in “firstresponder” Mayday as a consequence of 9/11, and policy regulations toward enhanced 911 (E-911) capabilities throughout the US. Statistics about system growth document this rise: Wireless 911 calls have grown from 22,000 per day in 1991 to 155,000 per day in 2001, and represent over 50% of emergency calls made (CTIA, 2002). In short, the mobile (cellular) phone has become the de facto safety lifeline, particularly for mobile travelers and especially in rural areas.

Ontology Handbook

825

While there are several policies, market, and technological pressures leading to emergency medical services growth, the full system is quite dynamic and still unfolding—hence, it is not very well understood. There has been a substantial amount of emergency response and crisis management research and literature aimed at improving the effectiveness of the emergency response infrastructure (Davis, 2002; Hale, 1997; Perrow, 2000). Effective response to “unexpected events” (health emergencies, crises) is highly dependent upon timely and accurate information to and from all participating organizations (Arens and Rosenbloom, 2002; Turoff et. al., 2004). Moreover, the need to improve EMS services is especially true for rural areas where approximately 60 percent of all vehicle fatalities in the United States occur and the average EMS response time between a rural crash and the arrival of the victim to a hospital is 52 minutes, compared to 34 minutes for an urban crash victim (NCSA, 2002).

3.

METHOD

3.1

Inductive approach

The ontology development methodology used in this study is a casebased approach applying inductive methods (Holsapple and Joshi, 2002). Development techniques ascribing to this approach require observing, examining, and analyzing a specific case in the domain in a non-static fashion. Inductive methods use case studies as references for ontology design and then the ontology is refined by evolving toward a more generalized ontology. The inductive approach to ontology design fits perfectly with our purpose to both contribute to and validate the conceptual framework for capturing EMS performance. This bottoms-up approach is particularly appropriate because of the emergent nature of wireless EMS services—that is, the system is growing rapidly and very dynamically due to a number of market, and technology considerations. As advanced by Markus, Majchrzak, and Gasser (2002) such a context lends itself to a design theory approach whereby the system is captured at a point in time, while its eventual functioning may be undetermined. Knowing this, it is our intention to focus on wireless emergency response management implementation with particular attention to “on the ground” performance. In this case, the “on the ground” dimension rural deployment in Minnesota has a distinctive approach for delivering emergency services to rural areas as compared to other states. The wireless EMS is not limited to E-911 infrastructure where PSAPs (Public Safety Answering Point) were established called TOCCs (Transportation Operation Communications Center), in different counties to aid emergency response agencies and

826


incident management dispatches. Our fieldwork involved in-depth field interviews and site visits to Brainerd, Minnesota.1 These interviews were focused on a local EMS services and data use therein. (The research team had previously performed a more global statewide set of interviews). 2 The findings and concepts from these field reviews were then integrated into the ontological software (Protégé) that includes its knowledgebase populated with collected data. Finally, these data were subjected to simulation analysis utilizing a business software simulation program (ARENA).

3.2

Ontology Design Using Protégé 2000

The ontological-development task is to review field interviews and data, and devise an ontology using a platform-independent ontological software product. Specific steps in the process included: local field interviews, conceptual version of the ontology, application of data to ontology (Factual Ontology), and use of ontology software (Protégé) to specify conceptual and factual ontology for EMS. The EMS ontology was developed using Protégé knowledge acquisition software. Protégé 2000 is developed by Stanford University’s Medical Informatics Group as an ontology editor and knowledgebase editor (Grosso et al., 1999). It is a java-based, platform-independent tool for developing ontology and knowledge bases. As an ontological knowledgebase editor, Protégé 2000 has its own knowledge acquisition features similar to available database solutions in the market. The factual wireless EMS ontology and knowledgebase were constructed assuming that data from other rural Minnesota areas in addition to Brainerd will be integrated later when it is available. The only constraint for a future expansion of the knowledgebase is the limited available memory of the Protégé software in the absence of a database running as a back-end. However, since this study is focusing on aggregate EMS data rather than event-specific data the usable memory space is adequate. Because of its graphical user-friendly interface, Protégé 2000 makes it easy to portray and modify the ontological classification of EMS systems in a visually oriented and structured manner.3 Moreover, web publishing of the outcome in the form of an ontology and knowledgebase can increase the accessibility to the domain knowledge. Protégé capabilities in this regard will allow researchers to browse EMS ontology and knowledge bases rather 1

2

3

A more detailed accounting of the field interviews and related EMS data acquisition effort can be found in Horan et al., (2005). This preliminary review is summarized in Horan and Schooley (2005). It resulting in the global framework presented in Figure 29-1 and discussed below under findings. For a more complete description on Protégé, see http://protege.stanford.edu/.

Ontology Handbook

827

than scanning hundreds of pages of technical consultancy papers and documents to quickly find and navigate domain specific knowledge. The research team made two local site visits, interviewing representatives at each site (PSAP, TOCC). During each meeting, the research team provided an overview of the study objectives and asked each interviewee to comment on the organization’s role in each step of the process and the availability of data for documenting the performance of their activity in each relevant step. Four sources were identified for use in creating a factual ontology: the Fatality Analysis Reporting System (FARS), Minnesota FARS, the Baxter Transportation and Operation Communication Center (TOCC), and the Crow Wing County PSAP. The year 2002 was identified as the most recent year with complete data at the time of the review (mid-late 2003).

3.3

Ontology Testing Using ARENA Simulation

EMS is an integrated system and, as such, a dynamic systems approach suggests the utility of portraying a system as an important conceptual step toward understanding how the system operates and evolves (Sussman, 2002). Included in this portrayal is the need to examine links across the various classes in the ontology. A means to portray and understand a system is to model and simulate its performance. Traditionally, it has been used as a technical systems and operations research tool (Sterman, 2000), but it has recently been used to understand more dynamic inter-organizational interactions (Black et. al., 2003). The ARENA simulation package was used by the research team for such as simulation because it is a robust, visually oriented software system that is well suited to processes such as call centers and emergency medical response (Kelton et al., 2004). Drawing from this factual ontology, a preliminary simulation was conducted for rural EMS performance. A judgmental sample of emergency response cases (called “ICRs”) was extracted from the raw data for 2002 with 36 days sampled at 10-day intervals.4 In order to fully capture the expected range of daily call frequencies, data from several days with expected high emergency activity (e.g., December 31, July 4) were included in the sample. The final sample of 40 days yielded a range of eight to thirtyfour incidents per day, with a mean of approximately 14 incidents per day. The distribution of call times was examined for the entire data set of 7,215 incidents, yielding an approximately uniform distribution of calls throughout the day, with the exception of markedly decreased incidence from approximately 3:00 a.m. to 6:00 a.m. 4

The data sampling interval period of 10 days across the 2002 data was chosen to ensure a range of weekdays and weekend days would be included in the final sample of 36 days.

828


4.

FINDINGS

4.1

A Socio-technical framework for EMS

Figure 29-1 provides a high-level overview of EMS systems in rural Minnesota. The framework helps to define the EMS system along several key strata: organizations, technology, and policy.5 A brief summary of each layer of the framework follows. • Organizations – The framework illustrates some of the public and private organizations involved in the Minnesota EMS and the general interorganizational relationships between these organizations. • Technology – The top layer of the framework illustrates some of the essential networks and communications technologies used by Minnesota EMS organizations to carry out their individual and interorganizational functions. • Policy – In order for EMS interorganizational relationships (i.e. partnerships, joint ventures, etc…) to succeed, policies need to be developed that facilitate the interorganizational use of new and existing communications technologies. The overarching EMS technology-related policies currently under development in the state are illustrated (e911, 800 Mhz radio).

Figure 29-1. EMS in Rural Minnesota 5

For more detail on state-wide EMS framework, see Horan and Schooley (2005).

Ontology Handbook

4.2

829

Wireless Emergency Call Routing

While this general system architecture is useful in defining system strata, for the purposes of developing the ontology, it was necessary to translate the overall EMS system architectures into a process that traces the information flows across the EMS system. Such an evaluation occurred through the process of interviewing representatives in Brainerd, Minnesota. Figure 29-2 below shows the wireless mayday call routing procedure in rural Minnesota designed for use in the preliminary design phase of the ontology. The information flow is charted from the originating emergency call to the 911 center (PSAP) and out to various emergency service providers. The linkages of this procedure to the ensuing conceptual and factual ontology are outlined below.

Figure 29-2. Wireless Mayday Call Routing Procedure in Rural Minnesota

830

4.3


Conceptual Ontology for Rural EMS

The socio-technical framework and the wireless mayday call routing procedure described above represents an architectural blueprint for conceptual EMS ontology. All or some of the components and their relationships in this architecture can be translated into an ontological framework. Figures 29-1 and 29-2 are used as the input for the development of the conceptual ontology for EMS using Protégé 2000. At the highest level, this conceptual EMS ontology provides a simple visual representation of the complex and inter-organizational EMS process through its superclass/subclass schema. Further defining subclasses for the knowledgebase implementation where instances (data) are linked with classes produced a comprehensive class hierarchy. The first step in developing the ontology is to establish classes. As a general proposition, these classes followed the process identified in Figure 29-2. Five super-classes were defined and these are: 1. Mayday or Distress Call, 2. Call Routing, 3. EMS Dispatch, 4. Response and Hospitalization, 5. Related System Information. The superclasses are defined first followed by subclasses. The number of superclasses may increase as the case study evolves. All superclasses had at least one subclass and some subclasses had additional classes. A detailed list of all superclasses and subclasses are shown in Table 29-1. Many super/subclasses are created as a manifestation from the wireless mayday call procedure described above. For example, the subclasses called “911 Calls,” “Automatic Crash Notification,” and “Radio Communication” under the Mayday/Distress Call superclass are part of the wireless mayday call routing procedure as well as the technical systems layer of the Sociotechnical framework. On the other hand, there are subclasses that are not directly related to both Socio-technical framework and mayday call routing procedure. One of them is “Benchmark Systems” under the Related System Information superclass. This subclass plays an important role in knowledgebase development with crucial nationwide information attached for comparison. That is, it establishes the basis for communicating performance, another key goal for the EMS ontology. Development of this conceptual ontology played an integral role in course of understanding the “end-to-end” performance issue in EMS. An “end-to-end” EMS system includes an inter-organizational network of service providers delivering time-information critical services. Looking from one “end” to the other “end,” a medical emergency response involves multiple government agencies and non-government organizations, from the

Ontology Handbook

831

time an emergency communication (911 phone call) is made, answered by a PSAP, dispatched to public agency resource (fire, police, ambulance), and treated at the scene and/or ambulanced to a hospital. This conceptual ontology codifies this “end-to-end” perspective and facilitates testing a performance-based ontology, through both its factual instantiation as well as fact-based simulation. Table 29-1. Rural Minnesota Wireless EMS Classes Superclass

Subclass (1st level)

Subclass (2nd level) Accidental Calls

Cellular Phone Call

Emergency Calls

Mayday/Distress Call

Call Routing

EMS Dispatch

Dropped Calls

Automatic Crash Notification

-

Radio Communication

-

Network Routing

-

Routing (Forwarding) Between Dispatching Agencies

-

PSAP Dispatch

-

TOCC Dispatch

-

Other Dispatchers

Assigning EMS Dispatch

Response Coordination and Hospitalization Response & Hospitalization

Arrival to Scene Clearing Emergency/Hospitalization Accidents Data

FARS Data

Injury Data Fatality Data Benchmark Systems

Technology Deployment Related System Information

Wireless Network Coverage Wireless E9-1-1 Deployment Data

Data Management IT Management

832

4.4


EMS Ontology Instances

Following the formation of ontology class schema, data obtained through field interviews and subsequent analysis were entered into the knowledge base. For some classes, data were not available and no instances are recorded. Nevertheless, since the ontological construction of Protégé is in place, data may be entered at any time. The collected data is then integrated and matched with the class schema of the conceptual ontology. The result was a factual ontology, shown in Table 29-2, that also shows some of the

Table 29-2. Conceptual and Factual Wireless EMS System Ontology

Wireless EMS Process

Mayday Call

Conceptual Wireless EMS Ontology o o o

Call Routing

EMS Dispatch

Response & Hospitalization

Related System Information

Mayday Call Cellular phone call Automatic crash notification Radio communication

Call Routing to EMS Dispatcher o Routing delay o Third party routing (GM OnStar) o Mayday Call Response o Response delay o Dispatch o Data Management Response to Incident o Response delay Response Coordination Hospitalization Fatalities E-911 Technology o Network based o Satellite based Wireless Coverage E-911 Deployment

Factual Ontology Brainerd, 2002 Around 21,745 wireless calls in 2002 at Brainerd (Baxter) TOCC Two of seven PSAPs in Brainerd (MSP-D2800) accept all local wireless 911 calls Brainerd PSAP made 36,488 dispatches; TOCC made 7,215 dispatches 71 Fatalities in Brainerd (MSP District 2800). Phase I – 100% complete Phase II – 33% complete

Ontology Handbook

833

data elements of constructed knowledgebase. In this case, the factual ontology pertains to the TOCC activities located in Brainerd, Minnesota. The process of data and/or knowledge entry is based on the previously defined slots and slot-value restrictions, relationships between classes, and properties of these relationships. Protégé 2000 uses its Slots and Forms tabs to enforce such restrictions and relationships. These tabs provide a default layout for capturing and storing data within the knowledgebase and during factual ontology development, and the research team based on the characteristics and applicability of the data at hand formulates them. Data findings used in knowledgebase are explained below.

4.5

Building a Knowledge Base

Protégé software enabled researchers to incorporate both conceptual and factual ontology of rural Minnesota wireless EMS under one ontological knowledgebase. The Protégé system created is called Minnesota Rural EMS. This system allowed researchers to explore conceptual aspects of the actual EMS process (See ontological class schema in Figure 29-3) and extract factual data reports (See Figure 29-4 presenting an instance from the knowledge base) when needed. A factual ontology is an inclusive reflection of the knowledge, which is populated by recording the statistical and qualitative data under instances. Such knowledgebase data includes, for example, FARS data providing the official fatality statistics for accidents, wireless enhanced 9-1-1 deployments in seven MSP districts, and EMS response times measured in minutes. Moreover, this data provided the basis for a simulation of end-to-end performance using ARENA. At the end of the knowledgebase development process, HTML files for the Minnesota Rural EMS ontology created by using Protege’s HTML generator6. This files were posted on the Web to make the results of the ontological–knowledge base accessible to all interested parties and available for future research enquiries.

4.6

Using Ontology to Simulate Performance

Figure 29-5 provides an overview of the ARENA simulation model that was constructed using a sample of the data from the Brainerd Factual Ontology. Each of the regional PSAPs will directly dispatch ambulance, fire, city police, or tow trucks, following the same rules employed by the TOCC emergency services. During the typical scenario, events and calls are generated according to the rules in the preceding discussion of the typical 6

Samples of HTML files generated by Protégé can be found at: http://www.cgu.edu/pages/ 1534.asp.

834


Figure 29-3. Protege 2000 interface showing classes of the ontology

Figure 29-4. A knowledge base instance for Response & Hospitalization class

Ontology Handbook

835

Figure 29-5. Overview of ARENA modeling parameters for factual data simulation

case. Key results of the 14-day run of this simulation scenario are included in Table 29-3. Several observations can be drawn from these results. First, a number of system bottlenecks can be clearly identified. In the model, TOCC dispatchers spend a minimum of 1.2 minutes handling a dispatch, and an average of 4.8 minutes per dispatch. The Brainerd TOCC dispatcher may be able to begin dispatching the necessary services immediately, but the Sample Case may take nearly an hour to dispatch the needed services. Moreover, in the visual model, it becomes clear that this dispatching event is tightly coupled with response service, as demonstrated by subsequent queues in Morrison Ambulance. Conversely, the long queues in dispatching Department of Transportation vehicles (MN/DOT) are not coupled to any subsequent events.

836


Table 29-3. Representative outcomes of sample case

Sample Case – 14 days Emergency Calls – TOCC Dispatcher 1

883

Emergency Calls – TOCC Dispatcher 2

264

Total Incidents

187 Sample Case (hours)

Activity / response unit / incident

Min

Mean

Max

Baxter Dispatcher 1

0.02

0.08

0.16

Crow Wing County Sheriff

0.54

1.16

1.90

Baxter Tow

0.61

1.18

1.95

Mn/DOT

1.31

3.64

7.61

Morrison Fire

1.83

3.47

7.09

Morrison Ambulance

0.63

1.01

1.56

Queue

Min

Mean

Max

Baxter Dispatcher 1

0.0

0.20

0.92

Baxter Tow

0.0

0.06

1.18

Crow Wing County Sheriff

0.0

0.11

2.19

Mn/DOT

0.0

0.15

3.54

5.

DISCUSSION

The ultimate goal of this research is to create a new means to measure, improve, and communicate about the performance of systems such as emergency medical services (EMS). Such an ontology seems particularly timely, as interest in EMS services continues to grow. With several hundred million dollars identified for building local emergency response capability (FHWA, 2002), an ontology can play an important role in organizing, documenting and communicating about system performance using a webbased, visually oriented system. Further, reporting is typically recorded in various types of paper or computer forms by individual agencies. Though there are recent national efforts to construct formalized methods and standards for emergency response reporting into a centralized database using XML and Internet protocols (see NEMSIS, 2004), there is much work to be done at the local level in terms of accepting and implementing such new systems.

Ontology Handbook

837

This study can aid such an effort by demonstrating the use of ontologicalknowledge bases to aggregate performance data from multiple organizations and to aid in diagnosing the existence or non-existence of key data elements in each step of the rural wireless EMS process. Such a system could be a useful step to designing an integrated approach to EMS performance reporting. While there are a variety of approaches available for instantiating ontology, this project demonstrated the utility of platform-independent approaches such as Protégé. Protégé was effective during this creation of a customized domain knowledge-based system because of its ease of use. Protégé capabilities for web posting also attend to the need for communicating the end results. Although Protégé supports a more text-based web appearance, the ability to provide a web-based posting of conceptual and factual ontology can facilitate discussions for further modifications to ontology that will lead to the reuse of the system and the knowledge it already contains. Finally, this provides demonstration of how an ontology-knowledgebase can be used to track and simulate end-to-end performance. While governmental evaluations traditionally involve extensive summative assessments, such an approach brings the performance information into the organizational information system. Such an approach represents an innovation in information systems to improve governmental processes; especially those that transcend traditional organizational boundaries. While several crises management simulation developments are underway (see Jain and McLean, 2003), our endeavor is noteworthy in its use of “off-the-shelf” ontology and business process software as a platform for looking EMS performance, both extant and simulated. The research team continues to evolve this research, including conduct of additional simulation performance under various scenarios (included typical and crises) and calibration with on the ground performance. It is hoped that such an approach will lead to a robust understanding of EMS performance, and position the ontology to play an important role in organizing, documenting and communicating about results.

ACKNOWLEDGEMENTS The authors gratefully acknowledge the support provided by the U.S. Department of Transportation, the Minnesota Department of Transportation, and the ITS Institute at the Center for Transportation Studies, University of Minnesota. Our research would not have been possible without their research and support. Earlier version of this research was presented at AMCIS 2004. Related findings on the EMS architecture and case study analysis were published in Horan and Schooley (2005) and Horan et al. (2005).

838


REFERENCES Arens, Y., and Rosenbloom, P., October 2002, Responding to the Unexpected, A Report of the Feb.27-March 1, 2002, workshop in New York, NY. Black, L.J., Cresswell, A.M., Pardo, T.A., Thompson, F., Canestraro, D.S., Cook, M., Luna, L.F., Marinez, I.J., Andersen, D.F., Richardson, G.P., 2003, A Dynamic Theory of Collaboration: A Structural Approach to Facilitating Intergovernmental Use of Information Technology, Proceedings of the 36th Hawaii International Conference on System Science, Kona, Hawaii. CTIA wireless 911 call stats, 2002, http://www.wow-com.com/industry/stats/e911/. Davis, S., 2002, Virtual emergency operations, Risk Management. 49(7):46-52. Department of Administration, Minnesota Statewide 911 Program, 2003, “Statewide 911 Emergency Telephone Service Program Report”. Federal Highway Administration, 2002, http://www.fhwa.dot.gov/pressroom/test020207.htm. Grosso, W.E., Eriksson, H., Fergerson, R.W., Gennari, J.H., Tu, S.W., and Musen, M.A., 1999, Knowledge Modeling at the Millennium: The Design and Evolution of Protégé 2000, in: 12th Workshop on Knowledge Acquisition, Modeling, and Management, Banff, Alberta. Gruber, T., 1993, A translation approach to portable ontology specifications, Knowledge Acquisition, 5(2):199-220. Gruninger, M., and Lee J., 2002, Ontology applications and design, CACM. 45(2):39-41. Hale, J., 1997, A layered communication architecture for the support of crisis response, JMIS, 14(1):235-255. Holsapple, C., and Joshi, K., 2002, A collaborative approach to ontology design, CACM, 45(2):42-47. Horan, T., McCabe, D., Burkhard, R., and Schooley B., 2005, Performance information systems for emergency response: Field examination and simulation of end-to-end rural response systems, JHSEM, 2(1): Article 4. http://www.bepress.com/jhsem/vol2/iss1/4. Horan, T. and Schooley, B., 2005, Interorganizational emergency medical services: Case study of rural wireless deployment and management, ISF, 7(2):2005 forthcoming. Jain, S., and McLean, C., 2003, “Modeling and Simulation for Emergency Response: Workshop Report, Standards, and Tools.” Washington D.C.: U.S. Department of Commerce. Jurisica, I. J. and Mylopoulos, E.Y., 1999, Using ontologies for knowledge management: A computational perspective, presented at Annual Conference of the American Society for Information Science, Washington, DC, 482-496. Kelton, W.D., Sadowski, R.P., and Sturrock, D.T., 2004, Simulation with Aren, 3rd ed., McGraw Hill, New York, NY. Leroy, G., Tolle, K. M. and Chen, H., 1999, Customizable and Ontology-Enhanced Medical Information Retrieval Interfaces, paper presented at the IMIA WG6 Triennial Conference on Natural Language and Medical Concept Representation, Phoenix, Arizona. Markus, M.L., Majchrzak, A., and Gasser, L., 2002, A design theory for systems that support emergent knowledge processes, MISQ, 26(3):179-212. McGuinness, D. L., 2002, Ontologies Come of Age, in D. Fensel, J.Hendler, H. Lieberman, and W. Wahlster (Eds.), Spinning the Semantic Web: Bringing the World Wide Web to Its Full Potential, Cambridge, MA.: MIT Press. National Center for Statistics and Analysis (NCSA), 2002, U.S. Department of Transportation, National Center for Statistics and Analysis Fatality Analysis Reporting System (FARS) Web-Based Encyclopedia, http://www-fars.nhtsa.dot.gov.

Ontology Handbook

839

National Emergency Medical Services Information System (NEMSIS), 2004, NEMSIS General Fact Sheet, , http://www.nemsis.org/PDFs/factsheetfinal.pdf. Noy, N.F., W. Grosso, W., and M. Musen, 2000, Knowledge-Acquisition Interfaces for Domain Experts: An Empirical Evaluation of Protege-2000, paper presented at Twelfth International Conference on Software Engineering and Knowledge Engineering (SEKE2000), Chicago, Illinois. Perrow, C., 2000, Extreme Events: A Framework for Organizing, Integrating and Ensuring The Public Value of Research, paper prepared for Extreme Events: Developing a Research Agenda for the 21st Century, Boulder, Colorado. Protégé 2000 website http://protege.stanford.edu/. Smith, B., 2003, Ontology and information systems, forthcoming in Stanford Encyclopedia of Philosophy, http://ontology.buffalo.edu/ontology(PIC).pdf. Sterman, J., 2000, Business Dynamics: Systems Thinking and Modeling for a Complex World, McGraw-Hill/Irwin. Sussman, J. M., 2002, Representing the transportation/environmental system in Mexico City as a CLIOS. paper presented at the 5th Annual US-Mexico Workshop on Air Quality, Ixtapan de la Sal, Mexico. Turoff, M., Chumer, M., Van de Walle, B., and Yao, X., 2004, The design of a dynamic emergency response management information System (DERMIS), JITTA, 5(4):1-36.