Modular system architecture for electronic device integration in police cruisers .... list audio object feedback. Application manager. Speech output coordinator.
Modular system architecture for electronic device integration in police cruisers Andrew L. Kun, W. Thomas Miller, William H. Lenharth {andrew.kun, tom.miller, whl}@unh.edu ECE Department, University of New Hampshire, Durham, NH 03824 Abstract In order to solve the problem of device integration for police cruisers Project54 created a modular, scalable system for the integration of in-car electronic devices. The system provides for both hardware and software integration of the devices. The system has a standard and safe speech-based user interface that allows hands-free and eyes-free operation of in-car devices. The New Hampshire State Police started using the system in its cruisers and will install it in all of the cruisers in its fleet starting in the summer of 2002. 1. Introduction The advancement of technology over the last several decades has had a large impact on police cruisers. Today’s cruisers are equipped with digital radios, GPS units, computers, radars, lights, sirens, etc. All of these devices are intended to help police officers perform their job. Law enforcement agencies purchase the devices and integrate them into their vehicles. Integration is needed in three areas. First, the devices have to be physically connected to the system. This means that their hardware has to interact with the cruiser and potentially with other devices in the cruiser. Next, the devices may require software control, or they may have input and output signals that can be put under software control. The software for a device may run on a central in-car computer, and it may have to interact with software controlling other devices. Finally, the user interfaces of the devices have to be installed. This has to be done while taking into account ergonomic requirements. One problem that law enforcement agencies face is that the aftermarket devices they install in cruisers are presently not built with integration in mind. This general problem can be divided into three
interconnected sub-problems that closely follow the areas of integration mentioned above: • The devices are not hardware compatible. The devices are not built using a standard for hardware interconnections. Some devices are only controlled using buttons. Others have sophisticated hardware interfaces (e.g. serial or parallel buses) which allow user input and output. Some provide feedback to the user through LEDs, others use built-in displays. As a result integrating the devices requires creating custom hardware interfaces. • The devices are not software compatible. Once the hardware compatibility issue has been solved, the devices can be controlled by software running on a computer. However, there is no software standard that would allow writing applications for devices meant for the law enforcement market. This means that computer control of devices, as well as any inter-device interaction, has to be dealt with on a case-by-case basis. • Each device has its own user interface. Since the devices are not hardware or software compatible, each of them needs its own user interface. This results in a multitude of individual interfaces in the cruiser. The main problem with having a large number of interfaces is that interacting with them is distracting to the officer. Undeniably using the devices presents a cognitive load in itself. However, this cognitive load is increased by dispersing the user interfaces around the car. An added problem is that most interfaces rely almost exclusively on switches or buttons for input from the user and LEDs or displays for output to the user. This creates a safety problem in the eyes and hands-busy car environment. The goal of Project54 [1] is to solve the problem of device integration for police cruisers. From the point of view of system assembly the project aims to create a modular, scalable system which is easily installed, modified, expanded, inspected, and
repaired. From the user point of view, the goal of the project is to create a system with a standard and safe user interface that will allow hands-free and eyes-free operation of in-car devices. Project54 approached the integration problem using a two-pronged approach. First, we decided to adopt publicly available hardware and software standards for in-car devices. If devices are built using these standards it will be possible to include them in incar systems without costly integration efforts. Second, we decided to create a standard, unified, speech-based user interface for the system. This user interface is shared by all in-car devices. By creating the system with a unified user interface we expected to minimize the cognitive load of the user. Speech was chosen to allow the users to keep their hands on the wheel and their eyes on the road. The standard nature of the user interface makes writing applications relatively easy. 2. Background Researchers at Texas A&M worked on improving the functionality of police cruisers using advanced electronics through the ALERT project [2, 3]. The project addressed the need to integrate various electronic devices in police cruisers. However the project did not result in the acceptance of a standard for integration; rather it was a proof-of-concept effort. Wahab et al. worked on creating a dashboard system for intelligent vehicles that would be able to handle data and speech communication [4]. They paid special attention to combating noise in the car that interferes with voice communication. Muthusamy et al. created a prototype system for information retrieval in cars [5]. The system has a voice interface (input and output), and it can be used for voice dialing, Internet operations, and help with navigation. Speech recognition in cars is of great interest to researchers as well as industry. An ambitious effort uniting car manufacturers, telecommunications equipment manufacturers, mobile telephone network operators, and universities is the SpeechDat-Car data collection effort. SpeechDat-
Car collected in-car speech data from multiple languages [6, 7]. The data is available in databases that can be used to improve the performance of speech recognition in cars. 3. Hardware integration The Project54 system realizes hardware integration by using the Intelligent Transportation System Data Bus (IDB). The IDB standard prescribes both the hardware components and the protocol needed to accomplish control tasks and to exchange information between the car’s systems [8].
Figure 1 The common IDB interface allows IDB messages to be passed to the aftermarket device and responses from the device to be put on the IDB. Most devices meant for police cruisers are not able to connect to the IDB, however most of them have serial and/or parallel inputs and outputs. Therefore we developed an embedded device that can interface to electronic products in a serial and parallel fashion, and can communicate with the IDB. Using this device, messages from the IDB can be translated and sent to a connected device, and responses from that device can be put on the IDB, as shown in Figure 1. We call the device the common IDB interface. The common IDB interface hardware uses Version 2.0B of the Control Area Network (CAN) protocol as the method of data transmission on the IDB. The CAN protocol, developed by Robert Bosch GmbH, uses differential signaling that is well suited for noisy environments such as an automobile. As a means to connect the aftermarket devices to the IDB the interface has two options. The first type of connection offered is a serial port that uses standard RS-232 signaling. The second type of
connection is a port that offers five parallel TTL level signals that can be used for a variety of applications including switching.
would also allow the upgrading of existing software components as more advanced ones become available.
The flow of data between the aftermarket device and the IDB is handled by a microcontroller. During initialization the microcontroller sets up the device’s unique IDB address, and sets up serial and/or parallel communication with the device.
The second goal was to create the software architecture such that speech can easily be used as the user interface for all applications. We decided to accomplish this by allowing developers to write applications for our system without having to learn how to use speech recognition (SR) and text-tospeech (TTS) engines directly. Initialization list
Grammar
Keyboard and mouse input Application manager
Command list
status messages
Message coordinator
control messages
Logging coordinator
recognition results
Speech output coordinator
logging messages
Speech recognition engine audio object feedback coordinated speech output Text-tospeech engine
Application 1 Application 2
Figure 2 Aftermarket devices connected to the IDB using the common interface (CI). Figure 2 shows some of the devices that are being integrated into the system as part of Project54. At the center of the system is an embedded PC. Control of all devices is performed by this PC. The PC uses the IDB to communicate with the devices. The aftermarket devices in a police cruiser range from standard serial devices, such as GPS, radar, etc., to relay driven light bars. The devices are connected to the IDB through the Project54 common IDB interface (CI in Figure 2). The PC is also connected to PC-native devices, such as a hard-drive, audio device, LCD touch screen, etc. 4. Software integration and user interface integration The software architecture was designed with two goals in mind. First, we wanted to create an open and modular architecture, which would allow the addition and exchange of components. Such a system would allow the easy addition of software components for the control of new hardware. It
Application n
speech output
Figure 3 Software system architecture Figure 3 shows a detailed view of the Project54 software system. At the center of the system is the application manager. The manager sends control messages to the individual applications. The applications control individual functions of the police cruiser (for example one application controls the light bar while a different application controls the radar unit). The applications talk to three modules in the system. The first one is the message coordinator. This module receives status messages from the applications. These messages are forwarded to the manager. The second module is the logging coordinator. This module provides a centralized means for all applications to log events and errors. Finally, the speech output coordinator is responsible for an orderly output of speech messages to the user. Multiple applications may want to send speech output. Without the coordinator they would have to find a way to negotiate which one of them can send output first. They would also have to know how to talk to the TTS engine, which is not necessary in this system. The manager receives input from a mouse, a keyboard, and an SR engine. Speech is the
preferred means of communication in the Project54 system. However, interacting with speech engines is not trivial. The Project54 software system centralizes this responsibility by designating the manager as the only part of the system that can interact with an SR engine. As a result individual applications do not need to know how to do this. In fact, they do not even need to know that the user uses speech as the input to the system. All they need to know is how to receive control input from the manager. Application developers have to be aware of the existence of a grammars used by the SR engine. The grammars are supplied to the engine in the form of text files. The format of this file is determined by the SR engine. A commonly used industry standard (SAPI 4) exists which describes a standard for grammar file format. The manager uses two system files in its operation. One of them is the initialization list file. This file lists all the dynamic link libraries (DLLs) that have to be loaded and all the executable files that have to be run in order for the system to work. The DLLs and the executable files contain the individual applications. The second file is the command list file. The command list is used to associate recognition results from the SR engine with actions of specific applications. Recognition results (words or phrases) are sent to the correct application using the command list. The application receives the recognition result in text format and decides on a proper course of action. Let us look at who is expected to develop which part of the system. Blocks in the cloud with the horizontal lines are to be implemented by Project54 developers. These blocks perform the central functions of the system. Blocks in the cloud with the box pattern are the applications controlling the electronic devices. The idea of the system architecture is that these applications can be changed easily. Thus a user may decide to use some applications written by Project54 developers and some applications written by other developers. Blocks in the clouds with cross hatched lines are system files that can be edited by both Project54 and other developers. However, the format of the files is determined by Project54. These files contain
startup information and command association information. Blocks in clouds with diagonal lines are supplied by a third party. These blocks represent the SR engine and the TTS engine. Since interaction with the engines is based on an industry standard (SAPI 4) the engines can be substituted easily. This allows the user to upgrade the system and take advantage of the fast development of new engines. Finally the grammar block is in a cloud with vertical lines, which signals that both Project54 and other developers can edit this file. However, the format of the file is given by a third party - the company whose SR engine is used in the system. Again, SAPI 4 gives a standard for the grammar file format. 5. Software implementation The Project54 software executes under the MS Windows 2000 operating system. In order to provide application developers with the convenience of being able to develop code using the tools and operating systems of their choice, the system architecture is implemented using the Component Object Model (COM) [9]. Figure 4 shows the implementation of the system using COM. The application manager is implemented by Project54 developers. It contains three COM objects that receive messages from the individual control applications. These objects are the logging coordinator object, the message coordinator object, and the speech output coordinator object. Every application (Application 1 through n in Figure 4) has to implement a message handler object. This object receives messages from the application manager. The application manager is also responsible for dealing with the SR and the TTS engines. This is done using audio-, speech recognition-, and text-tospeech COM objects. However, this means that from the perspective of developers the software system consists of: • the application manager, which is provided by Project54, and implements three objects: the logging coordinator object, the message
•
coordinator object, and the speech output coordinator object, and applications written by developers, each implementing a message handler object. Application manager
SR or TTS object 2
...
Message coordinator object
Application 1
Application 2
Message handler object
Message handler object
Audio object m
...
...
...
Logging coordinator object
Audio object 2
...
...
SR or TTS object 1
...
...
Audio object 1
SR or TTS object n
Speech output coordinator object
Application n ...
Message handler object
Figure 4 Software implementation using COM 6. Using speech recognition Within the software system the application manager is responsible for dealing with SR engines. This involves identifying the appropriate grammar for a given instance of time. Grammars provide a set of words and phrases that can be recognized by an engine. The performance of an SR engine will be better the smaller the grammar is. Therefore, it is advantageous to create several small grammars and load them according to the situation, as opposed to having an all-encompassing grammar. An example would be the case when the user is asked to tell the computer what gender the subject of a search is. The only valid answers are “male” or “female”. The SR engine can use a grammar containing these two words and a small number of other utility words. The Project54 software system allows applications to control which grammar is currently being used. This is accomplished by sending a message requesting a change of grammar to the application manager. The message is of the form: GRAMMAR grammar_file where grammar_file is the filename of the grammar to be used. If grammar_file is NULL the default grammar is used. Recall that the system uses a
command list file to associate recognized commands with applications. This file is used when the default grammar is in use. If an applicationspecific grammar is loaded all recognition results are passed to the application that requested the loading of that grammar. 7. Results Figure 5 shows the Project54 system hardware installed in a New Hampshire State Police (NHSP) cruiser. The center console contains the embedded computer, the radio, the radar, and miscellaneous electronics. The vehicle is equipped with an LCD touch screen. The touch screen can be used as a second input and output modality in case the speech interface is not available. There is also a directional microphone on the dashboard, as well as a push-totalk button on the steering wheel. The directional microphone reduces the influence of sounds that are not coming from the driver of the vehicle – this improves SR engine performance. The push-to-talk button is used to start recognition by the SR engine.
Figure 5 System hardware Figure 6 shows the graphical user interface (GUI) used by the application manager. This GUI appears on the LCD touch screen. On the left there are two columns of virtual buttons. These buttons can be used to move from the application manager GUI to the GUI of any other application. Of course, the user can start up the desired application by issuing a voice command – the GUI is only a backup. Figure 6 shows the application manager GUI but all GUIs
have the same look and feel. This is accomplished by providing a GUI service DLL which all applications call when creating their GUIs.
Another area of interest is the inclusion of handheld computers in the system. A handheld computer could be used to access some of the functionality of the car from outside the car. It could also be used as a backup to the embedded computer – if the embedded computer was damaged the handheld computer could be connected to the IDB and it could take control of the system. Acknowledgement The authors wish to thank the US DOJ for funding Project54 through grant #1999-DD-BX-0082.
Figure 6 Graphical user interface 8. Conclusion and future work We created a modular, scalable system for the integration of in-car devices. The system is easily installed, modified, expanded, inspected, and repaired. The system uses a standard speech-based user interface which is safe because it allows handsfree and eyes-free operation of in-car devices. As of April 2002, Project54 has equipped three police cruisers with the new system. One of these cruisers is being used for testing purposes at the University of New Hampshire, while the other two are currently on the road in everyday use by police officers. Four to five more cruisers will be put on the road and tested in the summer of 2002. The entire fleet of the New Hampshire State Police (250 cruisers) will be equipped with the system, starting in the summer of 2002. The development work continues with work on several issues, including speech recognition. While our system can use SAPI 4 engines, there are other industry standards that we would like to support. We are designing a new way for the application manager to interact with SR engines that would allow the use of SAPI 4 as well as other SR engines. We are also testing various microphones and commercially available SR engines.
References [1] http://www.catlab.unh.edu/ [2] http://alert.tamu.edu/ [3] J. A. Ochoa, J. Witz, M. Eshani, and J. Morgan, "Advanced law enforcement vehicle electronics and associated power," 18th Digital Avionics Systems Conference, St. Louis, MO, 1999 [4] A. Wahab and T. E. Chong, "Intelligent dashboard with speech enhancement," Proc. Int’l Conf. on Information, Communications and Signal Processing, Singapore, 1997 [5] Y. Muthusamy, R. Agarwal, Y. Gong, and V. Viswanathan, "Speech-enabled information retrieval in the automobile environment," ICASSP, Phoenix, AZ, 1999 [6] M. Sala, F. Sanchez, H. Wengelnik, H. van den Heuvel, A. Moreno, E. Deregibus, G. Richard, and E. le Chevalier, "Speechdat-Car: Speech databases for voice driven teleservices and control of in-car applications," EAEC 99, Barcelona, 1999 [7] P. Heeman, D. Cole, and A. Cronk, "The U.S. SpeechDat-Car data collection," Eurospeech, Aalborg, Denmark, 2001 [8] M. Martin, "The Project54 Common Interface for the Intelligent Transportation Systems Data Bus Version 4," Technical Report ECE.P54.2001.3, ECE Department, University of New Hampshire, 2001 [9] Dale Rogerson, Inside COM, Microsoft Press, Redmond, WA, 1997
