ActiveCube and its 3D Applications - Semantic Scholar

IEEE VR 2004 Workshop Beyond Wand and Glove Based Interaction, March 28, Chicago, IL USA

ActiveCube and its 3D Applications Hiroyasu Ichida, Yuichi Itoh, Yoshifumi Kitamura, Fumio Kishino Graduate School of Information Science and Technology, Osaka University 2-1 Yamadaoka, Suita,Osaka 565-0871, Japan E-mail: {ichida, itoh, kitamura, kishino}@ist.osaka-u.ac.jp http://www-human.ist.osaka-u.ac.jp/ActiveCube/ Abstract ActiveCube is a novel device that allows a user to construct and interact with a 3D environment by using cubes with a bi-directional user interface. A computer recognizes the 3D structure of connected cubes in real time by utilizing the real-time communication network among cubes. Also, ActiveCube is equipped with both input and output devices, at where the user expects to be, and this makes the interface intuitive and helps to clarify the causal relationship between the input of the user's operational intention and the output of simulated results. Consistency is always maintained between the real object and its corresponding representation in the computer in terms of object shape and functionalities. A variety of applications can be developed by utilizing ActiveCube. In this paper,we introduce a retrieval system of 3D shape models, a story-telling system and assessment of human’s cognitive spatial ability as examples.

1. Introduction 3D object shape modeling and interaction are essential elsewhere, however, it is still difficult if ordinary interaction devices are used because of the complexity of 3D space. If these processes are accomplished by assembling and interacting physical blocks, this will be a solution to solve the problem of complexity in 3D space by offering an easy way to recognize the spatial configuration in a 3D environment. Recent research efforts attempted to design user interfaces that use physically meaningful objects to improve the intuitiveness of 3D object modeling or interactive manipulation. If users were able to construct 3D objects by simply combining physical blocks, the user interface for 3D object shape modeling would become intuitive. Moreover, if the constructed object were to have a functionality to accept the user’s input and to express the simulated output results, the user could directly interact with the 3D environment by using the constructed object instead of using ordinary interaction devices such as the mouse, keyboard and display monitor. Consequently, the user interface would become more intuitive, and it would be easier to understand what is

happening in the virtual environment, because the constructed object would act as a physical replica of the virtual structure. To achieve these features, we proposed and implemented a user interface called ActiveCube [1]. It allows users to construct and interact with 3D environments using physical cubes with a bi-directional user interface. A variety of applications can be developed by utilizing ActiveCube, and as examples, a novel user interface which allows users to retrieve a 3D shape model [2], a story-telling system [3] and assessment of human’s cognitive spatial ability [4] using ActiveCube are introduced.

2. Implementation of ActiveCube ActiveCube is a set of rigid cubes with 5-cm length edge, and users can construct various 3D structures by combining the cubes as they desire. The faces of the cubes are the same so that each cube can be connected to any other cube. The 3D structure of connected cubes is recognized by a computer in real time; therefore, users can construct a 3D structure in the computer (i.e., in the virtual environment) that exactly corresponds to the physical structure of the physical cubes in the real environment at the moment.

2.1. ActiveCube Hardware To achieve the functionality of real-time interaction, a real-time network management system called LON (Local Operating Network) technology (Echelon Corporation) is used. Neuron Chip is incorporated into each ActiveCube for the communications among the cubes. An ID number is assigned to each cube (cube ID) for the unique identification of cubes. An ID number is also assigned to each face (face ID) of the cube to identify the connecting face. Connected cubes constitute a network where a parallel RS-485 port is used for communications between cubes. The cubes are connected to the host PC through a special cube called a base cube.

IEEE VR 2004 Workshop Beyond Wand and Glove Based Interaction, March 28, Chicago, IL USA

2.2. Real-time Recognition of 3D Structure When a new cube (a child cube) is connected to a cube that has been already connected to the network (a parent cube), the child cube is supplied with power when it is connected, which allows it to broadcast its cube ID and connected face ID. The host PC updates the status of the connection of cubes from the received information; therefore, real-time recognition of connection/disconnection is possible.

Figure 1: An example of interaction with ActiveCube.

The 3D structure of the connected cubes can also be recognized. Through repeated analyses from the base cube to the leaves, the entire 3D structure of the connected cubes is recognized. When a cube is disconnected, its parent cube broadcasts the disconnection, and the host PC recognizes the change of object structure.

2.3. Input/Output devices of ActiveCube Because each cube is equipped with a Neuron Chip, it is possible to control the sensors to obtain the user's operational intention and other environmental information. The sensors can be ultrasonic, gyroscopic, tactile, infrared, luminous, and temperature sensors. It is also possible for a cube to be equipped with actuator/display systems controlled by a Neuron Chip to execute actions or show simulated results. Examples of such devices are lights, buzzers, motors and vibrators. The program to control the sensing device or actuators is installed on the non-volatile memory on each Neuron Chip.

3. How ActiveCube Works The programs for each cube consist of two parts: one for recognizing the connected face and the other for controlling the sensor or actuator. The software on the host PC determines the operation of the entire ActiveCube system. This program analyzes the 3D structure, shows the simulation results on the display, and transmits the operational commands to the cubes connected to the network. The causal relationships between the input devices and the output devices can also be determined by the program on the host PC. For example, brightness of the light varies with the distance measured by the ultrasonic sensor (see Figure 1), and the color of the light varies with the measured orientation by the gyroscopic sensor. The distance measured by the ultrasonic sensor can also change the rotational speed of the motor, frequency of the vibrator, and so on. Here, the combinations among multiple input and multiple output devices can be considered simultaneously.

Figure 2: An example of 3D structure with 39 cubes. Each cube has a unique function, and an object that has various purposes as a whole can be constructed by assembling them. The function of each cube can be flexibly changed with the connected positions/orientations or the assembled object shape. Figure 2 shows a snapshot of real-time recognition of a 3D structure with 39 cubes.

4. Retrieval of 3D Shape Models 4.1. Use of Physical Objects If a user could construct 3D shapes by simply combining physical blocks, the user interface that generates the query shape would become intuitive. This leads to other benefits: the user interface would not depend on the language, culture or age group of the users. In addition, users could operate and manipulate the 3D structure manually without encountering the difficulty of transforming it into 2D images. Following, the user does not have to be a computer graphics or Computer Aided Design (CAD) expert. Moreover, the physical objects used for a query shape could be recognized as the 3D model in the virtual world. Consequently, such an interface might become more usable across the boundary of real space and virtual space. To realize the system described above, we implemented a query interface for retrieval of 3D shape models with physical objects by using the ActiveCube system [3].

IEEE VR 2004 Workshop Beyond Wand and Glove Based Interaction, March 28, Chicago, IL USA

4.2. Implementation of Retrieval System When creating a query shape, the user is constructing a volume representation of the intended 3D model out of the ActiveCube system. Another possible technique can be based on recognition of the shape based on the skeleton of the 3D model. However, because each cube has a considerable 3D volume, with a 5-cm side, we assumed that the former method is more appropriate than the latter. To represent the 3D volume, we decided to use a voxel data representation for the 3D constructed shape of ActiveCube and for the 3D shape polyhedral models in the database. Our system recognizes the 3D shape structure constructed with ActiveCube and converts it into a voxel data representation in real time. Figure 4 shows this conversion from query shape to voxel data. In the ActiveCube system, there is a specific cube called the BaseBlock which is equipped with a connector to the Host PC. In the process of converting to voxels, the origin of three coordinate axes is based on the position of the BaseBlock. When a user constructs the query structure as shown in figure4 (a), the extracted voxel data is defined as shown in figure4 (b). The polyhedral models are also converted into voxel data representations in the 3D shape model database. Each polyhedral model is represented with n (n=1, 2,..., N) voxels in advance. To do this, first the system obtains all of the voxel data represented by n voxels. Then, the system compares a polyhedral model with all of the voxel data obtained above, calculates the level of similarity, and picks the highest level of voxel data as the voxel representation of the polyhedral model. The system repeats the above process for all polyhedral models. Figure 5 shows an example of this approach. In the process of calculating similarities, the system uses the 3D models’ voxel data represented by the same number of voxels as that of the constructed shape in its current form. These similarities (Voxel-Voxel Similarity) are calculated with Eq. (1) i (1) VV _ Sim. =   *100 [%], n where i is the number of voxels in the constructed structure that corresponds to voxels of the 3D virtual model, and n is the number of all voxels in the constructed shape. Here, VV_Sim. is maximized over all possible intersections Vi produced by rotating or translating voxels extracted from the constructed shape. We also observed a child using our system. The child successfully constructed query shapes by connecting and disconnecting cubes and retrieved 3D shape models in a play-like activity. So we also assume that novice users,

Figure 3: Flow of retrieval system.

(a) Constructed shape (b) Voxel data

Figure 4: Voxel data representation of constructed shape. including children and the older users, will be able to retrieve 3D shape models using our proposed system.

5. Other 3D Applications 5.1. Story-telling System We have developed an interactive story-telling system for stimulating children’s creativity and imagination intuitively [3]. First, by using ActiveCube, users construct a shape with which they want to play in virtual space. The computer automatically recognizes the constructed structure in real time, and then retrieves some candidate 3D shape models closely matching the constructed structure by using the method described above. After that, users select one of the candidates, and the computer starts to play the virtual model. Users can play in virtual space while holding the constructed object in their hands. This interaction is supported by input and output devices fitted to each block, and by ActiveCube structure being selfaware of its geometry. To enable users to play and interact with interest and excitement, we emphasize the gaming elements of this scenario. After selecting the virtual object, users have to get through some events within a certain time indicated by the timer. As shown in figure 6, even a very young child could construct a shape easily by combining blocks at their pleasure. We still didn’t evaluate that this system

IEEE VR 2004 Workshop Beyond Wand and Glove Based Interaction, March 28, Chicago, IL USA

Figure 6: Shape selection from several candidates. Figure 5: Examples of voxel representations for a 3D shape model.

data

stimulates users’ creativity and imagination considerably. However, we do believe our proposed system can stimulate children’s creativity and imagination because of its scalability and usability. This system provides a unique and innovative “edutainment” (educationalentertainment) experience for users, especially children.

5.2. Cognitive Assessment Cognitive Cubes is an automated tool for examination of 3D spatial cognitive ability [4], by making use of ActiveCubes. With ActiveCubes, users attempt to construct a target 3D shape, while each change of shape they make is automatically recorded and scored for assessment. Cognitive Cubes is the first computerized tool for 3D spatial constructional assessment, combining the increased sensitivity of 3D constructional tasks with the efficiency, consistency, flexibility and detailed data collection of automation. Preliminary results indicate that Cognitive Cubes is sensitive to mild Alzheimer’s disease (AD). Though small numbers of mild AD participants are examined, outcomes show strong differences between unaffected elderly and AD participants. Figure 7 graphs similarity vs. time for all participants in one Cognitive Cubes task. As experimental results, Cognitive Cubes offers improved sensitivity and reliability in assessment of cognitive ability and ultimately, reduced cost. Experimental evaluation with 43 participants confirms the sensitivity and reliability of the system.

Figure 7: Similarity vs. time graphs for all participants in one task. Black shows young participants, gray elderly, and the thin line is one AD participant. As application examples, we described a retrieval system of 3D shape models, a story-telling system and a system for cognitive assessment using ActiveCube. Acknowledgement This research was supported in part by “The 21st Century Center of Excellence Program” of the Ministry of Education, Culture, Sports, Science and Technology, Japan and by Exploratory Software Project grant of Information-technology Promotion Agency, Japan.

References [1]

[2]

[3]

5. Summary We proposed the ActiveCube system to allow users to construct and interact with a 3D environment by using cubes with a bi-directional user interface. A computer recognizes the constructed 3D structure in real time, so consistency is always maintained between the real object and its corresponding representation in the computer in terms of object shape and functionality.

[4]

Y. Kitamura, Y. Itoh and F. Kishino, “Real-time 3D interaction with ActiveCube,” CHI 2001 Extended Abstracts, pp.355–356, 2001. H. Ichida, Y. Itoh, Y. Kitamura and F. Kishino. “Interactive retrieval of 3D virtual shapes using physical objects,” in Proc. of IEEE Virtual Reality, 2004. Y. Itoh, S. Akinobu, H. Ichida, R. Watanabe, Y. Kitamura and F. Kishino. “Stimulating children's creativity and imagination with interactive blocks,” in Proc. of The Second International Conference on Creating, Connecting and Collaborating through Computing (C5), pp. 60-67, 2004. E. Sharlin, Y. Itoh, B. A. Watson, Y. Kitamura, S. Sutphen and L. Liu. “Cognitive Cubes: a tangible user interface for cognitive assessment,” in Proc. of Conference on Human Factors in Computing Systems (CHI 2002), pp. 347-354, 2002.