Intuitive 3D Computer-Aided Design (CAD) System With Multimodal ...

Proceedings of the ASME 2013 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference IDETC/CIE 2013 August 4-7, 2013, Portland, OR, USA

DETC2013-12365 (DRAFT)

INTUITIVE 3D COMPUTER-AIDED DESIGN (CAD) SYSTEM WITH MULTIMODAL INTERFACES

Vishwas G. Nanjundaswamy, Amit Kulkarni, Zhuo Chen, Prakhar Jaiswal, Sree Shankar S, Anoop Verma, Rahul Rai∗ DART Laboratory Department of Mechanical and Aerospace Engineering University at Buffalo Buffalo, NewYork, 14260 Email: [email protected]

ABSTRACT The existing interfaces for 3D CAD modeling softwares use 2D subspace inputs such as x and y axes of mouse to create 3D models. These existing interfaces are inherently modal because one needs to switch between subspaces, and disconnects the input space from modeling space. This makes existing interfaces tedious, complex, non-intuitive and difficult to learn. In this paper, a multi-sensory, interactive, and intuitive 3D CAD modeling interface is presented to address these shortcomings. Three different modalities (gestures, brain-computer interface, and speech) have been used for creating interactive and intuitive 3D CAD modeling interface. DepthSense® camera from SoftKinetic is used to recognize gestures, EEG Neuro-headset from Emotiv® is used for acquiring, and processing neuro-signals and CMU Sphinx is used for recognizing and processing speech. Multiple CAD models created by several users using the proposed multimodal interface are presented. In conclusion, the proposed system is easier to learn and use as compared to the already existing systems.

and growing area in software markets with market cap of $7 billion in 2011 [1]. CAD is used for representation, knowledge management, communication, and visualization of design information at various stages of engineering design process. The present work is focused on 3D modeling paradigm of CAD and its usage in the conceptual phase of design. Increase in computational power, as governed by Moore’s law, has led to the situation in which computers have become cheaper, powerful, and accessible. However, the modern day 3D CAD softwares involve ‘very steep learning curve’ to comprehend the complex task of using a dense set of toolbars and menus. In addition, the user interaction in 3D CAD softwares is non-intuitive as the 3D modeling subspace has to be mapped to the x and y axes of input devices such as mouse. This metaphor is inherently and cognitively complex because one needs to switch between 3D to 2D subspaces, and disconnects the input space from the modeling space. This increases novice users’ learning time. In addition to increase in computational power, the technology push driving the integration of natural and intuitive sensors into everyday consumer devices such as Kinect® in our homes and Brain Computer Interfaces (BCI) in commercial applications is setting the scene for the deployment of intuitive, natural and people-centric applications over the next two decades. Such people-centric devices combined with the ability of computational pipelines that can process information emanating from these devices to provide a natural interface portends a revolution

1. MOTIVATION AND OBJECTIVES Computer Aided Design (CAD) is omnipresent in multiple industrial domains such as automative and aerospace. In terms of market size, the CAD market is one of the established, largest,

∗ Address

all correspondence to this author.

1

Copyright © 2013 by ASME

for next generation 3D CAD. Natural and intuitive interfaces based on hand gesture, speech recognition, and BCI can replace WIMP (windows, icons, menus, pointer) based user interfaces. Multi-Modal natural interfaces can lower application learning time and can be especially useful for conceptual 3D CAD modeling. During the conceptual design phase the user’s focus is on overall shape and visual appearance of the 3D model and not so much on exact dimensions and tolerances. Most commercially available 3D modeling software are based on a structured procedure that requires an user to define the exact shape parameters early on. What is still needed is 3D conceptual CAD modeling tools that are more natural and intuitive to its users. Conventional 3D modeling systems and interfaces have yet to attain this. The intent of the present work is to lay the foundation for next generation of multimodal interfaces that can enable 3D model creation, especially in the conceptual phase of design. The importance is on creating modeling tools that not only offers 3D modeling functionality but also offers creative freedom to the user. To the best of our knowledge, this paper presents a developed system that for the first time combines the usage of BCI, Gesture, and Speech (BCIGS) for creating and interacting with CAD models. Additionally, several aspects learnt from preliminary human factors study related to the developed multimodal CAD system is also outlined. The paper is organized as follows. At first, the related work is reviewed. Secondly, the overall computational methodology and technical aspects of the developed multimodal interface and CAD system is described. The utility of developed system is then demonstrated through creation of several 3D models. A preliminary human factor study conducted on the use of the developed system is outlined next. Finally, the conclusions and discussions are presented.

CAD package.

2.2. Gesture Based CAD The importance of natural gesticulation in description of spatial objects has been emphasized in recent work of Holz and Wilson [4, 5]. Horvath has investigated the natural use of hand gestures for shape conceptualization [6, 7]. In order to exploit the use of gesture in CAD modeling, it is necessary to develop methods to help CAD applications recognize and interpret them efficiently. Various methods have been proposed to recognize gestures such as using gloves or markers [8, 9, 10], orientation histograms [11], disparity in stereo pair of images [12] etc. Researchers have also used depth data for recognizing gestures [13, 14, 15, 16, 17, 18]. Identifying gesture through depth data provides fast and efficient recognition and thus allows for natural and intuitive interaction [16, 5]. Recently, Vinayak et al [19] has used a Kinect and gesture based system for specific application of pottery design. While depth cameras are not conceptually new, Kinect has made natural and affordable sensor modalities accessible to all.

2.3. BCI in CAD Modeling

2. RELATED WORK In the present section, four key areas related to presented research namely (1) speech based interaction in CAD, (2) gesture based CAD, (3) BCI in CAD modeling, and (4) multimodal CAD interfaces and other relevant modalities are briefly reviewed.

Use of BCI in CAD modeling is relatively new. Brain activity associated with any thinking process has to be quantified and converted to tangible intentions and commands. The very first advancements in this respect were made in virtual reality domain by Pfurtscheller et al. [20] who for the first time demonstrated that it is possible to move through a virtual street without muscular activity when the participant only imagines feet movements. Leeb et al. [21] also carried out similar studies where the user was able to navigate in a virtual environment by using his/her EEG brain signals. Fabiani et al. [22] and Trejo et al. [23] took this one step further with their work on cursor movement using BCI. These studies show that the BCI can be used in many applications that involved the human thought process [24].

2.1. Speech Based Interaction in CAD As an interactional modality for CAD modeling, speech has been rarely used independently. Speech has been mostly used in conjunction with other interaction modalities. The pioneering work done by Bolt in creating ‘Put that There’ system used speech as one its input modalities. Boeing’s ‘Talk and Draw’ [2] was one of the first multimodal drawing applications, which allowed users to draw with a mouse and use speech input to change UI modes. Weimer and Ganapathy [3] used speech and glove based gesture inputs which were integrated with a stand-alone

In CAD systems, BCI offers a more intuitive and natural form of interaction between the user and a CAD application. An important aspect for BCI based 3D CAD modeling is the user’s comprehension of geometrical features that he/she intends to create [25,26,27]. The user must also be able to distinguish between different geometrical features [28, 29]. In this regard, Esfahani and Sundarajan [30] carried out experiments to explore the potential of BCI in distinguishing primitive shapes. In another recent work by Esfahani and Sundarajan [31], the authors report the use of BCI on the selection of geometrical surfaces in a 3D CAD model. 2

Copyright © 2013 by ASME

2.4. Multimodal CAD Interfaces And Other Relevant Modalities In terms of usage of multimodal interfaces in graphical application, the pioneering work done by Bolt in creating ‘Put that There’ system showed the value of combining different modalities such as speech and gesture [32]. This was followed by work of Ganapaty [3] in which speech and glove based gesture input were integrated with a stand-alone CAD package. These initial work led to development of many systems that typically used mouse, pen, speech or glove based input devices for creating 2D drawings or simple 3D shapes such as cubes [2, 33, 34]. Arangarasan and Gadh [35] used Head-Mounted Displays (HMDs) for multimodal input and output to work with 3D models. Stylus input was used in ILoveSketch [36] to allow users to create accurate 3D models primarily by sketching using an electronic stylus. Schmidt et al [37] developed a system to be used with a stylus to draw new objects or parts of an object. SenStylus [38]demonstrates a stylus based 6DOF input system for 3D modeling. Kara and co authors have created multiple computational techniques that converts 2d sketches on tablets to editable 3D CAD format [39, 40, 41]. While pen and stylus based sketching interfaces are good for creating 2D artifacts like sketches and drawings, the creation of 3D shapes using these interfaces limits the capability of designers to experiment at conceptual and artistic levels [42]. Scali et al [43] experimented with a 6 degrees of freedom haptic feedback device for 3D modeling. Creation of curved, free-form and organic shapes had also been shown in literature using glove-based [44] and augmented reality interfaces [45]. These interfaces provide 3D interaction capabilities for creating a wide variety of 3D shapes. Recently, Sharma et al [46] created a speech and touch based system for creating 3D models. The 3D modeling systems have been growing in functionality and effectiveness through-out the years. However, novice user generally need more time to learn these tools due to their advanced UI elements. In contrast to these efforts, our emphasis is on supporting 3D modeling by novice users using a minimalist and natural multimodal interface. The concept of natural multimodal interface refers to a user interface, that is imperceptible and natural to use [47, 48]. In regular UIs, devices need to be used to interact with a system and a learning period is usually needed. By contrast, in natural user interfaces, the device that separates the user and the system should be as unobtrusive as possible or hidden so that the user does not notice it. Natural user interfaces such as gesture, brain computer interface, speech, touch and haptics are promising candidates to replace current generation UI enabled by keyboard and mouse. Our emphasis is on combining the usage of BCI, gesture and speech for interacting with CAD models. In this paper, we introduce a new paradigm for reforming the early-stage design process by developing natural user interfaces (NUIs) based on combined BCI, gesture, and speech modalities

which facilitate cognitively simple interactions towards creative and exploratory design and 3D without the need for extensive training. 3. MULTI-MODAL INTERFACE BASED CAD SYSTEM COMPONENTS In this section, detailed description of the underlying multimodal interface and related hardware/software components are discussed. Figure 1 depicts the main components of the developed system. Gestures, speech, and BCI are the input modalities through which a user can interact with the developed system. DepthSense camera from SoftKinetic ® is used for gesture recognition, EEG Neuro-headset from Emotiv ® is used for acquiring and processing Neuro-signals, and CMU Sphinx is used for speech recognition and processing. Google Sketchup is used as the CAD modeling platform. Google Sketchup was chosen due to its ease of use [46]1 . Speech and BCI are used for creating 2D and 3D models in CAD, whereas, gesture is primarily used for manipulation operations like rotate and zoom. Specific details of three modalities and CAD GUI is briefly discussed next.

3.1. Speech Recognition In the present work, the speech recognition algorithm is based on a CMU Sphinx speech recognition system developed at Carnegie Mellon University (CMU). Sphinx use discrete hidden marker methods (HMM) with linear predictive coding (LPC)derived parameters to recognize speech command and offer numerous advantages in terms of: (1) speaker independency, (2) large vocabulary, and (3) isolated words. The speech recognition process starts with automatic segmentation, classification, and clustering. It then uses three recognition passes consisting of a Viterbi decoding using beam search and a best path search of the Viterbi word lattice, N-best list generation and rescoring [49]. Acoustic adaptation using a transformation of the mean vectors based on linear regression (MLLR) is performed between each recognition pass. Using CMU Sphinx, CAD specific functions such as creation of 2D geometry (circle, rectangle, arc), extrusion of 2D geometry (extrude), transformational (orbit) etc. are replaced with speech by recognizing speech commands such as circle, extrude, rotate, zoom etc. 3.2. Recognizing Brain Activity Emotiv® EEG is used to record and recognize brain signals (electroencephalography (EEG)) and facial muscles movement (electromyogram (EMG)). It is a low cost brain-computer interface that comprise of (1) neuroheadset hardware device, and (2)

3

1 Ease of use of Google Sketchup have recently gain credibility among online users and somewhat established by survey conducted by NUI group and SktechUp forums.

Copyright © 2013 by ASME

FIGURE 1.

Components of the multi-modal interface based CAD system

software development kit (SDK). The neuroheadset records the brain activity and facial muscle movements of the users, which is interpreted by its Software Development Kit (SDK). The neuroheadset acquires brain signals using 14 sensors (and 2 additional gyroscopes) placed on the user scalp which communicate wirelessly with the computer device via USB dongle. The Emotiv EEG can capture and process brainwaves in the Delta (0.54 Hz), Theta (48 Hz), Alpha (814 Hz) and Beta (1426 Hz) bands [50]. The Emotiv toolkit includes the Emotiv API, a C++ API, which acts as an interface between a CAD system and user. It provides (1) communication with the Emotiv headset, (2) reception of preprocessed EEG/EMG and gyroscope data, and (3) transformation of user-specific action commands into an easy-to-use structure (also known as EmoState).

a gesture recognition software development kit (SDK) and middleware known as iisu™, which is used to analyze and process raw data from DS311. iisu™ can locate people or hands in the scene and removes all the other objects and then calculates information about each person/hand and provides relevant data to the end application. iisu™ has a Close Interaction (CI) layer to detect and track hand gestures performed near the camera inside the distance bounds ranging from 0.15 to 1.0 meters from the camera. It can detect one or both hands in the camera’s field of view (FoV). C++ Application Programming Interface (API) of iisu ™ is used in the developed multimodal CAD system to recognize gestures and extract relevant information about user’s hand pose. The recognition of gesture is followed by assignment of CAD tool functionalities to the recognized gestures. CAD handling and manipulation functions such as select, deselect, and curser movements are assigned to the recognized gestures. Figure 2 displays a recognized gesture and associated functionality to be used in the developed 3D CAD modeling system.

3.3. Recognizing Gestures In present work, gestures are recognized using depth sensing camera from SoftKinetic, namely DepthSense® 311 (DS311). It is a time-of-flight (ToF) 3D camera that includes a RGB sensor, a depth sensor, and 2 microphones. The DS311 captures 3D real-time distance data of a scene for up to 60 frames per second (fps). The distance information is calculated based on the time required by infrared light (emitted by camera) to reach the scene and come back. The sensor transforms the ToF positional data into real-time 3D RGB image, which includes confidence and depth maps. The images and process maps are processed and analyzed to recognize gestures. SoftKinetic also provides

3.4. Combined Modalities The list of functionalities handled by all the three modalities is tabulated in Table 1. Usage of all three modalities is required to create/modify 3D part. There is overlap in set of commands that can be implemented through different modalities. For example, commands such as draw line, and extrude can be executed using more than one modality. In such cases, it is up to 4

Copyright © 2013 by ASME

FIGURE 3.

FIGURE 2.

Comparison of developed Multi-modal CAD interface with the existing CAD interface

3.5. Google SketchUp CAD GUI The CAD platform of the developed system is based on Google SketchUp software [51]. Google SketchUp provides an easy to use 3D modeling environment and also has a large online repository of model assemblies for efficient analysis. Google SketchUp is used in the developed system as a 3D modeling environment for creating and visualizing 3D models. The graphical user interface (GUI) of Google SketchUp is used as platform to display the current state of 3D models. Based on visual feedback from the current state of 3D models the user generates a new set of commands using the multimodal interface. The new commands are sent to Google SketchUp GUI for altering the state of the 3D models. The Google SketchUP GUI’s WIMP toolbars and menu were disabled. The modeling, visualization, and transformation functions of Google SketchUP was integrated with multimodal interface. The user interacted with overall system using speech, gesture, and thought command to create, modify, and manipulate 3D models (see Fig. 3). In next section, the overall experimental set up is discussed. A preliminary human factor study conducted on the use of the developed system is also outlined.

Gestures pose replicating mouse functions

the user to select the modality they prefer. The two-handed gestures replicates mouse functions such as left click, right click, mouse movement to manipulate the 3D CAD model. The recognized speech is used to replicate functionalities such as selection of circle, rectangle, pan, and zoom. Both the electroencephalography (EEG) and electromyography (EMG) functionalities of Emotiv headset based BCI are utilized. BCI functionalities includes, sketching a circle, undo, line etc. Certain BCI enabled functions overlap with gestures enabled functions. This kind of overlap provides added flexibility and improves the robustness of the interface. For example, extrusion of 2D drawings can be done by both BCI enabled command (using pull thought) and gesture enabled command (using right hand movements). However, the efforts needed to utilize these modalities may vary from one user to another user. The selection of best modality for any given function is subject of an extensive human factors study (a preliminary study is discussed in section 4).

3.6. Generated CAD Models Using Developed System The major components of developed system include: Emotiv headset, DepthSense camera, microphone, and Google SketchUp based 3D CAD environment. The process of creating 3D CAD models starts with BCI or speech modality where certain 2D CAD commands are invoked by the users. Users’ then utilize gestures or BCI thought process to create 3D models or manipulate the existing 3D models. Figure 4 depicts a snapshot 5

Copyright © 2013 by ASME

TABLE 1.

CAD functionalities associated with three modalities

Modality

Action/command

CAD function

CAD Objective

BCI (EEG)

think pull

Activate extrude

Create/Modify object

BCI (EMG)

line

Activate line

Create object

BCI (EMG)

zoom

Activate zoom

Move object

BCI (EMG)

undo

Activate undo

Object disappear

BCI (EMG)

extrude

Activate extrude

Create/Modify object

BCI (EMG)

move

Activate move

Move object

BCI (EMG)

Clinch

Activate undo

Object disappear

BCI (EMG)

Look left

Activate circle

Create object

BCI (EMG)

Look right

Activate line

Create object

BCI (EMG)

Blink

Mouse left click

Select object

Gesture

Left palm close

Mouse left click

Select object

Gesture

Left palm open

Mouse left button release

Deselect object

Gesture

Right palm movement

Curser movement

Modify object

Speech

circle

Activate circle

Create object

Speech

rectangle

Activate rectangle

Create object

Speech

circle

Activate circle

Create object

Speech

orbit

Activate orbit

Change object view

Speech

arc

Activate arc

Create object

of the developed system usage. In the Fig. 4, user is using left hand gesture as left mouse click (to hold the extrusion tool), and extruding the circle with right hand gesture. User utilized BCI to invoke the extrusion tool and later used gesture to modify the 3D model. Figure 5 depicts the step-wise process of creating a sample 3D part and associated modalities used. In Fig. 5 all modalities are equallty utilized by the users while creating the the given 3D model. A demo video of the developed system can be found here Multi-modal CAD system.

study. All the participants had correct vision, hearing, and no known medical issues. The study started with the training session where they were trained to operate the BCI, gesture, and speech modalities. Participants were initially trained to perform a set of commands such as draw a line, draw a circle, extrude the circle, and orbit the model. The training process lasted for about 35-40 minutes. The participants were then asked to perform four CAD modeling tasks with increasing difficulty as shown in Fig. 6. Following information about the tasks was also provided to the users: Task1: Create a rectangle (using speech or BCI). Task 2: Crate a triangle and extrude it using BCI or gestures. Create a circle on visible face using BCI or speech and extrude it. Task 3: Create a rectangle and extrude it using BCI/speech and gesture. Use BCI to create a circle and extrude it. Similarly, create 2 joined arcs, rectangles and extrude it. Use orbit command through speech modality to select the appropriate face. Task 4: Recreate the given 3D model. Figures 7- 10 display the models created by the participants

4. HUMAN FACTOR STUDY In order to analyze the usability of the developed multimodal system, a small scale human factor study was carried out. The aim of the study was to: (1) assess the effort required by users’ in creating the 3D models, (2) analyze users’ preferred modalities from available options to perform an action, and (3) identify limitations of the present system. Five graduate students (gender: male, age: 24-30 years) with no prior information about the interface were selected for the 6

Copyright © 2013 by ASME

FIGURE 4.

FIGURE 5.

The overall system setup

Step-wise process of creating a 3D part using developed system

during the human factors study. Using all three modalities, participants created both 2D and 3D models. As no feedback was given in Task 4, the models created by participants in Task 4 were found least similar to the target model (Fig. 10). The results from the study indicate that participants spent relatively less time fin-

ishing task 4 when compared with task 3. This indicates a notion of learning which could be continuously improved (Fig. 11). Figure 12 compares the performance of individual participants for all four tasks. Figure 12 indicate almost similar performance by all the users in the study. The average completion time of the 7

Copyright © 2013 by ASME

FIGURE 6.

FIGURE 8.

Taks 2: comparison of objects created by participants

FIGURE 9.

Taks 3: comparison of objects created by participants

FIGURE 10.

Taks 4: comparison of objects created by participants

Set of tasks given to the participants

tasks for different users were also found to be similar.

FIGURE 7.

Taks 1: comparison of objects created by participants

5. CONCLUSIONS We developed a multi-modal interface, utilizing speech, gestures, and BCI based modalities for intuitive 3D CAD modeling and exploration. Present work contributes another paradigm of natural and interactive CAD modeling through action, speech, and imagination for creating and manipulating CAD objects. The components required for the developed system is low cost and easy to setup. The strength of developed system is showcased with sample CAD parts created by the users. A preliminary human factor study was also performed to analyze the usability and robustness of the developed system. Participants were

FIGURE 11.

8

Performance of the participants across set of tasks

Copyright © 2013 by ASME

[9]

[10]

[11] FIGURE 12.

Comparison of individual participants’ performance

[12] able to replicate the CAD objects given in the task with mid-high precision within reasonable time. As anticipated, participants’ preferred speech over BCI to invoke optional CAD functionality. Study results indicate the potential of multi-modal system in terms of adaptability, and continous improvement. In future work, certain aspects of developed interface will be improved. These improvements include: (1) developing a better and robust speech recognition system, (2) improving the precision of gestures recognized by depth sense camera to facilate stable cursor movement. Another research direction includes a lagre scale user study to better unserstand the usability aspects of the developed system.

[13]

[14]

[15]

REFERENCES [1] , 2012. http://jonpeddie.com/pressreleases/details/worldwide-cad-market-report-2012/, October. [2] Salisbury, M., Hendrickson, J., Lammers, T., Fu, C., and Moody, S., 1990. “Talk and draw: Bundling speech and graphics”. Computer, 23(8), pp. 59–65. [3] Weimer, D., and Ganapathy, S., 1989. “A synthetic visual environment with hand gesturing and voice input”. In ACM SIGCHI Bulletin, Vol. 20, ACM, pp. 235–240. [4] Holz, C., and Wilson, A., 2011. “Data miming: inferring spatial object descriptions from human gesture”. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’11, ACM, pp. 811–820. [5] Murugappan, S., Piya, C., and Ramani, K. “Handy-potter: Rapid 3d shape exploration through natural hand motions”. [6] HorvÃath, ˛ I., and RusÃak, ˛ Z., 2001. “Collaborative shape conceptualization in virtual design environments”. Communications of ACM, pp. 59–63. [7] Horvath, I., Vergeest, J. S. M., and Rusak, Z., 2002. “Six ingredients of collaborative virtual design environments”. In in Proc. Int. Design Conf, pp. 67–74. [8] Nishino, H., Utsumiya, K., and Korida, K., 1998. “3d ob-

[16]

[17]

[18]

[19]

[20]

[21]

9

ject modeling using spatial and pictographic gestures”. In Proceedings of the ACM symposium on Virtual reality software and technology, ACM, pp. 51–58. Sturman, D., and Zeltzer, D., 1994. “A survey of glovebased input”. Computer Graphics and Applications, IEEE, 14(1), pp. 30–39. Baudel, T., and Beaudouin-Lafon, M., 1993. “Charade: remote control of objects using free-hand gestures”. Communications of the ACM, 36(7), pp. 28–35. Freeman, W., and Roth, M., 1995. “Orientation histograms for hand gesture recognition”. In International Workshop on Automatic Face and Gesture Recognition, Vol. 12, pp. 296–301. Jojic, N., Brumitt, B., Meyers, B., Harris, S., and Huang, T., 2000. “Detection and estimation of pointing gestures in dense disparity maps”. In Automatic Face and Gesture Recognition, 2000. Proceedings. Fourth IEEE International Conference on, IEEE, pp. 468–475. Muñoz-Salinas, R., Medina-Carnicer, R., Madrid-Cuevas, F., and Carmona-Poyato, A., 2008. “Depth silhouettes for gesture recognition”. Pattern Recognition Letters, 29(3), pp. 319–329. Liu, X., and Fujimura, K., 2004. “Hand gesture recognition using depth data”. In Automatic Face and Gesture Recognition, 2004. Proceedings. Sixth IEEE International Conference on, IEEE, pp. 529–534. Doliotis, P., Athitsos, V., Kosmopoulos, D., and Perantonis, S., 2012. “Hand shape and 3d pose estimation using depth data from a single cluttered frame”. Advances in Visual Computing, pp. 148–158. Suryanarayan, P., Subramanian, A., and Mandalapu, D., 2010. “Dynamic hand pose recognition using depth data”. In Pattern Recognition (ICPR), 2010 20th International Conference on, IEEE, pp. 3105–3108. Keskin, C., Kiraç, F., Kara, Y., and Akarun, L., 2011. “Real time hand pose estimation using depth sensors”. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, IEEE, pp. 1228–1234. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A., 2011. “Real-time human pose recognition in parts from single depth images”. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, IEEE, pp. 1297– 1304. Vinayak, Sundar, M. C. P., and Karthik, R., 2012. “Handypotter: Rapid 3d shape exploration through natural hand motions”. ASME. Pfurtscheller, G., Leeb, R., Keinrath, C., Friedman, D., Neuper, C., Guger, C., and Slater, M., 2006. “Walking from thought”. Brain Research, 1071(1), pp. 145–152. Leeb, R., Scherer, R., Lee, F., Bischof, H., and Pfurtscheller, G., 2004. “Navigation in virtual environCopyright © 2013 by ASME

[22]

[23]

[24]

[25] [26]

[27]

[28] [29]

[30]

[31]

[32] [33]

[34]

[35]

ments through motor imagery”. In 9th Computer Vision Winter Workshop, CVWW, Vol. 4, pp. 99–108. Fabiani, G., McFarland, D., Wolpaw, J., and Pfurtscheller, G., 2004. “Conversion of eeg activity into cursor movement by a brain-computer interface (bci)”. Neural Systems and Rehabilitation Engineering, IEEE Transactions on, 12(3), pp. 331–338. Trejo, L., Rosipal, R., and Matthews, B., 2006. “Braincomputer interfaces for 1-d and 2-d cursor control: designs using volitional control of the eeg spectrum or steady-state visual evoked potentials”. Neural Systems and Rehabilitation Engineering, IEEE Transactions on, 14(2), pp. 225– 229. Kübler, A., Kotchoubey, B., Kaiser, J., Wolpaw, J., and Birbaumer, N., 2001. “Brain–computer communication: Unlocking the locked in.”. Psychological bulletin, 127(3), p. 358. Kosslyn, S., and Osherson, D., 1995. An Invitation to Cognitive Science, -Vol. 2: Visual Cognition, Vol. 2. Mit Press. Moulton, S., and Kosslyn, S., 2009. “Imagining predictions: mental imagery as mental emulation”. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1521), pp. 1273–1280. Farah, M., Hammond, K., Levine, D., and Calvanio, R., 1988. “Visual and spatial mental imagery: Dissociable systems of representation”. Cognitive psychology, 20(4), pp. 439–462. Kosslyn, S., 1996. Image and brain: The resolution of the imagery debate. MIT press. Alexiou, K., Zamenopoulos, T., Johnson, J., and Gilbert, S., 2009. “Exploring the neurological basis of design cognition using brain imaging: some preliminary results”. Design Studies, 30(6), pp. 623–647. Esfahani, E., and Sundararajan, V., 2011. “Classification of primitive shapes using brain–computer interfaces”. Computer-Aided Design. Esfahani, E., and Sundararajan, V., 2011. “Using brain computer interfaces for geometry selection in cad systems: P300 detection approach”. ASME. ˇ Voice and gesture at Bolt, R., 1980. ¸SPut-that-thereÂT: the graphics interface, Vol. 14. ACM. Billinghurst, M., 1998. “Put that where? voice and gesture at the graphics interface”. ACM SIGGRAPH Computer Graphics, 32(4), pp. 60–63. Hauptmann, A., 1989. “Speech and gestures for graphic image manipulation”. In ACM SIGCHI Bulletin, Vol. 20, ACM, pp. 241–245. Arangarasan, R., and Gadh, R., 2000. “Geometric modeling and collaborative design in a multi-modal multi-sensory virtual environment”. In Proceeding of the ASME 2000 Design Engineering Technical Conferences and Computers and Information in Engineering Conference, pp. 10–13.

[36] Bae, S., Balakrishnan, R., and Singh, K., 2008. “Ilovesketch: as-natural-as-possible sketching system for creating 3d curve models”. In Proceedings of the 21st annual ACM symposium on User interface software and technology, ACM, pp. 151–160. [37] Schmidt, R., Wyvill, B., Sousa, M., and Jorge, J., 2006. “Shapeshop: Sketch-based solid modeling with blobtrees”. In ACM SIGGRAPH 2006 Courses, ACM, p. 14. [38] Fiorentino, M., Monno, G., and Uva, A., 2005. “The senstylus: a novel rumble-feedback pen device for cad application in virtual reality”. In Proceedings of the 13th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision’2005 (WSCG 2005), Citeseer. [39] Kara, L., D’Eramo, C., and Shimada, K., 2006. “Pen-based styling design of 3 d geometry using concept sketches and template models”. In ACM Symposium on Solid and Physical Modeling: Proceedings of the 2006 ACM symposium on Solid and physical modeling, Vol. 6, Citeseer, pp. 149– 160. [40] Kara, L., and Stahovich, T., 2005. “An image-based, trainable symbol recognizer for hand-drawn sketches”. Computers & Graphics, 29(4), pp. 501–517. [41] Kara, L., and Shimada, K., 2007. “Sketch-based 3d-shape creation for industrial styling design”. Computer Graphics and Applications, IEEE, 27(1), pp. 60–71. [42] Varga, E., 2008. “Using hand motions in conceptual shape design: theories, methods and tools”. Product Engineering, pp. 367–382. [43] Scali, S., Wright, M., and Shillito, A., 2003. “3d modelling is not for wimps”. In Proceedings of HCI International. [44] Schkolne, S., Pruett, M., and Schröder, P., 2001. “Surface drawing: creating organic 3d shapes with the hand and tangible tools”. In Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, pp. 261–268. [45] Fuge, M., Yumer, M., Orbay, G., and Kara, L., 2011. “Conceptual design and modification of freeform surfaces using dual shape representations in augmented reality environments”. Computer-Aided Design. [46] Sharma, A., Madhvanath, S., Shekhawat, A., and Billinghurst, M., 2011. “Mozart: a multimodal interface for conceptual 3d modeling”. In Proceedings of the 13th international conference on multimodal interfaces, ACM, pp. 307–310. [47] Lee, J., 2010. “In search of a natural gesture”. XRDS, 16(4), pp. 9–12. [48] Pentland, A., 2000. “Perceptual user interfaces: perceptual intelligence”. Communications of the ACM, 43(3), pp. 35– 44. [49] Chen, S., Chen, S., Doh, S., Eskenazi, M., Ea, E. G., Raj, B., Ravishankar, M., Rosenfeld, R., Siegler, M., Stern, R., and Thayer, E., 1998. “The 1997 cmu sphinx-3 english 10

Copyright © 2013 by ASME

broadcast news transcription system”. In In Proceedings of the 1998 DARPA Speech Recognition Workshop. DARPA, pp. 55–59. [50] Gomez-Gil, J., San-Jose-Gonzalez, I., Nicolas-Alonso, L. F., and Alonso-Garcia, S., 2011. “Steering a tractor by means of an emg-based human-machine interface”. Sensors, 11(7), pp. 7110–7126. [51] http://sitescontent.google.com/google-sketchup-foreducators/home. [52] http://www.youtube.com/watch?v=jp44vte5z4kfeature=youtu.be.

11

Copyright © 2013 by ASME