AaltoWindow: Open Multi-Touch Framework for Campus Learning Centers 1
Petri Vuorimaa and Jari Kleimola
2
1
Department of Media Technology, Aalto University, FI-00076 AALTO, Finland
[email protected] 2 Department of Signal Processing and Acoustics, Aalto University, FI-00076 AALTO, Finland
[email protected] Keywords Multi-touch, University Applications, Learning Centers.
ABSTRACT Learning centers are becoming the hubs of modern university campuses. They promote collaborative learning. The emphasis is on ICT supported interaction between the students, teachers, researchers, and visiting industry experts. New interaction technologies can enhance the collaboration. In this paper, we focus on multi-touch. We present an open-source multi-touch framework—called AaltoWindow—for university campus learning centers. Compared to other multi-touch frameworks the AaltoWindow supports multiple programming paradigms and languages. In addition, AaltoWindow supports networked multi-touch stations, and thus improves collaboration between different university buildings and multiple campuses.
1.
INTRODUCTION
Aalto University was established in 2010 as a combination of three existing universities: Helsinki University of Economics, Helsinki University of Technology, and University of Art and Design. The objective was to create a new kind of interdisciplinary university combining economics, technology, and design. Each field brings its own expertise and traditions. Aalto University has currently three separate campus areas. To break the barriers between different organizations, Aalto University has reorganized as six schools and 38 departments. So-called factories combine activities on three joint areas: design, media, and services. The factories bring together the students and faculty working in different units. The biggest obstacle for collaboration, however, has stayed the same—three different campuses and many buildings limit the cooperation. In spring 2011, Aalto University created a new campus vision, which aims to enhance collaboration between different campuses and units. The idea is to build the new university as a network of learning centers. Each node contains a learning center as a hub for collaboration between students, faculty, international visitors, and industrial experts. The learning centers offer different kinds of services, such as library, cafeteria, meeting rooms, seminar halls, Internet access, student projects, galleries, etc. Each learning center shares the common Aalto brand, but provides its unique selection of services. The learning centers promote openness so that all internal and external members of Aalto University can access the campus services. In addition to the fixed locations, pop-up learning centers can be created at student fairs, scientific conferences, international partner universities, etc. Effective usage of the learning centers requires that the different services are easily accessible. The learning centers are physical spaces, but often the services are at least to some extent digital. Thus, the learning centers are so called smart spaces—each learning space is a window, which allows easy access to Aalto University campus.
Figure 1. AaltoWindow is a networked digital front desk of Aalto learning centers. The aims of the AaltoWindow project are summarized in the Figure. 1. The university is composed of one main campus and two additional campuses. Each campus has several nodes, which provide learning center services. AaltoWindow is the digital front desk of the learning centers. Since the learning centers strive to network and open the university, the different AaltoWindow stations are connected to each other with ultra fast network. AaltoWindow stations can also be located in public, satellite, and pop-up locations. In this paper, we present the first results of the AaltoWindow project. Based on the Aalto campus strategy, Section 2 presents the AaltoWindow requirements. After that, we define the AaltoWindow hardware and software architecture in Sections 3 and 4, respectively. In Section 5, we present the first services implemented on top of the AaltoWindow platform. In fall 2011, 21 students implemented altogether nine AaltoWindow applications on Multimedia Programming course supervised by the authors. Section 6 discusses the results of the student feedback. After that, we discuss related work in Section 7. Finally, Section 8 gives conclusions.
2.
REQUIREMENTS
The AaltoWindow platform has three main requirements: 1. Visibility: a virtual window into our university 2. Interaction: dynamic virtual campus structure 3. Cooperation: support for interdisciplinary collaboration As discussed in the Introduction, AaltoWindow is a virtual window into our university. It opens the different premises and visualizes the activities happening in each learning center. Thus, the system enables access to the existing Aalto services and content servers, and presents them in a consistent manner. The content delivered by the AaltoWindow platform needs to be dynamic, real-time, and interactive.
Interaction is a key issue. The User Interface (UI) has to be logical and immediately usable, but at the same time enjoyable and engaging to invite prolonged use of the system. The UI needs to support single-user kiosk-style applications, as well as scenarios where several users are interacting together within a single learning center. To increase the feeling of unity, selected applications need to be able to share their UIs in real-time with other AaltoWindow stations. Thus, AaltoWindow acts as a virtual extension of the physical campus. The most important requirement is interdisciplinary cooperation. The AaltoWindow platform must support co-creation and discovery of knowledge. It has to allow easy accessing and sharing of information.
3.
HARDWARE ARCHITECTURE
The AaltoWindow architecture is based on the requirements defined in the previous section. First, we included IP connectivity to the Aalto intranet and outside Internet resources (Visibility). A full HD resolution display and stereophonic speakers provide the output. Second, we used natural user Interaction techniques and multimodal input (see Figure 2). The input modalities are captured using 1. large 46-inch tabletop multi-touch surface with unlimited number of 2D touch points 2. infrared camera tracking touchless bodily gestures in 3D from up to two users 3. mobile sensors from handheld devices 4. microphone array for acoustic source localization and aural input These input devices were connected locally into a powerful but still off-the-shelf PC using USB or WiFi.
Figure 2. Each AaltoWindow station can be used locally, but the support also collaboration between geographically separated users. Third, the local stations were connected together using 10 Gbps, low latency (4 ms) network (Cooperation). Although the network is certainly capable of transmitting high-resolution audiovisual media, we are primarily interested in streaming interaction data between the stations. The streams flow either in a peer-to-peer fashion—from station to station—or via a centralized AaltoWindow
server. The server doubles also as a cloud-based shared data model for traditional client-server applications. Collaborative interaction happens thus locally around a single table, or between several geographically separated stations.
4.
SOFTWARE IMPLEMENTATION
Fast network provides connectivity, while input and output devices support cooperation. However, this requires also software support. The AaltoWindow software framework consists of the AaltoWindow shell and the platform services, running in the local Windows 7 PC of each station. The platform services form a thin API that provides access to multimedia content, multimodal input devices, and interconnected stations. This service layer is callable from any of the 14 supported software frameworks (browsers, Surface, WPF/WinForms, Silverlight, Flash, PyMT/kivy, Processing, MT4J, Unity, vvvv, SDL/WGL/Win32, Cinder, and Cornerstone), allowing development using programming languages such as C/C++, C#, Java, Python, and HTML5/Javascript. The actual applications are implemented on top of the supported frameworks (see Figure 3). The applications describe their properties in XML-format, and register to the AaltoWindow shell, which launches and terminates them. During launching, the shell grabs the application content windows from their native framework environments, and hosts them in a consistent chromeless form inside the AaltoWindow UI. The shell contains also a Launchpad UI for browsing, selecting, launching, and terminating the hosted applications.
Figure 3. AaltoWindow software framework. The applications subscribe to the local services using RESTful URL-like topic strings (e.g., “/touch/gesture/swipe”). The remote services are accessible by prefixing the topic string with the address of a remote station (e.g., “/aaltowindow/2/touch/gesture/swipe”). The subscribed interaction streams are transported using XMPP1 or ZeroMQ2, with Open Sound Control (OSC)3 (Wright, 2005) format payload. The AaltoWindow framework decodes the streams, and invokes appropriate event handlers in the native software framework of the application. In addition to the described publish-subscribe mechanism, traditional client-server and push-pull patterns are also available, e.g., when accessing the centralized AaltoWindow server. 1
http://www.xmpp.org/
2
http://www.zeromq.org/
3
http://opensoundcontrol.org/
5.
CONTENT
The AaltoWindow content is produced by the Aalto community: 1) students, 2) researchers, and 3) Aalto services. Students can use AaltoWindow as an exercise platform, or as a platform for artistic installations. We are also planning design competitions for the students. The researchers can use AaltoWindow to explore novel research themes, in collaboration, for extended visibility, demos, and pitching for funding. Finally, Aalto service providers can use AaltoWindow as a platform for lecture streams, Aalto information mediation, library services, cloud mashups such as campus maps, and social media related applications. In fall 2011, we used the AaltoWindow as a development platform on T-111.5350 Multimedia Programming course4. The course consists of lectures on multimedia frameworks, student presentations, and a programming exercise. We instructed the students that they need to develop an AaltoWindow based multi-touch application. Furthermore, the application had to support the campus vision of Aalto University. Altogether 21 students participated in the course. They were divided into nine groups of two or three students. We gave some ideas for the applications, but the students were allowed to select the applications themselves. The selected applications are shown in Figure 4.
Figure 4. AaltoWindow applications created by the students of the T-111.5350 Multimedia Programming course. Three applications were map based: (2) Weather Map, (7) Campus Building Info, and (8) Journey Planner. Other three applications were document based: (3) Document Reader, (6) Media Scrapbook, and (9) Ideawall. Finally, three applications were miscellaneous: (1) RFID/2D tag Reader, (4) “Neppis” Car Game, and (5) Music Synthesizer. The development work was divided into five phases: Topic Selection, Workplan, Mid-Term Demo, Final Demo, and Final Submission. The whole course lasted about 2,5 months and each student was supposed to use about 50 hours for the programming exercise, which was half of the course credits (4 ECTS). All groups were able to finalize the applications. Only one group (1) had to modify their topic from tag reader to mobile public display communication. 4
https://noppa.aalto.fi/noppa/kurssi/t-111.5350/
6.
RESULTS
In the final workshop of the course, we collected feedback from the student groups using a two page questionnaire. The questions were divided into five groups: general questions, development, working style, content, and future expectations. Altogether, 18 students filled the questionnaire. In general questions, we asked, for example, about the programming experience of the students. On scale 1 (beginner) to 5 (expert), the students estimated themselves as advanced programmers (mean 3.6 and stdev 1.0). They had good knowledge on mouse based UIs (mean 3.3 and stdev 0.8) as well, but their multi-touch programming experience was at novice level (mean 1.7 and stdev 0.7). In development section, students told that support material for multi-touch development was quite difficult to find. Half of the students were satisfied with the support material provided at the course website, but the rest wanted more documentation, tutorials, and, especially, code examples. In working style section, students responded that they did most of the development using mouse simulator (mean 62% of time, stdev 35%); however, two students used primarily remote multi-touch source. Students used the multi-touch table for development on average 12% of the time (stdev 11 %). Thus, it seems that they did most of the development on their own computers and only tested the application on the multi-touch table. We also asked how the students would divide their time if they would start again. Surprisingly, the students told that they would use less simulator (mean 41% of time, stdev 33%) and more multitouch table (mean 30% of time, stdev 15%). One explanation is that—although the mouse simulator was convenient—the students realized that it is impractical. The physical dimensions of a 46-inch multi-touch table afford wider gestural space than mouse-based interaction. Therefore, the user experience is different. Also, the optical touch tracking mechanism contains more noise in comparison to the precise input stream acquired from the mouse device. In content section, we asked whether the students would use AaltoWindow once they are installed in public places around the campus. On scale 0 (no) to 5 (yes), the students told that they would use the table (mean 4.1, stdev 1.0). The preferred types of applications running in the platform comprised kiosk-style information applications (e.g., schedules, lunch lists, public transportation timetables, maps, journey planners, and weather information), collaborative applications, and infotainment (e.g., games, musical instruments, and demos), etc. Many students would also like to continue developing multi-touch-related applications (mean 3.2, stdev 1.4). In future expectations section, the students believed that multi-touch is going to become more popular. On scale 0 (less used) to 5 (more popular), the mean was 4.2 and stdev 0.9. Finally, the students thought that they had learned useful skills on the course (mean 3.4, stdev 1.0).
7.
RELATED WORK
In this section, we compare our results with related work on 1) multi-touch frameworks, 2) educational multi-touch applications, and 3) university applications.
7.1. Multi-touch frameworks Echtler and Klinker (Echtler, 2008) proposed layered software architecture for multi-touch applications. The architecture has four layers (from bottom to top): 1) Hardware Abstraction Layer, 2) Transformation Layer, 3) Interpretation Layer, and 4) Widget Layer. The lowest layer generates a stream of positions of fingers, hands, and/or objects from the raw input data. The next layer transforms the position data from device to screen coordinates. The interpretation layer translates the movements of hands and fingers into gestures. The widget layer generates the visible output for the user. Hansen et al. proposed a multi-touch framework—called Python Multi-Touch (PyMT)—based on Python programing language (Hansen 2009). The architecture has three components: 1) Input Providers, 2) Scene Graph, and 3) PyMT Main Loop. The architecture supports multiple input providers, for example, TUIO (Kaltenbrunner, 2005), mouse input, Microsoft Windows 7 touch screen driver, and Nintendo Wii Remote. The Scene Graph is defined using toolkit widgets that are rendered in OpenGL. The PyMT Main Loop is a modified version of the main event loop used in many graphical
applications. Instead of generating the event object for each event, they found useful to keep a persistent event object, whose data is modified just before it is dispatched. Laufs et al. list requirements and architectural design principles for a cross-platform multi-touch development framework called MT4j (Laufs, 2010). The architecture resembles LibTisch (Echtler, 2008), albeit the interpretation layer of MT4j is further decomposed into global and component level input processing layers. The global level dispatches unified touch points to the underlying scene graph components, which continue gestural processing in their own coordinate systems. Special attention is also given to extensibility and reusability at each layer. Recently, Kammer et al. (Kammer, 2010) defined taxonomy of nine multi-touch frameworks: 1. MT4j5 (Laufs, 2010) 2. Sparsh-UI6 3. Surface SDK7 4. Breezemultitouch8 5. Miria9 6. Grafiti10 7. LibTISCH11 (Echtler, 2008) 8. PyMT12 (Hansen 2009) 9. GestureWorks13 The analysis focused on the architecture, scope, and features of the nine different frameworks. The architectural analysis covers platform independence, hardware independence, and event system. The scope perspective looks at the recognition of touches and tangible objects. Finally, the features viewpoint examines visualization support, gesture extensibility, and standard gestures. According to the results, most frameworks support hardware independence by working with the TUIO protocol. Windows 7 and device adapters are also common. Four of the frameworks have centralized gesture recognition, while five have decentralized it. TUIO is implemented using the OSC, which is used also in AaltoWindow. Only Surface SDK and Grafiti focus on tangible objects. The frameworks differ also in touch and gesture event parameterization. All frameworks support online processing of gestures. Five frameworks support creation and registration of new gestures, while the rest allow access to raw data. Four frameworks use custom widgets for visualization, while three rely on Windows Presentation Foundation (WPF). Compared to the above presented frameworks, the main difference in AaltoWindow is that it supports multiple multi-touch frameworks (i.e., of the above, 1. MT4j, 3. Surface SDK, 8. PyMT, and 9. GestureWorks). Thus, the developers have more freedom in choosing the most suitable programming paradigm and language. Also, different widget and rendering engines are available, e.g., web browsers, WPF, and OpenGL. Tagged objects are supported with hosted Surface and Cornerstone applications.
5
http://www.mt4j.org/
6
http://code.google.com/p/sparsh-ui/
7
http://msdn.microsoft.com/en-us/library/ee804845.aspx
8
http://code.google.com/p/breezemultitouch/
9
http://miria.codeplex.com/
10
http://grafitiproject.wordpress.com/
11
http://tisch.sf.net/
12
http://pymt.eu/
13
http://gestureworks.com/
AaltoWindow has also a layered input model, where the lowest level consists of raw touch data encapsulated either as native Windows 7 or TUIO event streams (thus combining the layers 1 and 2 of Echtler et al.). For efficiency in communication, the gesture level of the AaltoWindow input model is decentralized for widget–based interaction, and distributed into the framework that is hosting the scene graph. However, a centralized widget-independent level is also available for global gestures such as whole page swipes.
7.2. Educational multi-touch applications Many authors have proposed educational multi-touch applications. However, most of them concentrate on children, and thus they are not suitable for university level of education. Rick and Rogers (Rick, 2008) published one of the first educational multi-touch papers. They adapted an existing desktop application DigiQuilt into a collaborative multi-touch application called DigiTile. DigiQuilt is a construction kit for learning math and art by designing patchwork quilt blocks. The DigiTile application suits better for collocated collaboration than the DigiQuilt. Ardito, Costabile, and Lanzilotti (Ardito, 2010) created another educational game for learning about historical sites. They used a multi-touch table developed by Multitouch Ltd., which was also utilized in AaltoWIndow. They created a History-Puzzle and Time-Voyager for young students. Schneider et al. (Schneider, 2010) created a general multi-touch multimedia interaction framework MuTable for collaborative learning. The MuTable supports searching, composing, and presenting information. They tested the framework in an open learning center, i.e., ICT classroom equipped with 13 computers and a smart board. The multi-touch table was positioned in the middle of the ICT classroom. Three secondary school students got an assignment to make a presentation about Leonardo Da Vinci. Other students created similar presentation with normal computers in groups of two.The outcome was that the multi-touch table supported better collaborative learning. The above describe applications are primarily targeted for learning. Games seem to be the prevalent format for younger kids, while tools are provided for older students. This is in accordance with our results. The applications developed at the Multimedia Programming course are primarily tools, while one application (4) is a game. The MuTable has similar functionality as the media scrapbook (6).
7.3. University applications Aldrich analyzed 111 members of Association of Research Libraries (Aldrich, 2010). He found out that 39 universities had mobile websites. 15 schools had university-wide mobile website, 14 had mobile web sites only for libraries, and 10 had separate mobile websites for university and its library. Most common university services were events calendar, directory, news, campus maps, videos, sports, images, course catalog, bus routes, etc. Most common library services were library hours, directory, catalog, contact us, main library website, databases, “ask a librarian”, news & events, renew material, library staff directory, and my account/patron information. According to Aldrich, universities need to identify and provide more student-centric services on their mobile websites. Also, mobile websites should address better educational needs. The applications developed at the Multimedia Programming course are similar to the university applications mentioned above. The student created applications contain also news (9), maps (2 and 7), bus routes (8), and media (3 and 6).
8.
CONCLUSIONS
In this paper, we have presented the AaltoWindow platform, which is a novel virtual extension of university learning centers. It provides an open platform for highly interactive and visual campus services, aiming to 1. make the learning center services visible 2. allow interaction with the students, faculty, international visitors, and alumni 3. support interdisciplinary collaboration between the different campus user groups
In addition to the concept, we presented the architecture of the AaltoWindow platform and described both the hardware and software implementation. The results of the project shall be made available as open source software with related documentation14. The platform supports crowd sourcing of the services. This was validated using 21 students as test developers. In fall 2011, the students implemented altogether nine AaltoWindow applications. The results show that 1. Working AaltoWindow applications can be developed in few weeks. 2. Advanced developers learn AaltoWindow development fast. Previous experience in UI development is beneficial, but multi-touch experience is unnecessary. 3. Better support material is needed: documentation, tutorials, and code examples. 4. Development can be done using a mouse simulator, but real multi-touch table is needed to understand the full user experience. 5. Students believe in multi-touch. They would use the AaltoWindow if the tables were publicly available. Many would like to continue developing AaltoWindow applications. In future work, we plan to continue the technical development of the AaltoWindow platform. There are several open research questions. Most student groups implemented the AaltoWindow applications using web technologies, and thus we believe that web based multi-touch applications is an important research direction. We aim to study, especially, multi-touch events in web applications. Additionally, we plan to integrate web of things sensors and RFID/NFC tags as part of the AaltoWindow platform. We are also planning to study how the AaltoWindow platform can be used in retailing and shopping center applications. We believe that the AaltoWindow network is another important research issue. A prominent topic is the protocol for interconnecting the AaltoWindow stations. We have been experimenting with ZeroMQ carrying RESTful OSC payload. In future work, we will compare the end-to-end latency of ZeroMQ to XMPP and WebSockets. Sensor fusion for 2D multi-touch and 3D touchless interaction was also left for future work. The development of the AaltoWindow project was started in spring 2011 with the concept design and prototype implementation. In fall 2011, we implemented the core platform and created the support material for application development. During the fall term, 21 students implemented nine pilot applications on top of the framework. In spring 2012, we aim to clone more stations; create second generation of applications together with different student and researcher groups; and make the public launch of the project.
9.
ACKNOWLEDGMENTS
This work was supported by Aalto IT and Aalto Media Factory. The authors would like to thank MA P. Mukhopadhyay and MA D. Sonawane for preparing Fig.2 of this publication, and for their contribution to the early concept prototype of the system.
10. REFERENCES Aldrich, A. W. (2010). Universities and libraries move to the mobile web. EDUCAUSE Quarterly, 33(2). Retrieved from http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazine Volum/UniversitiesandLibrariesMoveto/206531 Ardito, C., Costabile, M. F., & Lanzilotti, R. (2010). Gameplay on a multitouch screen to foster learning about historical sites. Proceedings of the International Conference on Advanced Visual Interfaces, AVI’10, 75-78. ACM. DOI:10.1145/1842993.1843006
14
https://noppa.aalto.fi/noppa/kurssi/t-111.5350/resources http://blogs.aalto.fi/aaltowindow http://aaltowindow.aalto.fi
Echtler, F., & Klinker, G. (2008). A multi-touch software architecture. Proceedings of the 5th Nordic conference on Human-computer interaction building bridges, NordiCHI’08, 463-466. ACM. DOI: 10.1145/1463160.1463220 Hansen, T. E., Hourcade, J. P., Virbel, M., Patali, S., & Serra, T. (2009). PyMT: A Post-WIMP MultiTouch User Interface Toolkit. Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, ITS’09, 17-24. ACM. DOI:10.1145/1731903.1731907 Kaltenbrunner, M., Bovermann, T., Bencina, R., & Costanza, E. (2005). TUIO : A Protocol for TableTop Tangible User Interfaces. The 6th Int’l Workshop on Gesture in Human-Computer Interaction and Simulation. Kammer, D., Keck, M., Freitag, G., & Wacker, M. (2010). Taxonomy and Overview of Multi-touch Frameworks: Architecture, Scope and Features. Workshop on Engineering Patterns for Multi-Touch Interfaces. Laufs, U., Ruff, C., & Zibuschka, J. (2010). MT4j—A Cross-platform Multi-touch Development Framework. Engineering Patterns for Multi-Touch Interfaces 2010 - Workshop at the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, June 20, 2010, Berlin, Germany. Rick, J., & Rogers, Y. (2008). From DigiQuilt to DigiTile: Adapting educational technology to a multitouch table. 3rd IEEE International Workshop on Horizontal Interactive Human Computer Systems, 73-80. IEEE. DOI:10.1109/TABLETOP.2008.4660186 Schneider, J., Derboven, J., Luyten, K., Vleugels, C., Bannier, S., De Roeck, D., Verstraete, M. (2010). Multi-user Multi-touch Setups for Collaborative Learning in an Educational Setting. 7th International Conference on Cooperative Design, Visualization, and Engineering, CDVE 2010, (Calvia, Mallorca, Spain, September 19-22, 2010), 181-188. Springer. DOI:10.1007/978-3-642-16066-0_28 Wright, M. (2005). Open Sound Control: an enabling technology for musical networking. Organised Sound, 10, 3, 193-200. DOI:10.1017/S1355771805000932
