image-based renderings, virtual displays, and other programs. To leverage our ... connects to a Mathematica program which is already running. We think we can ...
Integrating the Collaborative Virtual Environment Protocol with Mathematica Anzu Nakada, Michael Cohen, & Rasika Ranaweera Spatial Media Group, Computer Arts Lab. University of Aizu Aizu-Wakamatsu, Fukushima 965-8580 Japan Email: {s1180027, mcohen, d8121104}@u-aizu.ac.jp Abstract—We have built interfaces featuring smartphones and tablets that use magnetometer-derived orientation sensing to control spatial sound, motion platforms, panoramic and turnoramic image-based renderings, virtual displays, and other programs. To leverage our Collaborative Virtual Environment (CVE), which is implemented in pure Java, against the power of Mathematica, we use J/Link middleware. As a result, we can exploit Mathematica features of graphics and calculation and control the Mathematica Kernel by data from, among other clients, mobile devices.
I.
I NTRODUCTION
Our group uses a locally developed Collaborative Virtual Environment (CVE) to synchronize distributed clients, as seen in Fig. 1 [1], [2]. Various CVE clients can display or control the positions and motion of avatars & virtual objects, panoramic & turnoramic rendering, rotary motion platforms, spatial sound, etc. [3]. Poi, originally a Maori performance art featuring whirled tethered weight, combines elements of dance and juggling. Padiddling is the skill of spinning flattish objects such as plates, as seen in Fig. 2. Multiple clients can connect to a session server and be controlled by data of electronic compass obtained from mobile device with poi or padiddling. To increase the scope of interaction, we add Mathematica to the client suite. Mathematica has various functions such as calculation, graphics, programability in many styles, and sophisticated interaction [4]. The CVE protocol is developed in pure Java. We developed a CVE program to integrate Mathematica and CVE to connected Mathematica to Java and write to Mathematica data of electronic the compass from mobile device. For such occasions, we use J/Link [5]. II.
and a 3D graphics object compass from a pillar and rotating radial arm (Fig. 5), although we plan to extend both to display multiple clients’ data, streaming “tails”, and other information. These displays show networked client data including azimuth, such as that sensed by the magnetometers (electronic compass) of mobile devices used for poi or padiddling. As shown in Figs. 7 & 8, the end-to-end data delivery is consistent enough, over the Wi-Fi network and even the cellular network, to allow smooth display of telemetric information.
I MPLEMENTATION
We tried experimenting with various linkmodes— launch, listen, and connect. If we use launch linkmode, a Java program starts up a Mathematica kernel. If we use listen or connect linkmode, a Java program connects to a Mathematica program which is already running. We think we can connect a Java Virtual Machine (JVM) to the Mathematica front end, as shown in Fig. 3, if listen or connect linkmode is used, but couldn’t figure out how to connect a Java program to an active Mathematica session. So instead we render the graphics through a Java (AWT) window. (AWT is a windowing toolkit, a predecessor of the more modern Java “Swing” framework for crafting user interfaces.) This window currently features a polar plot (Fig. 4)
III.
I MPLICATIONS
J/Link integrates Mathematica and Java. We can use the power of Mathematica in Java programs. We can integrate Mathematica and Java programs, including CVE server, and therefore, indirectly, CVE clients. We have developed a J/Link bridge featuring CVE control as “write only” invocation of Mathematica. We can use Mathematica powers of graphics and calculation, including the back-end kernel and, eventually, the front-end user interface. If the program is run, the azimuth of smartphone or tablet which is streaming data can be displayed as, for instance, the arrow or needle of a compass. However, the rendering and process is heavy, especially for 3D graphics, and currently unsuitable for fast spinning. We hope to improve the throughput by optimizing the graphical refresh. We also hope to architect a way to read from Mathematica for full duplex “read and write,” control and display. If successful, we will be able control spatial media such as audio and other displays from Mathematica. R EFERENCES [1]
M. Cohen, R. Ranaweera, K. Nishimura, Y. Sasamoto, T. Oyama, Y. Nishikawa, T. Ohashi, R. Kanno, A. Nakada, J. Yamadera, S. Holesch, Y. P. Chen, A. Sasaki, and H. Ito, ““Whirled Worlds”: twirling interface for “mobile ambient,” “practically panoramic” whole-body entertainment,” in DCE: Digital Contents Expo. Tokyo: Digital Content Association of Japan, Oct. 2012, www.dcexpo.jp/2012/en/program/exhibition/detail.php#IT201222. [2] Digital Information, “Whirled Worlds,” www.diginfo.tv/v/120195-d-en.php, www.youtube.com/watch?v=mQf89AR3Tbg, http://gizmotv.blogspot.jp/2013/01/twirling-interface-developed-formobile.html, http://pann.nate.com/video/221245229. [3] M. Cohen, R. Ranaweera, K. Nishimura, Y. Sasamoto, S. Endo, T. Oyama, T. Ohashi, Y. Nishikawa, R. Kanno, A. Nakada, J. Villegas, Y. P. Chen, S. Holesch, J. Yamadera, H. Ito, Y. Saito, and A. Sasaki, ““Tworlds”: Twirled Worlds for Multimodal ‘Padiddle’ Spinning & Tethered ‘Poi’ Whirling,” in SIGGRAPH (Poster), Anaheim, California, Jul. 2013, http://dl.acm.org/citation.cfm?doid=2503385.2503459, ISBN 9781-4503-2342-0.
Fig. 1.
System Architecture
Fig. 2.
Rigged augmented reality scene allows controlling the motion of avatars and virtual objects with poi.
Fig. 3.
Mathematica front-end showing juxtaposed 2D and 3D widgets
Fig. 4.
2D graphics display featuring Java-rendered, Mathematica-generated objects
Fig. 5.
3D graphics display featuring Java-rendered, Mathematica-generated objects
Fig. 6. Data Flow: The dashed line indicates the originally intended invocation of Mathematica session from Java. However, we weren’t able to connect to the front-end of an ongoing Mathematica session, so adjusted our deployment to display Mathematica-generated graphics in a Java AWT-spawned window (ultimately rendered via OpenGL on computers running OS X, and via DirectX on Windows computers).
[4] [5]
S. Wolfram, Mathematica, 2008, sixth Edition. W. Research, J/Link User Guide, 2008, ISBN 978-1-57955-071-4, www.wolfram.com/learningcenter/tutorialcollection/JLinkUserGuide/JLinkUserGuide.pdf.
Fig. 7.
Wi-Fi delivered CVE message arrival time vs sending time
Fig. 8.
CDMA-delivered message arrival time vs sending time
