Augmented and Virtu Augmented and Virtual Reality ...

Proceedings of the IASS Annual Symposium 2017 “Interfaces: architecture . engineering . science” September 25 - 28th, 2017, Hamburg, Germany Annette Bögle, Manfred Grohmann (eds.)

Augmented and Virtual Reality Structures Gregory QUINN*, Adrian GALEAZZI, Christoph GENGNAGEL *Department for Structural Design & Engineering, Berlin University of Arts Hardenbergstrasse 33, 10623 Berlin, Germany [email protected]

Abstract This paper presents a prototypical tool in which users interact with physical objects (educational structural systems such as portal frames) which are visually augmented by means of projected images of a synchronous simulation of the same system. The visual augmentation provides the user with immediate insight into the systems’ structural performance by displaying internal forces (bending moments, axial forces and shear forces) as well as reactions re resulting sulting directly from the user-imposed deflections. This prototype provides an arguably more engaging and intuitive way in which to interact with and learn about structural systems and principals than comparable teaching aids. Keywords: augmented reality, virtual reality, dynamic relaxation, structures, education

1. Concept Many digital and physical aids exist for the teaching of structural engineering principals. Physical aids such as sponge blocks or hanging chains can help students develop an intuition for structural behaviour via tactile response whereas computer-based aids such as PushMePullMe [1] and Stiff [2] can be very effective in developing an intuition for the distribution of internal forces and reactions of structural systems and can support the teaching of theoretical principals. The augmented reality (AR) prototype presented in this paper succeeds in combining physical interactions with digital simulations in a colourful, intuitive and engaging educational experience. Although the prototype could in theory be realised using any number of software environments, this was made feasible and inspired by the precision, speed and customizability of the Kangaroo dynamic relaxation (DR) solver and the flexibility of the Rhino / Grasshopper software environment.

Early feedback on the prototype’s potential has been very positive and now the system is being trialled systematically in the architecture departments of the Berlin University of Arts as well as the University of Bath. While the prototype is currently limited to two di dimensions mensions and implemented within an educational context, the potential of augmented and virtual reality structures are far reaching.

Copyright © 2017 by Published by the International Association for Shell and Spatial Structures (IASS) with permission.

Proceedings of the IASS Annual Symposium 2017 Interfaces: architecture.engineering.science

Figure 1: StructAR prototype showing a user physically interacting with small structural frame made from GFRP rods. The physical objects are augmented by a projected overlay of internal forces from the same structural system.

Figure 2: Projected axial forces on a portal frame forcibly displaced by the user. The tracked marker can be seen by the blue circle overlay.

2


1.1 Origins and development The concept and technical realisation for StructAR was developed by the author as original research but also as part of the “Form-Active Hybrid Structures” workshop at the 2016 SmartGeometry conference. Original ideas and contributions were also provided by Daniel Piker with valuable feedback from Anders Holden Delueran and Cecilie Brandt-Olsen.

Figure 3: First ever application of an early version of StructAR at the SmartGeometry 2016 cluster on Form Active Hybrid Structures.

2. Technical realisation The software for the prototype was written in the Rhino / Grasshopper environment in which the physical simulations are powered by Kangaroo [3] (a vector-projection-based drift-damped dynamic relaxation solver), the tracking of physical objects is powered by reacTIVision (an open source tracking framework for fiducial markers), and custom scripts were written in Python and C#. The Kangaroo DR solver and its feature permitting the user to script custom goals played a central and catalytic role in this experiment. The versatility of Grasshopper and the many free plug-ins available for the software environment facilitated a fast development and realisation of the concept.

2.1. Simulation 2.1.1. DR solver The dynamic relaxation method for solving physical simulations was first developed in the 1960s [4] and has since become established in many fields. In the built environment, DR is most commonly associated with stiffness-independent (purely force based) membrane form form--finding or with the manipulation of architectural geometry (e.g. meshing, panelisation). In structural engineering implicit integration methods are more common, typically featuring a full six degrees of freedom (DOF) at each node which can accurately describe mechanica mechanicall stresses and displacements in a discretized continuum under the assumption of small deflections. If equilibrium is being sought in a system where deflections are large, the stiffness matrix must be updated over multiple iterations in a non-linear analysis which can be computationally demanding and often unstable. Generally a prerequisite for implicitly integrated methods is that the systems must be statically determinate or indeterminate. Mechanisms cause numerical instability and cannot be calculated. DR on the other hand, does not require the computation and inversion of a global stiffness matrix, but instead seeks equilibrium in each node explicitly and simultaneously by assigning mass, acceleration and a method of damping to the nodes. This means that DR methods are insensitive to the static determinacy of the structural system such that mechanisms and large deformations are not an issue, provided the solver is able to remain stable (as is the case with Kangaroo). This insensitivity to static

3


determinacy and large deformations is highly suitable and forgiving in an experimental and interactive design environment such as this one. Imposing large deformations on the fly and observing the immediate impact they have on structural performance (e.g. deflections, internal forces and reactions) can be an illuminating experience for the user.

Figure 4: A typical demo featuring four different portal frame systems. Proximity sensors were scripted to detect when the user’s marker is close to a particular portal frame thereby activating only that system and avoiding wasted processing power on inactive stru structural ctural systems in the demo.

2.1.2. Quantitatively accurate stiffness Since Kangaroo minimises the total energy of the system it is possible to simulate quantitatively accurate structural behaviour by providing appropriate stiffness values as input for the goals. At the time of writing, the Kangaroo solver is based on the manipulation of vertices with three degrees of freedom. This has certain implications on which mathematical models can and can’t be used for the simulation of structural behaviour. For the modelling of beams in Kangaroo, axial and bending stiffness are defined by two separate goals based on Hooke’s Law and the Barnes / Adriaenssens model [5] respectively. The bending model defines bending radii on a plane of three sequential nodes and does not account for orientation or anisotropy of cross sections. Nor is beam torsion accounted for. As such the beam model is simple and fast to compute. While more accurate 4 and 6 DOF solutions exist to describe beam behaviour using DR [6], [7] [7],, their increased computational demands significantly reduce their speed and hence suitability for this method which depends on fast and interactive feedback. In order to further increase the speed of simulations, the axial stiffness (EA/L) of the beam elements was reduced to one tenth of its actual value so that it was numerically closer to the value for bending stiffness (EI). If stiffness values (or in Kangaroo terminology goal weightings) are numerically different by multiple orders of magnitude this causes the convergence to slow down. When multiple goals act on a single vertex, Kangaroo minimizes the sum of the vectors’ squared errors. More highly waited goals then effectively cause the vertex to “drag” on its path to convergence.

4


2.1.3. Custom goals A great strength of the latest release of Kangaroo is the ability to script custom goals in C# or Python which define move vectors and weightings for particles in the system to be solved. One such goal scripted especially for this method was a point-line collision goal which was used for the roller supports as well as the user-controlled pointer. The overall simulation makes use of predominantly native Kangaroo goals in combination with some custom goals. 2.1.4. Graphical User Interface (GUI) A central objective for the GUI of StructAR is that once it has been set up, the user must not be required to interact in any way with the computer interface but instead may simply and intuitively interact with the projection board and the pointer. This was achieved by simple coloured circles which permit the user to control Boolean inputs such as visualising internal forces. Rendered mesh geometries were scripted using native Grasshopper and Python components to display internal forces (bending moments, axial and shear forces) providing immediate visual feedback on structural behaviour.

Figure 5: Custom GUI and visualisation of internal and external forces. From left to right: bending moments, axial force, shear force and reactions.

2.2. Physical objects In this prototype the physically interactive structural objects were entirely passive in terms of the simulation. Their deflections were not recorded or fed back to the computer in any way. Instead the computer simulates the same system in parallel base on the recorded position of the pointer. The scale of the objects as well as the user being forced to interact with them in large deliberate movements effectively eliminates any chance of significant deflection discrepancies between the two. Using the pointer also ensures a single (and computationally manageable) imposed load. Ultimately however a developmental target for this system should be direct tactile interaction by hand. The projection surface of the system is a 1.4m wide and 11m m high plywood board with an array of holes every 20cm in which custom pin, roller and rigid supports can be attached. The pin support is made from steel eyelets welded to m6 bolts. The roller support iiss ball bearing roller from the curtain industry. 3mm GFRP rods were used for the structural systems. These feature a suitably high level of stiffness which provides the user with good amount of haptic resistance upon interaction. Experimental systems were also made using PVC foam board which in a dark colour offers better visibility but results in substantial material wastage due to the parts being cut out from larger panels.

5


Figure 6: Physical solutions for a pin joint (left), a rolling support (centre) and a rigid corner (right) in combination with 3mm GFRP rods.

2.3. Interfacing The interfacing between the physical objects and the digital simulation is achieved by tracking printed fiducial markers by means of a standard optical webcam. The Firefly plug-in for Grasshopper features many gateways to multiple forms of digital input including components for the interfacing between Grasshopper and the open source reacTIVision framework for fiducial markers. reacTIVision is able to detect the position and ID of these markers from a webcam image feed. The projection board has a printed marker at each corner so that even if the recorded webcam image is distorted or misaligned, the rectangular plane of the board is tracked and the position of the user-controlled marker is remapped to u/v coordinates independent of any optical distortions. This setup determines coordinates in two degrees of freedom; other fiducial libraries (such as ARTag) are able to record further DOF which are not necessary here. Fiducial markers are just one of many potentially viable tracking methods for a user-controlled marker on a 2D plane. Methods which require no (or less demanding) image processing, such as infrared LEDs or capacitive signals, could speed up the processing required for tracking which is currently the computational bottleneck of the prototype.

Figure 7: The user held marker for physical interactions, aka the “magic lollipop”. A future iteration of the wand will feature rolling disc to enable frictionless contact (as digitally simulated). Also shown is an alternative material (PVC foam board) to the 3mm GFRP rods which offers greater visibility due to its thicker section and more contrasting colour.

3. Future developments 3.1. StructAR The current prototype works based on predefined structural systems in combination with complementary educational objectives. A more engaging and enlightening extension of this system would be to allow the user to assemble and test their own structural systems from a selection of parts

6


(beams, joints and supports) which the digital system is able to read and replicate on the fly. User experimentation should still be guided however by educational cues and objectives. While the scale of the prototypical projection board and the magnitude of the haptic resistance from the structural systems are highly appropriate for human interaction, miniaturising the system to the size of a tablet could potentially increase its potential to be disseminated in classrooms. A solution here may involve suction cups, snap-to-fit joints and a capacitive detection system. 3.2. StructVR A natural extension of the presented concept is virtual reality structures (StructVR) which is currently an active area of research for the authors. At the time of writing a working prototype is not yet complete but initial tests and the broader concept will be explained here. A virtual reality environment offers the advantages of versatile scale, high fidelity and engaging visuals, greater potential for directed user experiences and of course the freedom of three dimensions. A disadvantage over StructAR is the limited ability to provide the user with force feedback. Similar to StructAR, the aim for StructVR is to allow the user to physically interact with structural systems, visualise their internal and external forces and to facilitate a “sandbox” environment for creating, discovering and learning about structural engineering in an engaging manner.

Figure 8: A mock-up image for StructVR which at the time of writing is near completion. Virtual reality requires the rendering of two separate scenes (one for each eye) at high resolution and a high frame rate (≥90Hz) which the Rhino environment is unable to deliver. Subsequently gaming engines which in contrast can provide the required fidelity and offer extensive VR toolkits are more suited to the actual rendering of visuals for this application. After initial attempts with the Unreal engine (C++) and Cryengine (C#), the Unity gaming engine was selected due to its use of the C# programming language, its strong online community presence and its established research credentials. While highly competent physics engines exist in Unity (e.g. Havok), the Kangaroo engine is still preferred for this application because it can facilitate quantitatively accurate stiffness. The Kangaroo library is written in C# and so, if it were suitably compiled, could be referenced directly from within Unity. Until then we must still compute simulations in parallel in the Grasshopper environment and exchange data between the two environments on the fly. For initial prototypes with the HTC Vive, this was achieved by sending coordinates of the VR controllers as bytes to Grasshopper via nodejs (a JavaScript runtime) resulting in 1:1 positional tracking data between the Vive and Grasshopper. Similarly lists of coordinate data are streamed back to Unity from Grasshopper.

7


Figure 9: On-the-fly tracking of HTC Vive controller positions and trigger button Boolean.

Acknowledgements Lukas Bodelschwingh, Jörg Rädler, Daniel Piker, Cecilie Brandt-Olsen, Anders Holden Deleuran

References [1] G. Senatore and D. Piker, “Interactive real-time physics,” Comput.-Aided Des., vol. 61, pp. 32–41, Apr. 2015. [2] K.-U. Bletzinger, F. Dieringer, R. Fisch, and B. Philipp, Eds., Aufgabensammlung zur Baustatik: Übungsaufgaben zur Berechnung ebener Stabtragwerke. München: fv, Fachbuchverl. Leipzig im Hanser Verl, 2015. [3] D. Piker, “Kangaroo: Form Finding with Computational Physics,” Archit. Des., vol. 83, no. 2, pp. 136–137, Mar. 2013. [4] A. S. Day, “An Introduction to Dynamic Relaxation,” Engineer, pp. 218–221, 1965. [5] M. R. Barnes, S. Adriaenssens, and M. Krupka, “A novel torsion/bending element for dynamic relaxation modeling,” Comput. Struct., vol. 119, pp. 60–67, Apr. 2013. [6] B. D’Amico, H. Zhang, and A. Kermani, “A finite-difference formulation of elastic rod for the design of actively bent structures,” Eng. Struct., vol. 117, pp. 518–527, Jun. 2016. [7] L. Du Peloux, F. Tayeb, B. Lefevre, O. Baverel, and J.-F. Caron, “Formulation of a 4-DoF torsion/bending element for the formfinding of elastic gridshells,” in IASS Symposium 2015:“ Future Visions,” 2015.

8