SOFA Native Spatializer Plugin for Unity

WWW.KFS.OEAW.AC.AT

SOFA Native Spatializer Plugin for Unity: Exchangeable HRTFs in Virtual Reality 1,2

2

1

Claudia Jenny , Piotr Majdak , Christoph Reuter 1 Musicology Institute, University of Vienna 2 Acoustics Research Institute, Austrian Academy of Sciences Background In order to provide a realistic audio for headphone reproduction in virtual reality devices, directional features commonly described by headrelated transfer functions (HRTFs; Møller et al., 1995) can be applied to audio signals. Most game engines and virtual reality devices manufacturer already have started to use HRTFs in their audio algorithms. For instance, in Unity, the native Spatializer Plugin allows to filter audio signals via factory-implemented HRTFs recorded from the KEMAR manikin. However, with these fixed HRTFs, proper spatial perception is not always guaranteed, as the individual HRTFs of the listeners are highly individual and do not always correspond to the implemented ones. There exists a variety of other audio spatialization solutions (e.g. 3Dception, RealSpace 3D, Rapture3D, MS Spatializer, Steam Audio, Oculus Spatializer, etc.) for different platforms (DAW Win, DAW Mac, FMOD, Unity, Wwise, Unreal, etc.), however, none of them are able to process different HRTF sets and switching between them in real time in audio-visual virtual environments. Hence, for research purposes, we have developed a spatialization plugin capable of the interchangeable use of any HRTFs in audio-visual virtual environments. The challenge was to be able to process HRTFs from SOFA files (spatially oriented format for acoustics, AES69-2015 standard, Majdak et al., 2013, also see Fig. 2) in real time with head-tracking involved and to be able to switch from one HRTF set to another on-the-fly.

The convolution results by multiplying pointwise and then carring out the inverse transformation (IFFT). For the selectable use within the SOFAlizer, we created a parameter called “SOFASelector“ allowing to set and get the parameter from the plugin via the public “SetSpatializer“ and ”GetSpatializer“ float variables. With the RegisterParameterfunction in the native plugin, it is possible to create such a new enumerator parameter that allows to affect data by setting and getting parameters entered from the “AudioSource”. At the moment, it is possible to compile the SOFAlizer plugin such that on start, it loads a maximum of ten HRTFs into the memory. SOFA allows to store not only HRTFs but also more complex data, e.g., directional room impulse responses (DRIRs). SOFAlizer is able to load also more than ten HRTFs or even other complex data which contain longer impulse responses. Still, because the instantiation of the spatializer effect is a very costly thing (in terms of memory allocations, precalculations, etc.), Unity suggests to consider keeping the plugin interface bindings very light-weight (Unity-Manual, 2017). Examplary Scene As an example for the usage of the SOFAlizer plugin, we have created a project demonstrating its capabilities: As an audio-visual object, a loudspeaker is presented from the first-person perspective, embedded in a freefield environment with a sky and a light source (see Fig. 4). The first-person can be moved freely (with a game controller) and the listener can rotate his/her head in 6 degrees of freedom. While playing, the listener can change HRTFs in real-time by pressing the keys from 0 to 9.

Unity Audio Data For the plugin, as a software framework, the software package Unity was selected. Unity is a development system for platform-independent games or for two- or three-dimensional virtual environment simulations. These environments are created in C# and can be paired with various virtual reality devices such as Oculus Rift, Samsung Gear, or HTC Vive. With respect to spatial audio in Unity, the Audio Spatializer SDK provides an open interface of an implementation of a spatializer. The Unity Audio Spatializer is a native plugin written in C/C++ and can be directly integrated within the Unity “AudioManager” and activated on any “GameObject” attached on a “AudioSource” component (see Fig. 1).

AudioClips

AudioSource

Spatializer Effects post

Fig. 4: Examplary Scene showing the usage of the SOFAlizer plugin for Unity3D. AudioListener

Conclusion and Outlook

Fig. 1: Basic structure of the workflow of the SOFAlizer for Unity.

SOFAlizer Algorithm Based on that Audio Spatializer SDK, we developed the SOFA Native Spatializer Plugin (SOFAlizer), a spatializer plugin capable of processing individualized HRTFs from SOFA files. In the SOFAlizer, any HRTF stored as SOFA (www.sofaconventions.org) can be integrated. Many SOFA files are available from various HRTF databases (Majdak et al., 2007). In the SOFAlizer, a SOFA file is read using the library “mylibsofa” (Hoene et al., 2017). The accepted HRTFs must use the convention “SimpleFreeFieldHRIR”. While filtering, the position and head orientation of the listener given by the tracking sensors from the virtual reality devices are taken into account.

The availability of the SOFAlizer enables more detailed investigations of multimodal perception with individual HRTFs. Experiments with exchangeable HRTFs can now be tested in audio-visual virtual environments in which listening situations can be included in the evaluation in a repeatable manner and with all variables controlled. With the SOFAlizer it is possible to use now any HRTFs real time interchangeably in audio-visual virtual environments and one can adjust for instance the HRTFs adaptively during a game by loading different angles with high or low spatial resolution. For MS Windows, the SOFAlizer was compiled into a dynamic link library (DLL) available for Unity version 5.5 or higher. The first release of the SOFAlizer plugin is available from github under the open source license European Union Public License version 1.1. Future steps will be an implementation of cross-fading buffers or segmentedFFT methods such as zero-delay convolution (Gardner, 1995) in order to reduce zipper artefacts in fast movements of a source. Spatial interpolation of HRTFs as well as the support of multiple sources will further enhance the usability of the SOFAlizer plugin in the future. References

ZP

FFT

Source Pos Listener Pos

Fig.2: Description of HRTFs in SOFA.

IFFT

HRTF

SOFASelector

Fig.3: Simplified illustration of the SOFAlizer fast convolution algorithm using zero-padding (ZP), fast Fourier transform (FFT), and inverse FFT (IFFT) excl. callback functions for loops.

Within the SOFAlizer algorithm, meta-data about the source and listener position are used to compute the matching impulse response position of respectively selected HRTF and these variable data are processed using fast convolution algorithm (see Fig. 3). In order to perform fast convolution via circular convolution using fast Fourier transform (FFT), the input sequence and the impulse response must be the same length and therefore zero-padding is additional implemented.

AES69-2015, “AES standard for file exchange - Spatial acoustic data file format,” 2015. Gardner, W. G., “Efficient Convolution without Input-Output Delay,” Journal of the Audio Engineering Society, 43(3), pp. 127–136, 1995. ˜ Mej´ıa, I. C., and Cacerovschi, A., “MySofa-Design Your Personal HRTF,” in Hoene, C., Patino Proceedings of the 142nd Convention of the Audio Engineering Society, Convention Paper 9764, Berlin, Germany, 2017. Majdak, P., Balazs, P., and Laback, B., “Multiple exponential sweep method for fast measurement of head-related transfer functions,” Journal of the Audio Engineering Society, 55(7/8), pp. 623–637, 2007. Majdak, P., Carpentier, T., Nicol, R., Roginska, A., Suzuki, Y., Watanabe, K., Wierstorf, H., Ziegelwanger, H., and Noisternig, M., “Spatially Oriented Format for Acoustics: A Data Exchange Format Representing Head-Related Transfer Functions,” in Proceedings of the 134th Convention of the Audio Engineering Society, Convention Paper 8880, Roma, Italy, 2013. Møller, H., Sørensen, M. F., Hammershøi, D., and Jensen, C. B., “Head-related transfer functions of human subjects,” Journal of the Audio Engineering Society, 43(5), pp. 300–321, 1995. Unity-Manual, “https://docs.unity3d.com/Manual/AudioSpatializerSDK.html,” 2017.

¨ SCHALLFORSCHUNG ISF - INSTITUT FUR

M I X E R