Musical Metaverse Playgrounds: exploring the design of shared virtual sonic experiences on web browsers

The "Musical Metaverse"(MM) promises a new dimension of musical expression, creation, and education through shared virtual environments. However, the research on the MM is in its infancy. Little work has been done to understand the MM musical capabilities and its user experience. One cause can be found in the lack of technologies capable of providing high-quality audio streaming, and complex enough musical interactions within shared virtual worlds. Two promising candidates for bridging these gaps are web technologies such as WebXR and Web Audio, whose combination can potentially allow more accessible and interoperable networked immersive musical experiences. To explore this possibility, we developed two prototypes of MM playgrounds. We leveraged WebXR through Networked-AFrame, and Web Audio with Tone.js and Essentia.js to create and test immersive sonic experiences that conveniently run on the web browsers integrated into commercially available standalone Head-Mounted Displays. The first playground focuses on facilitating musical creation in a multi-user immersive application through real-time sound synthesis and binaural rendering. The second explores real-time audio analysis and music information retrieval for creating audio-reactive virtual shared environments. A preliminary evaluation of these two playgrounds is also presented, which revealed some usability issues such as: accessing the immersive experiences through URLs on a headset, the ambiguity in ownership of shared musical tools, and the impact of real-time audio analysis algorithms on perceived audio-visual latency. Finally, the paper outlines future work and discusses possible developments and applications of web-based MM environments.