A journey through the musical metaverse
SUPSI Image Focus
Music production and consumption have always evolved hand in hand with technological development. From vinyl to streaming platforms, we have witnessed the rise and fall of media such as audio cassettes, mini-discs and compact discs. Today, some technologies promise increasingly immersive and multi-sensory production and listening experiences. One of these is the musical metaverse, the focus of the MUSMET project, funded by the EIC Pathfinder programme of Horizon Europe and the State Secretariat for Education, Research and Innovation (SETI), led by a consortium of 15 partners, including the Institute of Information Systems and Networking (ISIN) of the Department of Innovative Technologies.
For some, the definition of ‘metaverse’ is quite clear: a virtual world where people can interact with each other and with digital environments. However, for experts in the field, the concept of the musical metaverse is still evolving and developing rapidly. It is increasingly emerging as a place to produce and listen to music and this project aims to lay the foundations for a technology capable of offering innovative systems and services, aimed at both musicians and the public.
At ISIN, MUSMET is being followed by senior researcher Omran Ayoub, who is keen to point out that it is a ‘use case’ project, ‘meaning that it goes beyond its specific sector. If, in the context of the musical metaverse, it is possible to recreate a high-performance multisensory environment, among the most precise and complex, then we expect to be able to apply the developed technology in other areas as well’.
The researcher illustrates some scenarios to better explain what it is: "Imagine three members of a band who are in three different cities - Geneva, Berlin and New York. With the appropriate hardware and software equipment, they can meet in the metaverse and rehearse for their next concert. They could do this in two ways: using purely virtual instruments (i.e. that only exist in virtual reality) such as digital guitars, basses and drums that faithfully reproduce the sounds of real instruments. Or playing real instruments connected to a signal processing system to convert sounds into data and transmit them in the shared virtual space. But you can go further and imagine an audience immersed in the metaverse to watch a pre-recorded concert by an artist, or participate in a live performance by a band, which in turn could perform with virtual or real instruments".
The order of the scenarios listed above is not random, it is based on increasing degrees of interaction and technological complexity, with an increase in the quantity and type of data to be transferred. This is where a fundamental concept comes into play: latency. For the experience in the musical metaverse to be truly realistic, it's necessary that everyone perceives the signals at the same time, sounds and actions must be synchronised. Reducing this delay to a minimum, i.e. the time it takes for a signal to travel from one point to another in a network, is essential for everything to happen in real time.
"Imagine we are among the audience as the singer encourages the audience with a “Hands up!” If people raise their hands at different times due to the delay in data transmission, the experience will be unengaging. For the musical metaverse to really work, it is essential that all connected people react to stimuli at the same time, just like in reality. This requires a network infrastructure capable of guaranteeing extremely low latency, a threshold that we are trying to define and which can be reported in the order of 30 milliseconds."
Within the project, ISIN is coordinating the ‘Experimental Evaluation’ work package, and in this context, together with the consortium partners, it will conduct a series of experiments with musicians and real audiences to test the rationing times of users in the metaverse.
Having an efficient network infrastructure is a necessary condition for data transmission to guarantee synchronicity of actions; this is what experts call a ‘Network Slice’. We can imagine it as a network infrastructure designed to route data according to its type, anticipating peak flow rates as needed, exploiting available channels and guaranteeing security, speed and stability for the entire system.
"Our task within the project is also to develop machine learning algorithms to create network models that predict which and how much data needs to be routed depending on the scenario. This is to ensure a smooth experience and an adequate flow of data when users connect to an event. The musical metaverse is a multimodal network, in which different types of data coexist: audio and video, but also physiological data. As we approach the drummer, we should physically feel the snare drum beats in our chests thanks to sensory vests provided by one of the project partners. Optimising these devices requires targeted co-design, taking into account latency, transmission processes and data management. But many other factors come into play that help make the experience feel real, such as spatial audio that allows you to hear sounds coming from different directions, like chatter from the audience or someone calling your name from behind."
SUPSI Image Focus
Together with ISIN, the academic institutions in the consortium include the KTH Royal Institute of Technology, the Polytechnic University of Turin and the University of Trento, where Professor Luca Turchet coordinates MUSMET. Turchet himself reveals that the great challenge of the project is to lay the foundations for a paradigm shift in the way music is performed and experienced: "Our goal is to experiment and have fun with music in these multisensory environments. Thanks to these new immersive spaces, we can imagine and create unprecedented experiences, exploiting their potential to give life to innovative artistic forms. With a little imagination, we can generate intense emotions, not only for the audience, but also for the musicians themselves. The goal is to develop new types of music and artistic formats that can exist exclusively in these environments, opening up new scenarios for creativity and musical expression."
Although the project has only been active for a month, the work is already in full experimentation mode: "As a video recorded for the occasion shows, we held an experimental concert in our laboratory. Traditional musical instruments were connected, including drums, keyboard and electric guitar, enhanced with artificial intelligence technologies, and mixed reality viewers worn by the spectators. ‘In addition to the visual component, we have integrated tactile devices, such as vests equipped with actuators, to offer an immersive multisensory experience that combines hearing, sight and touch in a complete immersion."
‘The project is undoubtedly very ambitious, but history teaches us that musicians, especially composers, have often been pioneers of innovation and new artistic ideas, even when they were not initially understood or appreciated,’ reflects the professor, concluding, "this is happening with artificial intelligence and I am sure it will also happen with the musical metaverse. It‘s just a matter of time. However, the effects of such systems on people involved in musical activities are not yet known, and it is of fundamental importance to adopt a design based on solid ethical principles so that the impact on society is indeed positive."