Interactive IoMusT-Based Concerts: Real-Time Pattern Recognition and Audience Experience

This paper explores the possibilities offered by the Internet of Musical Things paradigm to create new interactive performance experiences. We introduce a novel performance ecosystem where performers utilize smart musical instruments (SMIs) equipped with embedded Real-time Pattern Detection (RTPD) algorithms to trigger wirelessly connected devices. To validate our approach, the system was ecologically validated with two concerts, which involved two groups of performers and two groups of audience members. Each concert featured three SMIs (electric guitar, keyboard, and electronic drumkit) which were used to control three stage lights, one smoke machine, 3D graphics and haptic vibrations on Mixed Reality headsets and smartphones worn ad held by the audience members. We compared the use of RTPD-triggered effects against randomized triggering, and we assessed the experience of a composer, performers and audiences in interacting with the ecosystem using qualitative and quantitative assessments. Audience members preferred the condition in which the RTPD system directly triggered effects rather than random triggering. Overall, composers, performers and audience valued this new art format. The results indicate that combining smart musical instruments with pattern detection in an Internet of Musical Things ecosystem can open new avenues for artistic performance and audience engagement.