Melody and Motion: Integrating Guitar Gestures with Musical Patterns for Extended Control in Live Performance and Metaverse Applications

This paper introduces a novel system for detecting gestural movements of guitarists during live performances, integrated with a pre-existing musical pattern detection framework. The system enables guitarists to trigger peripheral devices through combined musical and physical gestures, extending instrumental control beyond traditional interfaces. We conducted a case study evaluation with five professional guitarists to explore the continuum between natural and theatrical instrument movements, examining how these gestures can be leveraged for expressive control. Technical evaluation of the system using a 500-event test corpus demonstrated a precision of 0.79, recall of 0.76, and F1 score of 0.78 for combined gesture and pattern detection. Through a set of interviews we further investigated practical applications within Musical Metaverse environments, highlighting opportunities for immersive performance experiences. Our findings reveal how combining musical content with physical expression creates intentional and novel performance controls, opening novel possibilities for interactive music that bridges traditional performances with virtual environments.