Session 6 — Paper Presentations

Chelys Sonorum: A Quasi-Autonomous Guitar

Guglielmo Diana

Keywords: electromagnetic sustain, hexaphonic pickup, motorized retuning, quasi-autonomous instrument, feedback control, HPS pitch estimation, sound installation, chroma analysis

Abstract

Chelys Sonorum is a hybrid electroacoustic guitar that combines hexaphonic electromagnetic sustain and motorized string-tension control to investigate quasi-autonomous sound production. Each string integrates an independent pickup-exciter pair for sustained self-resonance, while six geared DC motors adjust tuning in real time via Arduino and a Max control patch. A control pipeline based on loudness, pitch, and spectral descriptors modulates feedback response and retuning behaviour during operation. The paper documents the design of the instrument and reports on its deployment in the installation I've Used to Play It, presented at the To Listen To 2025 festival in Turin, as a case study of long-duration operation in a public installation context. In this setting, a database of popular music riffs was used to generate evolving sonic behaviours. The installation demonstrated robust long-duration operation with limited technical intervention, while also revealing limitations in chroma-based analysis under feedback-driven conditions.

 

InvisiblePiano: Enabling Gesture-Based Right-Hand Monophonic Piano Performance using a Smartphone Camera via Statistical Machine Translation

Chihiro Mita, Kento Watanabe, Takayuki Nakatsuka, Tomoyasu Nakano, Masataka Goto and Takayuki Itoh

Keywords: Digital Musical Instruments, Gesture-based Interaction, Pitch Estimation, Piano Performance

Abstract

This paper proposes InvisiblePiano, an interactive system that enables gesture-based right-hand monophonic piano performance without an actual piano keyboard. The system captures the user's hand movements using an existing hand tracking method, without requiring a physical interface or virtual visual guides such as VR overlays. To achieve musically natural interaction, InvisiblePiano incorporates a pitch estimation model that infers pitch sequences (i.e., individual MIDI note numbers) directly from the tracking results. This model combines an N-gram pitch language model with a translation model based on key press changes and fingering, enabling the mapping of piano-like hand gestures to structured musical output. Experiments using known melodies to evaluate the pitch estimation model achieved a pitch accuracy of 98.43\%. A user study with 18 participants confirmed the system's intuitiveness and usability. These results suggest that InvisiblePiano enables accessible and highly accurate gesture-based right-hand monophonic piano performance without an actual piano keyboard or visual guidance.

 

ESP-NOW MIDI Protocol and Enomik (ESP-NOW MIDI Kit): Wired and Wireless MIDI Devices with an Optional Code-Free Workflow

Thomas Geissl

Keywords: Wireless MIDI, ESP-NOW, Digital Musical Instruments, Rapid Prototyping, Embedded Music Systems

Abstract

ESP-NOW MIDI is a lightweight wireless layer for low-latency links between distributed devices in musical instruments and interactive systems. It uses Espressif's ESP-NOW to carry MIDI directly between devices, without relying on external network infrastructure. Wireless hops are converted to standard MIDI streams at the edges, preserving compatibility with existing software and hardware. The API follows established MIDI libraries closely and can often be used as a near drop-in replacement. The evaluation section reports round-trip latency measurements. Enomik adds utilities for multi-device setups: a USB dongle bridging wireless nodes to a host, a WebMIDI-based configuration interface, Pure Data patches, and modular PCBs with standardized connectors and common sensors and actuators. Together, these aim to reduce infrastructural overhead, support interplay among networked devices, and keep physical controllers and digital sound-making environments tightly coupled in musical exploration.

 

A Multi-Modal Adaptive Microtuning Architecture: Bridging Mts, Mpe, And Midi 2.0 For Struck-String Just Intonation

Rui Su, Joseph Timoney and Damián Keller

Keywords: MIDI, Microtuning, Real-Time processing, Adaptive Just Intonation, Ubiquitous Music

Abstract

Current implementations of Real-Time Just Intonation (JI) are faced with a conundrum of precision versus compatibility. Tuning within MIDI is supported by the MIDI Tuning Standard (MTS) and pitch bend-based methods. MTS features good resolution but suffers from limited hardware support. Pitch bend-based methods provide ample compatibility but not enough granular resolution for microtonal tuning. This paper outlines an adaptive, multi-stage tuning architecture that extends the capabilities of the Web MIDI API and solves both of these problems. Within ubimus research agenda for struck string interactions, this proposed system utilizes a hierarchical fall back strategy: (1) MTS for scale-based tuning, (2) An MPE mode with precision optimized for locking the sensitivity of pitch bend to ±2 semitones for high resolution tuning, and (3) A forward compatible prototype engine that complies with the emerging MIDI 2.0 standard. Both MTS and MPE modes provide resolution that is at least one order of magnitude finer than those deviation intervals present in equal temperament, which enables a perceptively transparent means of achieving adaptive tuning via native deployment in the browser.

 

TidalBind: Towards Interactive Real-Time Control and Declarative Mapping In Live Coding

Gordan Kreković and Antonio Pošćić

Keywords: live coding, MIDI control, semantic markup, ergodynamics, Tidal Cycles, computer music

Abstract

TidalBind bridges the gap between symbolic logic and gestural immediacy in live coding by enabling real-time MIDI control in Tidal Cycles. Its novelty is the use of declarative in-code mapping of MIDI interactions based on semantic markups. Being an integral part of the code, mapping configurations become dynamic material, enabling interactivity, while reducing performer’s cognitive load. TidalBind provides triggering functions that initiate certain actions, such as evaluating specific blocks or navigating the code sequence, based on conditional MIDI events. Additionally, it includes a mapping function that dynamically substitutes semantic markups with raw or linearly scaled MIDI values immediately before code execution. The variety of mappable MIDI events implies a great control flexibility. TidalBind improves the ergodynamics of live coding and maintains the “show us your screens” principle by displaying incoming MIDI data directly within the text buffer. While current limitations include the initialization of continuous parameters, TidalBind offers a more embodied and transparent live coding workflow.

 

Programming a music performance instrument for a wandering mind and clumsy hands

Jean-Philippe Drecourt

Keywords: digital musical instrument, second-order control, stochastic triggering, live improvisation, performer-centred design

Abstract

The Chaos Conductor is a live performance instrument for concrete sounds, designed around human fuzziness. Most humans have average dexterity and a cognitive need for novelty that limit their ability to perform conventional musical instruments. The Chaos Conductor addresses both through second-order parameter control. Stochastic triggering, MIDI virtualisation, and a three-stage workflow (design, practice, performance) replace the precision-and-repetition model of classical instrument practice with accessible bounded exploration. This paper presents the design rationale, the control architecture, and reflections from solo and group performance. It argues that shifting the performer's agency to shaping conditions for sonic emergence can broaden the accessibility of performance interfaces.

 

Humanoid Musical Robots as Experimental Interfaces for Music-Evoked Emotion

Vincent Cheung and Jia-Yeu Lin

Keywords: emotions, robotics, cognition

Abstract

Advances in technology have led to increasingly sophisticated musical humanoid robots. However, their use has largely been limited to performance and related research in human–robot interaction. In this position paper, we propose a novel perspective: musical humanoid robots as experimental interfaces for investigating music-evoked emotions. We argue that current research is constrained by paradigms relying on pre-recorded auditory stimuli, which fail to capture the multimodal, embodied, and interactive nature of real-world musical experience. Building on existing theories of music cognition and emotion, we identify mechanisms that require controlled manipulation of both acoustic and non-acoustic variables. We show that humanoid robots are well-suited as they enable parametric control of performance variables, reproducibility across trials, and the decoupling and recombination of auditory, visual, and interactive components. We illustrate the technical feasibility of this perspective through a case study of the WAseda Saxophonist Robot 5 (WAS-5), demonstrating reproducible control of acoustic and interaction variables that are prerequisites for future music-emotion experiments. Our work positions musical humanoid robots as a methodological platform that enables future controlled investigations of music-evoked emotions.

 

Effect-Time Cue Scheduling for Distributed Gaze-Based Musical Performance

Doohyun Jung

Keywords: gaze-based musical interaction, distributed musical performance, telematic music, effect-time cue scheduling, interactive performance systems, multimodal musical interfaces

Abstract

We present a distributed gaze-based musical performance system designed as a prototype for cross-distance gaze-driven co-performance. Rather than executing cues on message arrival, the system coordinates performers with pre-cues and scheduled effect times so that gaze-driven sound generation and shared musical moments can be maintained across distance. Three real-world conductor--performer sessions were analyzed through the same cloud-routed architecture: a co-located Korea--Korea baseline with an experienced participant, and two first-time remote sessions in India and South Africa. Under an effect-time design, cue-error p95 is expected to remain stable as long as cues arrive before their scheduled effect times. The observed data confirm this: cue-error p95 stayed near 10 ms even as round-trip time (RTT) ranged from 164 ms to 357 ms. The harder problems appeared elsewhere. Gaze acquisition depended heavily on calibration familiarity, and interaction reliability broke down under reconnect conditions, producing missing-fire and duplicate-fire failures in the first-time remote sessions. Taken together, the sessions suggest that distributed gaze performance should be evaluated not only as a latency problem, but also as a problem of onboarding, calibration familiarity, and session-state management.

 

Challenges for Eye-Tracking In Naturalistic Music-Reading Setups: A Pilot Study

Charlotte Schneider

Keywords: Music-reading, Music-cognition, Music-psychology, eye-tracking, Notation

Abstract

Music-reading research rarely employs naturalistic, corpus-based stimuli that are standart in language-reading research. To better understand how musicians interact with scores, a corpus-derived stimulus database for sight-singing experiments was developed, designed to balance experimental control with ecological validity by providing structured, idiom-specific materials that better reflect the musical knowledge trained performers bring to score reading. Preliminary results from an exploratory pilot eye- tracking study suggest that singers tended to remain visually close to the current score position rather than reading ahead, in contrast to common findings from language-reading research. Correlation analyses revealed no consistent associations between gaze measures and choir experience or professional singing experience. The only significant association was a negative correlation between formal singing training and number of fixations, although this effect appeared to be influenced by two individual participants and should therefore be interpreted cautiously. A further non-significant trend suggested that greater formal singing training might be associated with longer fixation durations. Overall, the findings indicate that established measures such as fixation duration, fixation count, and eye-voice span may not transfer straightforwardly to musical settings. The study thus provides both a methodological resource and an exploratory foundation for future research.

 

Inclusive Performative Ecosystems: Ai, Bci, And Biofeedback for Accessible Musical Interaction

Sara Montagni

Keywords: BCI, EEG, machine learning, face tracking, inclusive musical interfaces

Abstract

This paper presents a hybrid interaction system engineered for inclusive musical performance, investigating how physiological signals and computer vision can enable real-time, hands-free control of musical structures. The architecture integrates two complementary modalities: EEG signals acquired via an OpenBCI headset and facial gesture tracking using MediaPipe FaceMesh. Raw EEG data are processed through a custom Python bridge that extracts spectral and statistical features to train machine learning models within Wekinator for classification and regression, with resulting parameters mapped via Open Sound Control (OSC) to generative processes in Ableton Live. In parallel, a webcam-based interface detects head movements to navigate a circular graphical UI, allowing discrete selection of root notes, scales, arpeggiator patterns, and tempo divisions through dwell-time interaction and specific eye-blink gestures. By combining continuous neural control with discrete facial selection, the system enhances interaction stability and expressive capabilities, offering a novel approach to neuroadaptive instruments, particularly for users with limited motor abilities. The paper delineates the system architecture, interaction design, and signal-processing strategies. Preliminary single-participant tests demonstrate system feasibility, confirming real-time OSC communication latency below 50 ms and validating the viability of this multi-modal approach.