Session 3 — Paper Presentations

Strategies for Musical Entanglement - Shaping Co-Creative Interplay through Harmonic and Melodic Visualization of Intent in Somax2

Marco Fiorini and Miles Okazaki

Keywords: Interplay, Human-Machine Improvisation, Co-Creativity, Interactive Systems, Agency

Abstract

This paper explores how visual feedback can actively shape real-time co-creative interaction between human improvisers and music AI systems. While corpus-based sys- tems such as Somax2 offer high reactivity and sophisticated responsiveness, the lack of transparency regarding the machine’s internal state and choices can limit the human response choices to a passive reactive dynamic. To address this issue, we have developed two new user interface (UI) modules that allow the performer to “sight-read the machine”: a chord-symbol-based module for harmonic navigation (used with Thelonious Monk’s Introspection as the corpus of source) and a staff-based reading mod- ule for melodic anticipation (using the Max/MSP library bach). We discuss case studies from two live performances (ICMC2025 in Boston and the Improtech Festi- val at IRCAM in Paris), demonstrating how the visualization of harmonic and melodic intent transforms the human- machine relationship into a mechanics of interplay that unfolds in real time, enabling new strategies for counterpoint, synchronization, and shared action.

 

Interactive Melody Generation and Robotic Saxophone Performance: The RoboSax Melody Slot Machine

Masatoshi Hamanaka and Gou Koutaki

Keywords: Robosax, Melody Slot Machine, Generative Theory of Tonal Music (GTTM), Time-span Tree, Melody Morphing, servo motor, MIDI

Abstract

This paper presents RoboSax Melody Slot Machine, a system that combines Melody Slot Machine, a dial-based interface for exploring melodically coherent variations, with RoboSax, a robotic saxophone that automates fingering with servo motors. Selected melodies are transmitted as MIDI to RoboSax, while the performer supplies breath and articulation. Because the melodic variations are generated by a GTTM-based morphing method, users can explore related phrases without losing structural coherence. The system therefore supports both accessible melody creation and embodied wind-instrument performance within a single interactive loop.

 

Curve2Melody: Sketch-Guided Melody Generation

Cristina Jorda i Custal, Tejaswinee Kelkar and Stefano Fasciani

Keywords: Sketch-based melody generation, Melodic contour, Spline curves, Variational autoencoder

Abstract

In this paper, we present Curve2Melody, a multimodal AI generative system that enables the creation of new melodies based on a melodic context and a hand-drawn sketch. Our work builds upon sketch-based melodic generative models, specifically the inpainting system “Draw and Listen”, optimizing its architecture by utilizing a simplified input data structure and conditioning the model on a single context measure. We introduce a novel methodology for representing melodic contours using spline curves, encoding them through their derivatives to extract relevant features. The model employs a variational autoencoder with two encoders and three decoders, predicting pitch range, rhythm, and key from the context measure, while pitch is generated from the input sketch. To evaluate the performance, we design three experiments using the same pitch-, rhythm-, and context-matching metrics as in the baseline. Results show that our approach improves contour and rhythm matching while achieving tonal consistency, outperforming the baseline model on the same dataset using a smaller model with more than thirty times fewer trainable parameters.

 

Toward Instruction-Guided Interactive Low-Latency Generation for Acoustic Music

Yinghao Ma and Hannes Gamper

Keywords: Interactive Music Generation, Low-Latency Music Generation, Flow Matching Models

Abstract

Real-time, controllable music generation is important for interactive applications, yet many state-of-the-art systems remain oriented toward offline synthesis, with limited support for responsive interaction. In this work, we present \textit{InteractiveMusicFlow} (IMF), a compact generative framework based on latent flow matching for low-latency music generation. IMF extends a text-to-audio flow matching backbone with audio-context conditioning and multi-task training, enabling a unified model that supports text-to-music generation, audio continuation, and text-instructed editing. Across MusicCaps and MusDB benchmarks, IMF demonstrates a consistent pattern: it achieves substantially lower inference latency than diffusion-based baselines, while producing outputs that remain broadly aligned with the target data distribution under standard evaluation metrics. At the same time, IMF does not yet match stronger offline models in perceptual fidelity or fine-grained control, particularly in tasks requiring precise source preservation. We view these results as evidence that flow matching provides a practical foundation for interactive music systems, where responsiveness and flexible control are essential.

 

 

Style-specific accompaniment generation via constrained discrete optimization

Stephanie Lew, Wee Kheng Leow and Kat R. Agres

Keywords: Style-Specific Accompaniment Generation, Constraint Satisfaction and Optimization, Discrete Optimization, Symbolic Music Generation

Abstract

Style-specific accompaniment generation aims to produce accompaniment for a given melody in a target style. Despite the success of recent methods including deep neural networks (DNNs) and hidden Markov models, they have several shortcomings. To adapt to different styles, they require separate training on style-specific datasets, increasing total training time. Training on mixed-style datasets can result in diluted, inconsistent outputs. Moreover, they cannot handle section closure and global music structure effectively. The proposed method, FlexFlow, overcomes the shortcomings of existing methods by formulating style-specific accompaniment generation as a discrete optimization problem. It defines style as the pitch progression in user-provided reference phrases, enabling adaptation to any target style expressed in those references. It can generate accompaniment using as little as one reference phrase. It adopts a training-free approach that directly adjusts individual pitches according to chord constraints while minimizing dissonance and preserving pitch direction and pitch difference of reference phrases. It can handle full-length melodies with multiple sections. It can maintain section consistency, produce contrast between sections and handle phrase repetitions. Performance tests and a user study indicate that FlexFlow's accompaniments are more harmonious than those of comparable DNNs and its quality is closer to human-composed accompaniments compared to that of DNNs.

Beat Tracking for Real-Time Human-Computer Live Improvisation

Joris Monnet, Michael Wollny and Martin Rohrmeier

Keywords: Beat tracking, Beat induction, Real-time music analysis, MIDI, Interactive music systems, Tempo estimation

Abstract

Beat tracking is a fundamental task in computational musicology, with applications ranging from music analysis to interactive performance systems. While many existing approaches achieve high accuracy in offline settings, they typically rely on access to the complete musical signal and are not designed for real-time use. In live performance and human–computer interaction scenarios, beat tracking must operate under strict latency constraints while remaining robust to expressive timing, tempo fluctuations, and structural variability. In this paper, we evaluate a set of beat tracking methods within a real-time framework, with a particular focus on MIDI-based input. We compare several classes of approaches, including inter-onset interval models, onset-driven methods, and frequency-domain techniques, within a modular streaming pipeline. Experiments are conducted on classical and jazz piano datasets, including augmented data to simulate tempo variation. Our results show that while most methods provide reliable tempo estimation, they struggle to maintain precise and stable beat localization under real-time constraints. This discrepancy highlights a limitation of current evaluation paradigms, which may overestimate performance by not fully accounting for temporal alignment in interactive settings. These findings underline the need for beat tracking approaches and evaluation frameworks specifically designed for real-time musical interaction.

 

A critical data case study in human-machine co-creation of electro-acoustic music

Arthur Flexer

Keywords: critical data, interview, electro-acoustic music, ethics, small data

Abstract

We present a critical data case study about human-machine interplay in the context of an electro-acoustic quartet performance. Based on a self-developed questionnaire, we conducted expert interviews with the four performing artists who also built and programmed the four instruments being played. The data being utilized by the instruments are a mix of self recorded and external sounds, with the latter often being of unclear provenance. Our results elucidate handling and documentation of data usage and show that encounters of legal and ethical concerns are often dominated by conscious artistic decisions, including data obfuscation or challenges to copyright conventions.

Neuralyzer: An AI Chatbot in Partiels

Pierre Guillot

Keywords: large language models, model context protocol, sound analysis, human-computer interaction, local inference, Partiels, conversational agent, audio software

Abstract

This paper introduces Neuralyzer, an extension to the sound analysis software Partiels developed at IRCAM. Implemented as a conversational agent coupled with a server based on the Model Context Protocol (MCP), Neuralyzer assists users in the analysis of digital audio files by facilitating parameter configuration and automating repetitive tasks. We investigate how large language models (LLMs) can introduce an intelligent assistance layer within a sound analysis environment. Particular attention is given to local inference, motivated by constraints related to accessibility, data privacy, and computational efficiency. The text retraces the technical evolution of the project, from an initial phase based on remote APIs to the integration of locally executed models and adoption of MCP, enabling structured and controlled interaction between the model and the software's internal features without compromising application stability. The different architectures tested are critically analyzed, and initial technical and functional results are presented. Overall, this work suggests that a controlled integration of language models can make sound exploration more accessible in research, pedagogy, and musical creation contexts. These features also make it possible to use them in place of a scripting language to perform automation.

 

 

 

Conversing for Coexistence

Patrik Lechner, Marco Döttlinger and Thomas Grill

Keywords: audio installation, ecological niche, speculative sociality, adaptive algorithms, reinforcement learning

Abstract

The proposed audio installation work explores emergent relations between distinct agents – human, technological, and environmental – within an adaptive, electroacoustic ecology. Sound becomes a medium for attuning to difference, negotiating tension, and practicing or simulating coexistence, thereby enabling an agent's potential of finding their individual niches within the system. To achieve this we developed a tool set, encapsulating expression, agency and perception of our machine agents. While the technical setup is presented in a prototypical form, we are contributing a small but versatile tool that allows for integrating reinforcement learning algorithms and multi-agent setups in common sound or media art workflows via OSC. The project foregrounds adaptive algorithms and dynamical adaptive system in both machine listening and synthesis as aesthetic strategy and proposes a performative ecology and ethical practice. We claim that the installation does not merely generate sounds; it enacts a speculative sociality – a sonic experiment in how heterogeneous beings might inhabit shared worlds without erasure but by 'Conversing for Coexistence' for making kin.

 

An Exploratory Study on the Effects of AI Song Generation on Agency, Authenticity, Creativity, and Connectedness in Musicians and Nonmusicians

Dana Swarbrick and Suvi Saarikallio

Keywords: AI Song Generation, Agency, Authenticity, Creativity, Connectedness, Wellbeing, Musicians, Nonmusicians

Abstract

Songwriting supports creative and emotional expression and social and emotional development through emotion regulation and wellbeing, self-esteem, and a stronger sense of identity. Recent advances in AI technology have provided access to music creation even for people without musical expertise, through AI song generation. We conducted an exploratory, single session study to examine these outcomes in musicians and nonmusicians. Participants were recruited during a public outreach event. There were 15 nonmusicians and 20 musicians with 13 participants having previous songwriting experience and 21 with no prior experience (1 chose not to answer). Participants were instructed to express themselves creatively and emotionally by 'composing' a song with an AI music generator, by being as descriptive in their prompts as possible. They listened to the two versions of their generated song and completed a questionnaire with items measuring liking, song quality, engagement, emotions, agency, authenticity, creativity, and connectedness using 5-point Likert scales. We explored how demographic variables of musical proficiency and attitudes towards AI affected experience. These exploratory findings suggest that nonmusicians may particularly benefit from AI song generation for emotional and creative expression, while musicians are more critical and prefer unaided composition.