童话 (Fairy Tale)
0:38Camera
0:00 / 0:38
StreamMUSE+
Towards more stable and interactive real-time piano accompaniment generation.
StreamMUSE+ combines BEAT-based music generation with a two-stage prompt and continuation process for more reliable accompaniment as a melody unfolds.
StreamMUSE+
How it works
Beat-wise generation, prompt-based startup, and live interaction.
StreamMUSE+ builds on StreamMUSE's synchronized playback framework. A BEAT-based model represents music in compact, synchronized beat units and generates accompaniment incrementally from the melody and accompaniment history.
The performer first plays eight beats without accompaniment. A prompt model generates candidate openings, screens out overly sparse, dense, or out-of-key candidates, and ranks the rest by pitch-distribution richness and pitch variation. The selected opening is used as context for continuation and is not played aloud.
The continuation model generates one beat at a time while catching up with the ongoing performance. Beats that miss their playback window remain as context; audible accompaniment begins once generation is ahead of the live melody. This startup process reduces errors from limited opening context, at the cost of an initial delay.
The live performance interface shown in the recorded demos provides tempo and session controls, together with a piano-roll display of the melody and generated accompaniment. Performers can follow the system's responses as they continue playing.
Recorded demos
Demo Videos
We tested StreamMUSE+ with a human performer playing melodies from Chinese and Japanese pop songs. Each round is a separate recording session, and each take is one performance of a song within that session. Not every song was tested in every round, and some songs have only one take in a round. Round 2 includes synchronized camera and screen recordings.
Lemon
0:55Camera
0:00 / 0:55
A Cruel Angel's Thesis
1:18Camera
0:00 / 1:18
告白气球 (Love Confession)
1:39Camera
0:00 / 1:39
青花瓷 (Blue and White Porcelain)
1:36Camera
0:00 / 1:36
月亮代表我的心 (The Moon Represents My Heart)
1:27Camera
0:00 / 1:27
天空之城 (Castle in the Sky)
1:38Camera
0:00 / 1:38
一路向北 (All the Way North)
1:11Camera
0:00 / 1:11
Playable assets
MIDI Examples
Ten recorded melodies, three systems, and three accompaniment samples per system.
We invited a musician to perform the melodies of ten different songs and recorded the melody part as MIDI. Each recording was then replayed through StreamMUSE+, StreamMUSE+ without the prompt model (w/o Prompt), and StreamMUSE in real-time simulation, feeding the melody to each system incrementally as a live performer would. We ran each system three times per song with different random seeds to produce the samples below.
The simulation uses the same generation and playback process as live performance. With the same melody input and system settings, these examples should therefore, in principle, reflect the accompaniment a real user would hear while playing. Different seeds produce variations in density, register, and texture.
StreamMUSE+StreamMUSE+ (w/o Prompt)StreamMUSE
Three samples per system, using seeds 0, 1 and 2. Card colors identify the system.
MelodyAccompaniment
Choose a melody to compare the three systems.
Click a piano roll to listen. All nine samples of a song share the same time and pitch scales. Download any MIDI to explore its notes.
Notes
Evaluation and limitations
Findings from controlled experiments and an initial performance pilot.
Across 40 held-out pieces with three seeds each, StreamMUSE+ produced non-empty accompaniment in all 120 trials, compared with 90 of 120 for its prompt-free variant.
On the 90 common non-empty trials, StreamMUSE+ achieved lower overall distributional distance and pitch-distribution divergence than the original StreamMUSE. It also improved all reported music metrics over the prompt-free variant on this matched subset.
Generation remains sensitive to the random seed, with variations in accompaniment density, register, and texture. Evaluation used controlled, quantized MIDI input, and the ten-song pilot involved one performer; robustness across performers and expressive playing styles remains untested.
Future work
- Broaden evaluation across performers, expressive playing styles, and free improvisation.
- Develop velocity control, more robust prompt selection, and recovery from degraded outputs.