Playing Along in Time and Key

Silent Metronome showed that giving a streaming accompaniment model an external clock more than triples its beat alignment. That clock came from labels precomputed for the dataset though, so it was still open whether a live system could work it out by itself. This demo closes that loop and adds harmony, with a chord chart as a second conditioning signal. To the best of our knowledge it is the first audio-domain accompaniment system that uses an explicitly tracked causal clock.

Kevin Bretz, Derya Soydaner, Aske Plaat · Leiden University

Related · 🥁 Silent Metronome · 📄 SiMe paper

Demo

The video shows the system playing along with held-out songs in real time. The first session, almost two minutes on drums, introduces the clock, the chord chart and the candidate selection one by one before the drums come in. After that it plays guitar, piano and bass, each first on its own and then with the band. Captions are available through the player.

How it works

Listening. A beat predictor and a chord predictor, both small causal transformers, read the tokens of the causal audio codec as the music comes in. They listen for two seconds before the generator joins, much like the count-in of a human band.

Keeping time. The raw beat predictions jitter too much to use directly, which would bring back the very drift we want to remove. A dynamic Bayesian network turns them into one steady tempo and phase that glides through corrections instead of jumping. It has no learned parameters and runs at under one percent of real time on the CPU.

Playing. The generator is the streaming model of Wu et al. as used in SiMe, scaled to 301M parameters. It receives the clock, a chord per half bar and a short reference clip of the sound it should aim for. Every half second it proposes eight continuations, and a small learned scorer predicts which one will fit best and commits it. The scorer never hears the audio under the chunk it scores.

In real time. Each chunk is generated half a second before it plays, so it never hears the audio it accompanies. The whole loop takes 340 ms per half-second chunk on a single RTX 2080 Ti.

Results

We follow the evaluation protocol of Wu et al. on 1,024 ten-second windows of the Slakh2100 test split, with one stem held out as the target. Beat-F compares the beats of the accompaniment with those of the input mix, CoCoLa measures harmonic and percussive coherence with the input, and FAD measures audio quality.

With predicted labels and selection the system keeps 93% of the oracle beat alignment and goes past the beat alignment of the held-out ground-truth stem. On the predicted labels, selection adds 0.32 Beat-F and improves coherence and audio quality at the same time. Retraining the generator without the clock removes most of its beat alignment, while without the chart it loses harmonic coherence and keeps its beat alignment, so the clock carries the rhythm and the chart the harmony.

These numbers have to be read next to the audio. By ear the accompaniment follows the beat and the chords but still sounds clearly worse than the real stem, with drum patterns that change from bar to bar and harmonic parts that follow the chart without phrasing. None of the metrics capture that, which is why the video matters more here than the table.

What comes next

The output often plays at regular intervals locked to the predicted beat, which is most noticeable on piano, and I want to find out why. I also want to give the harmonic side more to work with than chords, and to improve the texture reference.

Poster

The poster walks through the live system in five steps, from listening to committing a chunk.

</a>

🖼️ Open the poster

Extended abstract

The camera-ready abstract has the full system description, the results table and the diagram of the live pipeline.

📄 Open the PDF