Silent Metronome

Every example is one ten-second window from the Slakh2100 test split. The model hears only the input mix and generates the missing stem strictly causally, committing to each second of output before hearing the chunk of audio it accompanies. Pick a target class and an example. The input mix sits on top, and below it each system’s stem is playable on its own and blended back into the input mix, so the baseline, SiMe+Aux, and the ground truth can be compared directly. Listen for how the baseline loses the rhythm while SiMe stays locked to it.

We build on the streaming accompaniment framework of Wu et al. (SMG), a prefix-decoder transformer over causal DAC tokens at 50 Hz that generates the accompaniment in one-second chunks with zero future visibility, so it never hears the input it is accompanying before committing to output. On top of this strictly causal baseline we add two components.

Metrical phase conditioning. Every frame carries a four-channel phase encoding, the sine and cosine of the phase within the beat and within the bar, extended with local and window-level log-tempo, tempo and meter change flags, and learned time-signature embeddings. A MLP projects these 24 channels to the model width, and the signal enters every decoder block through adaptive layer normalization in the AdaLN-Zero style. Because the signal is external to the audio pathway, it cannot drift with the model’s own mistakes. The conditioning path is precomputable one chunk ahead and adds roughly 5% to per-chunk generation time on an A100.

In all experiments the phase, tempo, and time-signature information comes from ground-truth annotations extracted in preprocessing. This is a deliberate oracle setting, it isolates the effect of an undrifting external reference from the separate question of estimating one live.

Latent-shaping auxiliary heads. Three shallow heads train jointly on the decoder’s hidden state and are dropped at inference. Two predict the target stem’s current multi-pitch activations and constant-Q spectrum. The third is a target-token future head, new to music generation, which predicts the model’s own output tokens 0.2, 0.5, and 0.8 seconds ahead, pressuring the representation to plan ahead even without seeing ahead.

Evaluation on 1,024 held-out Slakh2100 windows, continued chunk-by-chunk at the one-second chunk size. Every reported value is the mean over five sampling seeds with the sample standard deviation as the ± term. All models except the last row are strictly causal, the non-causal reference is granted one second of look-ahead into the input mix and serves as an upper reference, not as a deployable model. Beat-F scores agreement between the pulse of the generated accompaniment and of the input mix, CoCoLa scores harmonic and percussive coherence against the input, and FAD (VGGish) measures distributional audio quality. Best model value per column in bold, the held-out ground-truth stem scored under the identical protocol serves as the attainable ceiling.

The ablations isolate the mechanism. All three auxiliary heads alone, including the future-token head, leave Beat-F at the baseline, and the conditioning stripped of its phase channels reaches only 0.141, so the metrical phase is the core mechanism and the auxiliary heads refine what it provides.

Targets are grouped into drums (97 windows), bass (144), and harmonic (783, all pitched non-bass classes). The ground-truth row is the held-out stem scored under the identical protocol and acts as the attainable ceiling per class.

Beat-F ↑

CoCoLa ↑

FAD (VGGish) ↓

The headline model reaches about 94%, 79%, and 71% of the attainable alignment ceiling on drums, bass, and harmonic targets respectively, up from the baseline's 23%, 17%, and 25%. Expressed as a single number, the SiMe + Aux additions close about 92%, 75%, and 61% of the gap between the baseline and the ceiling per class. The advantage over the non-causal reference holds in every class, so there is no class where look-ahead outperforms the metrical signal. Additionally, there appears to be a tradeoff between the contrastive and distributional metrics when the future token auxiliary head is introduced on top of metrical conditioning as well as pitch and spectral prediction heads, present across all classes.

The standard protocol matches the ten-second training horizon, so it cannot show what happens when generation runs past it. We therefore generate twenty-second continuations with a sliding window at the same one-second chunk size and score the two ten-second halves separately, on the same 1,024 test windows and five sampling seeds, with loudness normalization applied once per twenty-second file before slicing. Because the halves are cut from these jointly normalized files, the FAD reference values are pooled across both halves, and the continuations are re-sampled, the first-half values are close to but not identical with the standard protocol above, and the two tables should not be compared cell by cell. The two-half drift protocol is adopted from LiveBand (arXiv:2606.03803). We do not compare our scores with LiveBand’s published numbers because their model is trained on different data under unreported conditions, so we adopt only their evaluation protocol.

Δ rel. is the Beat-F change from the first to the second half relative to the first half. Δ vs. floor subtracts the ground-truth drift from each model’s Δ rel. For CoCoLa and FAD, the second half is reported as the absolute change from the first.

The ground truth has no FAD entry under the standard protocol because the reference set there is the ground truth itself, making the value zero by construction. Against the pooled background it earns a small nonzero FAD instead, the measurement floor of the protocol.

The second half is intrinsically harder material, the ground truth itself gives up 4% of its score, so that −4% is the content floor every model must be read against, not zero. The conditioned models mostly track the floor, SiMe improves on it by 1%, the headline model trails it by 1%, and the phase-stripped conditioning matches it exactly. The one exception is the aux heads without the future-token head, which drifts 3% below the floor, worse than SiMe alone, and it is specifically the future-token head that pulls the headline model back toward the floor. The unconditioned models lose 14 to 16% and the non-causal reference loses 13% despite its full second of look-ahead, three to four times the floor. The non-causal reference is also the only model whose CoCoLa declines across the horizon (−0.57), every conditioned model holds or gains, although the ground truth also drops slightly. The headline model consistently reaches about 75% of the Beat-F ground-truth ceiling in both halves, up from the baseline’s 23% and 21%. The external signal carries alignment past the training horizon that a second of look-ahead does not.

The pattern that stands out across the results is where the method helps. The per-instrument split shows the advantage is largest on percussive targets and smallest on instruments whose coherence lives in spectral harmony, with the achievable alignment falling from 94% on drums to 71% on harmonic targets, and with the non-causal reference retaking the CoCoLa lead on harmonic targets only. Put as one number per class, the SiMe + Aux additions close about 92%, 75%, and 61% of the baseline-to-ceiling beat alignment gap on drums, bass, and harmonic targets. Informal listening agrees with this ordering, the gains are unmistakable on drums and subtler on harmonic material.

Part of the explanation is built into the method. The SiMe conditioning, which contributes the largest gains across all metrics, carries no musical or harmonic content at all, it provides purely temporal grounding. A drum part is very nearly a direct expression of that temporal structure, so an unambiguous metrical reference addresses most of what the target demands. A harmonic part must also agree with the input in pitch and harmony, which the phase signal carries no information about, so a larger share of the task is left to the audio pathway that the conditioning does not touch.

These interpretations are supported by informal listening rather than a formal evaluation, and a listening study with participants is the natural next step for the perceptual claims. The rhythmic reference itself is an oracle in the current experiments. In practice it can be supplied by a click track or DAW clock, exactly as it is for human musicians, and replacing the annotations with a causal online beat tracker is the natural step toward a fully self-contained live system. Conditioning signals that carry harmonic information alongside the temporal ones are an equally significant next step for the method.