ImprovMusicGen

This is the project closest to what I actually want to build, an agent that plays along with you. Given a backing track with one instrument missing, it improvises that missing part a beat at a time. Solo project for the Audio Processing and Indexing course at Leiden University.

Links · 📄 Report

Listen

The model hears a backing track with one instrument taken out of a precomposed mix, and improvises that missing part. The top row contains only the backing track, without any generated music. There is an audible dropout of the target instrument at the halfway point. The second row contains a mix of the backing track and the generated accompaniment, which begins playing after the halfway point. It holds together best on drums, gets shakier on bass, and on lead it tends to drift into silence, which is the honest limit of where it is right now.

How it works

I started from MusicGen, Meta’s pretrained music generation model, and fine-tuned a LoRA adapter on top of it. I built the training data using the open-source Slakh2100 dataset, containing MIDI-generated multi-track songs. One instrument is held out as the target and the rest is fed in as context, with some pitch shifting and deliberate corruption so the model learns to recover from its own mistakes. It generates a single beat at a time from twenty beats of context, a deliberate hard ceiling to limit inference time for real-time feasibility.

Honest notes

It is a proof of concept, and the limits are real. Most of the shakiness on bass and lead comes down to the base model wanting to generate a full mix rather than one instrument, as well as to the ambiguity of what a “lead” part actually is in the data. The direction feels right even though the results are not there yet.