VAE Drum Machine

A small drum sound designer you can steer. It learns an embedding space of drum one shots and lets you sample new sounds from it and morph between two existing sounds to hear everything in between. Group project of three for the AI and Art course at Leiden University.

Links · 💻 Code · 🤗 Model and data · 📄 Report

Listen

We first present the model reconstructing a drum sound.

Then two interpolations, each with the two reconstructed drums on the left and right, and in the middle the brand new sound decoded from the point halfway between them.

How it works

Rather than work on raw audio, every drum sample is first turned into discrete tokens by Meta’s EnCodec audio codec, and the model lives entirely in that token space. An encoder squeezes a sound down to a small latent vector, a decoder rebuilds the codec tokens from it, and a separate small head predicts how long the output should be so the model learns how to correctly handle audio of differing lengths.

Honest notes

It works as a general drum generator, but it is not perfect. Reconstructions are recognisable rather than exact, and the latent space is not always smooth to slide through. A couple of things surprised us. Making the model bigger barely helped, and simplifying the training set down to just kicks, snares and hats did not help either.