Abstract

We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for cascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MusicGen can generate high-quality samples, while being conditioned on textual description or melodic features, allowing better controls over the generated output. We conduct extensive empirical evaluation, considering both automatic and human studies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark. Through ablation studies, we shed light over the importance of each of the components comprising MusicGen. Music samples can be found on the supplemental materials. Code and models are available on our repo github.com/facebookresearch/audiocraft.

Check out our a paper on Simple and Controllable Music Generation for more information.

Samples: comparison to prior work

In the following, we compare MusicGen 3.3B to a number of prior work detailed in the paper:
MusicLM,
using the public AI Test Kitchen demo,
Riffusion using the provided pre-trained modes,
and Mousai, which we retrained
on the same dataset as our proposed MusicGen model.

desc MusicGen MusicLM Riffusion Musai
Pop dance track with catchy melodies, tropical percussion, and upbeat rhythms, perfect for the beach
A grand orchestral arrangement with thunderous percussion, epic brass fanfares, and soaring strings, creating a cinematic atmosphere fit for a heroic battle.
classic reggae track with an electronic guitar solo
earthy tones, environmentally conscious, ukulele-infused, harmonic, breezy, easygoing, organic instrumentation, gentle grooves
lofi slow bpm electro chill with organic samples
drum and bass beat with intense percussions
A dynamic blend of hip-hop and orchestral elements, with sweeping strings and brass, evoking the vibrant energy of the city.
violins and synths that inspire awe at the finiteness of life and the universe
80s electronic track with melodic synthesizers, catchy beat and groovy bass
reggaeton track, with a booming 808 kick, synth melodies layered with Latin percussion elements, uplifting and energizing
a piano and cello duet playing a sad chambers music
smooth jazz, with a saxophone solo, piano chords, and snare full drums
a light and cheerly EDM track, with syncopated drums, aery pads, and strong emotions
a punchy double-bass and a distorted guitar riff
acoustic folk song to play during roadtrips: guitar flute choirs
rock with saturated guitars, a heavy bass line and crazy drum break and fills.

Melody conditioning

We now experiment with our novel chroma-based melody conditioning.
We condition on famous melodies from classical music along with new text description to provide
interpretations in any genre or style. We use our MusicGen 1.5B with melody and text conditioning.

Source of Melody desc MusicGen
90s rock song with electric guitar and heavy drums
An 80s driving pop song with heavy drums and synth pads in the background
An energetic hip-hop music piece, with synth sounds and strong bass. There is a rhythmic hi-hat patten in the drums.
90s rock song with electric guitar and heavy drums
An 80s driving pop song with heavy drums and synth pads in the background
An energetic hip-hop music piece, with synth sounds and strong bass. There is a rhythmic hi-hat patten in the drums.
90s rock song with electric guitar and heavy drums
An 80s driving pop song with heavy drums and synth pads in the background
An energetic hip-hop music piece, with synth sounds and strong bass. There is a rhythmic hi-hat patten in the drums.

Long generation

It is possible to generate longer sequence using a fixed 30 seconds windows. We then slide the window by chunks of 10 seconds, keeping the last 20 seconds
that were generated as context. We use MusicGen 3.3B.

desc MusicGen
lofi slow bpm electro chill with organic samples
a light and cheerly EDM track, with syncopated drums, aery pads, and strong emotions
A grand orchestral arrangement with thunderous percussion, epic brass fanfares, and soaring strings, creating a cinematic atmosphere fit for a heroic battle.

Read More