Masked Audio Generation using a Single Non-Autoregressive Transformer

Alon Ziv1,3*, Itai Gat1, Gael Le Lan1, Tal Remez1, Felix Kreuk1, Alexandre Defossez2, Jade Copet1,
Gabriel Synnaeve1, Yossi Adi1,3

1FAIR, Meta AI

2Kyutai

3The Hebrew University of Jerusalem

[paper] [code] [bib]

Samples

Your browser does not support the video tag.

Abstract


We introduce MAGNeT, a masked generative sequence modeling method that operates directly over several streams of audio tokens. Unlike prior work, MAGNeT is comprised of a single-stage, non-autoregressive transformer. During training, we predict spans of masked tokens obtained from a masking scheduler, while during inference we gradually construct the output sequence using several decoding steps. To further enhance the quality of the generated audio, we introduce a novel rescoring method in which, we leverage an external pre-trained model to rescore and rank predictions from MAGNeT, which will be then used for later decoding steps. Lastly, we explore a hybrid version of MAGNeT, in which we fuse between autoregressive and non-autoregressive models to generate the first few seconds in an autoregressive manner while the rest of the sequence is being decoded in parallel. We demonstrate the efficiency of MAGNeT for the task of text-to-music and text-to-audio generation and conduct an extensive empirical evaluation, considering both objective metrics and human studies. The proposed approach is comparable to the evaluated baselines, while being significantly faster (x7 faster than the autoregressive baseline). Through ablation studies and analysis, we shed light on the importance of each of the components comprising MAGNeT, together with pointing to the trade-offs between autoregressive and non-autoregressive modeling, considering latency, throughput, and generation quality.

Text-to-Music

In the following, we present samples for MAGNeT MusicGen, MusicLM, using the public AI Test Kitchen demo, AudioLDM2, and Mousai, which we retrained on the same dataset as MAGNeT.

Description MAGNeT MusicGen MusicLM AudioLDM2 Mousai
Earthy tones, environmentally conscious, ukulele-infused, harmonic, breezy, easygoing, organic instrumentation, gentle grooves
80s electronic track with melodic synthesizers, catchy beat and groovy bass
Smooth jazz, with a saxophone solo, piano chords, and snare full drums
A grand orchestral arrangement with thunderous percussion, epic brass fanfares, and soaring strings, creating a cinematic atmosphere fit for a heroic battle
Rock with saturated guitars, a heavy bass line and crazy drum break and fills

Text-to-Audio

In the following, we present samples for MAGNeT AudioGen, and AudioLDM2.

Description MAGNeT AudioGen AudioLDM2
Whistling with wind blowing
A toilet flushing as music is playing and a man is singing in the distance
Pigeons are making grunting sounds and snapping beaks
Seagulls squawking as ocean waves crash while wind blows heavily into a microphone

Hybrid-MAGNeT

We present samples of Hybrid-MAGNeT where the first 5-seconds were generated using an autoregressive mode, while the rest were generated in a non-autoregressive manner.

Description Hybrid-MAGNeT
Hypnotic and bouncy, with hip hop trap elements featuring trippy synthesizer and synth drums to create a content and chill mood
Funky and confident, featuring groovy electric guitar, keyboards that create a chill, laid-back mood
Heavy, hard and driving, in the style of Pop Punk, featuring edgy electric guitar that creates a bold, rebellious mood
Contemporary Jazz Waltz featuring a fabulous guitar solo
Bright and groovy, featuring a Tropical House feel and warm synth textures that create an enthusiastic mood.

Restricted Temporal Context - Analysis

We present 10-second samples from MAGNeT trained with and without the temporal context restriction as defined in our paper.

Description MAGNeT w.o. restricted context MAGNeT
House track with pads and synths creating a tripping harmony
House track with pads and synths creating a tripping harmony
House track with pads and synths creating a tripping harmony
Funky groove with electric piano playing blue chords rhythmically
Funky groove with electric piano playing blue chords rhythmically
Funky groove with electric piano playing blue chords rhythmically

BibTex:

              @misc{ziv2024masked,
                title={Masked Audio Generation using a Single Non-Autoregressive Transformer}, 
                author={Alon Ziv and Itai Gat and Gael Le Lan and Tal Remez and Felix Kreuk and Alexandre Défossez and Jade Copet and Gabriel Synnaeve and Yossi Adi},
                year={2024},
                eprint={2401.04577},
                archivePrefix={arXiv},
                primaryClass={cs.SD}
              }