Tiny Audio Diffusion: Waveform Diffusion That Does not Require Cloud Computing | by Christopher Landschoot

The tactic of educating a mannequin to carry out this denoising course of may very well be a bit counter-intuitive from an preliminary thought. The mannequin really learns to denoise a sign by doing the precise reverse — including noise to a clear sign again and again till solely noise stays. The concept is that if the mannequin can learn to predict the noise added to a sign at every step, then it may well additionally predict the noise eliminated at every step for the reverse course of. The essential factor to make this potential is that the noise being added/eliminated must be of an outlined probabilistic distribution (sometimes Gaussian) in order that the noising/denoising steps are predictable and repeatable.

There’s way more element that goes into this course of, however this could present a sound conceptual understanding of what’s taking place beneath the hood. In case you are eager about studying extra about diffusion fashions (mathematical formulations, scheduling, latent area, and so forth.), I like to recommend studying this blog post by AssemblyAI and these papers (DDPM, Improving DDPM, DDIM, Stable Diffusion).

Understanding Audio for Machine Studying

My curiosity in diffusion stems from the potential that it has proven with generative audio. Historically, to coach ML algorithms, audio was transformed right into a spectrogram, which is principally a heatmap of sound vitality over time. This was as a result of a spectrogram illustration was just like a picture, which computer systems are distinctive at working with, and it was a major discount in knowledge dimension in comparison with a uncooked waveform.

Nonetheless, with this transformation come some tradeoffs, together with a discount of decision and a lack of section info. The section of an audio sign represents the place of a number of waveforms relative to 1 one other. This may be demonstrated within the distinction between a sine and a cosine operate. They symbolize the identical actual sign relating to amplitude, the one distinction is a 90° (π/2 radians) section shift between the 2. For a extra in-depth clarification of section, try this video by Akash Murthy.

Section is a perpetually difficult idea to understand, even for many who work in audio, however it performs a essential position in creating the timbral qualities of sound. Suffice it to say that it shouldn’t be discarded so simply. Section info may technically be represented in spectrogram type (the advanced portion of the rework), similar to magnitude. Nonetheless, the result’s noisy and visually seems random, making it difficult for a mannequin to study any helpful info from it. Due to this downside, there was latest curiosity in refraining from remodeling audio into spectrograms and slightly leaving it as a uncooked waveform to coach fashions. Whereas this brings its personal set of challenges, each the amplitude and section info are contained inside the single sign of a waveform, offering a mannequin with a extra holistic image of sound to study from.

It is a key piece of my curiosity in waveform diffusion, and it has proven promise in yielding high-quality outcomes for generative audio. Waveforms, nonetheless, are very dense indicators requiring a major quantity of knowledge to symbolize the vary of frequencies people can hear. For instance, the music trade commonplace sampling charge is 44.1kHz, which implies that 44,100 samples are required to symbolize simply 1 second of mono audio. Now double that for stereo playback. Due to this, most waveform diffusion fashions (that don’t leverage latent diffusion or different compression strategies) require excessive GPU capability (normally at the least 16GB+ VRAM) to retailer the entire info whereas being skilled.

Motivation

Many individuals shouldn’t have entry to high-powered, high-capacity GPUs, or don’t need to pay the charge to lease cloud GPUs for private initiatives. Discovering myself on this place, however nonetheless eager to discover waveform diffusion fashions, I made a decision to develop a waveform diffusion system that might run on my meager native {hardware}.

{Hardware} Setup

I used to be outfitted with an HP Spectre laptop computer from 2017 with an eighth Gen i7 processor and GeForce MX150 graphics card with 2GB VRAM — not what you’ll name a powerhouse for coaching ML fashions. My objective was to have the ability to create a mannequin that might practice and produce high-quality (44.1kHz) stereo outputs on this technique.

I leveraged Archinet’s audio-diffusion-pytorch library to construct this mannequin — thanks to Flavio Schneider for his assist working with this library that he largely constructed.

Consideration U-Web

The bottom mannequin structure consists of a U-Web with consideration blocks which is commonplace for contemporary diffusion fashions. A U-Web is a neural community that was initially developed for picture (2D) segmentation however has been tailored to audio (1D) for our makes use of with waveform diffusion. The U-Web structure will get its identify from its U-shaped design.

Similar to an autoencoder, consisting of an encoder and a decoder, a U-Web additionally accommodates skip connections at every stage of the community. These skip connections are direct connections between corresponding layers of the encoder and decoder, facilitating the switch of fine-grained particulars from the encoder to the decoder. The encoder is answerable for capturing the necessary options of the enter sign, whereas the decoder is answerable for producing the brand new audio pattern. The encoder steadily reduces the decision of the enter audio, extracting options at totally different ranges of abstraction. The decoder then takes these options and upsamples them, steadily growing the decision to generate the ultimate audio pattern.

This U-Web additionally accommodates self-attention blocks on the decrease ranges which assist preserve the temporal consistency of the output. It’s essential for the audio to be downsampled sufficiently to take care of effectivity for sampling through the diffusion course of in addition to keep away from overloading the eye blocks. The mannequin leverages V-Diffusion which is a diffusion method impressed by DDIM sampling.

To keep away from operating out of GPU VRAM, the size of the info that the bottom mannequin was to be skilled on wanted to be quick. Due to this, I made a decision to coach one-shot drum samples because of their inherently quick context lengths. After many iterations, the bottom mannequin size was decided to be 32,768 samples @ 44.1kHz in stereo, which leads to roughly 0.75 seconds. This will likely appear significantly quick, however it’s loads of time for many drum samples.

Transforms

To downsample the audio sufficient for the eye blocks, a number of pre-processing transforms had been tried. The hope was that if the audio knowledge might be downsampled with out dropping important info previous to coaching the mannequin, then the variety of nodes (neurons) and layers might be maximized with out growing the GPU reminiscence load.

The primary rework tried was a model of “patching”. Initially proposed for images, this course of was tailored to audio for our functions. The enter audio pattern is grouped by sequential time steps into chunks which might be then transposed into channels. This course of might then be reversed on the output of the U-Web to un-chunk the audio again to its full size. The un-chunking course of created aliasing points, nonetheless, leading to undesirable excessive frequency artifacts within the generated audio.

The second rework tried, proposed by Schneider, known as a “Discovered Rework” which consists of single convolutional blocks with massive kernel sizes and strides firstly and finish of the U-Web. A number of kernel sizes and strides had been tried (16, 32, 64) coupled with accompanying mannequin variations to appropriately downsample the audio. Once more, nonetheless, this resulted in aliasing points within the generated audio, although not as prevalent because the patching rework.

Due to this, I made a decision that the mannequin structure would must be adjusted to accommodate the uncooked audio with no pre-processing transforms to supply adequate high quality outputs.

This required extending the variety of layers inside the U-Web to keep away from downsampling too shortly and dropping necessary options alongside the best way. After a number of iterations, the very best structure resulted in downsampling by solely 2 at every layer. Whereas this required a discount within the variety of nodes per layer, it finally produced the very best outcomes. Detailed details about the precise variety of U-Web ranges, layers, nodes, consideration options, and so forth. could be discovered within the configuration file within the tiny-audio-diffusion repository on GitHub.

Pre-Educated Fashions

I skilled 4 separate unconditional fashions to supply kicks, snare drums, hi-hats, and percussion (all drum sounds). The datasets used for coaching had been small free one-shot samples that I had collected for my music manufacturing workflows (all open-source). Bigger, extra assorted datasets would enhance the standard and variety of every mannequin’s generated outputs. The fashions had been skilled for a varied variety of steps and epochs relying on the scale of every dataset.

Pre-trained fashions can be found for obtain on Hugging Face. See the coaching progress and output samples logged at Weights & Biases.

Outcomes

General, the standard of the output is sort of excessive regardless of the diminished dimension of the fashions. Nonetheless, there’s nonetheless some slight excessive frequency “hiss” remaining, which is probably going because of the restricted dimension of the mannequin. This may be seen within the small quantity of noise remaining within the waveforms beneath. Most samples generated are crisp, sustaining transients and broadband timbral traits. Generally the fashions add additional noise towards the top of the pattern, and that is doubtless a value of the restrict of layers and nodes of the mannequin.

Hearken to some output samples from the fashions here. Instance outputs from every mannequin are proven beneath.