Procedural sound generation in computer games is on the rise, and making spicy music with synthesizers should be no exception! In this article, I present an approach for implementing a native synthesizer in the Unity game engine with modular signal flow and audio-rate modulation for the true synthesizer madness that modular synth builders know and love. We will use the multimedia-oriented visual programming environment Pure Data to design synth modules and the online Heavy compiler to transform those modules into native code with C# interfaces that we can implement in Unity.

Here is an example of what we will be making. A modular wavetable oscillator, here performed with seven concurrent oscillators in VR. As you can see, lots of fun can be had, even with only one type of synth module:

For years, I have been an avid user of Max/MSP and its open source equivalent Pure Data (PD) - They are excellent environments to intuitively prototype ideas for audio and multimedia applications. However, when one wants to implement such ideas in shippable products, a longer, more cumbersome process awaits of integrating each component in native code. While it's possible to generate C code with Gen, the recently introduced low level patcher in Max/MSP, most other functionality is not so easily integrated in other projects. Needless to say, I was a very happy man when I discovered that Heavy allowed me to convert PD patches to optimized native code with relative ease. With the time saved in implementation, it allows me to spend more time designing interesting audio modules and interactions. In this article, I will assume familiarity with PD and a fundamental understanding of synthesis on the part of the reader, so we can focus on the elements of implementation. If you're new to modular synthesizers, Modular Landing provides good video introduction.

We begin by creating a synth module in PD.

For this article, I have created a simple wavetable oscillator and prepared it for interfacing with Heavy. This patch and scripts relating to this article can be accessed via GitHub. It plays back a table called waveform using tabosc4~ and has controls for coarse tuning, fine tuning, LFO mode, amplitude and two CV inputs (which is short for Control Voltage, referencing the lingo of modular synth design) for audio-rate frequency and amplitude modulation.

Exposing parameters to Heavy, so we can control them via script later, is done by formatting a receive (r) object with the name of a parameter and the attribute @hv_param followed by minimum, maximum and default values. The two audio-rate CV inputs each use an audio channel from adc~. More information on exposing parameters and audio input/output can be found in the online documentation.

Now, we can upload the patch to Heavy and compile it to code that Unity likes.

When uploading a patch, Heavy provides platform-specific audio libraries and a C# scripting interface for Unity which provides useful methods for passing audio buffers and setting parameters. If the script is dragged onto a GameObject, it will create an AudioSource and begin processing when the scene is started. In order to enable this code for modular signal flow across GameObjects, we must make a few changes.

In realizing the modular signal flow, we can conceptualize an entry point as a speaker or AudioOutput. This is the final destination of the audio that delivers a playable clip to an AudioSource in OnAudioFilterRead. An AudioOutput knows only an input in the form of a SynthModule, from which it can get audio. When Unity asks an AudioOutput to provide audio to be played by the AudioSource, we can imagine a process of calling for sound from the front of the signal chain to the back - The AudioOutput will ask a connected SynthModule (for the sake of example, we could imagine this was a filter) to provide its sound, which will, in turn, ask any connected modules (such as an oscillator) to provide theirs before returning sound to the AudioOutput.

The AudioOutput class is very simple and looks like this:

[RequireComponent( typeof (AudioSource))] public class AudioOutput : MonoBehaviour { public SynthModule signalIn; private void OnAudioFilterRead ( float [] data, int channels ) { if (signalIn != null ) { signalIn.ProcessBuffer(data, channels); } } }

The SynthModule is an abstract base class that provides the guarantee of a module being able to process an audio buffer as well as providing an entry to setting parameters by index, without further knowledge of a connected module. This provides the base functionality with which SynthModules can interact with each other:

public abstract class SynthModule : MonoBehaviour { public abstract void ProcessBuffer ( float [] buffer, int numChannels ) ; public abstract void SetParameter ( int paramIndex, float value ) ; public abstract float GetParameter ( int paramIndex ) ; }

Now we can implement the oscillator. We make a new class inherited from SynthModule which takes care of loading the Heavy library and implements patch-specific functionality. In my case, the library is called Hv_osv_wavetable_single_AudioLib - this will differ if you upload your own patch to Heavy. In the Start method, it makes a reference to this library and fills the waveform table with an AudioClip.

Then, we override the ProcessBuffer method: If CV inputs are connected, we process them and store their audio in a temporary buffer. Then we merge the incoming audio buffer with the two additional channels of CV input and pass that to the Heavy oscillator. In Unity audio buffers, channels are interleaved so we have to do some loop nesting to merge them properly. After the merged buffer has been processed, we extract the two audio channels and return them. This process is not strictly necessary for the oscillator as it does not have audio inputs, but is nice to have figured out for future modules that take both audio and CV input, such as a modulate-able filter. Finally, we override the SetParameter and GetParameter methods to interface directly with the exposed parameters in the Heavy oscillator.

[RequireComponent( typeof (Hv_osc_wavetable_single_AudioLib))] public class OscWtSource : SynthModule { Hv_osc_wavetable_single_AudioLib osc; public AudioClip defaultWaveTable; public SynthModule cvFreqIn; public SynthModule cvAmpIn; float [] cvFreqBuffer = new float [ 2048 ]; float [] cvAmpBuffer = new float [ 2048 ]; float [] mergedBuffer = new float [ 2048 ]; int totalChannels = 4 ; void Start () { osc = GetComponent<Hv_osc_wavetable_single_AudioLib>(); osc.FillTableWithMonoAudioClip( "waveform" , defaultWaveTable); } public override void ProcessBuffer ( float [] buffer, int numChannels ) { ﻿ if (cvFreqIn != null ) { if (cvFreqBuffer.Length != buffer.Length) { cvFreqBuffer = new float [buffer.Length]; } cvFreqIn.ProcessBuffer(cvFreqBuffer, numChannels); } if (cvAmpIn != null ) { if (cvAmpBuffer.Length != buffer.Length) { cvAmpBuffer = new float [buffer.Length]; } cvAmpIn.ProcessBuffer(cvAmpBuffer, numChannels); } totalChannels = numChannels + 2 ; if (mergedBuffer.Length != buffer.Length / numChannels * totalChannels) mergedBuffer = new float [buffer.Length / numChannels * totalChannels]; for ( int i = 0 ; i < mergedBuffer.Length / totalChannels; i++) { for ( int channel = 0 ; channel < totalChannels; channel++) { if (channel < numChannels) { mergedBuffer[i * totalChannels + channel] = 1 f; } else if ( channel < numChannels + 1 ) { if (cvFreqIn != null ) mergedBuffer[i * totalChannels + channel] = cvFreqBuffer[i / totalChannels]; else mergedBuffer[i * totalChannels + channel] = 0 f; } else if ( channel < numChannels + 2 ) { if (cvAmpIn != null ) mergedBuffer[i * totalChannels + channel] = cvAmpBuffer[i / totalChannels]; else mergedBuffer[i * totalChannels + channel] = 0 f; } } } osc.ProcessBuffer(mergedBuffer, totalChannels); for ( int i = 0 ; i < mergedBuffer.Length / totalChannels; i++) { for ( int channel = 0 ; channel < numChannels; channel++) { buffer[i * numChannels + channel] = mergedBuffer[i * totalChannels + channel]; } } } public enum Parameter { Freq_Coarse, Freq_Fine, Amp, Cv_Freq_Amt, Cv_Amp_Amt } public override void SetParameter ( int paramIndex, float value ) { switch (paramIndex) { case ( int ) Parameter.Freq_Coarse: osc.SetFloatParameter(Hv_osc_wavetable_single_AudioLib.Parameter.Freqcoarse, value ); break ; case ( int ) Parameter.Freq_Fine: osc.SetFloatParameter(Hv_osc_wavetable_single_AudioLib.Parameter.Freqfine, value ); break ; case ( int ) Parameter.Amp: osc.SetFloatParameter(Hv_osc_wavetable_single_AudioLib.Parameter.Amp, value ); break ; case ( int ) Parameter.Cv_Freq_Amt: osc.SetFloatParameter(Hv_osc_wavetable_single_AudioLib.Parameter.Cvfreqamt, value ); break ; case ( int )Parameter.Cv_Amp_Amt: osc.SetFloatParameter(Hv_osc_wavetable_single_AudioLib.Parameter.Cvampamt, value ); break ; } } public override float GetParameter ( int paramIndex ) { switch (paramIndex) { case ( int )Parameter.Freq_Coarse: return osc.GetFloatParameter(Hv_osc_wavetable_single_AudioLib.Parameter.Freqcoarse); case ( int )Parameter.Freq_Fine: return osc.GetFloatParameter(Hv_osc_wavetable_single_AudioLib.Parameter.Freqfine); case ( int )Parameter.Amp: return osc.GetFloatParameter(Hv_osc_wavetable_single_AudioLib.Parameter.Amp); case ( int )Parameter.Cv_Freq_Amt: return osc.GetFloatParameter(Hv_osc_wavetable_single_AudioLib.Parameter.Cvfreqamt); case ( int )Parameter.Cv_Amp_Amt: return osc.GetFloatParameter(Hv_osc_wavetable_single_AudioLib.Parameter.Cvampamt); default : return 0.0 f; } } }

The final step is to make a correction to the oscillator's C# interface provided by Heavy. By default, it requires an AudioSource component and processes the patch in OnAudioFilterRead. We need to remove [RequireComponent(typeof(AudioSource))] and the OnAudioFilterRead method and replace it with a ProcessBuffer method:

public void ProcessBuffer ( float [] buffer, int numChannels) { Assert.AreEqual(numChannels, _ context.GetNumOutputChannels()); _ context.Process(buffer, buffer.Length / numChannels); }

This marks the end of our C# implementation. Now we can insert our AudioOutput on a GameObject in the Unity scene, and insert the OscWtSource component onto any number of additional GameObjects. We draw audio connections between them by setting it in the inspector. If we have more than one oscillator, we can connect one oscillator's audio output to the CV input of another. With this structure set up, we can make numerous new modules and connect them in new ways for hours of fun and synthesizer madness.

I hope this article has been helpful to you! If you have questions or feedback for improvement, please leave it in the comments. I'm working on a bigger implementation of these ideas to make a fun modular synth simulator. If you'd like to have a chat, feel free to get in touch at daniel(at)danielrothmann.com!