AudioUnitRender produces empty audio buffers

Question

I'm trying to mix audio from a couple of audio sources using Audio Units. The structure is quite simple, I have two inputs, a mixer audio unit and a generic output. The problem is that the buffers from the output AudioUnit are only filled with zeros.

• Inputs are provided through AURenderCallbackStruct directly to the mixer.
• Mixer subtype is kAudioUnitSubType_MultiChannelMixer.
• Output subtype is kAudioUnitSubType_GenericOutput to pull the mixed audio manually without playing it. Output is connected through AudioUnitConnection to avoid deprecated AUGraph. AVAudioSession cannot be used either for ReplayKit compatibility.

I removed all OSStatus checks to provide shorter version of the code; however, all Audio Unit methods return noErr. The setup method is:

private func setupAudioUnits() {
    // Setting up mixer unit
    var mixerComponentDescription = AudioComponentDescription(
        componentType: kAudioUnitType_Mixer,
        componentSubType: kAudioUnitSubType_MultiChannelMixer,
        componentManufacturer: kAudioUnitManufacturer_Apple,
        componentFlags: 0,
        componentFlagsMask: 0)
    let mixerComponent = AudioComponentFindNext(nil, &mixerComponentDescription)
    AudioComponentInstanceNew(mixerComponent!, &mixerAudioUnit)
    guard let mixerAudioUnit else { fatalError() }

    var streamFormat = outputFormat.streamDescription.pointee
    AudioUnitSetProperty(mixerAudioUnit,
                         kAudioUnitProperty_StreamFormat,
                         kAudioUnitScope_Output,
                         0,
                         &streamFormat,
                         UInt32(MemoryLayout<AudioStreamBasicDescription>.size))                                
                            
    AudioUnitInitialize(mixerAudioUnit)

    var busCount = UInt32(inputs)
    AudioUnitSetProperty(mixerAudioUnit,
                         kAudioUnitProperty_ElementCount,
                         kAudioUnitScope_Input,
                         0,
                         &busCount,
                         UInt32(MemoryLayout<UInt32>.size))

    // Setting up mixer's inputs
    for input in 0..<inputs {
        var callbackStruct = AURenderCallbackStruct(inputProc: inputRenderCallback, 
                                                    inputProcRefCon: Unmanaged.passUnretained(self).toOpaque())
        AudioUnitSetProperty(mixerAudioUnit,
                             kAudioUnitProperty_SetRenderCallback,
                             kAudioUnitScope_Input,
                             UInt32(input),
                             &callbackStruct,
                             UInt32(MemoryLayout<AURenderCallbackStruct>.size))
    }

    // Setting up output unit
    var outputDescription = AudioComponentDescription(
          componentType: kAudioUnitType_Output,
          componentSubType: kAudioUnitSubType_GenericOutput,
          componentManufacturer: kAudioUnitManufacturer_Apple,
          componentFlags: 0,
          componentFlagsMask: 0)
    let outputComponent = AudioComponentFindNext(nil, &outputDescription)
    AudioComponentInstanceNew(outputComponent!, &outputAudioUnit)
    guard let outputAudioUnit else { fatalError() }

    AudioUnitSetProperty(outputAudioUnit,
                         kAudioUnitProperty_StreamFormat,
                         kAudioUnitScope_Input,
                         0,
                         &streamFormat,
                         UInt32(MemoryLayout<AudioStreamBasicDescription>.size))

    AudioUnitInitialize(outputAudioUnit)

    // Setting up a connection between the mixer and output units
    var connection = AudioUnitConnection(sourceAudioUnit: mixerAudioUnit,
                                         sourceOutputNumber: 0,
                                         destInputNumber: 0)

    AudioUnitSetProperty(outputAudioUnit,
                         kAudioUnitProperty_MakeConnection,
                         kAudioUnitScope_Input,
                         0,
                         &connection,
                         UInt32(MemoryLayout<AudioUnitConnection>.size))

    AudioOutputUnitStart(outputAudioUnit)
}

Here is the part that initiates the rendering. I'm providing nil for mData to use the internal buffer of the audio unit.

private func render(numberOfFrames: AVAudioFrameCount) {
    timeStamp.mFlags = .sampleTimeValid
    timeStamp.mSampleTime = Float64(sampleTime)

    let channelCount = outputFormat.channelCount
    let audioBufferList = AudioBufferList.allocate(maximumBuffers: Int(channelCount))
    for i in 0..<Int(channelCount) {
        audioBufferList[i] = AudioBuffer(mNumberChannels: 1,
                                          mDataByteSize: outputFormat.streamDescription.pointee.mBytesPerFrame,
                                          mData: nil)
    }

    // always returns noErr
    AudioUnitRender(outputAudioUnit,
                    nil,
                    &timeStamp,
                    0,
                    numberOfFrames,
                    audioBufferList.unsafeMutablePointer)

    for channel in 0..<Int(channelCount) {
        let outputBufferData = audioBufferList[channel].mData?.assumingMemoryBound(to: Float.self)
        // Inspecting the data
        // Audio buffer has only zeros
    }

    sampleTime += Int64(numberOfFrames)
}

AudioUnitRender always returns noErr and the buffers have correct size that is based on the provided AVAudioFormat. Input callback is being called and ioData is filled with data:

However, the output buffer is always filled with zeros:

Is there something wrong with my output audio unit setup or with the connection? Or maybe the mixer is not passing the data forward? Any help is appreciated.

Answer 1

So the main issue was that kAudioUnitSubType_MultiChannelMixer always produces empty buffers on macOS. Not sure if it's a bug, but replacing the subtype with kAudioUnitSubType_StereoMixer fixed the issue:

let mixerSubType: OSType
#if os(macOS)
mixerSubType = kAudioUnitSubType_StereoMixer
#else
mixerSubType = kAudioUnitSubType_MultiChannelMixer
#endif

There were also some other things worth mentioning:

• If you set a render callback to the output unit with kAudioUnitProperty_SetRenderCallback, then AudioRenderUnit always produces empty buffers.
• Each mixer input needs correct audio format set with kAudioUnitProperty_StreamFormat.

Answer 2

The crosspoint gains for AUMultiChannelMixer default to 0. You can set the gains as desired using the property kAudioUnitProperty_MatrixLevels on kAudioUnitScope_Input or alternatively use kMultiChannelMixerParam_Volume.

The relevant documentation on kAudioUnitProperty_MatrixLevels from AudioUnitProperties.h is:

@constant       kAudioUnitProperty_MatrixLevels
@discussion         This property can be used for both the AUMatrixMixer and AUMultiChannelMixer.

                    AUMatrixMixer
                    Scope:          Global
                    Value Type:     Float32 array
                    Access:         read/write
                    
                    This property is used to retrieve the entire state of a matrix mixer. The size required is
                    the number of (input  channels + 1) * (output channels + 1) - see _MatrixDimensions
                    
                    So a matrix mixer that has 2 input channels and 2 output channels, will need a 3 x 3 array of Float32
                      
                    Global volume is stored at volumes[2][2]
                    Input volumes are stored in the last column (volumes[0][2] for the first input channel,  volumes[1][2] for the second)
                    Output volumes are stored in the last row (volumes [2][0] and [2][1])
                    Cross point volumes are stored at their expected locations ([0][1], etc)
                    
                    AUMultiChannelMixer
                    Scope:          Input
                    Value Type:     Float32 array
                    Access:         read/write
                    
                    Gets/sets the matrix levels for one input element. This allows arbitrary mixing configurations
                    from all input channels to all output channels.
                    The size required is the number of (input channels) * (output channels).
                    The matrix stores only the crosspoint gains, there are no overall input or output channel gains.