Toggle Speaker and ear-speaker for TwilioVideo SDK, long delay in switching devices

Question

I am trying to implement the ability for a user to toggle between speaker and earpiece speaker for a audio/video calling app using the TwilioVideo SDK. I have an instance of a DefaultAudioDevice that I set as the TwilioVideoSDK audioDevice, so I can change the AVAudioSession when user toggles between speaker and earpiece. I am then listening to the AVAudioSession.routeChangeNotification notification so when the device actually changes to the different audio route, I can update the UI with the change. The issue I am having is after a user toggles, the time between then and when the audio route actually changes is multiple seconds. Is there an approach to switch between devices that doesn't so long? Here is some code of how I change between devices

    func audioSessionBlock(speaker: Bool) -> (() -> Void) {
        return {
            let audioSession = AVAudioSession.sharedInstance()
            DefaultAudioDevice.DefaultAVAudioSessionConfigurationBlock()
            let options: AVAudioSession.CategoryOptions = speaker == true ? [.defaultToSpeaker, .allowBluetooth] : [.allowBluetooth]
            try? audioSession.setCategory(.playAndRecord, mode: .voiceChat, options: options)
            if speaker == true {
                try? audioSession.overrideOutputAudioPort(AVAudioSession.PortOverride.speaker)
            } else {
                try? audioSession.overrideOutputAudioPort(AVAudioSession.PortOverride.none)
            }
        }

Here is toggling the audio:

func toggleAudio(speaker: Bool) {
    self.audioDevice.block = self.mode.audioSessionBlock(speaker)
    self.audioDevice.block()
}