CSCore & Unity: How to input Microphone Data (Sample Source) into Audio Source in Unity

Question

Im having troubles figuring out how to send microphone data into unity's audiosource. Doing it mostly for the better latency. Unity's native "Microphone..." solution takes almost a second to get through.

if (selected_device == null)
            {
                using (var deviceEnumerator = new MMDeviceEnumerator())
                using (var deviceCollection = deviceEnumerator.EnumAudioEndpoints(DataFlow.Capture, DeviceState.Active))
                {
                    selected_device = deviceCollection[0];
                    Debug.Log("Got Device!: " + selected_device.FriendlyName);
                }
            }

            m_SoundKeeper = new WasapiCapture();
            m_SoundKeeper.Device = selected_device;
            m_SoundKeeper.Initialize();
            
            soundInSource = new SoundInSource(m_SoundKeeper);

With this im starting the objects, getting the mic, everything is fine here.

This one is in the update method of Unity:

if (!isRecordingStarted)
            {
                m_SoundKeeper.Start();
                isRecordingStarted = true;
            }

            Debug.Log("Mic Data available");

            ISampleSource convertSoundSource = soundInSource.ToSampleSource();
            convertSoundSource.Read(_microphoneBuffer, 0, _microphoneBuffer.Length);

            AudioClip audioClip = AudioClip.Create("bruhhhh", _microphoneBuffer.Length, 1, 48000, false);
            audioClip.SetData(_microphoneBuffer, 0);
            _audioSource.clip = audioClip;
            _audioSource.Play();

I get like a second of very distorted silent sound and then it throws errors object reference bla bla I think it might be because im trying to read data that is not existing yet.

I am very confused at what im doing wrong and would love some advice!

I also am using a bit modified Caress.Unity because since im trying to create a multiplayer, I have to send the byte array after encoding, but its a whole another topic.