Pinned GCHandle of array corrupted only when Debugger is attached

Question

I am currently writing a KMDF driver for an FPGA. Communication between the device and the driver is standard memory mapped I/O using the PCIe BAR spaces. The high-level UI is written in C#, it communicates with a user-mode library written in C/C++ over PInvoke, which then relays to the driver over IOCTL commands.

The device is equipped with 16GB DDR4 memory, which is exposed to the driver as a 64kB window, that can be read/written via BAR. The base address of the window can be changed by writing to a configuration register that is also exposed through the BAR.

I know that this is not an ideal setup performance-wise and ultimately memory transfers will be handled by DMA, but at the current stage I am debugging the internal DDR4 wirings of the FPGA design and want to have that kind of memory mapped access to the RAM for debugging purposes. I have written a RAM test that basically writes random bytes to the memory and then reads them out to compare with the original.

These random test-bytes are created on the C# side of things as an array of uint. This array is then pinned as a GCHandle and passed to the usermode library. The C# call looks like this:

        internal void ReadMemory_Segmented(ulong segment, ulong offset, uint[] data, uint index, uint count)
        {
            GCHandle handle = GCHandle.Alloc(data, GCHandleType.Pinned);
            try
            {
                nint hData = handle.AddrOfPinnedObject();
                if (!API.DDR4_Read_segmented(m_Handle, segment, offset, hData, index, count))
                    throw new InvalidOperationException();
            }
            finally
            {
                handle.Free();
            }
        }

The relevant C/C++ Code in DDR4_Read_segmented looks like this:

    PCHAR pInBuffer = (PCHAR)_malloca(2 * sizeof(UINT64));
    if (!pInBuffer)
        return FALSE;
    PCHAR pCurrent = pInBuffer;
    PUINT64 pSegment = (PUINT64)pCurrent;
    pCurrent += sizeof(UINT64);
    PUINT64 pOffset = (PUINT64)pCurrent;
    *pSegment = segmentIndex;
    *pOffset = segmentRegister;
    status = DeviceIoControl(hDevice, IOCTL_DDR4_READ_SEGMENTED, pInBuffer, 2 * sizeof(UINT64), pOutData + index, (DWORD)(countRegisters * sizeof(UINT32)), &bytesRead, NULL);

Everything works fine when I have no debugger attached. Both, Debug and Release builds run the test over the entire 16GB range with random addresses and random data without any error.

However, to my total surprise, whenever I attach a debugger (Visual Studio 2022) the contents of the array are incorrect and the test fails reproducibly on the second call to DDR4_Read_segmented. Analog code that copies the contents of the array to memory allocated by Marshal.AllocHGlobal instead of passing the pinned array directly seems to have no issues even with the debugger attached. I could use that as a work-around, since it's for debugging purposes only. But... what the heck is going on there? I suspect I am doing something wrong with the pinned pointers, but I am honestly clueless.