Github action claims failure but pytest passed

Question

I'm using tox to call pytest inside a github action on MacOS:

name: Mac CI
on: [push, pull_request]

jobs:
  CI:
    runs-on: macos-latest
    strategy:
      matrix:
        fortran-compiler:  ['8','9','10','11','12']
        python-version: ['3.7', '3.8', '3.9', '3.10', '3.11']
      fail-fast: false
    env:
      FC: gfortran
      CC: gcc

    steps:
    - uses: actions/checkout@v4
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}

    - name: Setup gfortran-${{ matrix.fortran-compiler }}
      run: |
        brew install gcc@${{ matrix.fortran-compiler }}
        ln -s /usr/local/bin/gfortran-${{ matrix.fortran-compiler }} /usr/local/bin/gfortran
        ln -s /usr/local/bin/gcc-${{ matrix.fortran-compiler }} /usr/local/bin/gcc

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        python -m pip install tox tox-gh-actions build wheel pytest

    - name: Test with tox
      run: tox --recreate -e py

When the action runs, it creates a build matrix of different Python and Fortran versions, where some combinations pass and some fail. Digging into the failures, however, shows no pattern. On some commits combinations of Fortran/Python pass only to fail on the next commit (and everything passes on a Linux). The output from the end of the github action log is:

 
  ======================= 205 passed, 21 skipped in 15.56s =======================
  py: exit -6 (27.37 seconds) /Users/runner/work/gfort2py/gfort2py> pytest pid=1572
.pkg: _exit> python /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
  py: FAIL code -6 (48.00=setup[20.63]+cmd[27.37] seconds)
  evaluation failed :( (48.21 seconds)
Error: Process completed with exit code 250.

So the pytest output shows all tests being passed/skipped so no failures but something after that is causing a failure and setting a non-zero exit code from the action.

Running tox with -vvv I got (now failing on a different python version):

 ======================= 205 passed, 21 skipped in 9.31s ========================
  py: 311971 C exit -6 (17.62 seconds) /Users/runner/work/gfort2py/gfort2py> pytest pid=3336 [tox/execute/api.py:279]
.pkg: 311974 W _exit> python /Users/runner/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta [tox/tox_env/api.py:427]
Backend: run command _exit with args {}
Backend: Wrote response {'return': 0} to /var/folders/3s/vfzpb5r51gs6y328rmlgzm7c0000gn/T/pep517__exit-3vp4qefz.json
.pkg: 311980 I exit None (0.00 seconds) /Users/runner/work/gfort2py/gfort2py> python /Users/runner/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta pid=1935 [tox/execute/api.py:279]
.pkg: 312049 D delete package /Users/runner/work/gfort2py/gfort2py/.tox/.tmp/package/1/gfort2py-0.1.dev1+ga7f592b.tar.gz [tox/tox_env/python/virtual_env/package/pyproject.py:206]
  py: FAIL code -6 (311.71=setup[294.10]+cmd[17.62] seconds)
  evaluation failed :( (311.83 seconds)
Error: Process completed with exit code 250.