cffi package broken in mac M2 arm chip - "but is an incompatible architecture (have 'arm64', need 'x86_64')"

Question

I have installed gcloud sdk for arm64 mac and I have set up the auth-plugin as well. When I try to run

kubectl version

I get the below error

F0212 08:33:03.283222    5921 cred.go:145] print credential failed with error: Failed to retrieve access token:: failure while executing gcloud, with args [config config-helper --format=json]: exit status 1 (err: ImportError: dlopen(/Users/vijaisha/.config/gcloud/virtenv/lib/python3.11/site-packages/_cffi_backend.cpython-311-darwin.so, 0x0002): tried: '/Users/vijaisha/.config/gcloud/virtenv/lib/python3.11/site-packages/_cffi_backend.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/vijaisha/.config/gcloud/virtenv/lib/python3.11/site-packages/_cffi_backend.cpython-311-darwin.so' (no such file), '/Users/vijaisha/.config/gcloud/virtenv/lib/python3.11/site-packages/_cffi_backend.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))
thread '<unnamed>' panicked at /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/err/mod.rs:790:5:
Python API call failed
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/Users/vijaisha/google-cloud-sdk/lib/gcloud.py", line 189, in <module>
    main()
  File "/Users/vijaisha/google-cloud-sdk/lib/gcloud.py", line 183, in main
    gcloud_main = _import_gcloud_main()
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vijaisha/google-cloud-sdk/lib/gcloud.py", line 90, in _import_gcloud_main
    import googlecloudsdk.gcloud_main
  File "/Users/vijaisha/google-cloud-sdk/lib/googlecloudsdk/gcloud_main.py", line 35, in <module>
    from googlecloudsdk.calliope import cli
  File "/Users/vijaisha/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 32, in <module>
    from googlecloudsdk.calliope import backend
  File "/Users/vijaisha/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 39, in <module>
    from googlecloudsdk.calliope import parser_extensions
  File "/Users/vijaisha/google-cloud-sdk/lib/googlecloudsdk/calliope/parser_extensions.py", line 85, in <module>
    from googlecloudsdk.core.updater import update_manager
  File "/Users/vijaisha/google-cloud-sdk/lib/googlecloudsdk/core/updater/update_manager.py", line 46, in <module>
    from googlecloudsdk.core.updater import installers
  File "/Users/vijaisha/google-cloud-sdk/lib/googlecloudsdk/core/updater/installers.py", line 31, in <module>
    from googlecloudsdk.core import requests as core_requests
  File "/Users/vijaisha/google-cloud-sdk/lib/googlecloudsdk/core/requests.py", line 27, in <module>
    from google.auth.transport import requests as google_auth_requests
  File "/Users/vijaisha/google-cloud-sdk/lib/third_party/google/auth/transport/requests.py", line 41, in <module>
    from google.oauth2 import service_account
  File "/Users/vijaisha/google-cloud-sdk/lib/third_party/google/oauth2/service_account.py", line 77, in <module>
    from google.auth import _service_account_info
  File "/Users/vijaisha/google-cloud-sdk/lib/third_party/google/auth/_service_account_info.py", line 20, in <module>
    from google.auth import crypt
  File "/Users/vijaisha/google-cloud-sdk/lib/third_party/google/auth/crypt/__init__.py", line 41, in <module>
    from google.auth.crypt import rsa
  File "/Users/vijaisha/google-cloud-sdk/lib/third_party/google/auth/crypt/rsa.py", line 20, in <module>
    from google.auth.crypt import _cryptography_rsa
  File "/Users/vijaisha/google-cloud-sdk/lib/third_party/google/auth/crypt/_cryptography_rsa.py", line 22, in <module>
    import cryptography.exceptions
  File "/Users/vijaisha/.config/gcloud/virtenv/lib/python3.11/site-packages/cryptography/exceptions.py", line 9, in <module>
    from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
pyo3_runtime.PanicException: Python API call failed
)
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Unable to connect to the server: getting credentials: exec: executable gke-gcloud-auth-plugin failed with exit code 1

To fix this, I have tried reinstalling gcloud sdk, I also tried to download the x86_64 version of cffi but mac was not allowing it complaining that it was incompatible.

I have tried using the export USE_GKE_GCLOUD_AUTH_PLUGIN=True but it still errors out.

The gcloud container clusters get-credentials works fine without error, its the kubectl related commands that error out with failed to retrieve access token

Answer 1

I was able to fix this problem with the following:

cd ~/.config/gcloud/virtenv
bin/pip3 uninstall cffi -y
bin/pip3 install --no-binary :all: cffi --ignore-installed

Answer 2

I ran into this same issue. Recompiling for x86_64 architecture like this would fix the issue:

arch -x86_64 pip install cffi --compile --force-reinstall --no-cache-dir

But other programs will say (have 'x86_64', need 'arm64').

What I ended up doing is creating a Mach-O universal binary with lipo using the arm64 file and x86_64 file.

Step 1 : copy the arm64 .so file

cp /Users/vijaisha/.config/gcloud/virtenv/lib/python3.11/site-packages/_cffi_backend.cpython-311-darwin.so ~/tmp_arm64.so

Step 2 : recompile for x86_64

arch -x86_64 pip install cffi --compile --force-reinstall --no-cache-dir

Step 3 : Create Universal Binary with lipo

lipo -create -output /Users/vijaisha/.config/gcloud/virtenv/lib/python3.11/site-packages/_cffi_backend.cpython-311-darwin.so ~/tmp_arm64.so /Users/vijaisha/.config/gcloud/virtenv/lib/python3.11/site-packages/_cffi_backend.cpython-311-darwin.so