K8s pod "Output: mount error(107): Transport endpoint is not connected" when mounting SMB share with cifs

Question

I'm trying to mount SMB share inside a k8s pod using the following:

A PV that connect to a SMB share using user and password inside a secret:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: smb-volume
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - vers=3.0
  csi:
    driver: smb.csi.k8s.io
    readOnly: false
    volumeHandle: smb-volume
    volumeAttributes:
      source: "//<SMB-SHARE-ADDRESS>/shared-data"
    nodeStageSecretRef:
      name: <SECRET-NAME>
      namespace: <SECRET-NAMESPACE>

A PVC that bounds to the PV:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: smb-pvc
  namespace: <PVC-NAMESPACE>
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  volumeName: smb-volume
  storageClassName: ""

A deploy that mounts the PVC with this code:

spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: <CONTAINER-NAME>
          volumeMounts:
          - name: share
            mountPath: /home/user/share
      volumes:
      - name: share
        persistentVolumeClaim:
          claimName: smb-pvc

I have smb.csi.k8s.io driver installed, as below:

I'm applying this solution in 5 clusters. Four of them successfully. One of them (production) presents the following error:

kind: Event
apiVersion: events.k8s.io/v1
metadata:
  name: <EVENT-NAME>
  namespace: <NAMESPACE>
eventTime: null
reason: FailedMount
regarding:
  kind: Pod
  namespace: <NAMESPACE>
  name: <POD-NAME>
  apiVersion: v1
note: >
  MountVolume.MountDevice failed for volume "smb-volume" : rpc error:
  code = Internal desc = volume(smb-volume) mount
  "//<SMB-SHARE-ADDRESS>/shared-data" on
  "/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/0f464fd4331cbcb83be21b563baad7a7227e7e6e6a0a6f33380040bd1e0bddfc/globalmount"
  failed with mount failed: exit status 32

  Mounting command: mount

  Mounting arguments: -t cifs -o dir_mode=0777,file_mode=0777,vers=3.0,<masked>
  //<SMB-SHARE-ADDRESS>/shared-data
  /var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/0f464fd4331cbcb83be21b563baad7a7227e7e6e6a0a6f33380040bd1e0bddfc/globalmount

  Output: mount error(107): Transport endpoint is not connected

  Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log
  messages (dmesg)
type: Warning
deprecatedSource:
  component: kubelet
  host: aks-system2-23099442-vmss000003
deprecatedFirstTimestamp: '2023-09-18T16:43:12Z'
deprecatedLastTimestamp: '2023-09-18T16:43:45Z'
deprecatedCount: 7

The things that I already checked:

• The SMB share is accessible, I can mount it through my machine or using one of the other environments
• The SMB share is open from inside the pod on this particular environment, I checked using nc -zv <SAMBA-SHARE> <PORT>
• I can't execute dmesg inside the pod

Any thoughts?

-- UPDATE --

I got this message from executing

dmesg | grep CIFS

inside the node where the pod is running:

[2869212.915113] CIFS: Attempting to mount \\<SMB-SHARE-ADDRESS>\shared-data
[2869212.927925] CIFS: VFS: \\<SMB-SHARE-ADDRESS> smb3_crypto_aead_allocate: Failed alloc encrypt aead
[2869212.931552] CIFS: VFS: \\<SMB-SHARE-ADDRESS> crypt_message: crypto alloc failed
[2869212.934458] CIFS: VFS: \\<SMB-SHARE-ADDRESS> failed to connect to IPC (rc=-2)
[2869212.937337] CIFS: VFS: session 00000000920e8221 has no tcon available for a dfs referral request
[2869212.942989] CIFS: VFS: \\<SMB-SHARE-ADDRESS> smb3_crypto_aead_allocate: Failed alloc encrypt aead
[2869212.946427] CIFS: VFS: \\<SMB-SHARE-ADDRESS> crypt_message: crypto alloc failed
[2869212.949261] CIFS: VFS: \\<SMB-SHARE-ADDRESS> __cifs_put_smb_ses: Session Logoff failure rc=-2
[2869212.952415] CIFS: VFS: cifs_mount failed w/return code = -107

I also able to mount the file share if I execute the following command from within the node:

mount -t cifs //<SMB-SHARE-ADDRESS>/shared-data temp -o username=<SMB-SHARE-USER>,password=<SMB-SHARE-PASSWORD>,domain=<SMB-SHARE>