How to pass data from Terraform Kubernetes cluster deployment to Kustomize app deployment?

Question

We build our cluster with Terraform. We provision EFS storage as part of the deployment, which needs to be referenced by our ArgoCD app deployment. The deployment is a two part process:

1. The infrastructure is deployed via Terraform, configuration is stored in its own repo.
2. We deploy our ArgoCD apps, configuration stored in a separate repo.

The Terraform deployment provisions the Kubernetes cluster and EFS storage and stores the file system ID in a ConfigMap at a known location inside the cluster. I want to use Kustomize to query the ConfigMap and inject the EFS file system ID into a StorageClass manifest that is part of the ArgoCD deployment.

I have a storage app in ArgoCD to manage the storage:

|- apps/
   - storage/
     - efs-sc.yaml
     - kustomization.yaml

# apps/storage/efs-sc.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: INJECT-FSID
  directoryPerms: "700"

# apps/storage/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - efs-sc.yaml

replacements:
  - source:
      kind: ConfigMap
      name: aws-storage
      namespace: kube-system
      fieldPath: data.efs-file-system-id
    targets:
      - select:
          kind: StorageClass
          name: efs-sc
        fieldPaths:
          - parameters.fileSystemId

The output of kubectl get cm -n kube-system aws-storage -o yaml is:

apiVersion: v1
data:
  efs-file-system-id: fs-abcdef12345678
immutable: false
kind: ConfigMap
metadata:
  creationTimestamp: "2024-02-14T11:16:47Z"
  name: aws-storage
  namespace: kube-system
  resourceVersion: "704"
  uid: 007d0b3f-e7ea-4b24-8eb3-8daa83cdc249

When I try to run kustomize build apps/storage > k.yaml to test this i get the error Error: nothing selected by ConfigMap.[noVer].[noGrp]/aws-storage.kube-system:data.efs-file-system-id.

Am I misunderstanding what replacements: can do? Can it read from the live cluster to get data to inject? Or does the data have to be included as a manifest somewhere in the Kustomize deployment? If I have misunderstood this feature, is there another way I can pass data from the Terraform deployment to the ArgoCD deployment?

Edit #1 (added relevant versions) Kustomize version: v5.3.0 Kubectl: Client Version: v1.28.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.0-eks-c417bb3

Answer 1

As stated in this GitHub issue, Kustomize is client side only. It cannot query any data from server side (cluster). Instead, I rearchitected infrastructure deployment to install the storage class via Terraform. The storage app in ArgoCD deployment is removed and the PVCs still reference the known storage class name.