flink RocksDB, EKS and EBS

89 Views Asked by At

I am currently trying to understand how RocksDB works in flink. We currently have an application that is running in flink inside AWS EKS.

  1. We currently have the state.backend.rocksdb.localdir commented out in the flink config file. Does that mean RocksDB is not being used? In the flink UI I can see keys getting added to RocksDB does that mean it’s only storing in memory and not on disk? Not sure if it still uses disk when using Kubernetes without any local storage enabled in flink.

  2. I am currently trying to add ebs persistent storage and I am able to do it with running a basic test pod in EKS. When I uncomment the state.backend.rocksdb.localdir in the flink config file it throws an error that it can not find the path. Below I attached a snippet of the task manager manifest files. Would also add I have added the PVC file and the storage class file. Any help would be appreciated.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: kubernetes.io/aws-ebs
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp2
  iopsPerGB: "10"
  fsType: ext4
reclaimPolicy: Retain
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 50Gi
      containers:
      - name: taskmanager
        image: 1234566.dkr.ecr.us-east-1.amazonaws.com/test123:latest
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 3
            memory: "6144Mi"
          limits:
            cpu: 3
            #memory: "4096Mi"
            memory: "6144Mi"
            #memory: "8192Mi"
        env:
        - name: STATE_BACKEND
          value: "rocksdb"
        - name: STATE_BACKEND_ROCKSDB_LOCALDIR
          value: "/rocksdb/data"
        args: ["taskmanager"]
        ports:
        - containerPort: 6122
          name: rpc
        - containerPort: 6125
          name: query-state
        livenessProbe:
          tcpSocket:
            port: 6122
          initialDelaySeconds: 30
          periodSeconds: 60
        volumeMounts:
        - name: persistent-storage
          mountPath: /rocksdb/data
        - name: flink-config-volume
          mountPath: /opt/flink/conf/
    #    - name: rocksdb-local-store
    #      mountPath: /mnt/flink/staleness/rocksdb
        securityContext:
          runAsUser: 9999  # refers to user _flink_ from official flink image, change if necessary
      volumes:
      - name: persistent-storage
        persistentVolumeClaim:
          claimName: ebs-claim
      - name: flink-config-volume
        configMap:
          name: inventory-staleness-stg-config
          items:
          - key: flink-conf.yaml
            path: flink-conf.yaml
          - key: log4j-console.properties
            path: log4j-console.properties
      - name: job-artifacts-volume
        hostPath:
          path: /host/path/to/job/artifacts      
0

There are 0 best solutions below