Spring Cloud Data Flow jar task on kubernetes

Question

How to run a task as a Java jar file in SCDF running in Kubernetes?

To determine the launch platform I use the following configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: scdf-server
  labels:
    app: scdf-server
data:
  application.yaml: |-
    spring:
      cloud:
        dataflow:
          task:
            platform:
              kubernetes:
                accounts:
                  default:
                    limits:
                      memory: 2048Mi

In this case Java files are launched in the namespace as a Docker container which they are not.

I tried to determine the local platform with this configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: scdf-server
  labels:
    app: scdf-server
data:
  application.yaml: |-
    spring:
      cloud:
        dataflow:
          task:
            platform:
              local:
                accounts:
                  default:
                    javaOpts: "-Xdebug -Xmx2048m"

Did not help.

I got the error:

kubelet  Failed to apply default image tag "/opt/dataflow/predefined_tasks/scdf_jdbc_exec_query-0.0.4.jar": couldn't parse image name "/opt/dataflow/predefined_tasks/scdf_jdbc_exec_query-0.0.4.jar": invalid reference format

File scdf-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: scdf
spec:
  selector:
    matchLabels:
      run: scdf
  replicas: 1
  template:
    metadata:
      labels:
        run: scdf
    spec:
      securityContext:
        runAsNonRoot: false
        runAsUser: 0
      containers:
        - name: scdf
          image: springcloud/spring-cloud-dataflow-server:2.11.1-jdk17
          resources:
            limits:
              memory: 10240Mi
            requests:
              memory: 10240Mi
          ports:
            - containerPort: 9393
          env:
            - name: JDK_JAVA_OPTIONS
              value: "-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Xmx8192m"
            - name: SPRING_CLOUD_DATAFLOW_APPLICATIONPROPERTIES_TASK_SPRING_CLOUD_TASK_CLOSECONTEXTENABLED
              value: "true"
            - name: SPRING_CLOUD_SKIPPER_CLIENT_SERVER_URI
              value: "http://skipper-server:7577/api"
            - name: SPRING_CLOUD_DATAFLOW_APPLICATIONPROPERTIES_STREAM_SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS
              value: "PLAINTEXT://kafka-broker:9092"
            - name: SPRING_CLOUD_DATAFLOW_APPLICATIONPROPERTIES_STREAM_SPRING_CLOUD_STREAM_KAFKA_STREAMS_BINDER_BROKERS
              value: "PLAINTEXT://kafka-broker:9092"
            - name: SPRING_CLOUD_DATAFLOW_APPLICATIONPROPERTIES_STREAM_SPRING_CLOUD_STREAM_KAFKA_BINDER_ZKNODES
              value: "zookeeper:2181"
            - name: SPRING_CLOUD_DATAFLOW_APPLICATIONPROPERTIES_STREAM_SPRING_CLOUD_STREAM_KAFKA_STREAMS_BINDER_ZKNODES
              value: "zookeeper:2181"
            - name: SPRING_CLOUD_DATAFLOW_APPLICATIONPROPERTIES_STREAM_SPRING_KAFKA_STREAMS_PROPERTIES_METRICS_RECORDING_LEVEL
              value: "ERROR"
            - name: SPRING_DATASOURCE_URL
              value: "jdbc:postgresql://scdf-postgres:5432/dataflow"
            - name: SPRING_DATASOURCE_USERNAME
              value: "postgres"
            - name: SPRING_DATASOURCE_PASSWORD
              value: "postgres"
            - name: SPRING_DATASOURCE_DRIVER_CLASS_NAME
              value: "org.postgresql.Driver"
            - name: SPRING_DATASOURCE_HIKARI_MAXIMUMPOOLSIZE
              value: "100"
          volumeMounts:
            - name: config
              mountPath: /workspace/config
              readOnly: true
            - name: scdf-tasks
              mountPath: /opt/dataflow/predefined_tasks
      serviceAccountName: scdf-sa
      volumes:
        - name: config
          configMap:
            name: scdf-server
            items:
            - key: application.yaml
              path: application.yaml
        - name: scdf-tasks
          persistentVolumeClaim:
            claimName: scdf-tasks-pvc
      hostname: scdf

How to configure the launch of Java files in SCDF on Kubernetes?

Answer 1

You will need to do the following:

• Create a container to run the Java application. Either use Spring Boot build image plugin or paketo pack.
• Push the container to a container registry.
• Provide access to container by configuring the container pull image credentials
• Register uri of docker container with SCDF as a task
• Create a task definition for the task
• Launch the task.