skaffold fails creating image names on mac book m3 with minikube

Question

skaffold dev -o DEBUG fails with ```The Deployment "hello-world" is invalid: spec.template.spec.containers[0].image: Required value```` And the output shows, that indeed, the generated k8s resources are missing the image name.

I am using a simple spring boot hello world REST service with the following pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.2.3</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>de.neubert.hello</groupId>
    <artifactId>hello-world</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>hello-world</name>
    <description>hello-world</description>
    <properties>
        <java.version>17</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>com.google.cloud.tools</groupId>
                <artifactId>jib-maven-plugin</artifactId>
                <version>3.4.1</version>
                <configuration>
                    <to>
                        <image>hello-world</image>
                    </to>
                    <container>
                        <mainClass>de.neubert.hello.helloworld.HelloWorldApplication</mainClass>
                        <ports>8080</ports>
                    </container>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

The skaffold.yml:

apiVersion: skaffold/v4beta9
kind: Config
metadata:
  name: hello-world
build:
  artifacts:
    - image: hello-world
      jib:
        project: de.neubert.hello:hello-world
manifests:
  rawYaml:
    - src/k8s/base/deployment.yml

The deployment.yml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/version: 0.0.1-SNAPSHOT
  name: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: hello-world
      app.kubernetes.io/version: 0.0.1-SNAPSHOT
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hello-world
        app.kubernetes.io/version: 0.0.1-SNAPSHOT
    spec:
      containers:
        - env:
            - name: KUBERNETES_NAMESPACE
              value: default
          image: hello-world
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
              scheme: HTTP
            successThreshold: 1
          name: hello-world
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
              scheme: HTTP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/version: 0.0.1-SNAPSHOT
  name: hello-world
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 8080
  selector:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/version: 0.0.1-SNAPSHOT
  type: ClusterIP

I am using minikube and podman on a mac book m3. skaffolds output when trying skaffold dev:

[INFO] Containerizing application to Docker daemon as hello-world:60d2d02-dirty...
[WARNING] Base image 'eclipse-temurin:17-jre' does not use a specific image digest - build may not be reproducible
[INFO] Getting manifest for base image eclipse-temurin:17-jre...
[INFO] Building dependencies layer...
[INFO] Building resources layer...
[INFO] Building classes layer...
[INFO] Building jvm arg files layer...
[INFO] The base image requires auth. Trying again for eclipse-temurin:17-jre...
[INFO] Using base image with digest: sha256:d9b347bd2bf6b849cfa2dbe8f1f32017217d086dec0a64a0746cd79be8a397c9
[INFO]
[INFO] Container entrypoint set to [java, -cp, @/app/jib-classpath-file, de.neubert.hello.helloworld.HelloWorldApplication]
[INFO] Loading to Docker daemon...
[INFO]
[INFO] Built image to Docker daemon as hello-world:60d2d02-dirty
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.461 s
[INFO] Finished at: 2024-03-08T13:23:29+01:00
[INFO] ------------------------------------------------------------------------
DEBU[0008] failed to get artifacts from store, err: failed to retrieve build result for image hello-world  subtask=hello-world task=Build
DEBU[0008] generating tag for hello-world:60d2d02-dirty: empty image id  subtask=hello-world task=Build
DEBU[0008] push value not present in isImageLocal(), defaulting to false because cluster.PushImages is false  subtask=-1 task=DevLoop
Build [hello-world] succeeded
INFO[0008] Build completed in 6.973 seconds              subtask=-1 task=Build
INFO[0008] Starting render...                            subtask=-1 task=DevLoop
INFO[0008] starting render process                       subtask=0 task=Render
DEBU[0008] Executing template &{envTemplate 0x1400060b440 0x14000c8bf40  } with environment map[COLORFGBG:7;0 COLORTERM:truecolor COMMAND_MODE:unix2003 DOCKER_CERT_PATH:/Users/U769336/.minikube/certs DOCKER_HOST:tcp://127.0.0.1:49899 DOCKER_TLS_VERIFY:1 HOME:/Users/U769336 HOMEBREW_CELLAR:/opt/homebrew/Cellar HOMEBREW_PREFIX:/opt/homebrew HOMEBREW_REPOSITORY:/opt/homebrew INFOPATH:/opt/homebrew/share/info: ITERM_PROFILE:Default ITERM_SESSION_ID:w0t0p0:AED05AE4-12B2-47E9-9B59-4BC5D5095F4E JAVA_HOME:/Users/U769336/.sdkman/candidates/java/current LANG:de_DE.UTF-8 LC_TERMINAL:iTerm2 LC_TERMINAL_VERSION:3.4.23 LESS:-R LOGNAME:U769336 LSCOLORS:Gxfxcxdxbxegedabagacad LS_COLORS:di=1;36:ln=35:so=32:pi=33:ex=31:bd=34;46:cd=34;43:su=30;41:sg=30;46:tw=30;42:ow=30;43 MANPATH:/Users/U769336/.nvm/versions/node/v20.11.1/share/man:/opt/homebrew/share/man:: MAVEN_HOME:/Users/U769336/.sdkman/candidates/maven/current MINIKUBE_ACTIVE_DOCKERD:minikube NVM_BIN:/Users/U769336/.nvm/versions/node/v20.11.1/bin NVM_CD_FLAGS:-q NVM_DIR:/Users/U769336/.nvm NVM_INC:/Users/U769336/.nvm/versions/node/v20.11.1/include/node OLDPWD:/Users/U769336 PAGER:less PATH:/Users/U769336/.nvm/versions/node/v20.11.1/bin:/Users/U769336/.rd/bin:/Users/U769336/.sdkman/candidates/maven/current/bin:/Users/U769336/.sdkman/candidates/java/current/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Applications/Privileges.app/Contents/Resources:/Users/U769336/Scripts PWD:/Users/U769336/Development/lht/lht-software-architecture/hello-world SDKMAN_CANDIDATES_API:https://api.sdkman.io/2 SDKMAN_CANDIDATES_DIR:/Users/U769336/.sdkman/candidates SDKMAN_DIR:/Users/U769336/.sdkman SDKMAN_PLATFORM:darwinarm64 SHELL:/bin/zsh SHLVL:1 SNC_LIB:/Applications/Secure Login Client.app/Contents/MacOS/lib/libsapcrypto.dylib SNC_LIB_64:/Applications/Secure Login Client.app/Contents/MacOS/lib/libsapcrypto.dylib SSF_LIBRARY_PATH:/Applications/Secure Login Client.app/Contents/MacOS/lib/libsapcrypto.dylib SSF_LIBRARY_PATH_64:/Applications/Secure Login Client.app/Contents/MacOS/lib/libsapcrypto.dylib SSH_AUTH_SOCK:/private/tmp/com.apple.launchd.82dshT1Wxy/Listeners TERM:xterm-256color TERM_PROGRAM:iTerm.app TERM_PROGRAM_VERSION:3.4.23 TERM_SESSION_ID:w0t0p0:AED05AE4-12B2-47E9-9B59-4BC5D5095F4E TMPDIR:/var/folders/rh/bz86_ljx76b0w2_fm1xb803c0000gq/T/ USER:U769336 XPC_FLAGS:0x0 XPC_SERVICE_NAME:0 ZSH:/Users/U769336/.oh-my-zsh _:/opt/homebrew/bin/skaffold __CFBundleIdentifier:com.googlecode.iterm2 __CF_USER_TEXT_ENCODING:0x1F7:0x0:0x3 export:]  subtask=-1 task=DevLoop
DEBU[0008] manifests with tagged images:apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/version: 0.0.1-SNAPSHOT
  name: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: hello-world
      app.kubernetes.io/version: 0.0.1-SNAPSHOT
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hello-world
        app.kubernetes.io/version: 0.0.1-SNAPSHOT
    spec:
      containers:
        - env:
            - name: KUBERNETES_NAMESPACE
              value: default
          image: ""
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
              scheme: HTTP
            successThreshold: 1
          name: hello-world
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
              scheme: HTTP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/version: 0.0.1-SNAPSHOT
  name: hello-world
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 8080
  selector:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/version: 0.0.1-SNAPSHOT
  type: ClusterIP  subtask=0 task=Render
DEBU[0008] manifests with labels apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/version: 0.0.1-SNAPSHOT
    skaffold.dev/run-id: 2cf63710-610f-49e5-9eec-e76abf37d1a0
  name: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: hello-world
      app.kubernetes.io/version: 0.0.1-SNAPSHOT
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hello-world
        app.kubernetes.io/version: 0.0.1-SNAPSHOT
        skaffold.dev/run-id: 2cf63710-610f-49e5-9eec-e76abf37d1a0
    spec:
      containers:
        - env:
            - name: KUBERNETES_NAMESPACE
              value: default
          image: ""
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
              scheme: HTTP
            successThreshold: 1
          name: hello-world
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
              scheme: HTTP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/version: 0.0.1-SNAPSHOT
    skaffold.dev/run-id: 2cf63710-610f-49e5-9eec-e76abf37d1a0
  name: hello-world
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 8080
  selector:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/version: 0.0.1-SNAPSHOT
  type: ClusterIP  subtask=-1 task=DevLoop
INFO[0008] Render completed in 1.92575ms                 subtask=-1 task=DevLoop
Tags used in deployment:
 - hello-world ->
DEBU[0008] push value not present in isImageLocal(), defaulting to false because cluster.PushImages is false  subtask=-1 task=DevLoop
DEBU[0008] Local images can't be referenced by digest.
They are tagged and referenced by a unique, local only, tag instead.
See https://skaffold.dev/docs/pipeline-stages/taggers/#how-tagging-works  subtask=-1 task=Deploy
Starting deploy...
DEBU[0008] getting client config for kubeContext: `minikube`  subtask=-1 task=DevLoop
DEBU[0008] Running command: [kubectl --context minikube get -f - --ignore-not-found -ojson]  subtask=0 task=Deploy
DEBU[0008] Command output: []                            subtask=0 task=Deploy
DEBU[0008] 2 manifests to deploy. 2 are updated or new   subtask=0 task=Deploy
DEBU[0008] Running command: [kubectl --context minikube apply -f -]  subtask=0 task=Deploy
 - service/hello-world created
 - The Deployment "hello-world" is invalid: spec.template.spec.containers[0].image: Required value
Cleaning up...
DEBU[0008] Running command: [kubectl --context minikube delete --ignore-not-found=true --wait=false -f -]  subtask=-1 task=DevLoop
 - service "hello-world" deleted
INFO[0008] Cleanup completed in 30.848167ms              subtask=-1 task=DevLoop
DEBU[0008] Running command: [tput colors]                subtask=-1 task=DevLoop
DEBU[0008] Command output: [256
]                        subtask=-1 task=DevLoop
kubectl apply: exit status 1
DEBU[0008] exporting metrics disabled                    subtask=-1 task=DevLoop

The commands I execute are:

minikube start
skaffold config set --global local-cluster true
eval $(minikube -p custom docker-env)
skaffold init

What is happening here? What am I doing wrong and how can I fix it?

Kind Regards, Tobias

Answer 1

Running an ARM64 image on an x86_64 node will result in a pull failure. Double-check the architecture of your Kubernetes cluster nodes. Make sure they are ARM64 compatible.

Use kubectl get nodes and kubectl describe <node\> to view your node architecture. Look for entries indicating "arm64" or "aarch64" in the architecture field.