ArgoCD Doesn't Trigger Kubernetes Job on Image Tag Update Using Hooks

Question

I'm using ArgoCD with an ApplicationSet to manage multiple applications in a Kubernetes cluster. One of these applications is a database migration job (main-db-migrator), which I intend to run whenever there's an update to its image tag in the deployment manifest (deployment.yaml). Despite the manifest being updated in the repository and ArgoCD successfully pulling these changes, the job does not get triggered. Here are the specifics of my setup:

ArgoCD ApplicationSet:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: staging-applications
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: 'https://github.com/myorg/myrepo.git'
        revision: HEAD
        directories:
          - path: "staging/apps/default/*"
  template:
    metadata:
      name: '{{.path.basename}}'
      finalizers:
      - resources-finalizer.argocd.argoproj.io
    spec:
      project: default
      source:
        repoURL: 'https://github.com/myorg/myrepo.git'
        targetRevision: HEAD
        path: '{{.path.path}}'
      destination:
        server: 'https://kubernetes.default.svc'
        namespace: '{{index .path.segments 2}}'
      syncPolicy:
        syncOptions:
        - CreateNamespace=true
        automated:
          selfHeal: true
          prune: true

Job Manifest (staging/apps/default/main-db-migrator/deployment.yaml):

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrator
  labels:
    app.kubernetes.io/name: db-migrator
    app.kubernetes.io/part-of: likvido-app
    app.kubernetes.io/component: server-utility

  # ArgoCD metadata:
  generateName: db-migrator-
  annotations:
    argocd.argoproj.io/hook: Sync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: db-migrator
        image: <my-image>
        volumeMounts:
          - name: secrets
            mountPath: "app/secrets"
            readOnly: true
      volumes:
        - name: secrets
          projected:
            sources:
            - secret:
                name: likvidoapp-dbmigrator
                items:
                  - key: app.json
                    path: app.json

Expected outcome:

I am very new to ArgoCD, but the expected outcome is that ArgoCD should deploy the job whenever the image tag changes in the deployment.yaml file.

What happens:

Nothing happens :) - I can see the application inside ArgoCD, where it is called main-db-migrator:

I read the documentation here: https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/ - but it seems lacking, as it does not show the Argo Application, and if anything else needs to be set up for this to work.

Can someone point me in the right direction?

Answer 1

OK, I think I figured out my problem. Nothing happens because of my ArgoCD application only consists of this "Sync" hook job and nothing else. This means ArgoCD will never sync anything.

I came up with a workaround, which works for me, and that is to create a fake cron-job, that will never trigger, and then let ArgoCD manage that as the "main" application. Then I can update an annotation in this cron job whenever I update the image in my db migration job, and then ArgoCD will sync the CronJob and run the migration :)

So I added these two files to the same folder as my deployment.yaml file:

fake-cronjob.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: fake-db-migrator
  annotations: {}
spec:
  schedule: "0 0 31 2 *" # This schedule means never (Feb 31st doesn't exist)
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: fake-job
            image: busybox
            command: ["echo", "This is a fake job for ArgoCD sync purposes."]
          restartPolicy: OnFailure

kustomization.yaml

resources:
  - fake-cronjob.yaml
  - deployment.yaml

Now, whenever I need to run the migration, I will update the image in the deployment.yaml, and also update an annotation in the fake-db-migrator cron job