How to execute a cloud run job and reschedule on successful execution?

Question

My requirement is to execute a cloud run job then schedule it so that execution of next job happens on successful execution of the first job.

How can I implement this in gcp as I have scheduled it through cloud schedular and disable the parallel execution.

For example:

• Job execution takes 5 minutes, and the schedule executes a job every 2 minutes.

Now there are two jobs executing in parallel for 2 minutes and third job is started to execute for 1 minute in parallel with 2 other jobs. Likewise, job one gets completed at 5 minutes and after 6 minute another job-4 started executing.

However, I need to execute job-1 first and complete its task and then job-2 will start executing once a job-1 gets completed likewise.

Please suggest any thing in Google cloud platform through which I can achieve this?

Answer 1

There are several way to implement this pattern and it depends on your requirements.

• You can use Cloud Workflow to link the job one by one and play them in sequence. But it requires to know the number of jobs, and their parameter at the beginning of the Workflow (when you start the execution)
• You can use PubSub also. You publish a message in PubSub with your scheduler. A Cloud Run (or a Cloud Functions) will consume the message: If there is no job running, a job is trigger with the message parameter and the PubSub message is ACK. Else the message is NACK (and republished by PubSub). The problem is that you can't manage the retry policy with PubSub, and your Cloud Run will be always on by waiting the end of the current job. But there is a low latency between jobs.
• A similar solution could be to use Cloud Task. This time the retry policy and the exponential backoff is configurable. Your task is agressive against your Cloud Run, but you will waste seconds/minutes to detect the end of a job and start the next ones.
• You could also imagine something event driven and to detect the end of a job and run the next one based on this event. But it required to know in advance the number of job to run and the parameters to use.