Question in brief: Does AEM have a jobs orchestration API or approach via listeners or similar that will let the orchestrator know when a topic's jobs are completed?
Question in detail: My first stack overflow post so be gentle. I have a scenario where a third party system is referenced each night by AEM via a scheduler, to identify any new closed user groups that need to be applied to specific folders in AEM. Then other CUGs are added to/removed from those folder-level CUGs.
The scheduler kicks off a job executor, and that job kicks off other jobs. So it's currently like this:
CURRENT STATE: AEM Scheduler (nightly)
-- Job Executor
-- Job Consumer 1: ordered queue, one job at a time. Around 1k jobs per run, takes hours.
-- Job Consumer 2: unordered queue, these jobs run in parallel. Around 1k jobs, takes minutes.
Things are working ok now but because of the matrix nature of some of our data, the Job Consumer 1 topic's jobs have to be run ordered, in sequence. That takes a very long time to run, and I'm looking at ways to speed it up significantly.
One way to speed up Job Consumer 1 is to break up that consumer into two, with each being changed to run in parallel vs in sequence. BUT.... after breaking it up into two, I wouldn't be able to safely start the second set until the first set is done. Here's what it could look like:
POSSIBLE FUTURE STATE: AEM Scheduler (nightly)
-- Job Executor
-- Job Consumer 1: unordered queue
-- Job Consumer 2: unordered queue, BUT MUST NOT START until all Job Consumer 1 jobs are done.
-- Job Consumer 3: unordered queue, can run whenever
My challenge is the "BUT MUST NOT START until all Job Consumer 1 jobs are done" part. If this were Azure Functions I'd use an actual Orchestrator to do that. But in AEM, I don't see an orchestration tool to manage a job set. I'm thinking I can use a resource change listener, listening to deletes from the topic's /var/eventing tree. That should work but I'd only do that if there isn't a better way like a jobs orchestration tool that I haven't come across yet.
Hopefully my question is coming across. Thanks in advance.
Have a functioning solution, but would like to speed it up.