How to ensure one sidekiq worker runs after another is finished in Ruby on Rails

Question

I'm working on a Ruby on Rails application that leverages Sidekiq (version 6.5.8) for background job processing. In my application, I have a specific requirement: I need to ensure that the second Sidekiq worker only starts its execution after the first one has successfully completed its task.

To provide more context, I have a service within my application that initiates the first worker. This worker is responsible for updating certain customer information. Additionally, there is a second worker designed to send an email to the user, indicating that the update process has been completed. My goal is to establish a sequential workflow, where the second worker is automatically invoked once the first worker has finished its work.

It's essential to note that my application's run configuration involves the concurrent execution of multiple workers. Given this setup, calling both workers using perform_async within the service is not a viable solution, as it may not guarantee the desired execution order.

Furthermore, the update service in question is invoked by a user through one of the application's endpoints. Therefore, it's user-triggered and requires a mechanism to ensure that the second worker is executed only after the first one successfully completes its task.

How can I achieve this ordered execution of Sidekiq workers in my Rails application, considering the concurrent nature of Sidekiq and the user-triggered nature of the update service?

I attempted to implement a callback function 'after_perform,' but it appears to be incompatible with my current Sidekiq version (6.5.8). Additionally, I'd prefer not to tightly couple these two workers if possible.

Answer 1

The most straightforward way to accomplish this is by simply calling the second job inside the first job:

class UpdateCustomerInfoJob
  include Sidekiq::Job

  def perform(data)
    # Some logic to update the customer info

    # Once it reaches this point, it means
    # the job completed successfully and will
    # execute the second job.
    EmailCustomerJob.perform_async(
      email: data[:email]
    )
  end
end

As far as I know, there are no other patterns / configuration that can be done purely in Sidekiq to execute jobs sequentially. This is also based on Sidekiq's FAQ:

Sidekiq is designed for asynchronous processing of jobs that can be completed in isolation and independent of each other.

If you really want to, you can try integrating a third-party gem like sidekiq-sequence:

# sequence class
class UpdateCustomerSequence < Sidekiq::Sequence::Base
  step UpdateCustomerInfoJob
  step EmailCustomerJob
end

# execute sequence
UpdateCustomerSequence.new(
  email: '[email protected]',
  first_name: 'John',
  last_name: 'Doe'
)

However, you need to consider if the benefits are worth the extra dependency. Also, just a warning: the gem is not being actively maintained.

Answer 2

Sidekiq Pro's Batches with a :success callback can also solve the problem.

https://github.com/sidekiq/sidekiq/wiki/Complex-Job-Workflows-with-Batches