Rails Background Jobs running twice

Question

I am using Rufus Scheduler to trigger the background jobs that need to run every 1 hour.

scheduler = Rufus::Scheduler.singleton

scheduler.every '1h' do
 JobName.perform_now
end

I have my Infra setup in AWS and for production, I have 2 Instances running the APP inside an ECS.

What happens is that the Scheduler schedules the jobs twice.

Instance A schedules the job at 00:00:05:01 and Instance B schedules at 00:00:05:05

The jobs are not failing. I am using ActiveJob. I was looking into other solutions like delayed Job but that has the same problem when there are multiple instances.

Can you guys provide an alternate approach to fix this issue? Or a workaround for the same?

Answer 1

https://github.com/jmettraux/rufus-scheduler#lockfile--mylockfiletxt

"This is useful in environments where the Ruby process holding the scheduler gets started multiple times."

Try this:

scheduler = Rufus::Scheduler.singleton(:lockfile => ".rufus-scheduler.lock")

scheduler.every '1h' do
 JobName.perform_now
end

Answer 2

You need a distributed lock since ECS instances don't share files, the most common ones being Zookeeper, Consul and Redis.

Below an example with Zookeeper, from the docs:

class ZookeptScheduler < Rufus::Scheduler

  def initialize(zookeeper, opts={})
    @zk = zookeeper
    super(opts)
  end

  def lock
    @zk_locker = @zk.exclusive_locker('scheduler')
    @zk_locker.lock # returns true if the lock was acquired, false else
  end

  def unlock
    @zk_locker.unlock
  end

  def confirm_lock
    return false if down?
    @zk_locker.assert!
  rescue ZK::Exceptions::LockAssertionFailedError => e
    # we've lost the lock, shutdown (and return false to at least prevent
    # this job from triggering
    shutdown
    false
  end
end

You could maybe use EFS to share a lockfile, but this is not the correct way.

Answer 3

Start only the scheduler on instance A.