I am new to queueing systems. I am using resque with my current app and I have three queues to monitor. I wonder what is the best way to run and manage resque jobs. The reason for the question is that I am slightly confused with so many options suggested by developers on the internet. More specifically my confusion is around these two strategies.
why do I need resque pool if I can run workers with god alone?
On
Resque was designed to handle background jobs, basically to handle jobs which stays in queue for shorter time.
It came into the picture as a replacement for Delayed Job. When your background jobs are increasing you should opt for Resque.
Now before talking about monitoring I would talk a little about how does Resque works.
Resque has processes which will take out the jobs and execute them, worker takes the job and fork a child which will perform the job and exits. So typically one fork per job is what Resque provide by default.
Now there is a vast variety of plugins are present to enhance the functionality of Resque. For Example: Resque Pool,
Why we need it and when: People have their requirement like one worker multiple jobs per fork. People want these things so these plugins are help full.
You can follow given links Which were help full to me: Redis Persistence: http://redis.io/topics/persistence
How Resque works with Redis : http://girders.org/blog/2011/10/30/how-queuing-with-resque-works/
Resque-web: https://github.com/resque/resque-web
Tutorial : http://www.sitepoint.com/simple-organized-queueing-with-resque/
Multiple redis servers : What's the Point of Multiple Redis Databases?
redis-databases
Resque-scheduler : https://github.com/resque/resque-scheduler
Resque-pool for managing workers : https://github.com/nevans/resque-pool
Resque-multi-jobs-fork : https://github.com/stulentsev/resque-multi-job-forks
Deploy in production: How to deploy resque workers in production? deploy-resque-workers-in-production?rq=1
Running jobs are usually asynchronous tasks. It means you dont wait for them to finish to keep going with your request or whatever called the job.
Because jobs nature is asynchronous, it may mean that at some point you are requesting several jobs at one time. Due to CPU restrictions, these jobs will either find a busy cpu and wait or fail.
A resque pool is a way to keep them in line, performed one at a time. If they fail, they go to a retry mode sometimes, etc.
God is a daemon monitoring tool. It will monitor the jobs 'server' to allow accepting jobs, but not necessarily handle failures, or handle jobs in the order they came in, etc.
A good tool that tries to encapsulate all these into one is sidekiq. There is also a railscast about it that explains well how to work with it (even though it may be a bit old): http://railscasts.com/episodes/366-sidekiq