I have a Rails project that needs to scrape the web every x hours. For that, I am using rufus-scheduler. When the job is running, however, the API slows way down, from under 1000ms to 30s or more. Checking the logs, Activerecord is not the cause of the extra latency. I can see that the passenger thread is using 100% of my CPU.
In another Rails project, I'm running an external script (youtube-dl) from rufus scheduler. When the script is running, the API blocks entirely sometimes (but other times is unphased).
In both cases, I am using Rails 6, Ruby 2.6.4, and Debian 9. I was thinking of installing sidekiq and having Rufus call it, so that my scheduled jobs would run in a separate process. Is there something I should be doing instead?