How to properly set up heavy work in concurrency?

135 Views Asked by At

There are 1 million customers. I need to send them in batches to a third-party service, get a response and process. the same code for 500_000 customers works.

thread_pool = Concurrent::FixedThreadPool.new(100)

executors = customers.each_slice(150).map do |batch|
  Concurrent::Future.execute(executor: thread_pool) do
    # hard work
  end
end

This code does not work, crashes or with an error can't create Thread: Resource temporarily unavailable. Or crashes with an error 137 (by memory) from k8s, when I work in the terminal.

How can I optimize my work with concurrent?

0

There are 0 best solutions below