best way to use node-worker-threads-pool with for loop

Question

I'm using node-worker-threads-pool to process a daily function on all the documents from one collection, for that I'm using a for loop to send the id of each document to the pool, I'm trying to get the time that it takes to process the whole process but it consoles before the execution of all the id's, so I have two questions, is better to run the for loop inside of the worker file so the pool.exec() function is just called once?, and how can I log the time with the actual code?. Here is my code:

Client.find({}, {_id: 1}).then(clients => {
        console.time('clientDayChange');

        const pool = new StaticPool({
            size: 8,
            task: "./worker.js"
        });
        
        for(let client of clients){
            let id = JSON.stringify(client._id);
            pool.exec(id);
        }
        
        console.timeEnd('clientDayChange');
   })

Answer 1

In your existing code, console.timeEnd('clientDayChange'); is executed before all the worker tasks are completed because pool.exec(id) is asynchronous. You have two options to get the accurate time it takes to process all the tasks:

Option 1: Run the loop inside the worker

If running the for-loop inside the worker file is possible and does not create any issue with the logic of your application, then it is a good approach. By doing so, you only invoke pool.exec() once, which can be efficient. Here's how you can do it:

In your worker.js:

// Existing worker logic
module.exports = async function worker(ids) {
  for(let id of ids) {
    // Process each id
  }
}

In your main file:

Client.find({}, {_id: 1}).then(clients => {
  console.time('clientDayChange');

  const pool = new StaticPool({
    size: 8,
    task: "./worker.js"
  });

  let ids = clients.map(client => client._id);
  pool.exec(ids).then(() => {
    console.timeEnd('clientDayChange');
  });
});

Option 2: Use Promise.all to wait for all tasks to finish

If you want to keep the for-loop in the main file, you can use Promise.all to wait for all the tasks to complete before you stop the timer. Here's how:

Client.find({}, {_id: 1}).then(async clients => {
  console.time('clientDayChange');

  const pool = new StaticPool({
    size: 8,
    task: "./worker.js"
  });

  const promises = [];
  for(let client of clients){
    let id = JSON.stringify(client._id);
    promises.push(pool.exec(id));
  }

  await Promise.all(promises);
  
  console.timeEnd('clientDayChange');
});