I want to launch a large number of jobs using an array. My sbatch script contains the following option: #SBATCH --array=0-10000%280 (each job takes 1 CPU). I have 280 CPUs available in my partition. My idea is to run these calculations in the background, as they are not a priority, to avoid interfering with other users who typically launch single calculations but with 128 CPUs.
My plan was to use the option #SBATCH --nice=10 on my array. I want that when a user with a single calculation requiring 128 CPUs enters the queue, the current batch of my array completes, allowing the large calculation to proceed, and then resumes the next batch once the large calculation is finished.
After conducting a test, it doesn't work as expected in practice because sbatch restarts small calculations as soon as there is available CPU, and the large calculation never gets through because there are never 128 free CPUs.
Is there a way to let the large calculation pass between two batches of an array?