Running Slurm array jobs one per virtual core instead of one per physical core

Question

we have a machine with 2x64-core CPU, each core consists of 2 virtual cores, so in htop we see 256 distinct (virtual) CPUs. We configured Slurm quality of service to better manage CPU usage per user. I.e. we have defined a --qos=cpus50 which, as far as I understand it, gives me a budget of 50 virtual cores to compute my jobs. I created a test.sbatch script with an array of 100 jobs. Each job takes 10s to compute. So with the following config, I would hope that my jobs will be finished in 20s + some small overhead.

#!/bin/bash
#SBATCH --job-name=test
#SBATCH --qos=cpus50
#SBATCH --array=1-100
#SBATCH --cpus-per-task=1
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --ntasks-per-core=1
#SBATCH --open-mode=append
#SBATCH --output=%x.out

python3 -c "import os; jobid = int(os.getenv('SLURM_ARRAY_TASK_ID')); start = f'JOBID:{jobid:04d} | Start {time.ctime()}'; time.sleep(10); print(f'{start} | End {time.ctime()} |')"

However,running the script above spawns only 25 jobs at once (according to squeue output) and finishes in 47seconds. (2x the desired duration). Running with --ntasks-per-core=2 results in the same behavior. Running with --ntasks=2 and --ntasks-per-core=2 results in the same behavior.

What am I doing wrong? I just want to run 50 jobs at once since I already have the virtual cores available. Thank you

Answer 1

Answering my own question. A member of our group found an answer here. The problem was in Slurm configuration. In short, for our setup, we had to change the relevant part in slurm.conf from

SelectTypeParameters=CR_Core
NodeName=nodename CPUs=256 Sockets=2 CoresPerSocket=64 ThreadsPerCore=2

to

SelectTypeParameters=CR_CPU
NodeName=nodename CPUs=256

Now the sbatch script from the question spawns 50 jobs and finishes in a bit more than 20s as expected.