For reference I am using Sun Grid Engine to run my snakemake pipeline.
A typical line to initiate would be:
snakemake --cluster "qsub -l mfree=8G -l h_rt=72:0:0" --jobs 20
This will submit 20 jobs simultaneously, where each job contains one rule and is sent to one slot. However, my cluster node topology enables SMT, where each slot contains a CPU with 2 threads. So my goal is to utilize both of these threads by running an independent snakemake rule on each thread. (The rules themselves only use one thread at a time)
The snakemake log files for each of these jobs also recognize "Provided cores: 2" - but since only one rule is scheduled per job, only one of these available cores is used at a time.
For additional context, when running this exact snakemake pipeline in a qlogin environment where I first set up an interactive session with 20 slots (40 threads) - snakemake recognizes that there are "Provided cores: 40" - and thus 40 rules are able to be run at one time.
So to reiterate, is there a way to send two rules at a time for each job so I can maximize the available resources on my cluster topology?
Thank you
I have tried specifying --cores 2, but this has the opposite functionality of what I want. This would request 2 slots (4 threads) per each job/rule and just exemplifies the problem.