Getting snakemake environment variables local to a compute node in cluster execution

271 Views Asked by At

Is there a way to lazy evaluate variables in the shell directive of snakemake?

Our PBS computing cluster creates a node-local scratch directory for every submitted job and sets the value of $TMPDIR to the path to this directory during the execution. After the job ends, this temporary scratch directory gets deleted. Since snakemake is creating the jobscripts (which are later qsub'ed to PBS) on the login node and at a point in time where the temporary scratch is not even created, I'm unable to use the scratch directory for efficient job grouping. This way I have to read and write the files back and forth on the NFS.

I tried using shadow and setting the shadow-dir to $TMPDIR, but again, the correct value is only set after the jobscript got parsed by PBS.

Update: workaround https://github.com/snakemake/snakemake/issues/522

1

There are 1 best solutions below

1
j0hn On

To the best of my knowledge, the answer is "no".

But if you were to give this feature request some love by commenting or upvoting then we might be able to get the attention of the devs a little sooner.