How to change dask job_name to SGECluster

Question

I am using dask_jobqueue.SGECluster() and when I submit jobs to the grid they are all listed as dask-worker. I want to have different names for each submitted job.

Here is one example:

futures = []
for i in range(1,10):
    res = client.submit(slow_pow, i,2)
    futures.append(res)
    
[future.result() for future in as_completed(futures)]

All 10 jobs appear with name dask-worker when checking their status with qsub.

I have tried adding client.adapt(job_name=f'job{i}') within the loop, but no success, name still dask-worker.

Any hints?

Answer 1

The dask-worker is a generic name for the compute allocation from the cluster, it can be changed by providing cluster-specific arguments at the time the cluster is created. For example, for SLURMCluster this would be:

cluster = SLURMCluster('job_extra': ['--job-name="func"'])

SGECluster might have a different syntax.

The actual tasks submitted to dask scheduler will have their names generated automatically by dask and can be viewed through the dashboard, by default on (http://localhost:8787/status).

It's possible to specify a custom name for each task submitted to scheduler by using key kwarg:

fut = client.submit(myfunc, my_arg, key='custom_key')

Note that if you are submitting multiple futures, you will want them to have unique keys:

futs = [client.submit(myfunc, i, key=f'custom_key_{i}') for i in range(3)]