Nextflow unable to create pugins dir

Question

I'm trying to run the nf-core RNAseq pipeline on my cluster (which uses SLURM) but I keep running into this error:

[33mWarning: Couldn't create cached classpath folder: /home/irvinng/.nextflow/tmp/launcher/nextflow-one_23.10.1/se439 -- Maybe NXF_HOME is not writable?(B[m curl: (28) Failed to connect to www.nextflow.io port 443 after 300967 ms: Timeout was reached N E X T F L O W  ~  version 23.10.1 ERROR ~ Unable to create plugins dir: /home/irvinng/.nextflow/plugins   -- Check '.nextflow.log' file for details 

and the nextflow.log shows the following

Mar-10 11:22:32.695 [main] DEBUG nextflow.cli.Launcher - $> nextflow run nf-core/rnaseq -r 3.14.0 -name RNAseq -profile singularity -params-file nf-core_RNAseq/nf-params.json -c nf-core_RNAseq/nextflow.config
Mar-10 11:23:53.787 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 23.10.1
Mar-10 11:23:53.833 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/irvinng/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Mar-10 11:23:53.836 [main] ERROR nextflow.cli.Launcher - Unable to create plugins dir: /home/irvinng/.nextflow/plugins
java.io.IOException: Unable to create plugins dir: /home/irvinng/.nextflow/plugins
    at nextflow.plugin.PluginsFacade.init(PluginsFacade.groovy:195)
    at nextflow.plugin.Plugins.init(Plugins.groovy:41)
    at nextflow.plugin.Plugins.init(Plugins.groovy)
    at nextflow.cli.CmdRun.run(CmdRun.groovy:314)
    at nextflow.cli.Launcher.run(Launcher.groovy:500)
    at nextflow.cli.Launcher.main(Launcher.groovy:672)

I tried doing several things:

1. I thought that maybe I had to change permissions in the .nextflow folder, so I went to /home/irvinng/.nexflow and then did chmod u+w+x+r. That didn't fix the problem and I was still getting the same error.

2. I then tried to get nextflow to write the plugin to a different location by adding a line in my job script before the nextflow line

   #!/bin/bash
   
   #SBATCH --time=10:00:00              # Maximum execution time (HH:MM:SS)
   #SBATCH --nodes=1                    # Number of nodes
   #SBATCH --ntasks=1                   # Number of tasks (MPI processes)
   #SBATCH --cpus-per-task=32           # Number of CPU cores per task
   #SBATCH --mem=300G                   # Total memory allocated (in GB)
   #SBATCH --mail-type=ALL              # Send email on job start, end, and abort
   
   
   ################################################################################
   # Load modules
   source /path/to/conda.sh
   conda activate /path/to/nf-core
   ################################################################################
   
   module load apptainer
   
   NXF_PLUGINS_DIR=nf-core_RNAseq/.nextflow/plugins
   nextflow run nf-core/rnaseq -r 3.14.0 -name RNAseq -profile singularity -params-file nf-params.json -c nextflow.config
   
   mamba deactivate
   

   But that didn't work. Nextflow kept trying to write to /home/irvinng/.nextflow/plugins

3. I then tried reinstalling nextflow to my nf-core_RNAseq directory because previously it was installed in my /home/irvinng/bin directory. That still didnt' work and I still kept getting the exact same message.

Answer 1

Don't sbatch your nextflow main process, just run it as your own user on the headnode and let nextflow handle submitting jobs to SLURM.