I created a nf-core project using nf-core create. I am trying to invoke python scripts using nf-core processes. This technique worked in my nextflow project as I was able to avoid having to write code in a process. Most of the imports are failing.
import sys --> works
import pymongo --> fails
from pymongo.mongo_client import MongoClient --> fails
From the terminal I can see the process is trying to invoke:
nf.prefetch_http.py SRR9686066 "$PWD/"SRR9686066".sra" "mongodb://xxxx:[email protected]:10255/?ssl=true&retrywrites=false&replicaSet=globaldb&maxIdleTimeMS=120000&appName=@xxxx@"
This results in error:
*File "/home/q/nf-core-pass/bin/mongodb_logger.py", line 1, in <module>
from google.cloud import secretmanager
ModuleNotFoundError: No module named 'google'*
However, if I run directly:
/home/q/nf-core-xxx/bin/nf.prefetch_http.py SRR9686066 "$PWD/"SRR9686066".sra" "mongodb://xxxx:[email protected]:10255/?ssl=true&retrywrites=false&replicaSet=globaldb&maxIdleTimeMS=120000&appName=@xxxx@"
This works proving the import(s) are installed.
If I move the imports to nf.prefetch_http.py, the failure simply moves to nf.prefetch_http.py.
nf-core-xxx/bin/nf.prefetch_http.py
#!/usr/bin/env python3
import sys
import mongodb_logger
import prefetch_http
sra = sys.argv[1]
sraPath = sys.argv[2]
mongodb_connection_string = sys.argv[3]
print(f"nf.prefetch_http: sra: {sra}")
print(f"nf.prefetch_http: sraPath: {sraPath}")
print(f"nf.prefetch_http: mongodb_connection_string: {mongodb_connection_string}")
mongodb_logger.initialize(None,mongodb_connection_string)
prefetch_http.prefetch(sra, sraPath)
nf-core-xxx/bin/prefetch_http.py
from google.cloud import secretmanager
from pymongo.mongo_client import MongoClient
from pymongo.server_api import ServerApi
import pymongo
import os
import json
import response
import globals
import jsons
import mongodb_logger
import pandas as pd
from dataclasses import dataclass
from typing import List
from typing import Optional, List
from datetime import datetime
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import globals
mongodb_connection_string = "NULL"
...
Is there a configuration in nf-core required to make python work correctly?