Snakemake: Solving dependencies between modules

204 Views Asked by At

I'm using Snakemake and tried to order my overall pipeline structure by utilizing modules as the modularization approach suggests. However, one of my modules has a rule which depends on the ouput of the other module and since both modules have their own namespace, the outputs and inputs are not connected (atleast so it seems to me). I tried a lot of different things, but did not find a solution yet.

So, my question in clear: Is there in Snakemake a way to utilize modules, whereby the respective rules can (dynamically) rely on the output of another module's rule?

Here's a minimal code to illustrate my situation:

Snakemake main

# module1
module module1:
    snakefile:  "modules/module1/Snakefile"
    prefix:     "output/module1"

use rule * from module1 as module1_*


# module2
module module2:
    snakefile:  "modules/module2/Snakefile"
    prefix:     "output/module2"

use rule * from module2 as module2_*


# ======== Main rule ===========
rule all:
    input:
        rules.module1_all.input,
        rules.module2_all.input     
    default_target: True    # Makes this rule the default rule

Module 1

rule create_txt:
    output:
        "output/test.txt"
    shell:
        "touch {output}"

rule all:
    input:
        "output/test.txt"

Module 2

rule create_txt2:
    input:
        rules.module1_create_txt.output
    output:
        "output/test2.txt"
    shell:
        "touch {output}"

rule all:
    input:
        "output/test2.txt"

If I then run the pipeline, this is the output (I dropped the following exceptions, since I don't think this would help...)

(snakemake) Olivers-MacBook-Pro-5:test_snakemake oliverkuchler$ snakemake -nr
Building DAG of jobs...
Traceback (most recent call last):
  File "/Users/oliverkuchler/opt/miniconda3/envs/snakemake/lib/python3.10/site-packages/snakemake/dag.py", line 1808, in collect_potential_dependencies
    yield PotentialDependency(file, known_producers[file], True)
KeyError: 'output/module2/output/module1/output/test.txt'

So, as you can see, Snakemake accepts referencing rules from the other module. However, this leads to the application of multiple prefixes to the input-file 'output/module2/output/module1/output/test.txt'. But even if I would solve this issue, still module2 would not be able to relate to rules from module1. Are there any solutions, that I do not see? Happy about any suggestions :)

0

There are 0 best solutions below