Snakemake: issues decompressing files in one rule and processing in another. Any solutions or guidance?

Question

I am currently facing an issue with Snakemake, and I hope someone could help me resolve it. I've searched the internet for a solution to a similar problem, but I haven't found anything that solves my specific issue. I am not an advanced user of Snakemake, and I would like to know if there is a built-in solution in Snakemake for this type of problem. If I could get an explanation of why and how, that would be really great.

Here are the details of my situation:

I would like my Snakemake code to simply allow me to decompress a file in one rule and retrieve the decompressed file in another rule to perform a simple processing task. Here is a snippet of code that reproduces the error I am facing (part 1). I improvised a solution to make my code work (part 2 of the code provided below). Not satisfied with this solution, which involves combining the file decompression with the data processing in the same rule (in this case, using the R language), I am seeking guidance on how to make option 1 functional. Are there options in Snakemake that allow me to separate these two rules as specified in the so-called part 1 of the following code ?

Many thanks for your help

# Import necessary modules
import os
import numpy as np
import pandas as pd
import zipfile
import shutil

# Create necessary directories and files
os.makedirs("path/to/my/file/", exist_ok=True)
os.makedirs("path/to/my/compressed", exist_ok=True)
os.makedirs("path/to/my/uncompressed", exist_ok=True)

# Generate random data and create a Pandas DataFrame
x = np.random.rand(3, 2)
df = pd.DataFrame(data=x.astype(float))
df.to_csv("path/to/my/file/data.csv")

# Compress the CSV file into a zip file
with zipfile.ZipFile('path/to/my/compressed/myfile.zip', 'w') as z:
    z.write("path/to/my/file/data.csv", "data.csv")

# Main rule (rule all) specifying expected output files
rule all:
    input:
        i1='path/to/my/uncompressed',
        i2='path/to/my/Routput/data.csv'

# Part 1: Set of rules causing an error
# The first rule unzips a file from the compressed directory and stores it in the uncompressed directory.
# The second rule reads the unzipped file in R and rewrites it in the Routput directory.
# This way of specifying the rule does not work and produces an error: MissingInputException at line 37 of the Snakefile.
rule uncompress:
    input:
        'path/to/my/compressed/myfile.zip'
    output:
        directory('path/to/my/uncompressed')
    shell:
        """
            unzip {input} -d {output}
        """

rule load_data:
    input:
        'path/to/my/uncompressed/data.csv'
    output:
        'path/to/my/Routput/data.csv'
    shell:
        """
            Rscript -e "x={input}; y={output}; X=read.csv(x, header=T, sep=','); write.csv(X, y)"
        """

# Part 2: Functional solution but needs to remove the second rule and embed the logic of the second rule within the first rule
# Comment part 1 and uncomment part 2 to execute.
# Data loading is declared with the former first rule. This way works fine without any error, but I want to avoid it.
# rule uncompress:
#     input:
#         'path/to/my/compressed/myfile.zip'
#     output:
#         o1=directory('path/to/my/uncompressed'),
#         o2='path/to/my/Routput/data.csv'
#     params:
#         p1='path/to/my/uncompressed/data.csv'
#     shell:
#         """
#             unzip {input} -d {output.o1}
#             Rscript -e "x='{params.p1}' ;y= '{output.o2}'; X=read.csv(x, header=T, sep=','); write.csv(X, file=y)"
#         """

Snakemake Version: 5.10.0

Python Version: 3.8.10

Execution Environment:

Linux distribution: Description: Ubuntu 20.04.6 LTS

Release: 20.04 Codename: focal

Answer 1

You want rule uncompress to produce data.csv since this is what later rule(s) are going to use. So add data.csv as an output file. To avoid hard-coding the path to data.csv twice in rule uncompress you can extract the directory name from the path of data.csv. E.g.:

import os

rule uncompress:
    input:
        'path/to/my/compressed/myfile.zip'
    output:
        csv='path/to/my/uncompressed/data.csv',
    params:
        d=lambda wc, output: os.path.dirname(output.csv),
    shell:
        r"""
        unzip -o {input} -d {params.d}
        """