Running parallel with changing inputs

21 Views Asked by At

A part of the code I am working with uses a ProcessPoolExecutor as below.

with ProcessPoolExecutor() as executor:
                    index = list(range(0,len(identifiers)))
              
                    results = executor.map(main_exec, *args, index) 

The *args (per identifier) are preprocessed and then main_exec is called. I need to be able to run multiple variations of *args. The code is set up to save results automatically. Is it advisable to wrap the whole code in a for loop that runs over all variations or is there a better way to do this?

Please let me know if more information is needed but I am unable to expose more than pseudo code.

Here's what I have tried:

def run_main(inputs):

    *args = preprocess_input(inputs)

    with ProcessPoolExecutor() as executor:
                    index = list(range(0,len(identifiers)))
              
                    results = executor.map(main_exec, *args, index) 

variations = [...variations...]
for variation in variations:
    run_main(variation)
0

There are 0 best solutions below