A part of the code I am working with uses a ProcessPoolExecutor as below.
with ProcessPoolExecutor() as executor:
index = list(range(0,len(identifiers)))
results = executor.map(main_exec, *args, index)
The *args (per identifier) are preprocessed and then main_exec is called. I need to be able to run multiple variations of *args. The code is set up to save results automatically. Is it advisable to wrap the whole code in a for loop that runs over all variations or is there a better way to do this?
Please let me know if more information is needed but I am unable to expose more than pseudo code.
Here's what I have tried:
def run_main(inputs):
*args = preprocess_input(inputs)
with ProcessPoolExecutor() as executor:
index = list(range(0,len(identifiers)))
results = executor.map(main_exec, *args, index)
variations = [...variations...]
for variation in variations:
run_main(variation)