How to use parallel for loop with scipy L-BFGS-B minimzer in python?

53 Views Asked by At

I am trying to do inversion using scipy L-BFGS-B optimizer which uses Pool inside for forward modelling function. In my forward modelling function I have a for loop of a function running over subset of arrays. Each for loop takes 7 min so I applied Pooling. Now because scipy is also using pooling so my code is unable to run. What can I do? With using threadpool code runs for 1 forward modeling and then starts to give memory error as it starts taking memory in GB. Can some one please help me to make the code run faster such that forward modeling also uses parallel coding in python and scipy also uses Pool?

I tried threadpool and pooling both did not work.

R = 5
C = 10
def rosenbrock(x, i):
    return sum(100.0 * (x[1:] - x[:-1]*2)2 + (1 - x[:-1])*2)

def poolexecutor(Vec):
    # with Pool(cpu_count()) as pool:
    #     res = sum(pool.starmap(rosenbrock, [(Vec[i:i+C], i) for i in range(0, RC, C)]))
    res = sum(Parallel(n_jobs=cpu_count())(delayed(rosenbrock)(Vec[i:i+C], i) for i in range(0, RC, C)))
    return res
x0 = np.random.random(size=(R,C))
res = minimize(poolexecutor, x0.flatten(), options={"disp": True, "maxiter": 5})
0

There are 0 best solutions below