Get Objective Function values in each iteration while using lmfit.minimize

Question

I am trying to use lmfit.minimize to minimize an objective function (which is a sum of squared error). I can use lmfit.minimize(function,params,args) to do it and it returns a Minimizer object with fit statistics. However, I want to see how the objective function value changes in each iteration. I could print values in each iteration or plot individual residual values using an iteration callback function but I want to get the values of objective function as an array that I can later use/plot. How can it be done ?

I am trying to get something like this:

Answer 1

Here's what you can do with the iterative callback function (using the example from the lmfit homepage):

from lmfit import minimize, Parameters
import numpy as np

def myfunc(x, amp, phase, freq, decay):
    return amp * np.sin(x*freq + phase) * np.exp(-x*x*decay)

def residual(params, x, data, uncertainty, *args, **kws):
    """Model a decaying sine wave and subtract data."""

    model = myfunc(x, params['amp'], params['phase'], params['freq'], params['decay'])

    return (data-model) / uncertainty

x = np.linspace(0, 100)
noise = np.random.normal(size=x.size, scale=0.2)
data = myfunc(x, 7.5, 2.5, 0.22, 0.01) + noise

# generate experimental uncertainties
uncertainty = np.abs(0.16 + np.random.normal(size=x.size, scale=0.05))

def callback(params, iter, resid, *args, **kws):
    itervalues = kws['itervalues']
    itervalues.append((resid**2).sum())

params = Parameters()
params.add('amp', value=10)
params.add('decay', value=0.007)
params.add('phase', value=0.2)
params.add('freq', value=3.0)

itervalues = []
result = minimize(residual, params, args=(x, data, uncertainty), iter_cb=callback, kws={'itervalues': itervalues})

and itervalues will have the values you're looking for, in progressive order. (There's often a high number at the start, when the algorithm is trying out different directions.)

This uses the mutability of Python lists, since that's the only way callback can change itervalues; its return value is used for something different, so that can't be used to change itervalues.