How are gradients calculated for DeepDream

Question

I have a very simple implementation of Deep Dream algorithm inspired by kadenze example https://github.com/pkmital/CADL/blob/master/session-4/lecture-4.ipynb:

layer = graph.get_tensor_by_name('inception/output2:0')
layer_size = layer.eval({x: img_4d}).shape

neuron_i = 110

layer_activation = np.zeros(layer_size)
layer_activation[..., neuron_i] = 1

grad = tf.gradients(layer[..., neuron_i], x)[0]

img_noise_4d = img_noise.copy()[np.newaxis]
img_noise_4d /=  255.0

grad_step = 4.0
n_steps = 100
for step in range(n_steps):
    print(step, end=', ')
    res = grad.eval({layer: layer_activation, x: img_noise_4d})
    res /= (np.max(np.abs(res)) + 1e-8)
    img_noise_4d += res * grad_step

plt.imshow(normalize(img_noise_4d[0]))

What I can't understand is how it works - I mean how can we replace actual layer activation with one which we generated (layer_activation) and get a correct gradient?

I made a simple experiment:

x = tf.Variable(3.0)
y = x**2

session = tf.InteractiveSession()
session.run(tf.global_variables_initializer())

session.run(tf.gradients(y, x)[0], {y: 100})

Whatever I substitute as y - I always get correct gradient for x in point 3.0 which is 6.0. I understand that I'm missing something, but what exactly?

Answer 1

I think I can answer my question now - it turns out I had a bad example.

This answer better shows how this works:

x = tf.Variable(3.0)
w = tf.Variable([6.0, 2.0])
y = x * w

session = tf.InteractiveSession()
session.run(tf.global_variables_initializer())

session.run(
    tf.gradients(tf.reduce_max(y), x)[0], 
    {y: [3, 9],
})

So basically by passing custom y to session.run we can suggest to back propagation algorithm which 'neuron' we expect to be the max one - and thus it will calculate gradient w.r.t. not actual one (y[0]), but to custom one (y[1]).

We can do even simpler if we know what particular neuron we are interested in:

session.run(
    tf.gradients(y[1], x)[0], 
)