Deep Dream Code does not generate recognizable patterns

Question

I've tried to create my own Deep Dream Algorithm with this Code:

import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np
import inception

img = np.random.rand(1,500,500,3)
net = inception.get_inception_model()
tf.import_graph_def(net['graph_def'], name='inception')
graph = tf.get_default_graph()
sess = tf.Session()
layer = graph.get_tensor_by_name('inception/mixed5b_pool_reduce_pre_relu:0')
gradient = tf.gradients(tf.reduce_mean(layer), graph.get_tensor_by_name('inception/input:0'))
softmax = sess.graph.get_tensor_by_name('inception/softmax2:0')
iters = 100
init = tf.global_variables_initializer()

sess.run(init)
for i in range(iters):
    prediction = sess.run(softmax, \
                          {'inception/input:0': img})
    grad = sess.run(gradient[0], \
                          {'inception/input:0': img})
    grad = (grad-np.mean(grad))/np.std(grad)
    img = grad
    plt.imshow(img[0])
    plt.savefig('output/'+str(i+1)+'.png')
    plt.close('all')

But even after running this loop for 100 iterations the resulting picture still looks random (I will attach said picture to this Question). Can someone please help me to optimize my code?

Answer 1

Using the Inception network for Deep Dream is a bit fiddly. On the CADL course that you have borrowed the helper library from, the instructor chooses to use VGG16 as the instruction network instead. If you use this and make a few small modifications to your code, you should get something that works (if you swap in the Inception network here it will kind of work, but results will look even more disappointing):

import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np
import vgg16 as vgg

# Note reduced range of image, your noise function was drowning
# out the few textures that you were getting
img = np.random.rand(1,500,500,3) * 0.1 + 0.45
net = vgg.get_vgg_model()
tf.import_graph_def(net['graph_def'], name='vgg')
graph = tf.get_default_graph()
sess = tf.Session()
layer = graph.get_tensor_by_name('vgg/pool4:0')
gradient = tf.gradients(tf.reduce_mean(layer),
   graph.get_tensor_by_name('vgg/images:0'))

# You don't need to define or use the softmax layer - TensorFlow
# is smart enough to resolve the computation graph for gradients 
# without explicitly running the whole network forward first
iters = 100
# You don't need to init the network variables, everything you need 
# is set by the import, plus the placeholder.

for i in range(iters):
    grad = sess.run(gradient[0], {'vgg/images:0': img})

    # You can use all sorts of normalisation, this one is from CADL
    grad /= (np.max(np.abs(grad))+1e-7)

    # You forgot to use += here, and it is best to use a 
    # step size even after gradient normalisation
    img += 0.25 * grad
    # Re-normalise the image, to prevent over-saturation
    img = 0.98 * (img - 0.5) + 0.5
    img = np.clip(img, 0.0, 1.0)
    plt.imshow(img[0])
    plt.savefig('output/'+str(i+1)+'.png')
    plt.close('all')
    print(i)

Doing all this gets images that are clearly working, but still need some refinement:

To get better, full colour images of the type you may have seen online requires more changes. For instance you could re-normalise or blur the image slightly between each iteration.

If you want to get more sophisticated, you could try the TensorFlow Jupyter notebook walk-through, although it is somewhat harder to understand from first principles due to combining multiple ideas.