Dropout cost becomes nan after 1500 iteration

Question

I was trying to implement the dropout model in the format of Andrew-NG deep learning course-1 week-4 format. Using data from course-2 week-1.

• “dropout_project.ipynb” is the main project file where when I ran it after 1500 iterations, the cost become nan
• “deep_nn.py” and “dropout_and_regularization.py” are the helper function file.
• I had tested my implementation for all the bugs
• And I have also one doubt, does the “d” variable change every iteration or fixed constant for every iteration. In my implementation I have kept the value of d1 and d2 to be fixed by recalling np.random.seed(1) at the start of the iteration.

Please someone help me Github Database Link

    def dropout_model(x, y,keep_probs = 0.86, learning_rate = 0.3, numm_iterations = 30000, print_cost = True):
    # 14% neurons will be dropped
    grads = {}
    costs = []
    m = x.shape[1]
    layer_dims = [x.shape[0], 20, 3,1]

    parameters = nn.initialize_paraeters_deep(layer_dims)


    for i in range(numm_iterations):
        np.random.seed(1)
        aL, caches, d_list = dr.dropout_L_model_forward(x, parameters ,keep_probs)
        cost = nn.cross_entropy_cost(aL, y)
        grads = dr.dropout_L_model_backward(aL, y , caches, d_list, keep_probs)
        parameters = nn.update_parameters(parameters, grads, learning_rate)

        if i % 1000 == 0:
            costs.append(cost)
            if print_cost:
                print(f'Cost after iteration {i} is {cost}')


    plt.plot(costs)
    plt.ylabel("Cost")
    plt.xlabel("iteration (x1000)")
    plt.title(f"Learning rate : {learning_rate}")
    plt.show()

    return parameters, costs

        dropout_parameters, dropout_cost = dropout_model(x_train, y_train)
    _ = nn.predict(x_train, y_train, dropout_parameters)
    _ = nn.predict(x_test, y_test, dropout_parameters, "Test")

Answer 1

There are two reasons for the cost going NaN.

• aL going 0
• aL going 1

which gives error in both cost and daL

• In above code, aL max value = 1 after 700 iterations and after 1500 iterations aL min value = 0
• Min-Max analysis of aL

• By adding this code after forward propagation makes sure the aL never becomes 0 and 1

    aL[aL == 0] = 1e-10
    aL[aL == 1] = 1 - 1e-10