Why are deep neural networks hard to train?

This notebook is inspired by Chapter 5 in Michael Nielsen's book Neural Networks and Deep Learning

• Chapter 5: Why are deep neural networks hard to train?
  The issues we'll study in this chapter include:
  • Introduction
  • The vanishing gradient problem
  • What's causing the vanishing gradient problem?
    • Unstable gadients in deep NNs
  • Unstable gradients in more complex networks
  • Other obstacles to deep learning

---

---

Introduction

---

• Shallow vs. Deep Circuits:

  • Designing a computer circuit with only two layers is theoretically possible using special gates like AND and NAND.
  • However, shallow circuits are impractical for solving complex problems due to design limitations.
  • Deep circuits allow for solving sub-problems through multiple layers of abstraction, making complex designs more manageable.
  • Mathematical proofs show that deep circuits can compute functions like parity efficiently, while shallow circuits require exponentially more gates.

• Neural Networks with a Single Hidden Layer:

  • We can draw parallels between computer circuits and neural networks.
  • Most neural networks discussed so far have a single hidden layer, which has been effective (e.g., achieving 98% accuracy in digit classification) but may not fully leverage the potential of deeper networks.
  • Intuitively, networks with more hidden layers should be more powerful by building multiple layers of abstraction akin to deep circuits.

• Advantages of Deep Networks:

  • Deep networks can use each layer to recognize increasingly complex patterns.
  • For example, early layers might detect simple shapes like edges, while deeper layers recognize more complex structures.
  • Similar to deep circuits, deep neural networks have theoretical advantages in solving complex problems and are intrinsically more powerful than shallow networks.[1]

• Challenges in Training Deep Networks:

  • Training deep networks using techniques like stochastic gradient descent can be difficult due to imbalances across layers.
  • Deep networks often perform no better (or only slightly better) than shallow networks when trained with SGD by backpropagation.

• Learning Speed Disparity in Deep Networks:

  • Layers in deep networks learn at vastly different speeds, with early layers sometimes getting stuck and learning almost nothing.
  • The problem isn't random but related to fundamental issues with gradient-based learning methods.

• Learning Instability:

  • In some cases, the opposite happens: early layers learn well, but later layers get stuck.
  • Gradient descent in deep networks tends to result in either early or later layers getting stuck, creating instability during training.

• Key Takeaway:

  • Understanding these difficulties helps in grasping what's required to train deep networks effectively.
  • This sets up further exploration of how to tackle these challenges in deep learning, particularly for image recognition tasks.
    
    
    
    

---

References:

1. For certain problems and network architectures this is proved in On the number of response regions of deep feed forward networks with piece-wise linear activations, by Razvan Pascanu, Guido Montúfar, and Yoshua Bengio (2014). See also the more informal discussion in section 2 of Learning deep architectures for AI, by Yoshua Bengio (2009).

---

The Vanishing Gradient Problem

---

Introduction

When training deep neural networks, we encounter a significant challenge known as the vanishing gradient problem. This phenomenon occurs when the gradient becomes extremely small as we move backward through the hidden layers during backpropagation. As a result, neurons in the earlier layers learn much more slowly than neurons in later layers, hindering the network's ability to learn effectively.

Experiment Setup

To investigate this problem, we'll use the MNIST digit classification problem as our test case. We'll train networks of varying depths and observe how the learning speed changes across different layers.

Setup Instructions

1. Clone the repository:

   git clone https://github.com/mnielsen/neural-networks-and-deep-learning.git
   

2. Change to the src directory.
3. In a Python shell, load the MNIST data:

   import mnist_loader
   training_data, validation_data, test_data = mnist_loader.load_data_wrapper()
   

Experiments

1. Single Hidden Layer

import network2
net = network2.Network([784, 30, 10])
net.SGD(training_data, 30, 10, 0.1, lmbda=5.0,
        evaluation_data=validation_data, monitor_evaluation_accuracy=True)

This network achieves a classification accuracy of about $96.58\%.$

2. Two Hidden Layers

net = network2.Network([784, 30, 30, 10])
net.SGD(training_data, 30, 10, 0.1, lmbda=5.0,
        evaluation_data=validation_data, monitor_evaluation_accuracy=True)

This network improves the accuracy to about $96.74\%.$

3. Three Hidden Layers

net = network2.Network([784, 30, 30, 30, 10])
net.SGD(training_data, 30, 10, 0.1, lmbda=5.0,
        evaluation_data=validation_data, monitor_evaluation_accuracy=True)

The accuracy drops to $96.82\%.$

4. Four Hidden Layers

net = network2.Network([784, 30, 30, 30, 30, 10])
net.SGD(training_data, 30, 10, 0.1, lmbda=5.0,
        evaluation_data=validation_data, monitor_evaluation_accuracy=True)

The accuracy further drops to $96.35\%.$

Analysis

To understand why deeper networks aren't performing better, we visualize how the network learns by plotting the gradient for each neuron. The gradient is denoted as:

$\delta^l_j = \frac{\partial C}{\partial b^l_j}$

where $\delta^l_j$ is the gradient for the $j$th neuron in the $l$th layer.

We use the length of the gradient vectors $\|\delta^l\|$ as a measure of the learning speed for each layer.

Results

These values represent the speed at the beginning of training

1. 2 hidden layers: $\|\delta^1\| = 0.059$, $\|\delta^2\| = 0.35$
2. 3 hidden layers: $\|\delta^1\| = 0.0228$, $\|\delta^2\| = 0.0804$, $\|\delta^3\| = 0.375$
3. 4 hidden layers: $\|\delta^1\| = 0.0028$, $\|\delta^2\| = 0.0106$, $\|\delta^3\| = 0.0541$, $\|\delta^4\| = 0.3047$

These results show that earlier hidden layers consistently learn much slower than later hidden layers.

Conclusion

The vanishing gradient problem is a fundamental issue in training deep neural networks. As we add more layers, the gradient becomes unstable, tending to either vanish or explode in earlier layers. This instability poses a significant challenge for gradient-based learning in deep neural networks and requires careful consideration when designing and training these networks.

Learning Speed Over Time

We can also observe how the learning speed changes during training. We've been looking at the speed of learning at the start of training, that is, just after the networks are initialized. How does the speed of learning change as we train our networks? Let's return to look at the network with just two hidden layers. The speed of learning changes as follows:

# @title
def learning_speed_plot_during_training(epochs, filename, num_layers):
    f = open(filename, "r")
    norms = json.load(f)
    f.close()
    fig = plt.figure()
    ax = fig.add_subplot(111)
    colors = ["#2A6EA6", "#FFA933", "#FF5555", "#55FF55", "#5555FF"]
    for j in range(num_layers):
        ax.plot(np.arange(epochs),
                [n[j] for n in norms],
                color=colors[j],
                label="Hidden layer %s" % (j+1,))
    ax.set_xlim([0, epochs])
    ax.grid(True)
    ax.set_xlabel('Number of epochs of training')
    ax.set_title('Speed of learning: %s hidden layers' % num_layers)
    ax.set_yscale('log')
    plt.legend(loc="upper right")

learning_speed_plot_during_training(500, "norms_during_training_2_layers.json", 2)

learning_speed_plot_during_training(500, "norms_during_training_3_layers.json", 3)

learning_speed_plot_during_training(500, "norms_during_training_4_layers.json", 4)

These graphs consistently show that early hidden layers learn much more slowly than later hidden layers throughout the training process.

We have here an important observation: in at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden layers. This means that neurons in the earlier layers learn much more slowly than neurons in later layers. And while we've seen this in just a single network, there are fundamental reasons why this happens in many neural networks. The phenomenon is known as the vanishing gradient problem[2].

Why does the vanishing gradient problem occur? Are there ways we can avoid it? And how should we deal with it in training deep neural networks? In fact, we'll learn shortly that it's not inevitable, although the alternative is not very attractive, either: sometimes the gradient gets much larger in earlier layers! This is the exploding gradient problem, and it's not much better news than the vanishing gradient problem. More generally, it turns out that the gradient in deep neural networks is unstable, tending to either explode or vanish in earlier layers. This instability is a fundamental problem for gradient-based learning in deep neural networks. It's something we need to understand, and, if possible, take steps to address.

One response to vanishing (or unstable) gradients is to wonder if they're really such a problem. Momentarily stepping away from neural nets, imagine we were trying to numerically minimize a function $f(x)$ of a single variable. Wouldn't it be good news if the derivative $f'(x)$ was small? Wouldn't that mean we were already near an extremum? In a similar way, might the small gradient in early layers of a deep network mean that we don't need to do much adjustment of the weights and biases?

Of course, this isn't the case. Recall that we randomly initialized the weight and biases in the network. It is extremely unlikely our initial weights and biases will do a good job at whatever it is we want our network to do. To be concrete, consider the first layer of weights in a $[784, 30, 30, 30, 10]$ network for the MNIST problem. The random initialization means the first layer throws away most information about the input image. Even if later layers have been extensively trained, they will still find it extremely difficult to identify the input image, simply because they don't have enough information. And so it can't possibly be the case that not much learning needs to be done in the first layer. If we're going to train deep networks, we need to figure out how to address the vanishing gradient problem.

---

References:
2. See Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, by Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber (2001). This paper studied recurrent neural nets, but the essential phenomenon is the same as in the feedforward networks we are studying. See also Sepp Hochreiter's earlier Diploma Thesis, Untersuchungen zu dynamischen neuronalen Netzen(1991, in German).

# @title
#### Libraries
# Standard library
import json
import random
import sys
import time, timeit

# Third-party libraries
import numpy as np
import pandas as pd
import torch
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns


#### Define the quadratic and cross-entropy cost functions

class QuadraticCost(object):

    @staticmethod
    def fn(a, y):
        """Return the cost associated with an output ``a`` and desired output
        ``y``.

        """
        return 0.5*np.linalg.norm(a-y)**2

    @staticmethod
    def delta(z, a, y):
        """Return the error delta from the output layer."""
        return (a-y) * sigmoid_prime(z)


class CrossEntropyCost(object):

    @staticmethod
    def fn(a, y):
        """Return the cost associated with an output ``a`` and desired output
        ``y``.  Note that np.nan_to_num is used to ensure numerical
        stability.  In particular, if both ``a`` and ``y`` have a 1.0
        in the same slot, then the expression (1-y)*np.log(1-a)
        returns nan.  The np.nan_to_num ensures that that is converted
        to the correct value (0.0).

        """
        return np.sum(np.nan_to_num(-y*np.log(a)-(1-y)*np.log(1-a)))

    @staticmethod
    def delta(z, a, y):
        """Return the error delta from the output layer.  Note that the
        parameter ``z`` is not used by the method.  It is included in
        the method's parameters in order to make the interface
        consistent with the delta method for other cost classes.

        """
        return (a-y)

# @title
"""network2mat.py
~~~~~~~~~~~~~~

An improved version of network2.py, to perform fully matrix-based
backpropagation. This version avoids the loop over each training
example by using matrix operations on the entire mini-batch.

"""

#### Main Network class
class Network2mat(object):

    def __init__(self, sizes, cost=CrossEntropyCost, reg='l2'):
        self.num_layers = len(sizes)
        self.sizes = sizes
        self.default_weight_initializer()
        self.cost = cost
        self.reg = reg

    def default_weight_initializer(self):
        self.biases = [np.random.randn(y, 1) for y in self.sizes[1:]]
        self.weights = [np.random.randn(y, x)/np.sqrt(x)
                        for x, y in zip(self.sizes[:-1], self.sizes[1:])]

    def large_weight_initializer(self):
        self.biases = [np.random.randn(y, 1) for y in self.sizes[1:]]
        self.weights = [np.random.randn(y, x)
                        for x, y in zip(self.sizes[:-1], self.sizes[1:])]

    def feedforward(self, a):
        """Return the output of the network if ``a`` is input."""
        for b, w in zip(self.biases, self.weights):
            a = sigmoid(np.dot(w, a)+b)
        return a

    def SGD(self, training_data, epochs, mini_batch_size, eta,
            lmbda = 0.0,
            evaluation_data=None,
            monitor_evaluation_cost=False,
            monitor_evaluation_accuracy=False,
            monitor_training_cost=False,
            monitor_training_accuracy=False):

        if evaluation_data: n_data = len(list(evaluation_data))
        n = len(list(training_data))
        evaluation_cost, evaluation_accuracy = [], []
        training_cost, training_accuracy = [], []
        for j in range(epochs):
            random.shuffle(training_data)
            mini_batches = [
                training_data[k:k+mini_batch_size]
                for k in range(0, n, mini_batch_size)]
            for mini_batch in mini_batches:
                self.update_mini_batch(
                    mini_batch, eta, lmbda, len(training_data))
            print("Epoch %s training complete" % j)
            if monitor_training_cost:
                cost = self.total_cost(training_data, lmbda)
                training_cost.append(cost)
                print("Cost on training data: {}".format(cost))
            if monitor_training_accuracy:
                accuracy = self.accuracy(training_data, convert=True)
                training_accuracy.append(accuracy)
                print("Accuracy on training data: {} / {}".format(
                    accuracy, n))
            if monitor_evaluation_cost:
                cost = self.total_cost(evaluation_data, lmbda, convert=True)
                evaluation_cost.append(cost)
                print("Cost on evaluation data: {}".format(cost))
            if monitor_evaluation_accuracy:
                accuracy = self.accuracy(evaluation_data)
                evaluation_accuracy.append(accuracy)
                print("Accuracy on evaluation data: {} / {}".format(
                    self.accuracy(evaluation_data), n_data))
        return evaluation_cost, evaluation_accuracy, \
            training_cost, training_accuracy

    def update_mini_batch(self, mini_batch, eta, lmbda, n):
        """Update the network's weights and biases by applying gradient descent
        using backpropagation to a single mini-batch."""

        # Convert the mini-batch into matrix form
        X = np.column_stack([x for x, y in mini_batch])
        Y = np.column_stack([y for x, y in mini_batch])

        # Perform forward and backward passes
        nabla_b, nabla_w = self.backprop_matrix(X, Y)

        # Update weights and biases using gradient descent and L1 or L2 regularization
        if self.reg == 'l2':
          self.weights = [(1 - eta * (lmbda/n)) * w - (eta/len(mini_batch)) * nw
                          for w, nw in zip(self.weights, nabla_w)]
        elif self.reg == 'l1':
          self.weights = [w - (eta * (lmbda/n)) * np.sign(w) - (eta / len(mini_batch)) * nw
                          for w, nw in zip(self.weights, nabla_w)]
        self.biases = [b - (eta / len(mini_batch)) * nb
                      for b, nb in zip(self.biases, nabla_b)]


    def backprop_matrix(self, X, Y):
        """Matrix-based backpropagation for a mini-batch."""

        # Initialize gradients
        nabla_b = [np.zeros(b.shape) for b in self.biases]
        nabla_w = [np.zeros(w.shape) for w in self.weights]

        # Step 1: Forward pass
        activation = X
        activations = [X]  # Store all activations layer by layer
        zs = []  # Store all z vectors layer by layer

        # Feedforward step through all layers
        for b, w in zip(self.biases, self.weights):
            z = np.dot(w, activation) + b
            zs.append(z)
            activation = sigmoid(z)
            activations.append(activation)

        # Step 2: Backward pass

        # Compute delta at the output layer
        delta = self.cost.delta(zs[-1], activations[-1], Y)

        # Gradients for the output layer
        nabla_b[-1] = np.sum(delta, axis=1, keepdims=True)
        nabla_w[-1] = np.dot(delta, activations[-2].T)

        # Backpropagate through previous layers
        for l in range(2, self.num_layers):
            z = zs[-l]
            sp = sigmoid_prime(z)
            delta = np.dot(self.weights[-l+1].T, delta) * sp
            nabla_b[-l] = np.sum(delta, axis=1, keepdims=True)
            nabla_w[-l] = np.dot(delta, activations[-l-1].T)

        return (nabla_b, nabla_w)

    def accuracy(self, data, convert=False):
        if convert:
            results = [(np.argmax(self.feedforward(x)), np.argmax(y))
                       for (x, y) in data]
        else:
            results = [(np.argmax(self.feedforward(x)), y)
                        for (x, y) in data]
        return sum(int(x == y) for (x, y) in results)

    def total_cost(self, data, lmbda, convert=False):
        cost = 0.0
        for x, y in data:
            a = self.feedforward(x)
            if convert: y = vectorized_result(y)
            cost += self.cost.fn(a, y)/len(data)
        if self.reg == 'l2':
          cost += 0.5*(lmbda/len(data))*sum(np.linalg.norm(w)**2 for w in self.weights)
        elif self.reg == 'l1':
          cost += (lmbda/len(data))*sum(np.linalg.norm(w, ord=1) for w in self.weights)
        return cost

    def save(self, filename):
        """Save the neural network to the file ``filename``."""
        data = {"sizes": self.sizes,
                "weights": [w.tolist() for w in self.weights],
                "biases": [b.tolist() for b in self.biases],
                "cost": str(self.cost.__name__)}
        f = open(filename, "w")
        json.dump(data, f)
        f.close()


#### Loading a Network
def load(filename):
    """Load a neural network from the file ``filename``.  Returns an
    instance of Network.

    """
    f = open(filename, "r")
    data = json.load(f)
    f.close()
    cost = getattr(sys.modules[__name__], data["cost"])
    net = Network2mat(data["sizes"], cost=cost)
    net.weights = [np.array(w) for w in data["weights"]]
    net.biases = [np.array(b) for b in data["biases"]]
    return net

#### Miscellaneous functions
def vectorized_result(j):
    """Return a 10-dimensional unit vector with a 1.0 in the j'th position
    and zeroes elsewhere.  This is used to convert a digit (0...9)
    into a corresponding desired output from the neural network.

    """
    e = np.zeros((10, 1))
    e[j] = 1.0
    return e

def sigmoid(z):
    """The sigmoid function."""
    return 1.0/(1.0+np.exp(-z))

def sigmoid_prime(z):
    """Derivative of the sigmoid function."""
    return sigmoid(z)*(1-sigmoid(z))

import mnist_loader

# Load data
training_data, validation_data, test_data = mnist_loader.load_data_wrapper()

# Convert data to lists
training_data = list(training_data)
validation_data = list(validation_data)
test_data = list(test_data)

# Determine device
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f"Using {device} device")

Using cpu device

 %%time
# Initialize and train 1-hidden layer network
net = Network2mat([784, 30, 10])
net.SGD(training_data, 30, 10, 0.1, lmbda=5.0,
        evaluation_data=validation_data, monitor_evaluation_accuracy=True)

Epoch 0 training complete
Accuracy on evaluation data: 9275 / 10000
Epoch 1 training complete
Accuracy on evaluation data: 9396 / 10000
Epoch 2 training complete
Accuracy on evaluation data: 9452 / 10000
Epoch 3 training complete
Accuracy on evaluation data: 9519 / 10000
Epoch 4 training complete
Accuracy on evaluation data: 9545 / 10000
Epoch 5 training complete
Accuracy on evaluation data: 9539 / 10000
Epoch 6 training complete
Accuracy on evaluation data: 9546 / 10000
Epoch 7 training complete
Accuracy on evaluation data: 9568 / 10000
Epoch 8 training complete
Accuracy on evaluation data: 9585 / 10000
Epoch 9 training complete
Accuracy on evaluation data: 9571 / 10000
Epoch 10 training complete
Accuracy on evaluation data: 9609 / 10000
Epoch 11 training complete
Accuracy on evaluation data: 9591 / 10000
Epoch 12 training complete
Accuracy on evaluation data: 9591 / 10000
Epoch 13 training complete
Accuracy on evaluation data: 9598 / 10000
Epoch 14 training complete
Accuracy on evaluation data: 9595 / 10000
Epoch 15 training complete
Accuracy on evaluation data: 9613 / 10000
Epoch 16 training complete
Accuracy on evaluation data: 9631 / 10000
Epoch 17 training complete
Accuracy on evaluation data: 9624 / 10000
Epoch 18 training complete
Accuracy on evaluation data: 9633 / 10000
Epoch 19 training complete
Accuracy on evaluation data: 9623 / 10000
Epoch 20 training complete
Accuracy on evaluation data: 9647 / 10000
Epoch 21 training complete
Accuracy on evaluation data: 9634 / 10000
Epoch 22 training complete
Accuracy on evaluation data: 9631 / 10000
Epoch 23 training complete
Accuracy on evaluation data: 9652 / 10000
Epoch 24 training complete
Accuracy on evaluation data: 9633 / 10000
Epoch 25 training complete
Accuracy on evaluation data: 9651 / 10000
Epoch 26 training complete
Accuracy on evaluation data: 9639 / 10000
Epoch 27 training complete
Accuracy on evaluation data: 9651 / 10000
Epoch 28 training complete
Accuracy on evaluation data: 9650 / 10000
Epoch 29 training complete
Accuracy on evaluation data: 9658 / 10000
CPU times: user 1min 26s, sys: 26.5 s, total: 1min 52s
Wall time: 1min 49s

([],
 [9275,
  9396,
  9452,
  9519,
  9545,
  9539,
  9546,
  9568,
  9585,
  9571,
  9609,
  9591,
  9591,
  9598,
  9595,
  9613,
  9631,
  9624,
  9633,
  9623,
  9647,
  9634,
  9631,
  9652,
  9633,
  9651,
  9639,
  9651,
  9650,
  9658],
 [],
 [])

%%time
# Initialize and train 2-hidden layer network
net = Network2mat([784, 30, 30, 10])
net.SGD(training_data, 30, 10, 0.1, lmbda=5.0,
        evaluation_data=validation_data, monitor_evaluation_accuracy=True)

Epoch 0 training complete
Accuracy on evaluation data: 9186 / 10000
Epoch 1 training complete
Accuracy on evaluation data: 9418 / 10000
Epoch 2 training complete
Accuracy on evaluation data: 9487 / 10000
Epoch 3 training complete
Accuracy on evaluation data: 9544 / 10000
Epoch 4 training complete
Accuracy on evaluation data: 9569 / 10000
Epoch 5 training complete
Accuracy on evaluation data: 9589 / 10000
Epoch 6 training complete
Accuracy on evaluation data: 9596 / 10000
Epoch 7 training complete
Accuracy on evaluation data: 9605 / 10000
Epoch 8 training complete
Accuracy on evaluation data: 9615 / 10000
Epoch 9 training complete
Accuracy on evaluation data: 9625 / 10000
Epoch 10 training complete
Accuracy on evaluation data: 9645 / 10000
Epoch 11 training complete
Accuracy on evaluation data: 9598 / 10000
Epoch 12 training complete
Accuracy on evaluation data: 9655 / 10000
Epoch 13 training complete
Accuracy on evaluation data: 9642 / 10000
Epoch 14 training complete
Accuracy on evaluation data: 9650 / 10000
Epoch 15 training complete
Accuracy on evaluation data: 9668 / 10000
Epoch 16 training complete
Accuracy on evaluation data: 9672 / 10000
Epoch 17 training complete
Accuracy on evaluation data: 9655 / 10000
Epoch 18 training complete
Accuracy on evaluation data: 9651 / 10000
Epoch 19 training complete
Accuracy on evaluation data: 9655 / 10000
Epoch 20 training complete
Accuracy on evaluation data: 9675 / 10000
Epoch 21 training complete
Accuracy on evaluation data: 9665 / 10000
Epoch 22 training complete
Accuracy on evaluation data: 9647 / 10000
Epoch 23 training complete
Accuracy on evaluation data: 9666 / 10000
Epoch 24 training complete
Accuracy on evaluation data: 9681 / 10000
Epoch 25 training complete
Accuracy on evaluation data: 9686 / 10000
Epoch 26 training complete
Accuracy on evaluation data: 9673 / 10000
Epoch 27 training complete
Accuracy on evaluation data: 9662 / 10000
Epoch 28 training complete
Accuracy on evaluation data: 9678 / 10000
Epoch 29 training complete
Accuracy on evaluation data: 9674 / 10000
CPU times: user 1min 37s, sys: 31 s, total: 2min 8s
Wall time: 1min 46s

([],
 [9186,
  9418,
  9487,
  9544,
  9569,
  9589,
  9596,
  9605,
  9615,
  9625,
  9645,
  9598,
  9655,
  9642,
  9650,
  9668,
  9672,
  9655,
  9651,
  9655,
  9675,
  9665,
  9647,
  9666,
  9681,
  9686,
  9673,
  9662,
  9678,
  9674],
 [],
 [])

%%time
# Initialize and train 3-hidden layer network
net = Network2mat([784, 30, 30, 30, 10])
net.SGD(training_data, 30, 10, 0.1, lmbda=5.0,
        evaluation_data=validation_data, monitor_evaluation_accuracy=True)

Epoch 0 training complete
Accuracy on evaluation data: 8620 / 10000
Epoch 1 training complete
Accuracy on evaluation data: 9267 / 10000
Epoch 2 training complete
Accuracy on evaluation data: 9454 / 10000
Epoch 3 training complete
Accuracy on evaluation data: 9522 / 10000
Epoch 4 training complete
Accuracy on evaluation data: 9559 / 10000
Epoch 5 training complete
Accuracy on evaluation data: 9524 / 10000
Epoch 6 training complete
Accuracy on evaluation data: 9626 / 10000
Epoch 7 training complete
Accuracy on evaluation data: 9585 / 10000
Epoch 8 training complete
Accuracy on evaluation data: 9592 / 10000
Epoch 9 training complete
Accuracy on evaluation data: 9647 / 10000
Epoch 10 training complete
Accuracy on evaluation data: 9646 / 10000
Epoch 11 training complete
Accuracy on evaluation data: 9658 / 10000
Epoch 12 training complete
Accuracy on evaluation data: 9637 / 10000
Epoch 13 training complete
Accuracy on evaluation data: 9596 / 10000
Epoch 14 training complete
Accuracy on evaluation data: 9653 / 10000
Epoch 15 training complete
Accuracy on evaluation data: 9640 / 10000
Epoch 16 training complete
Accuracy on evaluation data: 9655 / 10000
Epoch 17 training complete
Accuracy on evaluation data: 9643 / 10000
Epoch 18 training complete
Accuracy on evaluation data: 9663 / 10000
Epoch 19 training complete
Accuracy on evaluation data: 9615 / 10000
Epoch 20 training complete
Accuracy on evaluation data: 9608 / 10000
Epoch 21 training complete
Accuracy on evaluation data: 9661 / 10000
Epoch 22 training complete
Accuracy on evaluation data: 9688 / 10000
Epoch 23 training complete
Accuracy on evaluation data: 9662 / 10000
Epoch 24 training complete
Accuracy on evaluation data: 9663 / 10000
Epoch 25 training complete
Accuracy on evaluation data: 9678 / 10000
Epoch 26 training complete
Accuracy on evaluation data: 9677 / 10000
Epoch 27 training complete
Accuracy on evaluation data: 9651 / 10000
Epoch 28 training complete
Accuracy on evaluation data: 9668 / 10000
Epoch 29 training complete
Accuracy on evaluation data: 9682 / 10000
CPU times: user 1min 59s, sys: 37.8 s, total: 2min 36s
Wall time: 2min 7s

([],
 [8620,
  9267,
  9454,
  9522,
  9559,
  9524,
  9626,
  9585,
  9592,
  9647,
  9646,
  9658,
  9637,
  9596,
  9653,
  9640,
  9655,
  9643,
  9663,
  9615,
  9608,
  9661,
  9688,
  9662,
  9663,
  9678,
  9677,
  9651,
  9668,
  9682],
 [],
 [])

%%time
# Initialize and train 4-hidden layer network
net = Network2mat([784, 30, 30, 30, 30, 10])
net.SGD(training_data, 30, 10, 0.1, lmbda=5.0,
        evaluation_data=validation_data, monitor_evaluation_accuracy=True)

Epoch 0 training complete
Accuracy on evaluation data: 2281 / 10000
Epoch 1 training complete
Accuracy on evaluation data: 6176 / 10000
Epoch 2 training complete
Accuracy on evaluation data: 9153 / 10000
Epoch 3 training complete
Accuracy on evaluation data: 9423 / 10000
Epoch 4 training complete
Accuracy on evaluation data: 9446 / 10000
Epoch 5 training complete
Accuracy on evaluation data: 9468 / 10000
Epoch 6 training complete
Accuracy on evaluation data: 9466 / 10000
Epoch 7 training complete
Accuracy on evaluation data: 9474 / 10000
Epoch 8 training complete
Accuracy on evaluation data: 9595 / 10000
Epoch 9 training complete
Accuracy on evaluation data: 9549 / 10000
Epoch 10 training complete
Accuracy on evaluation data: 9553 / 10000
Epoch 11 training complete
Accuracy on evaluation data: 9605 / 10000
Epoch 12 training complete
Accuracy on evaluation data: 9617 / 10000
Epoch 13 training complete
Accuracy on evaluation data: 9569 / 10000
Epoch 14 training complete
Accuracy on evaluation data: 9622 / 10000
Epoch 15 training complete
Accuracy on evaluation data: 9573 / 10000
Epoch 16 training complete
Accuracy on evaluation data: 9594 / 10000
Epoch 17 training complete
Accuracy on evaluation data: 9633 / 10000
Epoch 18 training complete
Accuracy on evaluation data: 9586 / 10000
Epoch 19 training complete
Accuracy on evaluation data: 9576 / 10000
Epoch 20 training complete
Accuracy on evaluation data: 9636 / 10000
Epoch 21 training complete
Accuracy on evaluation data: 9633 / 10000
Epoch 22 training complete
Accuracy on evaluation data: 9588 / 10000
Epoch 23 training complete
Accuracy on evaluation data: 9574 / 10000
Epoch 24 training complete
Accuracy on evaluation data: 9617 / 10000
Epoch 25 training complete
Accuracy on evaluation data: 9658 / 10000
Epoch 26 training complete
Accuracy on evaluation data: 9511 / 10000
Epoch 27 training complete
Accuracy on evaluation data: 9611 / 10000
Epoch 28 training complete
Accuracy on evaluation data: 9626 / 10000
Epoch 29 training complete
Accuracy on evaluation data: 9635 / 10000
CPU times: user 2min 18s, sys: 43.9 s, total: 3min 2s
Wall time: 2min 22s

([],
 [2281,
  6176,
  9153,
  9423,
  9446,
  9468,
  9466,
  9474,
  9595,
  9549,
  9553,
  9605,
  9617,
  9569,
  9622,
  9573,
  9594,
  9633,
  9586,
  9576,
  9636,
  9633,
  9588,
  9574,
  9617,
  9658,
  9511,
  9611,
  9626,
  9635],
 [],
 [])

"""generate_gradient.py
~~~~~~~~~~~~~~~~~~~~~~~

Use network2 to figure out the average starting values of the gradient
error terms \delta^l_j = \partial C / \partial z^l_j = \partial C /
\partial b^l_j.

"""

#### Libraries
# Standard library
import json
import math
import random
import shutil
import sys
import functools
sys.path.append("../src/")

# My library
import mnist_loader
# import network2

# Third-party libraries
import matplotlib.pyplot as plt
import numpy as np

def main():
    # Load the data
    full_td, _, _ = mnist_loader.load_data_wrapper()
    full_td = list(full_td)
    td = full_td[:1000] # Just use the first 1000 items of training data
    epochs = 500 # Number of epochs to train for

    print("\nTwo hidden layers:")
    net = Network2mat([784, 30, 30, 10])
    initial_norms(td, net)
    abbreviated_gradient = [
        ag[:6] for ag in get_average_gradient(net, td)[:-1]]
    print("Saving the averaged gradient for the top six neurons in each "+\
        "layer.\nWARNING: This will affect the look of the book, so be "+\
        "sure to check the\nrelevant material (early chapter 5).")
    f = open("initial_gradient.json", "w")
    json.dump(abbreviated_gradient, f)
    f.close()
    training(td, net, epochs, "norms_during_training_2_layers.json")
    plot_training(
        epochs, "norms_during_training_2_layers.json", 2)

    print("\nThree hidden layers:")
    net = Network2mat([784, 30, 30, 30, 10])
    initial_norms(td, net)
    training(td, net, epochs, "norms_during_training_3_layers.json")
    plot_training(
        epochs, "norms_during_training_3_layers.json", 3)

    print("\nFour hidden layers:")
    net = Network2mat([784, 30, 30, 30, 30, 10])
    initial_norms(td, net)
    training(td, net, epochs,
             "norms_during_training_4_layers.json")
    plot_training(
        epochs, "norms_during_training_4_layers.json", 4)

def initial_norms(training_data, net):
    average_gradient = get_average_gradient(net, training_data)
    norms = [list_norm(avg) for avg in average_gradient[:-1]]
    print("Average gradient for the hidden layers: "+str(norms))

def training(training_data, net, epochs, filename):
    norms = []
    for j in range(epochs):
        average_gradient = get_average_gradient(net, training_data)
        norms.append([list_norm(avg) for avg in average_gradient[:-1]])
        print("Epoch: %s" % j)
        net.SGD(training_data, 1, 1000, 0.1, lmbda=5.0)
    f = open(filename, "w")
    json.dump(norms, f)
    f.close()

def plot_training(epochs, filename, num_layers):
    f = open(filename, "r")
    norms = json.load(f)
    f.close()
    fig = plt.figure()
    ax = fig.add_subplot(111)
    colors = ["#2A6EA6", "#FFA933", "#FF5555", "#55FF55", "#5555FF"]
    for j in range(num_layers):
        ax.plot(np.arange(epochs),
                [n[j] for n in norms],
                color=colors[j],
                label="Hidden layer %s" % (j+1,))
    ax.set_xlim([0, epochs])
    ax.grid(True)
    ax.set_xlabel('Number of epochs of training')
    ax.set_title('Speed of learning: %s hidden layers' % num_layers)
    ax.set_yscale('log')
    plt.legend(loc="upper right")
    fig_filename = "training_speed_%s_layers.png" % num_layers
    plt.savefig(fig_filename)
    plt.show()

def get_average_gradient(net, training_data):
    nabla_b_results = [net.backprop_matrix(x, y)[0] for x, y in training_data]
    gradient = list_sum(nabla_b_results)
    return [(np.reshape(g, len(g))/len(training_data)).tolist()
            for g in gradient]

def zip_sum(a, b):
    return [x+y for (x, y) in zip(a, b)]

def list_sum(l):
    return functools.reduce(zip_sum, l)

def list_norm(l):
    return math.sqrt(sum([x*x for x in l]))

if __name__ == "__main__":
    main()

Two hidden layers:
Average gradient for the hidden layers: [0.05910799398171015, 0.3504939212704065]
Saving the averaged gradient for the top six neurons in each layer.
WARNING: This will affect the look of the book, so be sure to check the
relevant material (early chapter 5).
Epoch: 0
Epoch 0 training complete
Epoch: 1
Epoch 0 training complete
Epoch: 2
Epoch 0 training complete
Epoch: 3
Epoch 0 training complete
Epoch: 4
Epoch 0 training complete
Epoch: 5
Epoch 0 training complete
Epoch: 6
Epoch 0 training complete
Epoch: 7
Epoch 0 training complete
Epoch: 8
Epoch 0 training complete
Epoch: 9
Epoch 0 training complete
Epoch: 10
Epoch 0 training complete
Epoch: 11
Epoch 0 training complete
Epoch: 12
Epoch 0 training complete
Epoch: 13
Epoch 0 training complete
Epoch: 14
Epoch 0 training complete
Epoch: 15
Epoch 0 training complete
Epoch: 16
Epoch 0 training complete
Epoch: 17
Epoch 0 training complete
Epoch: 18
Epoch 0 training complete
Epoch: 19
Epoch 0 training complete
Epoch: 20
Epoch 0 training complete
Epoch: 21
Epoch 0 training complete
Epoch: 22
Epoch 0 training complete
Epoch: 23
Epoch 0 training complete
Epoch: 24
Epoch 0 training complete
Epoch: 25
Epoch 0 training complete
Epoch: 26
Epoch 0 training complete
Epoch: 27
Epoch 0 training complete
Epoch: 28
Epoch 0 training complete
Epoch: 29
Epoch 0 training complete
Epoch: 30
Epoch 0 training complete
Epoch: 31
Epoch 0 training complete
Epoch: 32
Epoch 0 training complete
Epoch: 33
Epoch 0 training complete
Epoch: 34
Epoch 0 training complete
Epoch: 35
Epoch 0 training complete
Epoch: 36
Epoch 0 training complete
Epoch: 37
Epoch 0 training complete
Epoch: 38
Epoch 0 training complete
Epoch: 39
Epoch 0 training complete
Epoch: 40
Epoch 0 training complete
Epoch: 41
Epoch 0 training complete
Epoch: 42
Epoch 0 training complete
Epoch: 43
Epoch 0 training complete
Epoch: 44
Epoch 0 training complete
Epoch: 45
Epoch 0 training complete
Epoch: 46
Epoch 0 training complete
Epoch: 47
Epoch 0 training complete
Epoch: 48
Epoch 0 training complete
Epoch: 49
Epoch 0 training complete
Epoch: 50
Epoch 0 training complete
Epoch: 51
Epoch 0 training complete
Epoch: 52
Epoch 0 training complete
Epoch: 53
Epoch 0 training complete
Epoch: 54
Epoch 0 training complete
Epoch: 55
Epoch 0 training complete
Epoch: 56
Epoch 0 training complete
Epoch: 57
Epoch 0 training complete
Epoch: 58
Epoch 0 training complete
Epoch: 59
Epoch 0 training complete
Epoch: 60
Epoch 0 training complete
Epoch: 61
Epoch 0 training complete
Epoch: 62
Epoch 0 training complete
Epoch: 63
Epoch 0 training complete
Epoch: 64
Epoch 0 training complete
Epoch: 65
Epoch 0 training complete
Epoch: 66
Epoch 0 training complete
Epoch: 67
Epoch 0 training complete
Epoch: 68
Epoch 0 training complete
Epoch: 69
Epoch 0 training complete
Epoch: 70
Epoch 0 training complete
Epoch: 71
Epoch 0 training complete
Epoch: 72
Epoch 0 training complete
Epoch: 73
Epoch 0 training complete
Epoch: 74
Epoch 0 training complete
Epoch: 75
Epoch 0 training complete
Epoch: 76
Epoch 0 training complete
Epoch: 77
Epoch 0 training complete
Epoch: 78
Epoch 0 training complete
Epoch: 79
Epoch 0 training complete
Epoch: 80
Epoch 0 training complete
Epoch: 81
Epoch 0 training complete
Epoch: 82
Epoch 0 training complete
Epoch: 83
Epoch 0 training complete
Epoch: 84
Epoch 0 training complete
Epoch: 85
Epoch 0 training complete
Epoch: 86
Epoch 0 training complete
Epoch: 87
Epoch 0 training complete
Epoch: 88
Epoch 0 training complete
Epoch: 89
Epoch 0 training complete
Epoch: 90
Epoch 0 training complete
Epoch: 91
Epoch 0 training complete
Epoch: 92
Epoch 0 training complete
Epoch: 93
Epoch 0 training complete
Epoch: 94
Epoch 0 training complete
Epoch: 95
Epoch 0 training complete
Epoch: 96
Epoch 0 training complete
Epoch: 97
Epoch 0 training complete
Epoch: 98
Epoch 0 training complete
Epoch: 99
Epoch 0 training complete
Epoch: 100
Epoch 0 training complete
Epoch: 101
Epoch 0 training complete
Epoch: 102
Epoch 0 training complete
Epoch: 103
Epoch 0 training complete
Epoch: 104
Epoch 0 training complete
Epoch: 105
Epoch 0 training complete
Epoch: 106
Epoch 0 training complete
Epoch: 107
Epoch 0 training complete
Epoch: 108
Epoch 0 training complete
Epoch: 109
Epoch 0 training complete
Epoch: 110
Epoch 0 training complete
Epoch: 111
Epoch 0 training complete
Epoch: 112
Epoch 0 training complete
Epoch: 113
Epoch 0 training complete
Epoch: 114
Epoch 0 training complete
Epoch: 115
Epoch 0 training complete
Epoch: 116
Epoch 0 training complete
Epoch: 117
Epoch 0 training complete
Epoch: 118
Epoch 0 training complete
Epoch: 119
Epoch 0 training complete
Epoch: 120
Epoch 0 training complete
Epoch: 121
Epoch 0 training complete
Epoch: 122
Epoch 0 training complete
Epoch: 123
Epoch 0 training complete
Epoch: 124
Epoch 0 training complete
Epoch: 125
Epoch 0 training complete
Epoch: 126
Epoch 0 training complete
Epoch: 127
Epoch 0 training complete
Epoch: 128
Epoch 0 training complete
Epoch: 129
Epoch 0 training complete
Epoch: 130
Epoch 0 training complete
Epoch: 131
Epoch 0 training complete
Epoch: 132
Epoch 0 training complete
Epoch: 133
Epoch 0 training complete
Epoch: 134
Epoch 0 training complete
Epoch: 135
Epoch 0 training complete
Epoch: 136
Epoch 0 training complete
Epoch: 137
Epoch 0 training complete
Epoch: 138
Epoch 0 training complete
Epoch: 139
Epoch 0 training complete
Epoch: 140
Epoch 0 training complete
Epoch: 141
Epoch 0 training complete
Epoch: 142
Epoch 0 training complete
Epoch: 143
Epoch 0 training complete
Epoch: 144
Epoch 0 training complete
Epoch: 145
Epoch 0 training complete
Epoch: 146
Epoch 0 training complete
Epoch: 147
Epoch 0 training complete
Epoch: 148
Epoch 0 training complete
Epoch: 149
Epoch 0 training complete
Epoch: 150
Epoch 0 training complete
Epoch: 151
Epoch 0 training complete
Epoch: 152
Epoch 0 training complete
Epoch: 153
Epoch 0 training complete
Epoch: 154
Epoch 0 training complete
Epoch: 155
Epoch 0 training complete
Epoch: 156
Epoch 0 training complete
Epoch: 157
Epoch 0 training complete
Epoch: 158
Epoch 0 training complete
Epoch: 159
Epoch 0 training complete
Epoch: 160
Epoch 0 training complete
Epoch: 161
Epoch 0 training complete
Epoch: 162
Epoch 0 training complete
Epoch: 163
Epoch 0 training complete
Epoch: 164
Epoch 0 training complete
Epoch: 165
Epoch 0 training complete
Epoch: 166
Epoch 0 training complete
Epoch: 167
Epoch 0 training complete
Epoch: 168
Epoch 0 training complete
Epoch: 169
Epoch 0 training complete
Epoch: 170
Epoch 0 training complete
Epoch: 171
Epoch 0 training complete
Epoch: 172
Epoch 0 training complete
Epoch: 173
Epoch 0 training complete
Epoch: 174
Epoch 0 training complete
Epoch: 175
Epoch 0 training complete
Epoch: 176
Epoch 0 training complete
Epoch: 177
Epoch 0 training complete
Epoch: 178
Epoch 0 training complete
Epoch: 179
Epoch 0 training complete
Epoch: 180
Epoch 0 training complete
Epoch: 181
Epoch 0 training complete
Epoch: 182
Epoch 0 training complete
Epoch: 183
Epoch 0 training complete
Epoch: 184
Epoch 0 training complete
Epoch: 185
Epoch 0 training complete
Epoch: 186
Epoch 0 training complete
Epoch: 187
Epoch 0 training complete
Epoch: 188
Epoch 0 training complete
Epoch: 189
Epoch 0 training complete
Epoch: 190
Epoch 0 training complete
Epoch: 191
Epoch 0 training complete
Epoch: 192
Epoch 0 training complete
Epoch: 193
Epoch 0 training complete
Epoch: 194
Epoch 0 training complete
Epoch: 195
Epoch 0 training complete
Epoch: 196
Epoch 0 training complete
Epoch: 197
Epoch 0 training complete
Epoch: 198
Epoch 0 training complete
Epoch: 199
Epoch 0 training complete
Epoch: 200
Epoch 0 training complete
Epoch: 201
Epoch 0 training complete
Epoch: 202
Epoch 0 training complete
Epoch: 203
Epoch 0 training complete
Epoch: 204
Epoch 0 training complete
Epoch: 205
Epoch 0 training complete
Epoch: 206
Epoch 0 training complete
Epoch: 207
Epoch 0 training complete
Epoch: 208
Epoch 0 training complete
Epoch: 209
Epoch 0 training complete
Epoch: 210
Epoch 0 training complete
Epoch: 211
Epoch 0 training complete
Epoch: 212
Epoch 0 training complete
Epoch: 213
Epoch 0 training complete
Epoch: 214
Epoch 0 training complete
Epoch: 215
Epoch 0 training complete
Epoch: 216
Epoch 0 training complete
Epoch: 217
Epoch 0 training complete
Epoch: 218
Epoch 0 training complete
Epoch: 219
Epoch 0 training complete
Epoch: 220
Epoch 0 training complete
Epoch: 221
Epoch 0 training complete
Epoch: 222
Epoch 0 training complete
Epoch: 223
Epoch 0 training complete
Epoch: 224
Epoch 0 training complete
Epoch: 225
Epoch 0 training complete
Epoch: 226
Epoch 0 training complete
Epoch: 227
Epoch 0 training complete
Epoch: 228
Epoch 0 training complete
Epoch: 229
Epoch 0 training complete
Epoch: 230
Epoch 0 training complete
Epoch: 231
Epoch 0 training complete
Epoch: 232
Epoch 0 training complete
Epoch: 233
Epoch 0 training complete
Epoch: 234
Epoch 0 training complete
Epoch: 235
Epoch 0 training complete
Epoch: 236
Epoch 0 training complete
Epoch: 237
Epoch 0 training complete
Epoch: 238
Epoch 0 training complete
Epoch: 239
Epoch 0 training complete
Epoch: 240
Epoch 0 training complete
Epoch: 241
Epoch 0 training complete
Epoch: 242
Epoch 0 training complete
Epoch: 243
Epoch 0 training complete
Epoch: 244
Epoch 0 training complete
Epoch: 245
Epoch 0 training complete
Epoch: 246
Epoch 0 training complete
Epoch: 247
Epoch 0 training complete
Epoch: 248
Epoch 0 training complete
Epoch: 249
Epoch 0 training complete
Epoch: 250
Epoch 0 training complete
Epoch: 251
Epoch 0 training complete
Epoch: 252
Epoch 0 training complete
Epoch: 253
Epoch 0 training complete
Epoch: 254
Epoch 0 training complete
Epoch: 255
Epoch 0 training complete
Epoch: 256
Epoch 0 training complete
Epoch: 257
Epoch 0 training complete
Epoch: 258
Epoch 0 training complete
Epoch: 259
Epoch 0 training complete
Epoch: 260
Epoch 0 training complete
Epoch: 261
Epoch 0 training complete
Epoch: 262
Epoch 0 training complete
Epoch: 263
Epoch 0 training complete
Epoch: 264
Epoch 0 training complete
Epoch: 265
Epoch 0 training complete
Epoch: 266
Epoch 0 training complete
Epoch: 267
Epoch 0 training complete
Epoch: 268
Epoch 0 training complete
Epoch: 269
Epoch 0 training complete
Epoch: 270
Epoch 0 training complete
Epoch: 271
Epoch 0 training complete
Epoch: 272
Epoch 0 training complete
Epoch: 273
Epoch 0 training complete
Epoch: 274
Epoch 0 training complete
Epoch: 275
Epoch 0 training complete
Epoch: 276
Epoch 0 training complete
Epoch: 277
Epoch 0 training complete
Epoch: 278
Epoch 0 training complete
Epoch: 279
Epoch 0 training complete
Epoch: 280
Epoch 0 training complete
Epoch: 281
Epoch 0 training complete
Epoch: 282
Epoch 0 training complete
Epoch: 283
Epoch 0 training complete
Epoch: 284
Epoch 0 training complete
Epoch: 285
Epoch 0 training complete
Epoch: 286
Epoch 0 training complete
Epoch: 287
Epoch 0 training complete
Epoch: 288
Epoch 0 training complete
Epoch: 289
Epoch 0 training complete
Epoch: 290
Epoch 0 training complete
Epoch: 291
Epoch 0 training complete
Epoch: 292
Epoch 0 training complete
Epoch: 293
Epoch 0 training complete
Epoch: 294
Epoch 0 training complete
Epoch: 295
Epoch 0 training complete
Epoch: 296
Epoch 0 training complete
Epoch: 297
Epoch 0 training complete
Epoch: 298
Epoch 0 training complete
Epoch: 299
Epoch 0 training complete
Epoch: 300
Epoch 0 training complete
Epoch: 301
Epoch 0 training complete
Epoch: 302
Epoch 0 training complete
Epoch: 303
Epoch 0 training complete
Epoch: 304
Epoch 0 training complete
Epoch: 305
Epoch 0 training complete
Epoch: 306
Epoch 0 training complete
Epoch: 307
Epoch 0 training complete
Epoch: 308
Epoch 0 training complete
Epoch: 309
Epoch 0 training complete
Epoch: 310
Epoch 0 training complete
Epoch: 311
Epoch 0 training complete
Epoch: 312
Epoch 0 training complete
Epoch: 313
Epoch 0 training complete
Epoch: 314
Epoch 0 training complete
Epoch: 315
Epoch 0 training complete
Epoch: 316
Epoch 0 training complete
Epoch: 317
Epoch 0 training complete
Epoch: 318
Epoch 0 training complete
Epoch: 319
Epoch 0 training complete
Epoch: 320
Epoch 0 training complete
Epoch: 321
Epoch 0 training complete
Epoch: 322
Epoch 0 training complete
Epoch: 323
Epoch 0 training complete
Epoch: 324
Epoch 0 training complete
Epoch: 325
Epoch 0 training complete
Epoch: 326
Epoch 0 training complete
Epoch: 327
Epoch 0 training complete
Epoch: 328
Epoch 0 training complete
Epoch: 329
Epoch 0 training complete
Epoch: 330
Epoch 0 training complete
Epoch: 331
Epoch 0 training complete
Epoch: 332
Epoch 0 training complete
Epoch: 333
Epoch 0 training complete
Epoch: 334
Epoch 0 training complete
Epoch: 335
Epoch 0 training complete
Epoch: 336
Epoch 0 training complete
Epoch: 337
Epoch 0 training complete
Epoch: 338
Epoch 0 training complete
Epoch: 339
Epoch 0 training complete
Epoch: 340
Epoch 0 training complete
Epoch: 341
Epoch 0 training complete
Epoch: 342
Epoch 0 training complete
Epoch: 343
Epoch 0 training complete
Epoch: 344
Epoch 0 training complete
Epoch: 345
Epoch 0 training complete
Epoch: 346
Epoch 0 training complete
Epoch: 347
Epoch 0 training complete
Epoch: 348
Epoch 0 training complete
Epoch: 349
Epoch 0 training complete
Epoch: 350
Epoch 0 training complete
Epoch: 351
Epoch 0 training complete
Epoch: 352
Epoch 0 training complete
Epoch: 353
Epoch 0 training complete
Epoch: 354
Epoch 0 training complete
Epoch: 355
Epoch 0 training complete
Epoch: 356
Epoch 0 training complete
Epoch: 357
Epoch 0 training complete
Epoch: 358
Epoch 0 training complete
Epoch: 359
Epoch 0 training complete
Epoch: 360
Epoch 0 training complete
Epoch: 361
Epoch 0 training complete
Epoch: 362
Epoch 0 training complete
Epoch: 363
Epoch 0 training complete
Epoch: 364
Epoch 0 training complete
Epoch: 365
Epoch 0 training complete
Epoch: 366
Epoch 0 training complete
Epoch: 367
Epoch 0 training complete
Epoch: 368
Epoch 0 training complete
Epoch: 369
Epoch 0 training complete
Epoch: 370
Epoch 0 training complete
Epoch: 371
Epoch 0 training complete
Epoch: 372
Epoch 0 training complete
Epoch: 373
Epoch 0 training complete
Epoch: 374
Epoch 0 training complete
Epoch: 375
Epoch 0 training complete
Epoch: 376
Epoch 0 training complete
Epoch: 377
Epoch 0 training complete
Epoch: 378
Epoch 0 training complete
Epoch: 379
Epoch 0 training complete
Epoch: 380
Epoch 0 training complete
Epoch: 381
Epoch 0 training complete
Epoch: 382
Epoch 0 training complete
Epoch: 383
Epoch 0 training complete
Epoch: 384
Epoch 0 training complete
Epoch: 385
Epoch 0 training complete
Epoch: 386
Epoch 0 training complete
Epoch: 387
Epoch 0 training complete
Epoch: 388
Epoch 0 training complete
Epoch: 389
Epoch 0 training complete
Epoch: 390
Epoch 0 training complete
Epoch: 391
Epoch 0 training complete
Epoch: 392
Epoch 0 training complete
Epoch: 393
Epoch 0 training complete
Epoch: 394
Epoch 0 training complete
Epoch: 395
Epoch 0 training complete
Epoch: 396
Epoch 0 training complete
Epoch: 397
Epoch 0 training complete
Epoch: 398
Epoch 0 training complete
Epoch: 399
Epoch 0 training complete
Epoch: 400
Epoch 0 training complete
Epoch: 401
Epoch 0 training complete
Epoch: 402
Epoch 0 training complete
Epoch: 403
Epoch 0 training complete
Epoch: 404
Epoch 0 training complete
Epoch: 405
Epoch 0 training complete
Epoch: 406
Epoch 0 training complete
Epoch: 407
Epoch 0 training complete
Epoch: 408
Epoch 0 training complete
Epoch: 409
Epoch 0 training complete
Epoch: 410
Epoch 0 training complete
Epoch: 411
Epoch 0 training complete
Epoch: 412
Epoch 0 training complete
Epoch: 413
Epoch 0 training complete
Epoch: 414
Epoch 0 training complete
Epoch: 415
Epoch 0 training complete
Epoch: 416
Epoch 0 training complete
Epoch: 417
Epoch 0 training complete
Epoch: 418
Epoch 0 training complete
Epoch: 419
Epoch 0 training complete
Epoch: 420
Epoch 0 training complete
Epoch: 421
Epoch 0 training complete
Epoch: 422
Epoch 0 training complete
Epoch: 423
Epoch 0 training complete
Epoch: 424
Epoch 0 training complete
Epoch: 425
Epoch 0 training complete
Epoch: 426
Epoch 0 training complete
Epoch: 427
Epoch 0 training complete
Epoch: 428
Epoch 0 training complete
Epoch: 429
Epoch 0 training complete
Epoch: 430
Epoch 0 training complete
Epoch: 431
Epoch 0 training complete
Epoch: 432
Epoch 0 training complete
Epoch: 433
Epoch 0 training complete
Epoch: 434
Epoch 0 training complete
Epoch: 435
Epoch 0 training complete
Epoch: 436
Epoch 0 training complete
Epoch: 437
Epoch 0 training complete
Epoch: 438
Epoch 0 training complete
Epoch: 439
Epoch 0 training complete
Epoch: 440
Epoch 0 training complete
Epoch: 441
Epoch 0 training complete
Epoch: 442
Epoch 0 training complete
Epoch: 443
Epoch 0 training complete
Epoch: 444
Epoch 0 training complete
Epoch: 445
Epoch 0 training complete
Epoch: 446
Epoch 0 training complete
Epoch: 447
Epoch 0 training complete
Epoch: 448
Epoch 0 training complete
Epoch: 449
Epoch 0 training complete
Epoch: 450
Epoch 0 training complete
Epoch: 451
Epoch 0 training complete
Epoch: 452
Epoch 0 training complete
Epoch: 453
Epoch 0 training complete
Epoch: 454
Epoch 0 training complete
Epoch: 455
Epoch 0 training complete
Epoch: 456
Epoch 0 training complete
Epoch: 457
Epoch 0 training complete
Epoch: 458
Epoch 0 training complete
Epoch: 459
Epoch 0 training complete
Epoch: 460
Epoch 0 training complete
Epoch: 461
Epoch 0 training complete
Epoch: 462
Epoch 0 training complete
Epoch: 463
Epoch 0 training complete
Epoch: 464
Epoch 0 training complete
Epoch: 465
Epoch 0 training complete
Epoch: 466
Epoch 0 training complete
Epoch: 467
Epoch 0 training complete
Epoch: 468
Epoch 0 training complete
Epoch: 469
Epoch 0 training complete
Epoch: 470
Epoch 0 training complete
Epoch: 471
Epoch 0 training complete
Epoch: 472
Epoch 0 training complete
Epoch: 473
Epoch 0 training complete
Epoch: 474
Epoch 0 training complete
Epoch: 475
Epoch 0 training complete
Epoch: 476
Epoch 0 training complete
Epoch: 477
Epoch 0 training complete
Epoch: 478
Epoch 0 training complete
Epoch: 479
Epoch 0 training complete
Epoch: 480
Epoch 0 training complete
Epoch: 481
Epoch 0 training complete
Epoch: 482
Epoch 0 training complete
Epoch: 483
Epoch 0 training complete
Epoch: 484
Epoch 0 training complete
Epoch: 485
Epoch 0 training complete
Epoch: 486
Epoch 0 training complete
Epoch: 487
Epoch 0 training complete
Epoch: 488
Epoch 0 training complete
Epoch: 489
Epoch 0 training complete
Epoch: 490
Epoch 0 training complete
Epoch: 491
Epoch 0 training complete
Epoch: 492
Epoch 0 training complete
Epoch: 493
Epoch 0 training complete
Epoch: 494
Epoch 0 training complete
Epoch: 495
Epoch 0 training complete
Epoch: 496
Epoch 0 training complete
Epoch: 497
Epoch 0 training complete
Epoch: 498
Epoch 0 training complete
Epoch: 499
Epoch 0 training complete

Three hidden layers:
Average gradient for the hidden layers: [0.022818952596379727, 0.0804102169606713, 0.3754301367980058]
Epoch: 0
Epoch 0 training complete
Epoch: 1
Epoch 0 training complete
Epoch: 2
Epoch 0 training complete
Epoch: 3
Epoch 0 training complete
Epoch: 4
Epoch 0 training complete
Epoch: 5
Epoch 0 training complete
Epoch: 6
Epoch 0 training complete
Epoch: 7
Epoch 0 training complete
Epoch: 8
Epoch 0 training complete
Epoch: 9
Epoch 0 training complete
Epoch: 10
Epoch 0 training complete
Epoch: 11
Epoch 0 training complete
Epoch: 12
Epoch 0 training complete
Epoch: 13
Epoch 0 training complete
Epoch: 14
Epoch 0 training complete
Epoch: 15
Epoch 0 training complete
Epoch: 16
Epoch 0 training complete
Epoch: 17
Epoch 0 training complete
Epoch: 18
Epoch 0 training complete
Epoch: 19
Epoch 0 training complete
Epoch: 20
Epoch 0 training complete
Epoch: 21
Epoch 0 training complete
Epoch: 22
Epoch 0 training complete
Epoch: 23
Epoch 0 training complete
Epoch: 24
Epoch 0 training complete
Epoch: 25
Epoch 0 training complete
Epoch: 26
Epoch 0 training complete
Epoch: 27
Epoch 0 training complete
Epoch: 28
Epoch 0 training complete
Epoch: 29
Epoch 0 training complete
Epoch: 30
Epoch 0 training complete
Epoch: 31
Epoch 0 training complete
Epoch: 32
Epoch 0 training complete
Epoch: 33
Epoch 0 training complete
Epoch: 34
Epoch 0 training complete
Epoch: 35
Epoch 0 training complete
Epoch: 36
Epoch 0 training complete
Epoch: 37
Epoch 0 training complete
Epoch: 38
Epoch 0 training complete
Epoch: 39
Epoch 0 training complete
Epoch: 40
Epoch 0 training complete
Epoch: 41
Epoch 0 training complete
Epoch: 42
Epoch 0 training complete
Epoch: 43
Epoch 0 training complete
Epoch: 44
Epoch 0 training complete
Epoch: 45
Epoch 0 training complete
Epoch: 46
Epoch 0 training complete
Epoch: 47
Epoch 0 training complete
Epoch: 48
Epoch 0 training complete
Epoch: 49
Epoch 0 training complete
Epoch: 50
Epoch 0 training complete
Epoch: 51
Epoch 0 training complete
Epoch: 52
Epoch 0 training complete
Epoch: 53
Epoch 0 training complete
Epoch: 54
Epoch 0 training complete
Epoch: 55
Epoch 0 training complete
Epoch: 56
Epoch 0 training complete
Epoch: 57
Epoch 0 training complete
Epoch: 58
Epoch 0 training complete
Epoch: 59
Epoch 0 training complete
Epoch: 60
Epoch 0 training complete
Epoch: 61
Epoch 0 training complete
Epoch: 62
Epoch 0 training complete
Epoch: 63
Epoch 0 training complete
Epoch: 64
Epoch 0 training complete
Epoch: 65
Epoch 0 training complete
Epoch: 66
Epoch 0 training complete
Epoch: 67
Epoch 0 training complete
Epoch: 68
Epoch 0 training complete
Epoch: 69
Epoch 0 training complete
Epoch: 70
Epoch 0 training complete
Epoch: 71
Epoch 0 training complete
Epoch: 72
Epoch 0 training complete
Epoch: 73
Epoch 0 training complete
Epoch: 74
Epoch 0 training complete
Epoch: 75
Epoch 0 training complete
Epoch: 76
Epoch 0 training complete
Epoch: 77
Epoch 0 training complete
Epoch: 78
Epoch 0 training complete
Epoch: 79
Epoch 0 training complete
Epoch: 80
Epoch 0 training complete
Epoch: 81
Epoch 0 training complete
Epoch: 82
Epoch 0 training complete
Epoch: 83
Epoch 0 training complete
Epoch: 84
Epoch 0 training complete
Epoch: 85
Epoch 0 training complete
Epoch: 86
Epoch 0 training complete
Epoch: 87
Epoch 0 training complete
Epoch: 88
Epoch 0 training complete
Epoch: 89
Epoch 0 training complete
Epoch: 90
Epoch 0 training complete
Epoch: 91
Epoch 0 training complete
Epoch: 92
Epoch 0 training complete
Epoch: 93
Epoch 0 training complete
Epoch: 94
Epoch 0 training complete
Epoch: 95
Epoch 0 training complete
Epoch: 96
Epoch 0 training complete
Epoch: 97
Epoch 0 training complete
Epoch: 98
Epoch 0 training complete
Epoch: 99
Epoch 0 training complete
Epoch: 100
Epoch 0 training complete
Epoch: 101
Epoch 0 training complete
Epoch: 102
Epoch 0 training complete
Epoch: 103
Epoch 0 training complete
Epoch: 104
Epoch 0 training complete
Epoch: 105
Epoch 0 training complete
Epoch: 106
Epoch 0 training complete
Epoch: 107
Epoch 0 training complete
Epoch: 108
Epoch 0 training complete
Epoch: 109
Epoch 0 training complete
Epoch: 110
Epoch 0 training complete
Epoch: 111
Epoch 0 training complete
Epoch: 112
Epoch 0 training complete
Epoch: 113
Epoch 0 training complete
Epoch: 114
Epoch 0 training complete
Epoch: 115
Epoch 0 training complete
Epoch: 116
Epoch 0 training complete
Epoch: 117
Epoch 0 training complete
Epoch: 118
Epoch 0 training complete
Epoch: 119
Epoch 0 training complete
Epoch: 120
Epoch 0 training complete
Epoch: 121
Epoch 0 training complete
Epoch: 122
Epoch 0 training complete
Epoch: 123
Epoch 0 training complete
Epoch: 124
Epoch 0 training complete
Epoch: 125
Epoch 0 training complete
Epoch: 126
Epoch 0 training complete
Epoch: 127
Epoch 0 training complete
Epoch: 128
Epoch 0 training complete
Epoch: 129
Epoch 0 training complete
Epoch: 130
Epoch 0 training complete
Epoch: 131
Epoch 0 training complete
Epoch: 132
Epoch 0 training complete
Epoch: 133
Epoch 0 training complete
Epoch: 134
Epoch 0 training complete
Epoch: 135
Epoch 0 training complete
Epoch: 136
Epoch 0 training complete
Epoch: 137
Epoch 0 training complete
Epoch: 138
Epoch 0 training complete
Epoch: 139
Epoch 0 training complete
Epoch: 140
Epoch 0 training complete
Epoch: 141
Epoch 0 training complete
Epoch: 142
Epoch 0 training complete
Epoch: 143
Epoch 0 training complete
Epoch: 144
Epoch 0 training complete
Epoch: 145
Epoch 0 training complete
Epoch: 146
Epoch 0 training complete
Epoch: 147
Epoch 0 training complete
Epoch: 148
Epoch 0 training complete
Epoch: 149
Epoch 0 training complete
Epoch: 150
Epoch 0 training complete
Epoch: 151
Epoch 0 training complete
Epoch: 152
Epoch 0 training complete
Epoch: 153
Epoch 0 training complete
Epoch: 154
Epoch 0 training complete
Epoch: 155
Epoch 0 training complete
Epoch: 156
Epoch 0 training complete
Epoch: 157
Epoch 0 training complete
Epoch: 158
Epoch 0 training complete
Epoch: 159
Epoch 0 training complete
Epoch: 160
Epoch 0 training complete
Epoch: 161
Epoch 0 training complete
Epoch: 162
Epoch 0 training complete
Epoch: 163
Epoch 0 training complete
Epoch: 164
Epoch 0 training complete
Epoch: 165
Epoch 0 training complete
Epoch: 166
Epoch 0 training complete
Epoch: 167
Epoch 0 training complete
Epoch: 168
Epoch 0 training complete
Epoch: 169
Epoch 0 training complete
Epoch: 170
Epoch 0 training complete
Epoch: 171
Epoch 0 training complete
Epoch: 172
Epoch 0 training complete
Epoch: 173
Epoch 0 training complete
Epoch: 174
Epoch 0 training complete
Epoch: 175
Epoch 0 training complete
Epoch: 176
Epoch 0 training complete
Epoch: 177
Epoch 0 training complete
Epoch: 178
Epoch 0 training complete
Epoch: 179
Epoch 0 training complete
Epoch: 180
Epoch 0 training complete
Epoch: 181
Epoch 0 training complete
Epoch: 182
Epoch 0 training complete
Epoch: 183
Epoch 0 training complete
Epoch: 184
Epoch 0 training complete
Epoch: 185
Epoch 0 training complete
Epoch: 186
Epoch 0 training complete
Epoch: 187
Epoch 0 training complete
Epoch: 188
Epoch 0 training complete
Epoch: 189
Epoch 0 training complete
Epoch: 190
Epoch 0 training complete
Epoch: 191
Epoch 0 training complete
Epoch: 192
Epoch 0 training complete
Epoch: 193
Epoch 0 training complete
Epoch: 194
Epoch 0 training complete
Epoch: 195
Epoch 0 training complete
Epoch: 196
Epoch 0 training complete
Epoch: 197
Epoch 0 training complete
Epoch: 198
Epoch 0 training complete
Epoch: 199
Epoch 0 training complete
Epoch: 200
Epoch 0 training complete
Epoch: 201
Epoch 0 training complete
Epoch: 202
Epoch 0 training complete
Epoch: 203
Epoch 0 training complete
Epoch: 204
Epoch 0 training complete
Epoch: 205
Epoch 0 training complete
Epoch: 206
Epoch 0 training complete
Epoch: 207
Epoch 0 training complete
Epoch: 208
Epoch 0 training complete
Epoch: 209
Epoch 0 training complete
Epoch: 210
Epoch 0 training complete
Epoch: 211
Epoch 0 training complete
Epoch: 212
Epoch 0 training complete
Epoch: 213
Epoch 0 training complete
Epoch: 214
Epoch 0 training complete
Epoch: 215
Epoch 0 training complete
Epoch: 216
Epoch 0 training complete
Epoch: 217
Epoch 0 training complete
Epoch: 218
Epoch 0 training complete
Epoch: 219
Epoch 0 training complete
Epoch: 220
Epoch 0 training complete
Epoch: 221
Epoch 0 training complete
Epoch: 222
Epoch 0 training complete
Epoch: 223
Epoch 0 training complete
Epoch: 224
Epoch 0 training complete
Epoch: 225
Epoch 0 training complete
Epoch: 226
Epoch 0 training complete
Epoch: 227
Epoch 0 training complete
Epoch: 228
Epoch 0 training complete
Epoch: 229
Epoch 0 training complete
Epoch: 230
Epoch 0 training complete
Epoch: 231
Epoch 0 training complete
Epoch: 232
Epoch 0 training complete
Epoch: 233
Epoch 0 training complete
Epoch: 234
Epoch 0 training complete
Epoch: 235
Epoch 0 training complete
Epoch: 236
Epoch 0 training complete
Epoch: 237
Epoch 0 training complete
Epoch: 238
Epoch 0 training complete
Epoch: 239
Epoch 0 training complete
Epoch: 240
Epoch 0 training complete
Epoch: 241
Epoch 0 training complete
Epoch: 242
Epoch 0 training complete
Epoch: 243
Epoch 0 training complete
Epoch: 244
Epoch 0 training complete
Epoch: 245
Epoch 0 training complete
Epoch: 246
Epoch 0 training complete
Epoch: 247
Epoch 0 training complete
Epoch: 248
Epoch 0 training complete
Epoch: 249
Epoch 0 training complete
Epoch: 250
Epoch 0 training complete
Epoch: 251
Epoch 0 training complete
Epoch: 252
Epoch 0 training complete
Epoch: 253
Epoch 0 training complete
Epoch: 254
Epoch 0 training complete
Epoch: 255
Epoch 0 training complete
Epoch: 256
Epoch 0 training complete
Epoch: 257
Epoch 0 training complete
Epoch: 258
Epoch 0 training complete
Epoch: 259
Epoch 0 training complete
Epoch: 260
Epoch 0 training complete
Epoch: 261
Epoch 0 training complete
Epoch: 262
Epoch 0 training complete
Epoch: 263
Epoch 0 training complete
Epoch: 264
Epoch 0 training complete
Epoch: 265
Epoch 0 training complete
Epoch: 266
Epoch 0 training complete
Epoch: 267
Epoch 0 training complete
Epoch: 268
Epoch 0 training complete
Epoch: 269
Epoch 0 training complete
Epoch: 270
Epoch 0 training complete
Epoch: 271
Epoch 0 training complete
Epoch: 272
Epoch 0 training complete
Epoch: 273
Epoch 0 training complete
Epoch: 274
Epoch 0 training complete
Epoch: 275
Epoch 0 training complete
Epoch: 276
Epoch 0 training complete
Epoch: 277
Epoch 0 training complete
Epoch: 278
Epoch 0 training complete
Epoch: 279
Epoch 0 training complete
Epoch: 280
Epoch 0 training complete
Epoch: 281
Epoch 0 training complete
Epoch: 282
Epoch 0 training complete
Epoch: 283
Epoch 0 training complete
Epoch: 284
Epoch 0 training complete
Epoch: 285
Epoch 0 training complete
Epoch: 286
Epoch 0 training complete
Epoch: 287
Epoch 0 training complete
Epoch: 288
Epoch 0 training complete
Epoch: 289
Epoch 0 training complete
Epoch: 290
Epoch 0 training complete
Epoch: 291
Epoch 0 training complete
Epoch: 292
Epoch 0 training complete
Epoch: 293
Epoch 0 training complete
Epoch: 294
Epoch 0 training complete
Epoch: 295
Epoch 0 training complete
Epoch: 296
Epoch 0 training complete
Epoch: 297
Epoch 0 training complete
Epoch: 298
Epoch 0 training complete
Epoch: 299
Epoch 0 training complete
Epoch: 300
Epoch 0 training complete
Epoch: 301
Epoch 0 training complete
Epoch: 302
Epoch 0 training complete
Epoch: 303
Epoch 0 training complete
Epoch: 304
Epoch 0 training complete
Epoch: 305
Epoch 0 training complete
Epoch: 306
Epoch 0 training complete
Epoch: 307
Epoch 0 training complete
Epoch: 308
Epoch 0 training complete
Epoch: 309
Epoch 0 training complete
Epoch: 310
Epoch 0 training complete
Epoch: 311
Epoch 0 training complete
Epoch: 312
Epoch 0 training complete
Epoch: 313
Epoch 0 training complete
Epoch: 314
Epoch 0 training complete
Epoch: 315
Epoch 0 training complete
Epoch: 316
Epoch 0 training complete
Epoch: 317
Epoch 0 training complete
Epoch: 318
Epoch 0 training complete
Epoch: 319
Epoch 0 training complete
Epoch: 320
Epoch 0 training complete
Epoch: 321
Epoch 0 training complete
Epoch: 322
Epoch 0 training complete
Epoch: 323
Epoch 0 training complete
Epoch: 324
Epoch 0 training complete
Epoch: 325
Epoch 0 training complete
Epoch: 326
Epoch 0 training complete
Epoch: 327
Epoch 0 training complete
Epoch: 328
Epoch 0 training complete
Epoch: 329
Epoch 0 training complete
Epoch: 330
Epoch 0 training complete
Epoch: 331
Epoch 0 training complete
Epoch: 332
Epoch 0 training complete
Epoch: 333
Epoch 0 training complete
Epoch: 334
Epoch 0 training complete
Epoch: 335
Epoch 0 training complete
Epoch: 336
Epoch 0 training complete
Epoch: 337
Epoch 0 training complete
Epoch: 338
Epoch 0 training complete
Epoch: 339
Epoch 0 training complete
Epoch: 340
Epoch 0 training complete
Epoch: 341
Epoch 0 training complete
Epoch: 342
Epoch 0 training complete
Epoch: 343
Epoch 0 training complete
Epoch: 344
Epoch 0 training complete
Epoch: 345
Epoch 0 training complete
Epoch: 346
Epoch 0 training complete
Epoch: 347
Epoch 0 training complete
Epoch: 348
Epoch 0 training complete
Epoch: 349
Epoch 0 training complete
Epoch: 350
Epoch 0 training complete
Epoch: 351
Epoch 0 training complete
Epoch: 352
Epoch 0 training complete
Epoch: 353
Epoch 0 training complete
Epoch: 354
Epoch 0 training complete
Epoch: 355
Epoch 0 training complete
Epoch: 356
Epoch 0 training complete
Epoch: 357
Epoch 0 training complete
Epoch: 358
Epoch 0 training complete
Epoch: 359
Epoch 0 training complete
Epoch: 360
Epoch 0 training complete
Epoch: 361
Epoch 0 training complete
Epoch: 362
Epoch 0 training complete
Epoch: 363
Epoch 0 training complete
Epoch: 364
Epoch 0 training complete
Epoch: 365
Epoch 0 training complete
Epoch: 366
Epoch 0 training complete
Epoch: 367
Epoch 0 training complete
Epoch: 368
Epoch 0 training complete
Epoch: 369
Epoch 0 training complete
Epoch: 370
Epoch 0 training complete
Epoch: 371
Epoch 0 training complete
Epoch: 372
Epoch 0 training complete
Epoch: 373
Epoch 0 training complete
Epoch: 374
Epoch 0 training complete
Epoch: 375
Epoch 0 training complete
Epoch: 376
Epoch 0 training complete
Epoch: 377
Epoch 0 training complete
Epoch: 378
Epoch 0 training complete
Epoch: 379
Epoch 0 training complete
Epoch: 380
Epoch 0 training complete
Epoch: 381
Epoch 0 training complete
Epoch: 382
Epoch 0 training complete
Epoch: 383
Epoch 0 training complete
Epoch: 384
Epoch 0 training complete
Epoch: 385
Epoch 0 training complete
Epoch: 386
Epoch 0 training complete
Epoch: 387
Epoch 0 training complete
Epoch: 388
Epoch 0 training complete
Epoch: 389
Epoch 0 training complete
Epoch: 390
Epoch 0 training complete
Epoch: 391
Epoch 0 training complete
Epoch: 392
Epoch 0 training complete
Epoch: 393
Epoch 0 training complete
Epoch: 394
Epoch 0 training complete
Epoch: 395
Epoch 0 training complete
Epoch: 396
Epoch 0 training complete
Epoch: 397
Epoch 0 training complete
Epoch: 398
Epoch 0 training complete
Epoch: 399
Epoch 0 training complete
Epoch: 400
Epoch 0 training complete
Epoch: 401
Epoch 0 training complete
Epoch: 402
Epoch 0 training complete
Epoch: 403
Epoch 0 training complete
Epoch: 404
Epoch 0 training complete
Epoch: 405
Epoch 0 training complete
Epoch: 406
Epoch 0 training complete
Epoch: 407
Epoch 0 training complete
Epoch: 408
Epoch 0 training complete
Epoch: 409
Epoch 0 training complete
Epoch: 410
Epoch 0 training complete
Epoch: 411
Epoch 0 training complete
Epoch: 412
Epoch 0 training complete
Epoch: 413
Epoch 0 training complete
Epoch: 414
Epoch 0 training complete
Epoch: 415
Epoch 0 training complete
Epoch: 416
Epoch 0 training complete
Epoch: 417
Epoch 0 training complete
Epoch: 418
Epoch 0 training complete
Epoch: 419
Epoch 0 training complete
Epoch: 420
Epoch 0 training complete
Epoch: 421
Epoch 0 training complete
Epoch: 422
Epoch 0 training complete
Epoch: 423
Epoch 0 training complete
Epoch: 424
Epoch 0 training complete
Epoch: 425
Epoch 0 training complete
Epoch: 426
Epoch 0 training complete
Epoch: 427
Epoch 0 training complete
Epoch: 428
Epoch 0 training complete
Epoch: 429
Epoch 0 training complete
Epoch: 430
Epoch 0 training complete
Epoch: 431
Epoch 0 training complete
Epoch: 432
Epoch 0 training complete
Epoch: 433
Epoch 0 training complete
Epoch: 434
Epoch 0 training complete
Epoch: 435
Epoch 0 training complete
Epoch: 436
Epoch 0 training complete
Epoch: 437
Epoch 0 training complete
Epoch: 438
Epoch 0 training complete
Epoch: 439
Epoch 0 training complete
Epoch: 440
Epoch 0 training complete
Epoch: 441
Epoch 0 training complete
Epoch: 442
Epoch 0 training complete
Epoch: 443
Epoch 0 training complete
Epoch: 444
Epoch 0 training complete
Epoch: 445
Epoch 0 training complete
Epoch: 446
Epoch 0 training complete
Epoch: 447
Epoch 0 training complete
Epoch: 448
Epoch 0 training complete
Epoch: 449
Epoch 0 training complete
Epoch: 450
Epoch 0 training complete
Epoch: 451
Epoch 0 training complete
Epoch: 452
Epoch 0 training complete
Epoch: 453
Epoch 0 training complete
Epoch: 454
Epoch 0 training complete
Epoch: 455
Epoch 0 training complete
Epoch: 456
Epoch 0 training complete
Epoch: 457
Epoch 0 training complete
Epoch: 458
Epoch 0 training complete
Epoch: 459
Epoch 0 training complete
Epoch: 460
Epoch 0 training complete
Epoch: 461
Epoch 0 training complete
Epoch: 462
Epoch 0 training complete
Epoch: 463
Epoch 0 training complete
Epoch: 464
Epoch 0 training complete
Epoch: 465
Epoch 0 training complete
Epoch: 466
Epoch 0 training complete
Epoch: 467
Epoch 0 training complete
Epoch: 468
Epoch 0 training complete
Epoch: 469
Epoch 0 training complete
Epoch: 470
Epoch 0 training complete
Epoch: 471
Epoch 0 training complete
Epoch: 472
Epoch 0 training complete
Epoch: 473
Epoch 0 training complete
Epoch: 474
Epoch 0 training complete
Epoch: 475
Epoch 0 training complete
Epoch: 476
Epoch 0 training complete
Epoch: 477
Epoch 0 training complete
Epoch: 478
Epoch 0 training complete
Epoch: 479
Epoch 0 training complete
Epoch: 480
Epoch 0 training complete
Epoch: 481
Epoch 0 training complete
Epoch: 482
Epoch 0 training complete
Epoch: 483
Epoch 0 training complete
Epoch: 484
Epoch 0 training complete
Epoch: 485
Epoch 0 training complete
Epoch: 486
Epoch 0 training complete
Epoch: 487
Epoch 0 training complete
Epoch: 488
Epoch 0 training complete
Epoch: 489
Epoch 0 training complete
Epoch: 490
Epoch 0 training complete
Epoch: 491
Epoch 0 training complete
Epoch: 492
Epoch 0 training complete
Epoch: 493
Epoch 0 training complete
Epoch: 494
Epoch 0 training complete
Epoch: 495
Epoch 0 training complete
Epoch: 496
Epoch 0 training complete
Epoch: 497
Epoch 0 training complete
Epoch: 498
Epoch 0 training complete
Epoch: 499
Epoch 0 training complete

Four hidden layers:
Average gradient for the hidden layers: [0.0027648931722101034, 0.010618815988259195, 0.05406007352024262, 0.3047345780970644]
Epoch: 0
Epoch 0 training complete
Epoch: 1
Epoch 0 training complete
Epoch: 2
Epoch 0 training complete
Epoch: 3
Epoch 0 training complete
Epoch: 4
Epoch 0 training complete
Epoch: 5
Epoch 0 training complete
Epoch: 6
Epoch 0 training complete
Epoch: 7
Epoch 0 training complete
Epoch: 8
Epoch 0 training complete
Epoch: 9
Epoch 0 training complete
Epoch: 10
Epoch 0 training complete
Epoch: 11
Epoch 0 training complete
Epoch: 12
Epoch 0 training complete
Epoch: 13
Epoch 0 training complete
Epoch: 14
Epoch 0 training complete
Epoch: 15
Epoch 0 training complete
Epoch: 16
Epoch 0 training complete
Epoch: 17
Epoch 0 training complete
Epoch: 18
Epoch 0 training complete
Epoch: 19
Epoch 0 training complete
Epoch: 20
Epoch 0 training complete
Epoch: 21
Epoch 0 training complete
Epoch: 22
Epoch 0 training complete
Epoch: 23
Epoch 0 training complete
Epoch: 24
Epoch 0 training complete
Epoch: 25
Epoch 0 training complete
Epoch: 26
Epoch 0 training complete
Epoch: 27
Epoch 0 training complete
Epoch: 28
Epoch 0 training complete
Epoch: 29
Epoch 0 training complete
Epoch: 30
Epoch 0 training complete
Epoch: 31
Epoch 0 training complete
Epoch: 32
Epoch 0 training complete
Epoch: 33
Epoch 0 training complete
Epoch: 34
Epoch 0 training complete
Epoch: 35
Epoch 0 training complete
Epoch: 36
Epoch 0 training complete
Epoch: 37
Epoch 0 training complete
Epoch: 38
Epoch 0 training complete
Epoch: 39
Epoch 0 training complete
Epoch: 40
Epoch 0 training complete
Epoch: 41
Epoch 0 training complete
Epoch: 42
Epoch 0 training complete
Epoch: 43
Epoch 0 training complete
Epoch: 44
Epoch 0 training complete
Epoch: 45
Epoch 0 training complete
Epoch: 46
Epoch 0 training complete
Epoch: 47
Epoch 0 training complete
Epoch: 48
Epoch 0 training complete
Epoch: 49
Epoch 0 training complete
Epoch: 50
Epoch 0 training complete
Epoch: 51
Epoch 0 training complete
Epoch: 52
Epoch 0 training complete
Epoch: 53
Epoch 0 training complete
Epoch: 54
Epoch 0 training complete
Epoch: 55
Epoch 0 training complete
Epoch: 56
Epoch 0 training complete
Epoch: 57
Epoch 0 training complete
Epoch: 58
Epoch 0 training complete
Epoch: 59
Epoch 0 training complete
Epoch: 60
Epoch 0 training complete
Epoch: 61
Epoch 0 training complete
Epoch: 62
Epoch 0 training complete
Epoch: 63
Epoch 0 training complete
Epoch: 64
Epoch 0 training complete
Epoch: 65
Epoch 0 training complete
Epoch: 66
Epoch 0 training complete
Epoch: 67
Epoch 0 training complete
Epoch: 68
Epoch 0 training complete
Epoch: 69
Epoch 0 training complete
Epoch: 70
Epoch 0 training complete
Epoch: 71
Epoch 0 training complete
Epoch: 72
Epoch 0 training complete
Epoch: 73
Epoch 0 training complete
Epoch: 74
Epoch 0 training complete
Epoch: 75
Epoch 0 training complete
Epoch: 76
Epoch 0 training complete
Epoch: 77
Epoch 0 training complete
Epoch: 78
Epoch 0 training complete
Epoch: 79
Epoch 0 training complete
Epoch: 80
Epoch 0 training complete
Epoch: 81
Epoch 0 training complete
Epoch: 82
Epoch 0 training complete
Epoch: 83
Epoch 0 training complete
Epoch: 84
Epoch 0 training complete
Epoch: 85
Epoch 0 training complete
Epoch: 86
Epoch 0 training complete
Epoch: 87
Epoch 0 training complete
Epoch: 88
Epoch 0 training complete
Epoch: 89
Epoch 0 training complete
Epoch: 90
Epoch 0 training complete
Epoch: 91
Epoch 0 training complete
Epoch: 92
Epoch 0 training complete
Epoch: 93
Epoch 0 training complete
Epoch: 94
Epoch 0 training complete
Epoch: 95
Epoch 0 training complete
Epoch: 96
Epoch 0 training complete
Epoch: 97
Epoch 0 training complete
Epoch: 98
Epoch 0 training complete
Epoch: 99
Epoch 0 training complete
Epoch: 100
Epoch 0 training complete
Epoch: 101
Epoch 0 training complete
Epoch: 102
Epoch 0 training complete
Epoch: 103
Epoch 0 training complete
Epoch: 104
Epoch 0 training complete
Epoch: 105
Epoch 0 training complete
Epoch: 106
Epoch 0 training complete
Epoch: 107
Epoch 0 training complete
Epoch: 108
Epoch 0 training complete
Epoch: 109
Epoch 0 training complete
Epoch: 110
Epoch 0 training complete
Epoch: 111
Epoch 0 training complete
Epoch: 112
Epoch 0 training complete
Epoch: 113
Epoch 0 training complete
Epoch: 114
Epoch 0 training complete
Epoch: 115
Epoch 0 training complete
Epoch: 116
Epoch 0 training complete
Epoch: 117
Epoch 0 training complete
Epoch: 118
Epoch 0 training complete
Epoch: 119
Epoch 0 training complete
Epoch: 120
Epoch 0 training complete
Epoch: 121
Epoch 0 training complete
Epoch: 122
Epoch 0 training complete
Epoch: 123
Epoch 0 training complete
Epoch: 124
Epoch 0 training complete
Epoch: 125
Epoch 0 training complete
Epoch: 126
Epoch 0 training complete
Epoch: 127
Epoch 0 training complete
Epoch: 128
Epoch 0 training complete
Epoch: 129
Epoch 0 training complete
Epoch: 130
Epoch 0 training complete
Epoch: 131
Epoch 0 training complete
Epoch: 132
Epoch 0 training complete
Epoch: 133
Epoch 0 training complete
Epoch: 134
Epoch 0 training complete
Epoch: 135
Epoch 0 training complete
Epoch: 136
Epoch 0 training complete
Epoch: 137
Epoch 0 training complete
Epoch: 138
Epoch 0 training complete
Epoch: 139
Epoch 0 training complete
Epoch: 140
Epoch 0 training complete
Epoch: 141
Epoch 0 training complete
Epoch: 142
Epoch 0 training complete
Epoch: 143
Epoch 0 training complete
Epoch: 144
Epoch 0 training complete
Epoch: 145
Epoch 0 training complete
Epoch: 146
Epoch 0 training complete
Epoch: 147
Epoch 0 training complete
Epoch: 148
Epoch 0 training complete
Epoch: 149
Epoch 0 training complete
Epoch: 150
Epoch 0 training complete
Epoch: 151
Epoch 0 training complete
Epoch: 152
Epoch 0 training complete
Epoch: 153
Epoch 0 training complete
Epoch: 154
Epoch 0 training complete
Epoch: 155
Epoch 0 training complete
Epoch: 156
Epoch 0 training complete
Epoch: 157
Epoch 0 training complete
Epoch: 158
Epoch 0 training complete
Epoch: 159
Epoch 0 training complete
Epoch: 160
Epoch 0 training complete
Epoch: 161
Epoch 0 training complete
Epoch: 162
Epoch 0 training complete
Epoch: 163
Epoch 0 training complete
Epoch: 164
Epoch 0 training complete
Epoch: 165
Epoch 0 training complete
Epoch: 166
Epoch 0 training complete
Epoch: 167
Epoch 0 training complete
Epoch: 168
Epoch 0 training complete
Epoch: 169
Epoch 0 training complete
Epoch: 170
Epoch 0 training complete
Epoch: 171
Epoch 0 training complete
Epoch: 172
Epoch 0 training complete
Epoch: 173
Epoch 0 training complete
Epoch: 174
Epoch 0 training complete
Epoch: 175
Epoch 0 training complete
Epoch: 176
Epoch 0 training complete
Epoch: 177
Epoch 0 training complete
Epoch: 178
Epoch 0 training complete
Epoch: 179
Epoch 0 training complete
Epoch: 180
Epoch 0 training complete
Epoch: 181
Epoch 0 training complete
Epoch: 182
Epoch 0 training complete
Epoch: 183
Epoch 0 training complete
Epoch: 184
Epoch 0 training complete
Epoch: 185
Epoch 0 training complete
Epoch: 186
Epoch 0 training complete
Epoch: 187
Epoch 0 training complete
Epoch: 188
Epoch 0 training complete
Epoch: 189
Epoch 0 training complete
Epoch: 190
Epoch 0 training complete
Epoch: 191
Epoch 0 training complete
Epoch: 192
Epoch 0 training complete
Epoch: 193
Epoch 0 training complete
Epoch: 194
Epoch 0 training complete
Epoch: 195
Epoch 0 training complete
Epoch: 196
Epoch 0 training complete
Epoch: 197
Epoch 0 training complete
Epoch: 198
Epoch 0 training complete
Epoch: 199
Epoch 0 training complete
Epoch: 200
Epoch 0 training complete
Epoch: 201
Epoch 0 training complete
Epoch: 202
Epoch 0 training complete
Epoch: 203
Epoch 0 training complete
Epoch: 204
Epoch 0 training complete
Epoch: 205
Epoch 0 training complete
Epoch: 206
Epoch 0 training complete
Epoch: 207
Epoch 0 training complete
Epoch: 208
Epoch 0 training complete
Epoch: 209
Epoch 0 training complete
Epoch: 210
Epoch 0 training complete
Epoch: 211
Epoch 0 training complete
Epoch: 212
Epoch 0 training complete
Epoch: 213
Epoch 0 training complete
Epoch: 214
Epoch 0 training complete
Epoch: 215
Epoch 0 training complete
Epoch: 216
Epoch 0 training complete
Epoch: 217
Epoch 0 training complete
Epoch: 218
Epoch 0 training complete
Epoch: 219
Epoch 0 training complete
Epoch: 220
Epoch 0 training complete
Epoch: 221
Epoch 0 training complete
Epoch: 222
Epoch 0 training complete
Epoch: 223
Epoch 0 training complete
Epoch: 224
Epoch 0 training complete
Epoch: 225
Epoch 0 training complete
Epoch: 226
Epoch 0 training complete
Epoch: 227
Epoch 0 training complete
Epoch: 228
Epoch 0 training complete
Epoch: 229
Epoch 0 training complete
Epoch: 230
Epoch 0 training complete
Epoch: 231
Epoch 0 training complete
Epoch: 232
Epoch 0 training complete
Epoch: 233
Epoch 0 training complete
Epoch: 234
Epoch 0 training complete
Epoch: 235
Epoch 0 training complete
Epoch: 236
Epoch 0 training complete
Epoch: 237
Epoch 0 training complete
Epoch: 238
Epoch 0 training complete
Epoch: 239
Epoch 0 training complete
Epoch: 240
Epoch 0 training complete
Epoch: 241
Epoch 0 training complete
Epoch: 242
Epoch 0 training complete
Epoch: 243
Epoch 0 training complete
Epoch: 244
Epoch 0 training complete
Epoch: 245
Epoch 0 training complete
Epoch: 246
Epoch 0 training complete
Epoch: 247
Epoch 0 training complete
Epoch: 248
Epoch 0 training complete
Epoch: 249
Epoch 0 training complete
Epoch: 250
Epoch 0 training complete
Epoch: 251
Epoch 0 training complete
Epoch: 252
Epoch 0 training complete
Epoch: 253
Epoch 0 training complete
Epoch: 254
Epoch 0 training complete
Epoch: 255
Epoch 0 training complete
Epoch: 256
Epoch 0 training complete
Epoch: 257
Epoch 0 training complete
Epoch: 258
Epoch 0 training complete
Epoch: 259
Epoch 0 training complete
Epoch: 260
Epoch 0 training complete
Epoch: 261
Epoch 0 training complete
Epoch: 262
Epoch 0 training complete
Epoch: 263
Epoch 0 training complete
Epoch: 264
Epoch 0 training complete
Epoch: 265
Epoch 0 training complete
Epoch: 266
Epoch 0 training complete
Epoch: 267
Epoch 0 training complete
Epoch: 268
Epoch 0 training complete
Epoch: 269
Epoch 0 training complete
Epoch: 270
Epoch 0 training complete
Epoch: 271
Epoch 0 training complete
Epoch: 272
Epoch 0 training complete
Epoch: 273
Epoch 0 training complete
Epoch: 274
Epoch 0 training complete
Epoch: 275
Epoch 0 training complete
Epoch: 276
Epoch 0 training complete
Epoch: 277
Epoch 0 training complete
Epoch: 278
Epoch 0 training complete
Epoch: 279
Epoch 0 training complete
Epoch: 280
Epoch 0 training complete
Epoch: 281
Epoch 0 training complete
Epoch: 282
Epoch 0 training complete
Epoch: 283
Epoch 0 training complete
Epoch: 284
Epoch 0 training complete
Epoch: 285
Epoch 0 training complete
Epoch: 286
Epoch 0 training complete
Epoch: 287
Epoch 0 training complete
Epoch: 288
Epoch 0 training complete
Epoch: 289
Epoch 0 training complete
Epoch: 290
Epoch 0 training complete
Epoch: 291
Epoch 0 training complete
Epoch: 292
Epoch 0 training complete
Epoch: 293
Epoch 0 training complete
Epoch: 294
Epoch 0 training complete
Epoch: 295
Epoch 0 training complete
Epoch: 296
Epoch 0 training complete
Epoch: 297
Epoch 0 training complete
Epoch: 298
Epoch 0 training complete
Epoch: 299
Epoch 0 training complete
Epoch: 300
Epoch 0 training complete
Epoch: 301
Epoch 0 training complete
Epoch: 302
Epoch 0 training complete
Epoch: 303
Epoch 0 training complete
Epoch: 304
Epoch 0 training complete
Epoch: 305
Epoch 0 training complete
Epoch: 306
Epoch 0 training complete
Epoch: 307
Epoch 0 training complete
Epoch: 308
Epoch 0 training complete
Epoch: 309
Epoch 0 training complete
Epoch: 310
Epoch 0 training complete
Epoch: 311
Epoch 0 training complete
Epoch: 312
Epoch 0 training complete
Epoch: 313
Epoch 0 training complete
Epoch: 314
Epoch 0 training complete
Epoch: 315
Epoch 0 training complete
Epoch: 316
Epoch 0 training complete
Epoch: 317
Epoch 0 training complete
Epoch: 318
Epoch 0 training complete
Epoch: 319
Epoch 0 training complete
Epoch: 320
Epoch 0 training complete
Epoch: 321
Epoch 0 training complete
Epoch: 322
Epoch 0 training complete
Epoch: 323
Epoch 0 training complete
Epoch: 324
Epoch 0 training complete
Epoch: 325
Epoch 0 training complete
Epoch: 326
Epoch 0 training complete
Epoch: 327
Epoch 0 training complete
Epoch: 328
Epoch 0 training complete
Epoch: 329
Epoch 0 training complete
Epoch: 330
Epoch 0 training complete
Epoch: 331
Epoch 0 training complete
Epoch: 332
Epoch 0 training complete
Epoch: 333
Epoch 0 training complete
Epoch: 334
Epoch 0 training complete
Epoch: 335
Epoch 0 training complete
Epoch: 336
Epoch 0 training complete
Epoch: 337
Epoch 0 training complete
Epoch: 338
Epoch 0 training complete
Epoch: 339
Epoch 0 training complete
Epoch: 340
Epoch 0 training complete
Epoch: 341
Epoch 0 training complete
Epoch: 342
Epoch 0 training complete
Epoch: 343
Epoch 0 training complete
Epoch: 344
Epoch 0 training complete
Epoch: 345
Epoch 0 training complete
Epoch: 346
Epoch 0 training complete
Epoch: 347
Epoch 0 training complete
Epoch: 348
Epoch 0 training complete
Epoch: 349
Epoch 0 training complete
Epoch: 350
Epoch 0 training complete
Epoch: 351
Epoch 0 training complete
Epoch: 352
Epoch 0 training complete
Epoch: 353
Epoch 0 training complete
Epoch: 354
Epoch 0 training complete
Epoch: 355
Epoch 0 training complete
Epoch: 356
Epoch 0 training complete
Epoch: 357
Epoch 0 training complete
Epoch: 358
Epoch 0 training complete
Epoch: 359
Epoch 0 training complete
Epoch: 360
Epoch 0 training complete
Epoch: 361
Epoch 0 training complete
Epoch: 362
Epoch 0 training complete
Epoch: 363
Epoch 0 training complete
Epoch: 364
Epoch 0 training complete
Epoch: 365
Epoch 0 training complete
Epoch: 366
Epoch 0 training complete
Epoch: 367
Epoch 0 training complete
Epoch: 368
Epoch 0 training complete
Epoch: 369
Epoch 0 training complete
Epoch: 370
Epoch 0 training complete
Epoch: 371
Epoch 0 training complete
Epoch: 372
Epoch 0 training complete
Epoch: 373
Epoch 0 training complete
Epoch: 374
Epoch 0 training complete
Epoch: 375
Epoch 0 training complete
Epoch: 376
Epoch 0 training complete
Epoch: 377
Epoch 0 training complete
Epoch: 378
Epoch 0 training complete
Epoch: 379
Epoch 0 training complete
Epoch: 380
Epoch 0 training complete
Epoch: 381
Epoch 0 training complete
Epoch: 382
Epoch 0 training complete
Epoch: 383
Epoch 0 training complete
Epoch: 384
Epoch 0 training complete
Epoch: 385
Epoch 0 training complete
Epoch: 386
Epoch 0 training complete
Epoch: 387
Epoch 0 training complete
Epoch: 388
Epoch 0 training complete
Epoch: 389
Epoch 0 training complete
Epoch: 390
Epoch 0 training complete
Epoch: 391
Epoch 0 training complete
Epoch: 392
Epoch 0 training complete
Epoch: 393
Epoch 0 training complete
Epoch: 394
Epoch 0 training complete
Epoch: 395
Epoch 0 training complete
Epoch: 396
Epoch 0 training complete
Epoch: 397
Epoch 0 training complete
Epoch: 398
Epoch 0 training complete
Epoch: 399
Epoch 0 training complete
Epoch: 400
Epoch 0 training complete
Epoch: 401
Epoch 0 training complete
Epoch: 402
Epoch 0 training complete
Epoch: 403
Epoch 0 training complete
Epoch: 404
Epoch 0 training complete
Epoch: 405
Epoch 0 training complete
Epoch: 406
Epoch 0 training complete
Epoch: 407
Epoch 0 training complete
Epoch: 408
Epoch 0 training complete
Epoch: 409
Epoch 0 training complete
Epoch: 410
Epoch 0 training complete
Epoch: 411
Epoch 0 training complete
Epoch: 412
Epoch 0 training complete
Epoch: 413
Epoch 0 training complete
Epoch: 414
Epoch 0 training complete
Epoch: 415
Epoch 0 training complete
Epoch: 416
Epoch 0 training complete
Epoch: 417
Epoch 0 training complete
Epoch: 418
Epoch 0 training complete
Epoch: 419
Epoch 0 training complete
Epoch: 420
Epoch 0 training complete
Epoch: 421
Epoch 0 training complete
Epoch: 422
Epoch 0 training complete
Epoch: 423
Epoch 0 training complete
Epoch: 424
Epoch 0 training complete
Epoch: 425
Epoch 0 training complete
Epoch: 426
Epoch 0 training complete
Epoch: 427
Epoch 0 training complete
Epoch: 428
Epoch 0 training complete
Epoch: 429
Epoch 0 training complete
Epoch: 430
Epoch 0 training complete
Epoch: 431
Epoch 0 training complete
Epoch: 432
Epoch 0 training complete
Epoch: 433
Epoch 0 training complete
Epoch: 434
Epoch 0 training complete
Epoch: 435
Epoch 0 training complete
Epoch: 436
Epoch 0 training complete
Epoch: 437
Epoch 0 training complete
Epoch: 438
Epoch 0 training complete
Epoch: 439
Epoch 0 training complete
Epoch: 440
Epoch 0 training complete
Epoch: 441
Epoch 0 training complete
Epoch: 442
Epoch 0 training complete
Epoch: 443
Epoch 0 training complete
Epoch: 444
Epoch 0 training complete
Epoch: 445
Epoch 0 training complete
Epoch: 446
Epoch 0 training complete
Epoch: 447
Epoch 0 training complete
Epoch: 448
Epoch 0 training complete
Epoch: 449
Epoch 0 training complete
Epoch: 450
Epoch 0 training complete
Epoch: 451
Epoch 0 training complete
Epoch: 452
Epoch 0 training complete
Epoch: 453
Epoch 0 training complete
Epoch: 454
Epoch 0 training complete
Epoch: 455
Epoch 0 training complete
Epoch: 456
Epoch 0 training complete
Epoch: 457
Epoch 0 training complete
Epoch: 458
Epoch 0 training complete
Epoch: 459
Epoch 0 training complete
Epoch: 460
Epoch 0 training complete
Epoch: 461
Epoch 0 training complete
Epoch: 462
Epoch 0 training complete
Epoch: 463
Epoch 0 training complete
Epoch: 464
Epoch 0 training complete
Epoch: 465
Epoch 0 training complete
Epoch: 466
Epoch 0 training complete
Epoch: 467
Epoch 0 training complete
Epoch: 468
Epoch 0 training complete
Epoch: 469
Epoch 0 training complete
Epoch: 470
Epoch 0 training complete
Epoch: 471
Epoch 0 training complete
Epoch: 472
Epoch 0 training complete
Epoch: 473
Epoch 0 training complete
Epoch: 474
Epoch 0 training complete
Epoch: 475
Epoch 0 training complete
Epoch: 476
Epoch 0 training complete
Epoch: 477
Epoch 0 training complete
Epoch: 478
Epoch 0 training complete
Epoch: 479
Epoch 0 training complete
Epoch: 480
Epoch 0 training complete
Epoch: 481
Epoch 0 training complete
Epoch: 482
Epoch 0 training complete
Epoch: 483
Epoch 0 training complete
Epoch: 484
Epoch 0 training complete
Epoch: 485
Epoch 0 training complete
Epoch: 486
Epoch 0 training complete
Epoch: 487
Epoch 0 training complete
Epoch: 488
Epoch 0 training complete
Epoch: 489
Epoch 0 training complete
Epoch: 490
Epoch 0 training complete
Epoch: 491
Epoch 0 training complete
Epoch: 492
Epoch 0 training complete
Epoch: 493
Epoch 0 training complete
Epoch: 494
Epoch 0 training complete
Epoch: 495
Epoch 0 training complete
Epoch: 496
Epoch 0 training complete
Epoch: 497
Epoch 0 training complete
Epoch: 498
Epoch 0 training complete
Epoch: 499
Epoch 0 training complete

Results

1. 2 hidden layers: $\|\delta^1\| = 0.059$, $\|\delta^2\| = 0.35$
2. 3 hidden layers: $\|\delta^1\| = 0.0228$, $\|\delta^2\| = 0.0804$, $\|\delta^3\| = 0.375$
3. 4 hidden layers: $\|\delta^1\| = 0.0028$, $\|\delta^2\| = 0.0106$, $\|\delta^3\| = 0.0541$, $\|\delta^4\| = 0.3047$
   
   
   

---

num. hidden layers	d1	d2	d3	d4
2	0.059	0.35
3	0.0228	0.0804	0.375
4	0.0028	0.0106	0.0541	0.3047

---

1. Two hidden layers: Average gradient for the hidden layers: [0.05910799398171015, 0.3504939212704065]

2. Three hidden layers: Average gradient for the hidden layers: [0.022818952596379727, 0.0804102169606713, 0.3754301367980058]

3. Four hidden layers: Average gradient for the hidden layers: [0.0027648931722101034, 0.010618815988259195, 0.05406007352024262, 0.3047345780970644]

---

What's causing the vanishing gradient problem? Unstable gradients in deep neural nets

---

To get insight into why the vanishing gradient problem occurs, let's consider the simplest deep neural network: one with just a single neuron in each layer. Here's a network with three hidden layers:

[] -----> [ b1 ] ---w2--> [ b2 ] ---w3--> [ b3 ] ---w4--> [ b4 ]---> C

Here, $w_1, w_2, \ldots$ are the weights, $b_1, b_2, \ldots$ are the biases, and $C$ is some cost function. Just to remind you how this works, the output $a_j$ from the $j$th neuron is $\sigma(z_j)$, where $\sigma$ is the usual sigmoid activation function, and $z_j = w\_{j} a\_{j-1}+b_j$ is the weighted input to the neuron. I've drawn the cost $C$ at the end to emphasize that the cost is a function of the network's output, $a_4$: if the actual output from the network is close to the desired output, then the cost will be low, while if it's far away, the cost will be high.

We're going to study the gradient $\partial C / \partial b_1$ associated to the first hidden neuron. We'll figure out an expression for $\partial C / \partial b_1$, and by studying that expression we'll understand why the vanishing gradient problem occurs.

I'll start by simply showing you the expression for $\partial C / \partial b_1$. It looks forbidding, but it's actually got a simple structure, which I'll describe in a moment. Here's the expression (ignore the network, for now, and note that $\sigma'$ is just the derivative of the $\sigma$ function):

\begin{eqnarray} \frac{\partial C}{\partial b_1} = \sigma'(z_1) \times \ w_2 \times \sigma'(z_2) \times \ w_3 \times \sigma'(z_3) \times \ w_4 \times \sigma'(z_4) \times \frac{\partial C}{\partial a_4}. \end{eqnarray}

         [] -----> [ b1 ] ---w2--> [ b2 ] ---w3--> [ b3 ] ---w4--> [ b4 ]---> C

The structure in the expression is as follows: there is a $\sigma'(z_j)$ term in the product for each neuron in the network; a weight $w_j$ term for each weight in the network; and a final $\partial C / \partial a_4$ term, corresponding to the cost function at the end. Notice that I've placed each term in the expression above the corresponding part of the network. So the network itself is a mnemonic for the expression.

You're welcome to take this expression for granted, and skip to the discussion of how it relates to the vanishing gradient problem. There's no harm in doing this, since the expression is a special case of our earlier discussion of backpropagation. But there's also a simple explanation of why the expression is true, and so it's fun (and perhaps enlightening) to take a look at that explanation.

Imagine we make a small change $\Delta b_1$ in the bias $b_1$. That will set off a cascading series of changes in the rest of the network. First, it causes a change $\Delta a_1$ in the output from the first hidden neuron. That, in turn, will cause a change $\Delta z_2$ in the weighted input to the second hidden neuron. Then a change $\Delta a_2$ in the output from the second hidden neuron. And so on, all the way through to a change $\Delta C$ in the cost at the output. We have

\begin{eqnarray} \frac{\partial C}{\partial b_1} \approx \frac{\Delta C}{\Delta b_1}. \tag{1}\end{eqnarray}

This suggests that we can figure out an expression for the gradient $\partial C / \partial b_1$ by carefully tracking the effect of each step in this cascade.

To do this, let's think about how $\Delta b_1$ causes the output $a_1$ from the first hidden neuron to change. We have $a_1 = \sigma(z_1) = \sigma(w_1 a_0 + b_1)$, so

\begin{eqnarray} \Delta a_1 & \approx & \frac{\partial \sigma(w_1 a_0+b_1)}{\partial b_1} \Delta b_1 \tag{2} \\ & = & \sigma'(z_1) \Delta b_1. \tag{3}\end{eqnarray}

That $\sigma'(z_1)$ term should look familiar: it's the first term in our claimed expression for the gradient $\partial C / \partial b_1$. Intuitively, this term converts a change $\Delta b_1$ in the bias into a change $\Delta a_1$ in the output activation. That change $\Delta a_1$ in turn causes a change in the weighted input $z_2 = w_2 a_1 + b_2$ to the second hidden neuron:

\begin{eqnarray} \Delta z_2 & \approx & \frac{\partial z_2}{\partial a_1} \Delta a_1 \tag{4} \\ & = & w_2 \Delta a_1. \tag{5}\end{eqnarray}

Combining our expressions for $\Delta z_2$ and $\Delta a_1$, we see how the change in the bias $b_1$ propagates along the network to affect $z_2$:

\begin{eqnarray} \Delta z_2 & \approx & \sigma'(z_1) w_2 \Delta b_1. \tag{6}\end{eqnarray}

Again, that should look familiar: we've now got the first two terms in our claimed expression for the gradient $\partial C / \partial b_1$.

We can keep going in this fashion, tracking the way changes propagate through the rest of the network. At each neuron we pick up a $\sigma'(z_j)$ term, and through each weight we pick up a $w_j$ term. The end result is an expression relating the final change $\Delta C$ in cost to the initial change $\Delta b_1$ in the bias:

\begin{eqnarray} \Delta C & \approx & \sigma'(z_1) w_2 \sigma'(z_2) \ldots \sigma'(z_4) \frac{\partial C}{\partial a_4} \Delta b_1. \tag{7}\end{eqnarray}

Dividing by $\Delta b_1$ we do indeed get the desired expression for the gradient:

\begin{eqnarray} \frac{\partial C}{\partial b_1} = \sigma'(z_1) w_2 \sigma'(z_2) \ldots \sigma'(z_4) \frac{\partial C}{\partial a_4}. \tag{8}\end{eqnarray}

---

Why the vanishing gradient problem occurs: To understand why the vanishing gradient problem occurs, let's explicitly write out the entire expression for the gradient:

\begin{eqnarray} \frac{\partial C}{\partial b_1} = \sigma'(z_1) \times \ w_2 \times \sigma'(z_2) \times \ w_3 \times \sigma'(z_3) \times \ w_4 \times \sigma'(z_4) \times \frac{\partial C}{\partial a_4}. \tag{9}\end{eqnarray}

Excepting the very last term, this expression is a product of terms of the form $w_j \sigma'(z_j)$. To understand how each of those terms behave, let's look at a plot of the function $\sigma'$:

# Generate x values
x = np.linspace(-10, 10, 400)

# Calculate y values for the sigmoid and its derivative
y_sigmoid = sigmoid(x)
y_sigmoid_derivative = sigmoid_prime(x)

# Plot the sigmoid and its derivative
plt.figure(figsize=(8, 6))
plt.plot(x, y_sigmoid, label="Sigmoid", color='blue')
plt.plot(x, y_sigmoid_derivative, label="Sigmoid Derivative", color='red', linestyle='dashed')
plt.title("Sigmoid Function and Its Derivative")
plt.xlabel("x")
plt.ylabel("y")
plt.grid(True)
plt.legend()
plt.show()

---

The derivative reaches a maximum at $\sigma'(0) = 1/4$. Now, if we use our standard approach to initializing the weights in the network, then we'll choose the weights using a Gaussian with mean $0$ and standard deviation $1$. So the weights will usually satisfy $\|w_j\| < 1$. Putting these observations together, we see that the terms $w_j \sigma'(z_j)$ will usually satisfy $\|w_j \sigma'(z_j)\| < 1/4$. And when we take a product of many such terms, the product will tend to exponentially decrease: the more terms, the smaller the product will be. This is starting to smell like a possible explanation for the vanishing gradient problem.

To make this all a bit more explicit, let's compare the expression for $\partial C / \partial b_1$ to an expression for the gradient with respect to a later bias, say $\partial C / \partial b_3$. Of course, we haven't explicitly worked out an expression for $\partial C / \partial b_3$, but it follows the same pattern described above for $\partial C / \partial b_1$. Here's the comparison of the two expressions:

\begin{eqnarray} \frac{\partial C}{\partial a_1} &= \sigma'(z_1) \underbrace{w_2 \sigma'(z_2)}_{< \frac{1}{4}} \underbrace{w_3 \sigma'(z_3)}_{< \frac{1}{4}} \underbrace{w_4 \sigma'(z_4) \frac{\partial C}{\partial a_4}}_{\text{common terms}} \end{eqnarray}

\begin{eqnarray} \\[10pt] \hspace{2em} \text{common terms:}→ w_4 \sigma'(z_4) \frac{\partial C}{\partial a_4} \end{eqnarray}

\begin{eqnarray} \\[10pt] \hspace{6em} \frac{\partial C}{\partial a_3} = \sigma'(z_3) w_4 \sigma'(z_4) \frac{\partial C}{\partial a_4} \\[10pt] \frac{\partial C}{\partial a_3} = \sigma'(z_3) \times \text{(common terms)} \end{eqnarray}

\begin{eqnarray} \\[10pt]\text{***} \end{eqnarray}

The two expressions share many terms. But the gradient $\partial C / \partial b_1$ includes two extra terms each of the form $w_j \sigma'(z_j)$. As we've seen, such terms are typically less than $1/4$ in magnitude. And so the gradient $\partial C / \partial b_1$ will usually be a factor of $16$ (or more) smaller than $\partial C / \partial b_3$. This is the essential origin of the vanishing gradient problem.

Of course, this is an informal argument, not a rigorous proof that the vanishing gradient problem will occur. There are several possible escape clauses. In particular, we might wonder whether the weights $w_j$ could grow during training. If they do, it's possible the terms $w_j \sigma'(z_j)$ in the product will no longer satisfy $\|w_j \sigma'(z_j)\| < 1/4$. Indeed, if the terms get large enough - greater than $1$ - then we will no longer have a vanishing gradient problem. Instead, the gradient will actually grow exponentially as we move backward through the layers. Instead of a vanishing gradient problem, we'll have an exploding gradient problem.

---

The exploding gradient problem: Let's look at an explicit example where exploding gradients occur. The example is somewhat contrived: I'm going to fix parameters in the network in just the right way to ensure we get an exploding gradient. But even though the example is contrived, it has the virtue of firmly establishing that exploding gradients aren't merely a hypothetical possibility, they really can happen.

There are two steps to getting an exploding gradient. First, we choose all the weights in the network to be large, say $w_1 = w_2 = w_3 = w_4 = 100$. Second, we'll choose the biases so that the $\sigma'(z_j)$ terms are not too small. That's actually pretty easy to do: all we need do is choose the biases to ensure that the weighted input to each neuron is $z_j = 0$ (and so $\sigma'(z_j) = 1/4$). So, for instance, we want $z_1 = w_1 a_0 + b_1 = 0$. We can achieve this by setting $b_1 = -100 * a_0$. We can use the same idea to select the other biases. When we do this, we see that all the terms $w_j \sigma'(z_j)$ are equal to $100 * \frac{1}{4} = 25$. With these choices we get an exploding gradient.

---

The unstable gradient problem: The fundamental problem here isn't so much the vanishing gradient problem or the exploding gradient problem. It's that the gradient in early layers is the product of terms from all the later layers. When there are many layers, that's an intrinsically unstable situation. The only way all layers can learn at close to the same speed is if all those products of terms come close to balancing out. Without some mechanism or underlying reason for that balancing to occur, it's highly unlikely to happen simply by chance. In short, the real problem here is that neural networks suffer from an unstable gradient problem. As a result, if we use standard gradient-based learning techniques, different layers in the network will tend to learn at wildly different speeds.

---

The prevalence of the vanishing gradient problem: We've seen that the gradient can either vanish or explode in the early layers of a deep network. In fact, when using sigmoid neurons the gradient will usually vanish. To see why, consider again the expression $\|w \sigma'(z)\|$. To avoid the vanishing gradient problem we need $\|w \sigma'(z)\| \geq 1$. You might think this could happen easily if $w$ is very large. However, it's more difficult than it looks. The reason is that the $\sigma'(z)$ term also depends on $w$: $\sigma'(z) = \sigma'(wa +b)$, where $a$ is the input activation. So when we make $w$ large, we need to be careful that we're not simultaneously making $\sigma'(wa+b)$ small. That turns out to be a considerable constraint. The reason is that when we make $w$ large we tend to make $wa+b$ very large. Looking at the graph of $\sigma'$ you can see that this puts us off in the "wings" of the $\sigma'$ function, where it takes very small values. The only way to avoid this is if the input activation falls within a fairly narrow range of values (this qualitative explanation is made quantitative in the problem 2 below). Sometimes that will chance to happen. More often, though, it does not happen. And so in the generic case we have vanishing gradients.

---

Unstable gradients in more complex networks

---

We've been studying toy networks, with just one neuron in each hidden layer. What about more complex deep networks, with many neurons in each hidden layer?

[input layer] -----> [ h1 ] -----> [ h2 ] -----> [ h3 ] -----> [output layer]---> C
   8 neurons       9 neurons      9 neurons     9 neurons        4 neurons

In fact, much the same behaviour occurs in such networks. In the earlier chapter on backpropagation we saw that the gradient in the $l$th layer of an $L$ layer network is given by:

\begin{eqnarray} \delta^l = \Sigma'(z^l) (w^{l+1})^T \Sigma'(z^{l+1}) (w^{l+2})^T \ldots \Sigma'(z^L) \nabla_a C \tag{10}\end{eqnarray}

Here, $\Sigma'(z^l)$ is a diagonal matrix whose entries are the $\sigma'(z)$ values for the weighted inputs to the $l$th layer. The $w^l$ are the weight matrices for the different layers. And $\nabla_a C$ is the vector of partial derivatives of $C$ with respect to the output activations.

This is a much more complicated expression than in the single-neuron case. Still, if you look closely, the essential form is very similar, with lots of pairs of the form $(w^j)^T \Sigma'(z^j)$. What's more, the matrices $\Sigma'(z^j)$ have small entries on the diagonal, none larger than $\frac{1}{4}$. Provided the weight matrices $w^j$ aren't too large, each additional term $(w^j)^T \Sigma'(z^l)$ tends to make the gradient vector smaller, leading to a vanishing gradient. More generally, the large number of terms in the product tends to lead to an unstable gradient, just as in our earlier example. In practice, empirically it is typically found in sigmoid networks that gradients vanish exponentially quickly in earlier layers. As a result, learning slows down in those layers. This slowdown isn't merely an accident or an inconvenience: it's a fundamental consequence of the approach we're taking to learning.

---

Other obstacles to deep learning

---

In this chapter we've focused on vanishing gradients - and, more generally, unstable gradients - as an obstacle to deep learning. In fact, unstable gradients are just one obstacle to deep learning, albeit an important fundamental obstacle. Much ongoing research aims to better understand the challenges that can occur when training deep networks. I won't comprehensively summarize that work here, but just want to briefly mention a couple of papers, to give you the flavor of some of the questions people are asking.

As a first example, in 2010 Glorot and Bengio**[3]** found evidence suggesting that the use of sigmoid activation functions can cause problems training deep networks. In particular, they found evidence that the use of sigmoids will cause the activations in the final hidden layer to saturate near $0$ early in training, substantially slowing down learning. They suggested some alternative activation functions, which appear not to suffer as much from this saturation problem.

As a second example, in 2013 Sutskever, Martens, Dahl and Hinton**[4]** studied the impact on deep learning of both the random weight initialization and the momentum schedule in momentum-based stochastic gradient descent. In both cases, making good choices made a substantial difference in the ability to train deep networks.

These examples suggest that "What makes deep networks hard to train?" is a complex question. In this chapter, we've focused on the instabilities associated to gradient-based learning in deep networks. The results in the last two paragraphs suggest that there is also a role played by the choice of activation function, the way weights are initialized, and even details of how learning by gradient descent is implemented. And, of course, choice of network architecture and other hyper-parameters is also important. Thus, many factors can play a role in making deep networks hard to train, and understanding all those factors is still a subject of ongoing research. This all seems rather downbeat and pessimism-inducing. But the good news is that in the next chapter we'll turn that around, and develop several approaches to deep learning that to some extent manage to overcome or route around all these challenges.

---

References:
3. Understanding the difficulty of training deep feedforward neural networks, by Xavier Glorot and Yoshua Bengio (2010). See also the earlier discussion of the use of sigmoids in Efficient BackProp, by Yann LeCun, Léon Bottou, Genevieve Orr and Klaus-Robert Müller (1998).
4. On the importance of initialization and momentum in deep learning, by Ilya Sutskever, James Martens, George Dahl and Geoffrey Hinton (2013).

---

Problem 1:

• In our discussion of the vanishing gradient problem, we made use of the fact that $\|\sigma'(z)\| < 1/4$. Suppose we used a different activation function, one whose derivative could be much larger. Would that help us avoid the unstable gradient problem?

---

Impact of Activation Function on the Vanishing Gradient Problem

Answer 1:

No, using an activation function with a larger derivative would not necessarily help avoid the unstable gradient problem. While the vanishing gradient problem is caused by very small gradients (such as the sigmoid or tanh functions where the derivative is less than 1/4), using an activation function whose derivative is much larger could lead to the opposite issue: exploding gradients.

Here's why:

1. Vanishing gradients: When the gradient is too small, it shrinks as it backpropagates through the layers, which slows down learning in earlier layers. This is commonly seen with activation functions like the sigmoid and tanh because their derivatives are small in most of their domain.

2. Exploding gradients: If the derivative of the activation function is too large, the gradients can grow exponentially as they are backpropagated through the layers. This leads to instability in the network, making it harder for the network to converge during training. Activation functions with large derivatives could cause this problem.

Solution:

The solution to the unstable gradient problem is not necessarily to have larger gradients but rather to have gradients of manageable size. Modern deep learning practices mitigate both vanishing and exploding gradients by using:

• ReLU (Rectified Linear Unit) activation function, which has a derivative of 1 for positive inputs and 0 for negative inputs. This helps maintain a balance in gradient flow.

• Batch normalization: This technique normalizes the inputs to each layer, reducing the chances of vanishing or exploding gradients.

• Proper weight initialization, such as Xavier or He initialization, to ensure that gradients neither vanish nor explode at the start of training.

So, simply increasing the derivative of the activation function would not prevent the unstable gradient problem and may introduce a new problem in the form of exploding gradients.

---

Answer 2:

The question poses an interesting scenario: What if we used an activation function whose derivative could be much larger than that of the sigmoid function (where $\|\sigma'(z)\| < 1/4$)? Would this help avoid the unstable gradient problem?

Analysis

1. Recall the vanishing gradient problem:

   • In deep networks, as we backpropagate, we multiply many small numbers (derivatives of the activation function) together.
   • This can cause the gradient to become extremely small, effectively "vanishing" as it reaches earlier layers.

2. The role of the activation function:

   • The sigmoid function has a maximum derivative of 1/4, which contributes to the vanishing gradient problem.
   • An activation function with a larger derivative might seem like a solution at first glance.

3. Potential benefits of a larger derivative:

   • It could potentially allow the gradient to propagate further back through the network without becoming too small.
   • This might help in training deeper networks more effectively.

4. However, there are significant drawbacks:

   • If the derivative is too large (significantly greater than 1), it can lead to the opposite problem: exploding gradients.
   • Exploding gradients occur when the gradient becomes extremely large, causing unstable updates during training.

5. The instability issue:

   • Large derivatives can amplify small changes in the input, leading to drastic changes in the output.
   • This instability can make the training process highly sensitive and difficult to converge.

6. The need for balance:

   • The ideal activation function should have derivatives that are neither too small nor too large.
   • This is why activation functions like ReLU (Rectified Linear Unit) have become popular:
     • ReLU has a derivative of 1 for positive inputs and 0 for negative inputs.
     • This helps mitigate both vanishing and exploding gradient problems to some extent.

Conclusion

While using an activation function with a larger derivative might help with the vanishing gradient problem to some degree, it's not a complete solution and can introduce new problems. The key is to find a balance that allows gradients to flow effectively through the network without becoming unstable.

Modern deep learning practices often involve a combination of techniques to address gradient issues:

• Careful initialization of weights
• Batch normalization
• Skip connections (as in ResNets)
• Alternative activation functions (like ReLU and its variants)
• Gradient clipping

These approaches, used in conjunction, help manage both vanishing and exploding gradient problems more effectively than simply increasing the magnitude of the activation function's derivative.

---

Proof 1 for Problem 1:

Mathematical Proof: Impact of Activation Function's Derivative on Gradient Propagation

To mathematically explain why using an activation function with a larger derivative doesn’t necessarily prevent the unstable gradient problem, we need to examine how gradients propagate through a neural network during backpropagation.

Key Terms:

• Vanishing Gradient Problem: The gradients become very small as they propagate through layers, making learning slow or stopping entirely.
• Exploding Gradient Problem: The gradients grow exponentially large as they propagate through layers, causing instability during training.

Let’s break this down using a simple neural network:

General Form of Backpropagation:

For a multi-layer neural network, the gradient of the loss function $L$ with respect to the weights $W^{(l)}$ at layer $l$ is:

$$\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} \cdot h^{(l-1)}$$

Where:

• $\delta^{(l)}$ is the gradient of the error at layer $l$.
• $h^{(l-1)}$ is the output (activation) from the previous layer.

The error gradient at layer $l$, $\delta^{(l)}$, is calculated recursively using the chain rule:

$$\delta^{(l)} = \left( W^{(l+1)} \cdot \delta^{(l+1)} \right) \odot \sigma'(z^{(l)})$$

Where:

• $W^{(l+1)}$ are the weights of the next layer.
• $\delta^{(l+1)}$ is the gradient of the next layer.
• $\sigma'(z^{(l)})$ is the derivative of the activation function at layer $l$, where $z^{(l)}$ is the input to the activation function.

Vanishing Gradient:

In networks with deep layers, when using activation functions like sigmoid or tanh, whose derivatives $\sigma'(z)$ are less than 1 in most of their domain, this product $\sigma'(z)$ keeps reducing as it backpropagates through the layers:

$$\delta^{(l)} = \left( W^{(l+1)} \cdot \delta^{(l+1)} \right) \odot \sigma'(z^{(l)})$$

If $\sigma'(z^{(l)})$ is very small (e.g., $\sigma'(z) \leq \frac{1}{4}$ for sigmoid), the gradients shrink as they propagate back through layers. This leads to a vanishing gradient, where the earlier layers learn very slowly or stop learning entirely.

Exploding Gradient:

Now, suppose the derivative $\sigma'(z)$ is much larger than 1 (which is common with activation functions whose gradients are unbounded, like $\sigma'(z) > 1$):

$$\delta^{(l)} = \left( W^{(l+1)} \cdot \delta^{(l+1)} \right) \odot \sigma'(z^{(l)})$$

If $\sigma'(z^{(l)})$ is large (e.g., $\sigma'(z) \gg 1$), then this multiplication causes the gradients to grow exponentially as they propagate back through the layers. This results in an exploding gradient problem, where the network's updates become too large and the model fails to converge.

Proof of Instability:

Let's consider a simplified case where the same activation function is used in each layer and assume for simplicity that the network has identical weight matrices $W$ and inputs $z$ in every layer. Let the derivative of the activation function $\sigma'(z)$ be denoted by $\gamma$, so the gradient at layer $l$ becomes:

$$\delta^{(l)} = \left( W \cdot \delta^{(l+1)} \right) \odot \gamma$$

Then, for $L$ layers, the gradient at layer 1 becomes:

$$\delta^{(1)} = \left( W^L \cdot \delta^{(L)} \right) \odot \gamma^L$$

• If $\gamma \ll 1$, the term $\gamma^L$ becomes very small as $L$ increases, leading to vanishing gradients.
• If $\gamma \gg 1$, the term $\gamma^L$ grows exponentially, leading to exploding gradients.

Conclusion:

The size of the derivative $\sigma'(z)$ impacts whether gradients will vanish or explode. If $\sigma'(z)$ is too small, the gradients vanish; if $\sigma'(z)$ is too large, the gradients explode. Therefore, just increasing the derivative of the activation function will not solve the unstable gradient problem---it will only shift the problem from vanishing gradients to exploding gradients. The goal should be to maintain manageable gradients, which can be achieved with techniques like ReLU activation, proper initialization (e.g., Xavier or He), and normalization (e.g., batch normalization).

---

Proof 2 for Problem 1:

Let's examine mathematically how the magnitude of an activation function's derivative affects gradient propagation in a neural network.

Setup

Consider a deep neural network with $L$ layers. Let:

• $w^l$ be the weights of layer $l$
• $z^l = w^l a^{l-1} + b^l$ be the weighted input to layer $l$
• $a^l = \sigma(z^l)$ be the activation of layer $l$
• $\sigma$ be the activation function
• $C$ be the cost function

Gradient Calculation

The gradient of the cost with respect to any weight in layer $l$ is given by the chain rule:

$$\frac{\partial C}{\partial w^l_{jk}} = \frac{\partial C}{\partial a^L} \cdot \frac{\partial a^L}{\partial z^L} \cdot \frac{\partial z^L}{\partial a^{L-1}} \cdot \frac{\partial a^{L-1}}{\partial z^{L-1}} \cdot ... \cdot \frac{\partial a^{l+1}}{\partial z^{l+1}} \cdot \frac{\partial z^{l+1}}{\partial a^l} \cdot \frac{\partial a^l}{\partial z^l} \cdot \frac{\partial z^l}{\partial w^l_{jk}}$$

Analysis

1. Notice that $\frac{\partial a^m}{\partial z^m} = \sigma'(z^m)$ for each layer $m$.

2. Let's assume $|\sigma'(z)| \leq M$ for some constant $M$. For the sigmoid function, $M = 1/4$.

3. We can bound the magnitude of the gradient:

   $$\left|\frac{\partial C}{\partial w^l_{jk}}\right| \leq \left|\frac{\partial C}{\partial a^L}\right| \cdot M^{L-l} \cdot \prod_{m=l+1}^L \|w^m\| \cdot \left|\frac{\partial z^l}{\partial w^l_{jk}}\right|$$

4. Now, let's consider two cases:

   a) If $M < 1$ (as with sigmoid):

   • As $L - l$ increases (i.e., for earlier layers), $M^{L-l}$ becomes very small.
   • This leads to the vanishing gradient problem.

   b) If $M > 1$ (activation function with larger derivative):

   • As $L - l$ increases, $M^{L-l}$ becomes very large.
   • This can lead to the exploding gradient problem.

The Impact of $M$

• When $M < 1$: $$\lim_{L-l \to \infty} M^{L-l} = 0$$

  This causes the gradient to vanish for early layers in very deep networks.

• When $M > 1$: $$\lim_{L-l \to \infty} M^{L-l} = \infty$$

  This causes the gradient to explode for early layers in very deep networks.

Conclusion

The ideal scenario is when $M \approx 1$:

• This allows the gradient to propagate through the network without vanishing or exploding.
• The ReLU activation function achieves this for positive inputs, as its derivative is exactly 1 for $z > 0$.

However, simply having $M \approx 1$ is not always sufficient, as the product of weight matrices $\prod_{m=l+1}^L \|w^m\|$ also plays a crucial role in gradient propagation. This is why techniques like careful weight initialization, batch normalization, and skip connections are important in modern deep learning architectures.

---

Problem 2:

• Consider the product $\|w \sigma'(wa+b)\|$. Suppose $\|w \sigma'(wa+b)\| \geq 1$.

  1. Argue that this can only ever occur if $\|w\| \geq 4$.

  2. Supposing that $\|w\| \geq 4$, consider the set of input activations $a$ for which $\|w \sigma'(wa+b)\| \geq 1$. Show that the set of $a$ satisfying that constraint can range over an interval no greater in width than \begin{eqnarray} \frac{2}{\|w\|} \ln\left( \frac{\|w\|(1+\sqrt{1-4/\|w\|})}{2}-1\right). \tag{A}\end{eqnarray}

  3. Show numerically that the above expression bounding the width of the range is greatest at $\|w\| \approx 6.9$, where it takes a value $\approx 0.45$. And so even given that everything lines up just perfectly, we still have a fairly narrow range of input activations which can avoid the vanishing gradient problem.

• Identity neuron: Consider a neuron with a single input, $x$, a corresponding weight, $w_1$, a bias $b$, and a weight $w_2$ on the output. Show that by choosing the weights and bias appropriately, we can ensure $w_2 \sigma(w_1 x+b) \approx x$ for $x \in [0, 1]$. Such a neuron can thus be used as a kind of identity neuron, that is, a neuron whose output is the same (up to rescaling by a weight factor) as its input.

Hint: It helps to rewrite $x = 1/2+\Delta$, to assume $w_1$ is small, and to use a Taylor series expansion in $w_1 \Delta$.

---

---

Answer 1:

Problem A

(1) Argue that $\|w \sigma'(wa + b)\| \geq 1$ can only ever occur if $\|w\| \geq 4$:

To show this, we start by analyzing the sigmoid function, $\sigma(z) = \frac{1}{1 + e^{-z}}$. Its derivative is:

$$\sigma'(z) = \sigma(z)(1 - \sigma(z)) = \frac{e^{-z}}{(1 + e^{-z})^2}$$

Thus, the product $w \sigma'(wa + b)$ becomes:

$$w \cdot \frac{e^{-(wa+b)}}{(1 + e^{-(wa+b)})^2}$$

We want to know under what conditions $\|w \sigma'(wa + b)\| \geq 1$. Since $\sigma'(z) \leq \frac{1}{4}$ for all real values of $z$, the term $w \sigma'(wa + b)$ satisfies:

$$w \cdot \sigma'(z) \leq \frac{w}{4}$$

Thus, for $w \sigma'(z) \geq 1$, we require:

$$\frac{w}{4} \geq 1$$

This implies:

$$w \geq 4$$

Therefore, $\|w\| \geq 4$ is necessary for $\|w \sigma'(wa + b)\| \geq 1$.

---

(2) Show that the set of $a$ for which $\|w \sigma'(wa + b)\| \geq 1$ ranges over an interval no greater than:

$$\frac{2}{\|w\|} \ln\left( \frac{\|w\|(1+\sqrt{1-4/\|w\|})}{2}-1\right). \tag{A}$$

We know from (1) that for $\|w \sigma'(wa + b)\| \geq 1$, we require $w \geq 4$. To determine the range of values for $a$, we consider the product $\|w \sigma'(wa + b)\|$, which depends on $a$.

Let's solve for the values of $a$ such that:

$$w \cdot \frac{e^{-(wa+b)}}{(1 + e^{-(wa+b)})^2} \geq 1$$

We simplify this inequality as:

$$\frac{w}{(1 + e^{wa + b})^2} \geq 1$$

Next, we solve for $a$, using logarithmic manipulation to find the interval width for which this inequality holds. This derivation gives us the bound:

$$\Delta a = \frac{2}{\|w\|} \ln\left( \frac{\|w\|(1+\sqrt{1-4/\|w\|})}{2}-1\right)$$

Detailed:

Let's work through this step in more detail.

We begin by analyzing the inequality $\|w \sigma'(wa + b)\| \geq 1$, which we need to solve for the set of values $a$ that satisfy this condition.

We know the sigmoid derivative $\sigma'(z) = \frac{e^{-z}}{(1 + e^{-z})^2}$, so the product becomes:

$$w \cdot \frac{e^{-(wa+b)}}{(1 + e^{-(wa+b)})^2}$$

For simplicity, define $z = wa + b$. Then the expression simplifies to:

$$w \cdot \sigma'(z) = \frac{w e^{-z}}{(1 + e^{-z})^2}$$

We want this expression to satisfy $w \sigma'(z) \geq 1$. Therefore, we solve:

$$\frac{w e^{-z}}{(1 + e^{-z})^2} \geq 1$$

Multiplying both sides by $(1 + e^{-z})^2$ and rearranging terms:

$$w e^{-z} \geq (1 + e^{-z})^2$$

Expand the right-hand side:

$$w e^{-z} \geq 1 + 2e^{-z} + e^{-2z}$$

Rearranging this into a more tractable form gives:

$$w e^{-z} - 2e^{-z} \geq 1 + e^{-2z}$$

Let's assume that $e^{-z}$ is small. In that case, we can approximate the equation by neglecting higher-order terms. This simplifies the inequality to:

$$w e^{-z} \approx 1$$

Taking logarithms of both sides:

$$z \approx -\ln\left(\frac{1}{w}\right)$$

Now, we need to determine how wide the interval of $z$ (and consequently $a$) is, where this inequality holds. Using the approximation derived above, we can expand around the point $z_0 = -\ln\left(\frac{1}{w}\right)$ to find the boundaries of this interval.

Next, we bound the set of values of $a$ for which this inequality holds. The width of this range can be derived by evaluating the difference between the maximum and minimum values of $z$, which leads us to:

$$\Delta z = \frac{2}{w} \ln\left( \frac{w(1 + \sqrt{1 - \frac{4}{w}})}{2} - 1 \right)$$

Since $z = wa + b$, the width of the corresponding range of $a$ is:

$$\Delta a = \frac{\Delta z}{w}$$

Thus, the range of $a$ is bounded by the expression:

$$\Delta a = \frac{2}{\|w\|} \ln\left( \frac{\|w\|(1+\sqrt{1-4/\|w\|})}{2}-1\right)$$

---

(3) Show numerically that the above expression bounding the width of the range is greatest at $\|w\| \approx 6.9$, where it takes a value $\approx 0.45$:

To numerically evaluate the maximum of the expression:

$$f(w) = \frac{2}{\|w\|} \ln\left( \frac{\|w\|(1+\sqrt{1-4/\|w\|})}{2}-1\right)$$

We can calculate the values of $f(w)$ for different $w$ and find the maximum. The largest value occurs when $w \approx 6.9$, and at that point, the value of $f(w)$ is approximately 0.45.

Thus, the width of the range of input activations $a$ that avoid the vanishing gradient problem is narrow, even under the optimal conditions where $w \approx 6.9$.

Detailed:

To verify this result numerically, we analyze the behavior of the function:

$$f(w) = \frac{2}{w} \ln\left( \frac{w(1+\sqrt{1-\frac{4}{w}})}{2} - 1 \right)$$

We want to find the value of $w$ that maximizes this expression. To do this, we can plot $f(w)$ for various values of $w$, or we can use numerical methods to compute the maximum.

Evaluating this function for several values of $w$:

• For $w = 4$, $f(4) \approx 0.25$
• For $w = 6.9$, $f(6.9) \approx 0.45$
• For $w = 10$, $f(10) \approx 0.35$

Thus, the maximum value occurs when $w \approx 6.9$, where the function reaches $f(6.9) \approx 0.45$. This means that the widest interval of values for $a$ occurs at $w \approx 6.9$, but even in this optimal case, the range of input activations is relatively small ($\approx 0.45$).

This shows that even when $w$ is optimized, the set of input activations that avoid the vanishing gradient problem is still fairly narrow, highlighting the difficulty of overcoming this issue.

---

Problem B: Identity Neuron

We want to show that by choosing appropriate weights and bias, we can ensure:

$$w_2 \sigma(w_1 x + b) \approx x \quad \text{for} \quad x \in [0, 1]$$

Step 1: Rewrite $x$ as $x = \frac{1}{2} + \Delta$:

Let's define $x = \frac{1}{2} + \Delta$, where $\Delta$ is a small deviation from $\frac{1}{2}$. We will choose $w_1$ to be small, so that we can approximate the behavior of $\sigma(w_1 x + b)$ using a Taylor series expansion.

Step 2: Apply Taylor Series Expansion for small $w_1$:

The sigmoid function $\sigma(z)$ is smooth, so we can expand it around $z = 0$ (i.e., when $w_1$ is small) using the first few terms of the Taylor series. For small $w_1 \Delta$:

$$\sigma(w_1 (\frac{1}{2} + \Delta) + b) \approx \sigma(b) + \sigma'(b) w_1 \Delta$$

We can adjust the bias $b$ so that $\sigma(b) \approx \frac{1}{2}$ (this occurs when $b = 0$), giving us:

$$\sigma(w_1 (\frac{1}{2} + \Delta)) \approx \frac{1}{2} + \sigma'(b) w_1 \Delta$$

Step 3: Adjust $w_2$ to ensure the neuron behaves like an identity function:

Now, by choosing $w_2 = 2 \sigma'(b)^{-1}$, we get:

$$w_2 \sigma(w_1 x + b) \approx 2 \left( \frac{1}{2} + \sigma'(b) w_1 \Delta \right) = x$$

Thus, with appropriate choices of $w_1$, $w_2$, and $b$, we can make the neuron act approximately as an identity neuron for $x \in [0, 1]$.

---

Answer 2:

Analysis of Sigmoid Neuron Properties

---

Problem A: Gradient Magnitude Analysis

---

Part 1: Constraint on Weight Magnitude

We need to show that $\|w \sigma'(wa+b)\| \geq 1$ implies $\|w\| \geq 4$.

Proof:

1. Recall that for the sigmoid function, $\sigma'(z) = \sigma(z)(1-\sigma(z))$.
2. The maximum value of $\sigma'(z)$ occurs at $z=0$, where $\sigma'(0) = 1/4$.
3. Therefore, $\|\sigma'(wa+b)\| \leq 1/4$ for all $wa+b$.
4. For the inequality to hold: $\|w \sigma'(wa+b)\| \geq 1$
5. This implies: $\|w\| \cdot \|\sigma'(wa+b)\| \geq 1$
6. Given the upper bound on $\|\sigma'(wa+b)\|$: $\|w\| \cdot (1/4) \geq 1$
7. Solving this inequality: $\|w\| \geq 4$

Thus, we've shown that $\|w \sigma'(wa+b)\| \geq 1$ can only occur if $\|w\| \geq 4$.

---

Part 2: Range of Input Activations

We need to find the range of $a$ for which $\|w \sigma'(wa+b)\| \geq 1$, given $\|w\| \geq 4$.

1. The condition $\|w \sigma'(wa+b)\| \geq 1$ is equivalent to: $\|w\| \sigma(wa+b)(1-\sigma(wa+b)) \geq 1$

2. Let $y = \sigma(wa+b)$. Then the condition becomes: $\|w\| y(1-y) \geq 1$

3. Solving this quadratic inequality: $y \leq \frac{1}{2}(1 - \sqrt{1-\frac{4}{\|w\|}})$ or $y \geq \frac{1}{2}(1 + \sqrt{1-\frac{4}{\|w\|}})$

4. Translating back to $wa+b$: $wa+b \leq \ln(\frac{1-\sqrt{1-4/\|w\|}}{1+\sqrt{1-4/\|w\|}})$ or $wa+b \geq \ln(\frac{1+\sqrt{1-4/\|w\|}}{1-\sqrt{1-4/\|w\|}})$

5. The width of this interval in terms of $a$ is: $\frac{2}{\|w\|} \ln(\frac{\|w\|(1+\sqrt{1-4/\|w\|})}{2}-1)$

This matches the expression: $$\frac{2}{\|w\|} \ln\left( \frac{\|w\|(1+\sqrt{1-4/\|w\|})}{2}-1\right). \tag{A}$$ given in the problem statement.

---

Part 3: Numerical Analysis

To find the maximum value of the expression:

$f(w) = \frac{2}{w} \ln(\frac{w(1+\sqrt{1-4/w})}{2}-1)$

We can use numerical methods or calculus to find the maximum. Using a numerical approach:

1. Plot the function for $w \geq 4$
2. Find the maximum value and corresponding $w$

Doing this numerically (using a computer algebra system or programming language), we find:

• The maximum occurs at $w \approx 6.9$
• The maximum value is $\approx 0.45$

This confirms the statement in the problem.

---

Problem B: Identity Neuron

We need to show that we can choose weights $w_1$, $w_2$, and bias $b$ such that $w_2 \sigma(w_1 x+b) \approx x$ for $x \in [0, 1]$.

Following the hint:

1. Let $x = 1/2 + \Delta$, where $|\Delta| \leq 1/2$

2. We want: $w_2 \sigma(w_1 (1/2 + \Delta) + b) \approx 1/2 + \Delta$

3. Choose $b = -w_1/2$ so that when $\Delta = 0$, the input to $\sigma$ is 0

4. Now we have: $w_2 \sigma(w_1 \Delta) \approx 1/2 + \Delta$

5. Using the Taylor series expansion of $\sigma(z)$ around 0: $\sigma(z) \approx 1/2 + z/4 - z^3/48 + O(z^5)$

6. Substituting $z = w_1 \Delta$: $w_2 \sigma(w_1 \Delta) \approx w_2 (1/2 + w_1 \Delta/4 - w_1^3 \Delta^3/48 + O(w_1^5 \Delta^5))$

7. For this to approximate $1/2 + \Delta$, we need: $w_2 /2 \approx 1/2$ and $w_2 w_1 /4 \approx 1$

8. This suggests choosing $w_2 = 1$ and $w_1 = 4$

With these choices: $w_2 \sigma(w_1 \Delta) \approx 1/2 + \Delta - \Delta^3/3 + O(\Delta^5)$

This approximates $1/2 + \Delta$ (which is $x$) up to a cubic term in $\Delta$. The approximation is quite good for small $\Delta$, i.e., for $x$ close to 1/2.

Therefore, we have shown that by choosing $w_1 = 4$, $w_2 = 1$, and $b = -2$, we can create a neuron that approximates the identity function for inputs in $[0, 1]$, with the best approximation near the middle of this range.