Neural Network

The following diagram shows a Back Propagation NN:

This NN consists of three layers:

1. Input layer with three neurons.
2. Hidden layer with two neurons.
3. Output layer with two neurons.

1 Supervised Training

The Back Propagation NN works in two modes, a supervised training mode and a production mode. The training can be summarized as follows:

Start by initializing the input weights for all neurons to some random numbers between 0 and 1, then:

1. Apply input to the network.
2. Calculate the output.
3. Compare the resulting output with the desired output for the given input. This is called the error.
4. Modify the weights and threshold q for all neurons using the error.
5. Repeat the process until error reaches an acceptable value (e.g. error < 1%), which means that the NN was trained successfully, or if we reach a maximum count of iterations, which means that the NN training was not successful.

The challenge is to find a good algorithm for updating the weights and thresholds in each iteration (step 4) to minimize the error.

Changing weights and threshold for neurons in the output layer is different from hidden layers. Note that for the input layer, weights remain constant at 1 for each input neuron weight.

Before we explain the training, let’s define the following:

1. l (Lambda) the Learning Rate: a real number constant, usually 0.2 for output layer neurons and 0.15 for hidden layer neurons.
2. D (Delta) the change: For example Dx is the change in x. Note that Dx is a single value and not D multiplied by x.

2 Output Layer Training

• Let z be the output of an output layer neuron as shown in section ‎4.
• Let y be the desired output for the same neuron, it should be scaled to a value between 0 and 1. This is the ideal output which we like to get when applying a given set of input.
• Then e (the error) will be:

e = z * (1 – z) * (y – z)
Dq = l * e                  … The change in q
Dw_i = Dq * x_i              … The change in weight at input i of the neuron

In other words, for each output neuron, calculate its error e, and then modify its threshold and weights using the formulas above.

3 Hidden Layer Training

Consider a hidden layer neuron as shown in the following figure:

• Let z be the output of the hidden layer neuron as shown in section ‎4.
• Let m_i be the weight at neuron N_i in the layer following the current layer. This is the weight for the input coming from the current hidden layer neuron.
• Let e_i be the error (e) at neuron N_i.
• Let r be the number of neurons in the layer following the current layer. (In the above diagram r=3).

g = S m_i * e_i                   … (for i=1 to r)
e = z * (1 – z) * g         … Error at the hidden layer neuron
Dq = l * e                        … The change in q
Dw_i = Dq * x_i                    … The change in weight i

Notice that in calculating g, we used the weight m_i and error e_i from the following layer, which means that the error and weights in this following layer should have already been calculated. This implies that during a training iteration of a Back Propagation NN, we start modifying the weights at the output layer, and then we proceed backwards on the hidden layers one by one until we reach the input layer. It is this method of proceeding backwards which gives this network its name Backward Propagation.

source: http://www.tek271.com/articles/neuralNet/IntoToNeuralNets.html