04 Backpropagation Algorithm Instructionsl

04_backpropagation-algorithm_instructionsl

User Manual:

Open the PDF directly: View PDF .
Page Count: 2

Backpropagation Algorithm

"Backpropagation" is neural-network terminology for minimizing our cost

function, just like what we were doing with gradient descent in logistic and linear

regression. Our goal is to compute:

That is, we want to minimize our cost function J using an optimal set of

parameters in theta. In this section we'll look at the equations we use to compute

the partial derivative of J(Θ):

To do so, we use the following algorithm:

Back propagation Algorithm

Given training set

Set := 0 for all (l,i,j), (hence you end up having a matrix full of zeros)

For training example t =1 to m:

1. Set

2. Perform forward propagation to compute for l=2,3,…,L

3. Using , compute

Where L is our total number of layers and is the vector of outputs of the

activation units for the last layer. So our "error values" for the last layer are simply

the differences of our actual results in the last layer and the correct outputs in y. To

get the delta values of the layers before the last layer, we can use an equation that

steps us back from right to left:

4. Compute using

The delta values of layer l are calculated by multiplying the delta values in the

next layer with the theta matrix of layer l. We then element-wise multiply that with

a function called g', or g-prime, which is the derivative of the activation function g

evaluated with the input values given by .

The g-prime derivative terms can also be written out as:

5. or with vectorization,

Hence we update our new matrix.

, if j≠0.

If j=0

The capital-delta matrix D is used as an "accumulator" to add up our values as we

go along and eventually compute our partial derivative. Thus we get =