04 Backpropagation Algorithm Instructionsl
04_backpropagation-algorithm_instructionsl
User Manual:
Open the PDF directly: View PDF 
.
Page Count: 2
| Download | |
| Open PDF In Browser | View PDF |
Backpropagation Algorithm
"Backpropagation" is neural-network terminology for minimizing our cost
function, just like what we were doing with gradient descent in logistic and linear
regression. Our goal is to compute:
min Θ J (Θ)
That is, we want to minimize our cost function J using an optimal set of
parameters in theta. In this section we'll look at the equations we use to compute
the partial derivative of J(Θ):
∂
∂Θ
(l)
J (Θ)
i,j
To do so, we use the following algorithm:
Back propagation Algorithm
Given training set {(x
Set Δ
(l)
i,j
(1)
,y
(1)
) ⋯ (x
(m)
,y
(m)
)}
:= 0 for all (l,i,j), (hence you end up having a matrix full of zeros)
For training example t =1 to m:
1. Set a
(1)
:= x
(t)
2. Perform forward propagation to compute a
(l)
for l=2,3,…,L
3. Using y , compute δ
(t)
(L)
= a
(L)
− y
(t)
Where L is our total number of layers and a is the vector of outputs of the
activation units for the last layer. So our "error values" for the last layer are simply
the differences of our actual results in the last layer and the correct outputs in y. To
get the delta values of the layers before the last layer, we can use an equation that
steps us back from right to left:
(L)
4. Compute δ
δ
(l)
= ((Θ
(l)
(L−1)
T
)
δ
,δ
(L−2)
(l+1)
,…,δ
) .∗ a
(l)
(2)
using
. ∗ (1 − a
(l)
)
The delta values of layer l are calculated by multiplying the delta values in the
next layer with the theta matrix of layer l. We then element-wise multiply that with
a function called g', or g-prime, which is the derivative of the activation function g
evaluated with the input values given by z .
(l)
The g-prime derivative terms can also be written out as:
′
g (z
5. Δ
(l)
i,j
(l)
) = a
:= Δ
(l)
i,j
(l)
. ∗ (1 − a
+ a
(l)
j
δ
(l+1)
i
(l)
)
or with vectorization, Δ
(l)
:= Δ
(l)
+ δ
(l+1)
(a
(l)
T
)
Hence we update our new Δ matrix.
D
D
(l)
i,j
(l)
i,j
1
:=
(Δ
m
1
:=
Δ
m
(l)
i,j
(l)
i,j
+ λΘ
(l)
i,j
)
, if j≠0.
If j=0
The capital-delta matrix D is used as an "accumulator" to add up our values as we
go along and eventually compute our partial derivative. Thus we get
J (Θ) =
∂
(l)
∂Θ
ij
D
(l)
ij
Source Exif Data:
File Type : PDF File Type Extension : pdf MIME Type : application/pdf PDF Version : 1.4 Linearized : No XMP Toolkit : Adobe XMP Core 5.6-c015 91.163280, 2018/06/22-11:31:03 Create Date : 2018:12:10 02:41:02Z Creator Tool : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36 Modify Date : 2018:12:10 10:58+08:00 Metadata Date : 2018:12:10 10:58+08:00 Producer : Skia/PDF m71 Format : application/pdf Document ID : uuid:14fc99cf-6188-4b9b-872c-4c17efd17f0d Instance ID : uuid:383c27dd-245b-4a0b-8b3d-b0c237f3fda2 Page Count : 2 Creator : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36EXIF Metadata provided by EXIF.tools