Introduction to Deep Learning week 5

这一周主要介绍了RNN，这里回顾下RNN的反向传播。

Recurrent Neural Network (RNN)

RNN主要运用于序列数据，其架构如下：

右边是实际的架构，左边是拆开后的架构，方便理解，计算公式为：

$$h_t= f_t(Vx_t+Wh_{t-1} +b_h),\hat y _t =f_y(Uh_t+ b_y)$$

损失函数为：

$$L = \sum_{i} L_i(y_i, \hat y_i)$$

Backpropagation Through Time (BPTT)

下面介绍RNN的反向传播，我们借助下图进行理解：

可以看出我们要对两个方向进行反向传播，下面分别计算上图的偏导数。

$\frac{\partial L}{\partial U}$：

$$\frac{\partial L}{\partial U} =\sum_{t=0}^T\frac{\partial L_t}{\partial U} =\sum_{t=0}^T\frac{\partial L_t}{\partial \hat y_t} \frac{\partial \hat y_t}{\partial U} \\ \hat y _t =f_y(Uh_t+ b_y),h_t= f_t(Vx_t+Wh_{t-1} +b_h)$$

由于$h_t,b_y$和$U$无关，上式计算到此即可。

$\frac{\partial L}{\partial W}$：

$$\frac{\partial L}{\partial W} =\sum_{t=0}^T\frac{\partial L_t}{\partial W} =\sum_{t=0}^T\frac{\partial L_t}{\partial \hat y_t} \frac{\partial \hat y_t}{\partial h_t}\frac{\partial h_t}{\partial W} \\ \hat y _t =f_y(Uh_t+ b_y),h_t= f_t(Vx_t+Wh_{t-1} +b_h)$$

注意$h_t$依旧和$W$有关，所以上式还要继续处理，我们重新处理$\frac{\partial L_t}{\partial W}$，首先处理$\frac{\partial h_t}{\partial W}$，记$a= Vx_t+Wh_{t-1} +b_h$，从而

$$\begin{aligned} \frac{\partial h_t}{\partial W} &=\frac{\partial h_t}{\partial a} \Big(\frac{\partial a}{\partial W} +\frac{\partial a}{\partial h_{t-1}} \frac{\partial h_{t-1}}{\partial W}\Big) \\ &=\frac{\partial h_t}{\partial a}\frac{\partial a}{\partial W} + \frac{\partial h_t}{\partial a}\frac{\partial a}{\partial h_{t-1}} \frac{\partial h_{t-1}}{\partial W} \end{aligned}$$

为了避免混淆，将$\frac{\partial h_t}{\partial a}\frac{\partial a}{\partial W}$记录为$\frac{\partial h_t}{\partial W_*}$，因此上式为

$$\frac{\partial h_t}{\partial W} = \frac{\partial h_t}{\partial W_*} + \frac{\partial h_t}{\partial h_{t-1}} \frac{\partial h_{t-1}}{\partial W}$$

这是一个递推方程，递推下去不难得到

$$\frac{\partial h_t}{\partial W} = \sum_{k=0}^t \frac{\partial h_t}{\partial h_{t-1}} ...\frac{\partial h_{k+1}}{\partial h_{k}}\frac{\partial h_{k}}{\partial W} = \sum_{k=0}^t \Big(\prod_{i=k+1}^t \frac{\partial h_i}{\partial h_{i-1}} \Big)\frac{\partial h_{k}}{\partial W}$$

因此

$$\frac{\partial L_t}{\partial W} =\frac{\partial L_t}{\partial \hat y_t} \frac{\partial \hat y_t}{\partial h_t} \sum_{k=0}^t \Big(\prod_{i=k+1}^t \frac{\partial h_i}{\partial h_{i-1}} \Big)\frac{\partial h_{k}}{\partial W}$$

同理也可以用这个方法计算其他的导数，这里就不一一列出了。