CS224N Natural Language Processing with Deep Learning Lecture 7

距离上次更新已经 1814 天了，文章内容可能已经过时。

课程主页：https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/

视频地址：https://www.bilibili.com/video/av46216519?from=search&seid=13229282510647565239

这里回顾CS224N Lecture 7的课程内容，这一讲介绍了梯度消失以及梯度爆炸的问题，GRU以及LSTM。

Vanishing Gradient and Gradient Explosion Problems

回顾梯度计算公式

$$\frac{\partial E}{\partial W}=\sum _{t=1}^T\frac{\partial E_t}{\partial W}$$

由链式法则不难得到

$$\frac{\partial E_t}{\partial W}=\sum _{k=1}^t\frac{\partial E_t}{\partial y_t}\frac{\partial y_t}{\partial h_t}\frac{\partial h_t}{\partial h_k}\frac{\partial h_k}{\partial W}$$

上式中最难计算的是$\frac{\partial h_t}{\partial h_k}$，注意到

$$\frac{\partial h_t}{\partial h_k}=\prod _{j=k+1}^t\frac{\partial h_j}{\partial h_{j-1}}=\prod _{j=k+1}^t{W}^T\times \mathrm{diag}\left[f^{\prime }\left(h_{j-1}\right)\right]$$

其中

$$f(h_{j-1})=\sigma (W_h{h}_{j-1}+W_e{e}^{(t)}+b_1)$$

上述等式中省略了下标。

结合以上几个式子可得

$$\frac{\partial E}{\partial W}=\sum _{t=1}^T\sum _{k=1}^t\frac{\partial E_t}{\partial y_t}\frac{\partial y_t}{\partial h_t}\left(\prod _{j=k+1}^t\frac{\partial h_j}{\partial h_{j-1}}\right)\frac{\partial h_k}{\partial W}$$

利用范数不等式可得

$$\Vert \frac{\partial h_j}{\partial h_{j-1}}\Vert \le \Vert W^T\Vert \Vert \mathrm{diag}\left[f^{\prime }\left(h_{j-1}\right)\right]\Vert \le {\beta }_W{\beta }_h$$

其中${\beta }_W,{\beta }_h$分别表示$W^T,\mathrm{diag}[f^{\prime }\left(h_{j-1}\right)]$的范数上限，从而

$$\Vert \frac{\partial h_t}{\partial h_k}\Vert =\Vert \prod _{j=k+1}^t\frac{\partial h_j}{\partial h_{j-1}}\Vert \le {\left({\beta }_W{\beta }_h\right)}^{t-k}$$

所以如果${\beta }_W{\beta }_h<1$，那么梯度的范数就会非常小，从而产生梯度消失的问题。

在实际中，也会产生梯度爆炸的问题，该问题的解决方式较为简单，只要将梯度按比例缩小即可：

$$\begin{array}{rl}& \hat{g}\leftarrow \frac{\partial E}{\partial W}\\ & \text{if }\Vert \hat{g}\Vert \ge \text{threshold then}\\ & \hat{g}\leftarrow \frac{\text{threshold}}{\Vert \hat{g}\Vert }\hat{g}\\ & \text{end if}\end{array}$$

Deep Bidirectional RNNs

之前介绍的RNN是单向的，这里介绍双向RNN：

计算公式如下

$$\begin{array}{rl}& {\overleftarrow{h}}_t=f(\overrightarrow{W}{x}_t+\overrightarrow{V}{\overrightarrow{h}}_{t-1}+\overrightarrow{b})\\ & {\overleftarrow{h}}_t=f(\overleftarrow{W}{x}_t+\overleftarrow{V}{\overleftarrow{h}}_{t+1}+\overleftarrow{b})\\ & y_t=g(U{h}_t+c)=g(u[{\overrightarrow{h}}_t;{\overleftarrow{h}}_t]+c)\end{array}$$

将双向RNN堆叠起来即可得到Deep Bidirectional RNN：

计算公式和双向RNN非常接近：

$$\begin{array}{rl}& {\overrightarrow{h}}_t^{(i)}=f(\overrightarrow{W^{(i)}}{h}_t^{(i-1)}+\overrightarrow{V^{(i)}}\overrightarrow{h_{t-1}^{(i)}}+\overrightarrow{b^{(i)}})\\ & {\overleftarrow{h}}_t^{(i)}=f(\overleftarrow{W^{(i)}}{h}_t^{(i-1)}+\overleftarrow{V^{(i)}}\overleftarrow{h_{t+1}^{(i)}}+\overleftarrow{b^{(i)}})\\ & {\hat{y}}_t=g(U{h}_t+c)=g(U[\overrightarrow{h_t^{(L)}};\overleftarrow{h_t^{(L)}}]+c)\end{array}$$

为了解决梯度消失或者梯度爆炸的问题，人们提出了一些结构，其中最常见的为GRU和LSTM。

Gated Recurrent Units(GRU)

$$\begin{array}{rlr}& z_t=\sigma (W^{(z)}{x}_t+U^{(z)}{h}_{t-1})& \text{ (Update gate) }\\ & r_t=\sigma (W^{(r)}{x}_t+U^{(r)}{h}_{t-1})& \text{ (Reset gate) }\\ & {\tilde{h}}_t=\tanh (r_t\circ U{h}_{t-1}+W{x}_t)& \text{ (New memory) }\\ & h_t=(1-z_t)\circ {\tilde{h}}_t+z_t\circ h_{t-1}& \text{ (Hidden state) }\end{array}$$

图示如下：

Long-Short-Term-Memories

$$\begin{array}{rlr}& i_t=\sigma (W^{(i)}{x}_t+U^{(i)}{h}_{t-1})& (\text{Input gate})\\ & f_t=\sigma (W^{(f)}{x}_t+U^{(f)}{h}_{t-1})& (\text{Forget gate})\\ & o_t=\sigma (W^{(0)}{x}_t+U^{(o)}{h}_{t-1})& (\text{Output/Exposure gate})\\ & {\tilde{c}}_t=\tanh (W^{(c)}{x}_t+U^{(c)}{h}_{t-1})& (\text{New memory cell})\\ & c_t=f_t\circ c_{t-1}+i_t\circ {\tilde{c}}_t& (\text{Final memory cell})\\ & h_t=o_t\circ \tanh \left(c_t\right)\end{array}$$

图示如下：