CS224N Natural Language Processing with Deep Learning Lecture 7

课程主页：https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/

视频地址：https://www.bilibili.com/video/av46216519?from=search&seid=13229282510647565239

这里回顾CS224N Lecture 7的课程内容，这一讲介绍了梯度消失以及梯度爆炸的问题，GRU以及LSTM。

Vanishing Gradient and Gradient Explosion Problems

回顾梯度计算公式

$$\frac{\partial E}{\partial W}=\sum_{t=1}^{T} \frac{\partial E_{t}}{\partial W}$$

由链式法则不难得到

$$\frac{\partial E_{t}}{\partial W}=\sum_{k=1}^{t} \frac{\partial E_{t}}{\partial y_{t}} \frac{\partial y_{t}}{\partial h_{t}} \frac{\partial h_{t}}{\partial h_{k}} \frac{\partial h_{k}}{\partial W}$$

上式中最难计算的是$ \frac{\partial h_{t}}{\partial h_{k}}$，注意到

$$\frac{\partial h_{t}}{\partial h_{k}}=\prod_{j=k+1}^{t} \frac{\partial h_{j}}{\partial h_{j-1}}=\prod_{j=k+1}^{t} W^{T} \times \operatorname{diag}\left[f^{\prime}\left(h_{j-1}\right)\right]$$

其中

$$f(h_{j-1}) = \sigma\left({W}_{h} {h}_{j-1}+{W}_{e} {e}^{(t)}+{b}_{1}\right)$$

上述等式中省略了下标。

结合以上几个式子可得

$$\frac{\partial E}{\partial W}=\sum_{t=1}^{T} \sum_{k=1}^{t} \frac{\partial E_{t}}{\partial y_{t}} \frac{\partial y_{t}}{\partial h_{t}}\left(\prod_{j=k+1}^{t} \frac{\partial h_{j}}{\partial h_{j-1}}\right) \frac{\partial h_{k}}{\partial W}$$

利用范数不等式可得

$$\left\|\frac{\partial h_{j}}{\partial h_{j-1}}\right\| \leq\left\|W^{T}\right\|\left\|\operatorname{diag}\left[f^{\prime}\left(h_{j-1}\right)\right]\right\| \leq \beta_{W} \beta_{h}$$

其中$\beta_W,\beta_h$分别表示$W^T,\operatorname{diag} [f^{\prime}\left(h_{j-1}\right)]$的范数上限，从而

$$\left\|\frac{\partial h_{t}}{\partial h_{k}}\right\|=\left\|\prod_{j=k+1}^{t} \frac{\partial h_{j}}{\partial h_{j-1}}\right\| \leq\left(\beta_{W} \beta_{h}\right)^{t-k}$$

所以如果$\beta_{W} \beta_{h}<1$，那么梯度的范数就会非常小，从而产生梯度消失的问题。

在实际中，也会产生梯度爆炸的问题，该问题的解决方式较为简单，只要将梯度按比例缩小即可：

$$\begin{aligned} &\hat{g} \leftarrow \frac{\partial E}{\partial W}\\ &\text{if } \|\hat{g}\| \geq \text{threshold then} \\ & \ \ \ \ \ \ \ \hat{g} \leftarrow \frac{\text {threshold}}{\|\hat{g}\|} \hat{g}\\ &\text{end if} \end{aligned}$$

Deep Bidirectional RNNs

之前介绍的RNN是单向的，这里介绍双向RNN：

计算公式如下

$$\begin{aligned} &\overleftarrow{h}_{t}=f\left(\overrightarrow{W} x_{t}+\overrightarrow{V} \overrightarrow{h}_{t-1}+\overrightarrow{b}\right) \\ &\overleftarrow{h}_{t}=f\left(\overleftarrow{W} x_{t}+\overleftarrow{V} \overleftarrow{h}_{t+1}+\overleftarrow{b}\right) \\ &y_{t}=g\left(U h_{t}+c\right)=g\left(u\left[\overrightarrow{h}_{t} ; \overleftarrow{h}_{t}\right]+c\right) \end{aligned}$$

将双向RNN堆叠起来即可得到Deep Bidirectional RNN：

计算公式和双向RNN非常接近：

$$\begin{aligned} &\overrightarrow{h}_{t}^{(i)}=f\left(\overrightarrow{W^{(i)}} h_{t}^{(i-1)}+\overrightarrow{V^{(i)}} \overrightarrow{h^{(i)}_{t-1}}+\overrightarrow{b^{(i)}}\right)\\ &\overleftarrow{h}_{t}^{(i)}=f\left(\overleftarrow{W^{(i)}} h_{t}^{(i-1)}+\overleftarrow{V^{(i)}} \overleftarrow{h^{(i)}_{t+1}}+\overleftarrow{b^{(i)}}\right)\\ &\hat{y}_{t}=g\left(U h_{t}+c\right)=g\left(U\left[\overrightarrow{ {h}_{t}^{(L)}} ; \overleftarrow{ {h}_{t}^{(L)}}\right]+c\right) \end{aligned}$$

为了解决梯度消失或者梯度爆炸的问题，人们提出了一些结构，其中最常见的为GRU和LSTM。

Gated Recurrent Units(GRU)

$$\begin{aligned} &z_{t}=\sigma\left(W^{(z)} x_{t}+U^{(z)} h_{t-1}\right) & \text { (Update gate) }\\ &r_{t}=\sigma\left(W^{(r)} x_{t}+U^{(r)} h_{t-1}\right) & \text { (Reset gate) }\\ &\tilde{h}_{t}=\tanh \left(r_{t} \circ U h_{t-1}+W x_{t}\right) & \text { (New memory) }\\ &h_{t}=\left(1-z_{t}\right) \circ \tilde{h}_{t}+z_{t} \circ h_{t-1} & \text { (Hidden state) } \end{aligned}$$

图示如下：

Long-Short-Term-Memories

$$\begin{aligned} &i_{t}=\sigma\left(W^{(i)} x_{t}+U^{(i)} h_{t-1}\right) &(\text{Input gate})\\ &f_{t}=\sigma\left(W^{(f)} x_{t}+U^{(f)} h_{t-1}\right)& (\text{Forget gate})\\ &o_{t}=\sigma\left(W^{(0)} x_{t}+U^{(o)} h_{t-1}\right) & (\text{Output/Exposure gate})\\ &\tilde{c}_{t}=\tanh \left(W^{(c)} x_{t}+U^{(c)} h_{t-1}\right) & (\text{New memory cell})\\ &c_{t}=f_{t} \circ c_{t-1}+i_{t} \circ \tilde{c}_{t} &(\text{Final memory cell}) \\ &h_{t}=o_{t} \circ \tanh \left(c_{t}\right) \end{aligned}$$

图示如下：