CS224N Natural Language Processing with Deep Learning Lecture 4

课程主页：https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/

视频地址：https://www.bilibili.com/video/av46216519?from=search&seid=13229282510647565239

这里回顾CS224N Lecture 4的课程内容，这一讲介绍了矩阵微积分以及反向传播。

反向传播

雅克比矩阵

假设我们有函数$f: \mathbb{R}^{n} \rightarrow \mathbb{R}^{m}$，其中

$\boldsymbol{f}(\boldsymbol{x})=\left[f_{1}\left(x_{1}, \dots, x_{n}\right), f_{2}\left(x_{1}, \dots, x_{n}\right), \dots, f_{m}\left(x_{1}, \dots, x_{n}\right)\right]$

那么雅克比矩阵为如下矩阵

$\frac{\partial \boldsymbol{f}}{\partial \boldsymbol{x}}=\left[\begin{array}{ccc} \frac{\partial f_{1}}{\partial x_{1}} & \cdots & \frac{\partial f_{1}}{\partial x_{n}} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_{m}}{\partial x_{1}} & \cdots & \frac{\partial f_{m}}{\partial x_{n}} \end{array}\right]$

即

$\left(\frac{\partial f}{\partial x}\right)_{i j}=\frac{\partial f_{i}}{\partial x_{j}}$

利用雅克比矩阵可以使用矩阵形式的链式法则，考虑如下两个函数

$f: \mathbb{R}^{n} \rightarrow \mathbb{R}^{m}, g: \mathbb{R}^{m} \rightarrow \mathbb R^l$

下面计算函数$h(x)=g(f(x))$的雅克比矩阵：

$\begin{aligned} \left(\frac{\partial h}{\partial x}\right)_{i j} &= \frac{\partial h_{i}}{\partial x_{j}}\\ &= \frac{\partial (gf)_{i}}{\partial x_{j}}\\ &= \sum_{k=1}^m \frac{\partial (gf)_{i}}{\partial f_k} \frac {\partial f_k}{\partial x_{j}}\\ &= \sum_{k=1}^m \left(\frac{\partial gf}{\partial f}\right)_{ik} \left(\frac{\partial f}{\partial x}\right)_{k j}\\ &= \left(\frac{\partial \boldsymbol{h}}{\partial \boldsymbol{f}} \frac{\partial \boldsymbol{f}}{\partial x} \right)_{ij} \end{aligned}$

所以

$\frac{\partial \boldsymbol{h}}{\partial \boldsymbol{x}} = \frac{\partial \boldsymbol{h}}{\partial \boldsymbol{f}} \frac{\partial \boldsymbol{f}}{\partial \boldsymbol x}$

对于输入为矩阵的函数

$f: \mathbb{R}^{n \times m} \rightarrow \mathbb{R}$

定义梯度为

$\left(\frac{\partial f}{\partial W}\right)_{i j}=\frac{\partial f}{\partial W_{ij}}$

有用的性质

下面介绍几个有用的性质，这里规定$W$表示矩阵，$x$表示向量

$z= Wx, \frac{\partial z}{\partial x}=W$
$z=x{W}, \frac{\partial z}{\partial x}={W}^{T}$
$z= x,\frac{\partial z}{\partial x}=I$
$z=f(x), \frac{\partial z}{\partial x}=\operatorname{diag}\left(f^{\prime}(x)\right)$
$z={W} {x}, {\delta}=\frac{\partial J}{\partial \boldsymbol{z}}, \frac{\partial J}{\partial {W}}=\delta^{T} x^{T}$
$z= {x}{W}, {\delta}=\frac{\partial J}{\partial \boldsymbol{z}}, \frac{\partial J}{\partial {W}}= x^{T}\delta$
$\hat{y}=\operatorname{softmax}(\boldsymbol{\theta}), J=CE(y,\hat y), \frac{\partial J}{\partial {\theta}}=\hat{y}-{y}$

下面来证明这些性质。

性质1：

$\left(\frac{\partial z}{\partial x}\right)_{i j}=\frac{\partial z_{i}}{\partial x_{j}}=\frac{\partial}{\partial x_{j}} \sum_{k=1}^{m} W_{i k} x_{k}=W_{i j}$

性质2：

$\left(\frac{\partial z}{\partial x}\right)_{i j} =\frac{\partial z_{i}}{\partial x_{j}} =\frac{\partial}{\partial x_{j}} \sum_{k=1}^{n}x_{k} W_{k i} =W_{ji}$

性质3：

在性质1中取$W=I$

性质4：

$\left(\frac{\partial z}{\partial x}\right)_{i j}=\frac{\partial z_{i}}{\partial x_{j}}=\frac{\partial}{\partial x_{j}} f\left(x_{i}\right)=\left\{\begin{array}{ll} f^{\prime}\left(x_{i}\right) & \text { if } i=j \\ 0 & \text { if otherwise } \end{array}\right.$

性质5：

$\begin{aligned} z_{k} &=\sum_{l=1}^{m} W_{k l} x_{l} \\ \frac{\partial z_{k}}{\partial W_{i j}} &=\sum_{l=1}^{m} x_{l} \frac{\partial}{\partial W_{i j}} W_{k l} \end{aligned}$

所以

$\begin{aligned} \frac{\partial J}{\partial W_{i j}} &=\frac{\partial J}{\partial z} \frac{\partial z}{\partial W_{i j}}\\ &=\delta \frac{\partial z}{\partial W_{i j}}\\ &=\sum_{k=1}^{m} \delta_{k} \frac{\partial z_{k}}{\partial W_{i j}}\\ &=\sum_{k=1}^{m}\sum_{l=1}^{m} x_{l} \delta_{k} \frac{\partial}{\partial W_{i j}} W_{k l}\\ &=\delta_{i} x_{j} \end{aligned}$

性质6：

$\begin{aligned} z_{k} &= \sum_{l=1}^{n}x_{l} W_{l k} \\ \frac{\partial z_{k}}{\partial W_{i j}} &=\sum_{l=1}^{n} x_{l} \frac{\partial}{\partial W_{i j}} W_{lk} \end{aligned}$

所以

$\begin{aligned}\frac{\partial J}{\partial W_{i j}}&=\frac{\partial J}{\partial z} \frac{\partial z}{\partial W_{i j}}\\&=\delta \frac{\partial z}{\partial W_{i j}}\\&=\sum_{k=1}^{m} \delta_{k} \frac{\partial z_{k}}{\partial W_{i j}}\\ &=\sum_{k=1}^{m}\sum_{l=1}^{n} x_{l}\delta_{k} \frac{\partial}{\partial W_{i j}} W_{lk}\\ &=\delta_{j} x_{i} \end{aligned}$

性质7：

$\begin{aligned} J&=CE(y, \hat y)\\ &=-\sum_{j=1}^{n} y_j \log \hat y_j\\ &=-\sum_{j=1}^{n} y_j \log \frac{\exp \left(\theta_j\right)} {\sum_{k=1}^{n} \exp \left(\theta_k\right)}\\ &= -\sum_{j=1}^{n} y_j \left (\theta_j - \log \sum_{k=1}^{n} \exp \left(\theta_k\right) \right)\\ &= -\sum_{j=1}^{n} y_j \theta_j + \left(\sum_{j=1}^{n} y_j \right) \log \sum_{k=1}^{n} \exp \left(\theta_k\right) \\ &= -\sum_{j=1}^{n} y_j \theta_j + \log \sum_{k=1}^{n} \exp \left(\theta_k\right) \end{aligned}$

所以

$\begin{aligned} \frac{\partial J}{\partial \theta_i} &= -y_i + \frac{ \exp \left(\theta_i\right)}{\sum_{k=1}^{n} \exp \left(\theta_k\right)}\\ &= \hat y_i -y_i \end{aligned}$

例子

考虑如下神经网络

$\begin{aligned} &\boldsymbol{x}=\text { input }\\ &z=\boldsymbol{W} \boldsymbol{x}+\boldsymbol{b}_{1}\\ &h=\operatorname{ReLU}(z)\\ &\boldsymbol{\theta}=\boldsymbol{U} \boldsymbol{h}+\boldsymbol{b}_{2}\\ &\hat{y}=\operatorname{softmax}(\boldsymbol{\theta})\\ &J=C E(\boldsymbol{y}, \hat{\boldsymbol{y}}) \end{aligned}$

其中

$\boldsymbol{x} \in \mathbb{R}^{D_{x} \times 1} \quad \boldsymbol{b}_{1} \in \mathbb{R}^{D_{h} \times 1} \quad \boldsymbol{W} \in \mathbb{R}^{D_{h} \times D_{x}} \quad \boldsymbol{b}_{2} \in \mathbb{R}^{N_{c} \times 1} \quad \boldsymbol{U} \in \mathbb{R}^{N_{c} \times D_{h}}$

显然我们有

$\operatorname{ReLU}^{\prime}(x)=\left\{\begin{array}{ll} 1 & \text { if } x>0 \\ 0 & \text { if otherwise } \end{array}=\operatorname{sgn}(\operatorname{ReLU}(x))\right.$

为了后续计算方便，定义

$\delta_{1}=\frac{\partial J}{\partial \boldsymbol{\theta}}\in \mathbb R^{1\times N_c} \quad \delta_{2}=\frac{\partial J}{\partial z}\in \mathbb R^{1\times D_h}$

那么

$\begin{aligned} \delta_{1} &=\frac{\partial J}{\partial \theta}=(\hat{y}-y)^{T} \\ \delta_{2} &=\frac{\partial J}{\partial z}=\frac{\partial J}{\partial \theta} \frac{\partial \theta}{\partial h} \frac{\partial h}{\partial z} \\ &=\delta_{1} \frac{\partial \theta}{\partial h} \frac{\partial h}{\partial z} \\ &=\delta_{1} U \frac{\partial h}{\partial z} \\ &=\delta_{1} U \circ \operatorname{ReLU}^{\prime}(z) \\ &=\delta_{1} U \circ \operatorname{sgn}(h) \end{aligned}$

利用之前的性质可得

$\begin{aligned} \frac{\partial J}{\partial U} &=\frac{\partial J}{\partial \theta} \frac{\partial \theta}{\partial U}=\delta_{1} \frac{\partial \theta}{\partial U}=\delta_{1}^{T} h^{T} \\ \frac{\partial J}{\partial b_{2}} &=\frac{\partial J}{\partial \theta} \frac{\partial \theta}{\partial b_{2}}=\delta_{1} \frac{\partial \theta}{\partial b_{2}}=\delta_{1}^{T} \\ \frac{\partial J}{\partial W} &=\frac{\partial J}{\partial z} \frac{\partial z}{\partial W}=\delta_{2} \frac{\partial z}{\partial W}=\delta_{2}^{T} x^{T} \\ \frac{\partial J}{\partial b_{1}} &=\frac{\partial J}{\partial z} \frac{\partial z}{\partial b_{1}}=\delta_{2} \frac{\partial z}{\partial b_{1}}=\delta_{2}^{T} \\ \frac{\partial J}{\partial x}&=\frac{\partial J}{\partial z} \frac{\partial z}{\partial x}=\left(\delta_{2} W\right)^{T} \end{aligned}$

梯度检查

可以利用数值梯度的方法判断梯度计算是否正确：

$f^{\prime}(x) \approx \frac{f(x+h)-f(x-h)}{2 h}$

正则化

正则化是为了防止过拟合，在神经网络组一般使用Frobenius范数：

$J_{R}=J+\lambda \sum_{i=1}^{L}\left\|W^{(i)}\right\|_{F}^2$

初始化

通过对权重适当的初始化，可以加快收敛速度，比较常用的初始化方式为

$W \sim U \left[-\sqrt{\frac{6}{n^{(l)}+n^{(l+1)}}}, \sqrt{\frac{6}{n^{(l)}+n^{(l+1)}}} \right]$

其中$W \in \mathbb{R}^{n^{(l+1)} \times n^{(l)}}$