Neural Networks for Machine Learning Lecture 4

距离上次更新已经 2390 天了，文章内容可能已经过时。

备注：笔记内容和图片均参考老师课件。

这一章的内容其实没怎么看懂，这里就把课件中没证明的公式简单证明一下。

首先是Softmax的公式

y_{i} = \frac{e^{z_{i}}}{\sum_{j \in g r o u p} e^{z_{j}}} \frac{\partial y_{i}}{\partial z_{i}} = \frac{e^{z_{i}} \sum_{j \in g r o u p} e^{z_{j}} - e^{z_{i}} e^{z_{i}}}{(\sum_{j \in g r o u p} e^{z_{j}})^{2}} = \frac{e^{z_{i}}}{\sum_{j \in g r o u p} e^{z_{j}}} \frac{\sum_{j \in g r o u p} e^{z_{j}} - e^{z_{i}}}{\sum_{j \in g r o u p} e^{z_{j}}} = y_{i} (1 - y_{i}) \frac{\partial y_{i}}{\partial z_{j}} = \frac{e^{z_{i}}}{(\sum_{j \in g r o u p} e^{z_{j}})^{2}} (- e^{z_{j}}) = - \frac{e^{z_{i}} e^{z_{j}}}{(\sum_{j \in g r o u p} e^{z_{j}})^{2}} = - y_{i} y_{j} (j \neq i)

Cross-entropy的公式

C = - \sum_{j} t_{j} log y_{j} \begin{aligned} \frac{\partial C}{\partial z_{i}} & = \sum_{j} \frac{\partial C}{\partial y_{j}} \frac{\partial y_{j}}{\partial z_{i}} \\ = \sum_{j} (- \frac{t_{j}}{y_{j}}) \frac{\partial y_{j}}{\partial z_{i}} \\ = \sum_{j = i} (- \frac{t_{j}}{y_{j}}) \frac{\partial y_{j}}{\partial z_{i}} + \sum_{j \neq i} (- \frac{t_{j}}{y_{j}}) \frac{\partial y_{j}}{\partial z_{i}} \\ = (- \frac{t_{i}}{y_{i}}) y_{i} (1 - y_{i}) + \sum_{j \neq i} (- \frac{t_{j}}{y_{j}}) (- y_{i} y_{j}) \\ = - t_{i} + t_{i} y_{i} + y_{i} \sum_{j \neq i} t_{j} \\ = - t_{i} + y_{i} \sum t_{j} \end{aligned}

课件中 $t_{j}$ 为目标值，表示的是概率，所以

\sum t_{j} = 1 \frac{\partial C}{\partial z_{i}} = y_{i} - t_{i}