CS236 Deep Generative Models Homework 1

课程主页：https://deepgenerativemodels.github.io/

课件资料：https://github.com/Subhajit135/CS236_DGM，https://github.com/deepgenerativemodels/notes

视频地址：https://www.bilibili.com/video/av81625948?from=search&seid=4636291486683332935

这里回顾CS236 Homework 1。

Problem 1

证明：

$$\begin{aligned} \mathbb{E}_{\hat{p}(x)}\left[D_{\mathrm{KL} }\left(\hat{p}(y | x) \| p_{\theta}(y | x)\right)\right] &=\mathbb{E}_{\hat{p}(x)}\left[\mathbb{E}_{\hat{p}(y|x)}[\log \hat{p}(y|x)-\log {p}_{\theta}(y|x)] \right]\\ &= \sum_x \hat p (x)\sum_y \hat{p}(y|x)\log \frac{\hat{p}(y|x)}{ {p_{\theta} }(y|x)} \\ &=\sum_x\sum_y \hat p(x, y)\log \frac{\hat{p}(y|x)}{ {p_{\theta} }(y|x)} \\ &= \mathbb{E}_{\hat{p}(x, y)}\left[\log \hat p(y | x)\right]- \mathbb{E}_{\hat{p}(x, y)}\left[\log p_{\theta}(y | x)\right] \end{aligned}$$

因此

$$\arg \max _{\theta \in \Theta} \mathbb{E}_{\hat{p}(x, y)}\left[\log p_{\theta}(y | x)\right]=\arg \min _{\theta \in \Theta} \mathbb{E}_{\hat{p}(x)}\left[D_{\mathrm{KL} }\left(\hat{p}(y | x) \| p_{\theta}(y | x)\right)\right]$$

Problem 2

$$\begin{aligned} p_{\theta}(y | x) &= \frac{p_{\theta}(x,y)}{p_{\theta}(x)}\\ &= \frac{p_{\theta}(x |y) p_{\theta } (y)}{\sum_{y=1}^kp_{\theta}(x |y) p_{\theta } (y)}\\ &=\frac{\mathcal{N}\left(x | \mu_{y}, \sigma^{2} I\right) \pi_y} {\sum_{y=1}^k \mathcal{N}\left(x | \mu_{y}, \sigma^{2} I\right) \pi_y}\\ &= \frac{ \frac{1}{(\sqrt{2\pi} \sigma)^n} \exp \left(-\frac{\|x-\mu_y \|^2}{2\sigma^2}\right) \pi_y}{ \sum_{y=1}^k\frac{1}{(\sqrt{2\pi} \sigma)^n} \exp \left(-\frac{\|x-\mu_y \|^2}{2\sigma^2}\right) \pi_y}\\ &=\frac{\exp \left(-\frac{\mu_y^{\top}\mu_y -2x^{\top} \mu_y}{2\sigma^2} + \ln \pi_y\right) } { \sum_{y=1}^k\exp \left(-\frac{\mu_y^{\top}\mu_y -2x^{\top} \mu_y}{2\sigma^2}+ \ln \pi_y\right) }\\ &=\frac{\exp \left( x^\top \frac{\mu_y}{\sigma^2} +(\ln \pi_y -\frac{\mu_y^{\top}\mu_y}{2\sigma^2} )\right)} { \sum_{y=1}^k\exp \left( x^\top \frac{\mu_y}{\sigma^2} +(\ln \pi_y -\frac{\mu_y^{\top}\mu_y}{2\sigma^2} )\right) } \end{aligned}$$

所以

$$\begin{aligned} w_y &= \frac{\mu_y}{\sigma^2}\\ b_y &= \ln\pi_y -\frac{\mu_y^{\top}\mu_y}{2\sigma^2} \end{aligned}$$

Problem 3

(a)由乘法原理可得数量为

$$\prod_{i=1}^n k_i - 1$$

(b)由条件可得

$$p\left(X_{1}, \ldots, X_{n}\right) = p(X_{1}, \ldots, X_{m}) \prod_{i=m + 1}^n p\left(X_{i} | X_{i-1}, X_{i-2}, \ldots X_{i-m}\right)$$

注意$p(X_{1}, \ldots, X_{m})$需要的参数数量为

$$\prod_{i=1}^m k_i - 1$$

而$p\left(X_{i} | X_{i-1}, X_{i-2}, \ldots X_{i-m}\right)$需要的参数数量为

$$(k_i - 1)\prod_{j=i-m}^{i-1} k_j$$

所以总共需要的参数数量为

$$\left(\prod_{i=1}^m k_i - 1\right) +\sum_{i=m +1}^n (k_i - 1)\prod_{j=i-m}^{i-1} k_j$$

(c)不难看到当且仅当下式成立时满足条件

$$p\left(X_{1}, \ldots, X_{n}\right)= \prod_{i=1}^n p(X_i)$$

Problem 4

结论不成立。

利用反证法，考虑$n=2$的情形，假设原结论成立，那么

$$\begin{aligned} &\frac{1}{\sqrt{2\pi} \sigma_1 (0)} \exp\left(\frac{(x_1-\mu_1(0))^2}{2\sigma_1(0)^2}\right) \frac{1}{\sqrt{2\pi} \sigma_2 (x_1)} \exp\left(\frac{(x_2-\mu_2(x_1))^2}{2\sigma_2^2 (x_1)}\right)\\ =&\frac{1}{\sqrt{2\pi} \hat \sigma_1 (x_2)} \exp\left(\frac{(x_1-\hat \mu_1 (x_2))^2}{2\hat \sigma_1^2 (x_2)}\right) \frac{1}{\sqrt{2\pi} \hat \sigma_2(0)} \exp\left(\frac{(x_2-\hat \mu_2(0))^2}{2 \hat \sigma_2^2 (0)}\right) \end{aligned}$$

对比指数上的系数不难得到

$$\begin{aligned} \sigma_1 (0)&=\hat \sigma_1(x_2)\\ \sigma_2(x_1)& = \hat \sigma_2(0) \end{aligned}$$

由$\sigma_2$的任意性可得第二个等式不可能恒成立，与原假设矛盾。

Problem 5

(a)

$$\begin{aligned} \mathbb E_{z^{(i)} \sim p(z)}[A] &= \frac {1}{k}\sum_{i=1}^k\mathbb E_{z^{(i)} \sim p(z)} [p(x | z^{(i)})]\\ &= \frac {1}{k}\sum_{i=1}^k \int p(x|z) p(z) dz\\ &= \frac {1}{k}\sum_{i=1}^k p(x)\\ &= p(x) \end{aligned}$$

(b)由Jason不等式可得

$$\begin{aligned} \mathbb E_{z^{(i)} \sim p(z)}[\log A] &\le \log \mathbb E_{z^{(i)} \sim p(z)}[A]\\ &= \log p(x) \end{aligned}$$

当且仅当$A$是常数时等号成立，所以不是无偏估计

Problem 6

1.

$$n=[\log_2 657] + 1=10$$

2.

$$(900-657) \times 64 + (900-657) \times (128 + 1) =46899$$

3.

logits, h_prev = self.forward(x)
x = logits.argmax().item()
texts.append(x)
x = torch.tensor([[x]])
for i in range(seq_len):
	logits, h_prev = self.forward(x, h_prev)
	x = logits.argmax().item()
	texts.append(x)
	x = torch.tensor([[x]])

4.

for i in range(len(string) - 1):
    logits, h_prev = self.forward(x, h_prev)
    prob = F.softmax(logits)[0]
    ll += np.log(prob[string[i + 1]])

    x = string[None, i + 1, None]
    x = torch.from_numpy(x).type(torch.int64).to(self.device)

5.

根据图像即可得到条件

if ll < -600:
	lbls.append(0)
elif ll < -200:
	lbls.append(1)
else:
	lbls.append(2)