#### 选择题 1

Why is a Deep Belief Network not a Boltzmann Machine ?

• A DBN is not a probabilistic model of the data.
• All edges in a DBN are directed.
• Some edges in a DBN are directed.
• A DBN does not have hidden units.

DBN最顶层是RBM，这一层为无向边，其余每层为有向边，第三个选项正确。

#### 选择题 2

Brian looked at the direction of arrows in a DBN and was surprised to find that the data is at the “output”. “Where is the input ?!”, he exclaimed, “How will I give input to this model and get all those cool features?” In this context, which of the following statements are true? Check all that apply.

• In order to get features $h$ given some data $v$, he must perform inference to find out $P(h|v)$. There is an easy exact way of doing this, just traverse the arrows in the opposite direction.
• A DBN is a generative model of the data and cannot be used to generate features for any given input. It can only be used to get features for data that was generated by the model.
• A DBN is a generative model of the data, which means that, its arrows define a way of generating data from a probability distribution, so there is no “input”.
• In order to get features hh given some data vv, he must perform inference to find out $P(h|v)$. There is an easy approximate way of doing this, just traverse the arrows in the opposite direction.

DBN是生成模型，不把inputs映射到outputs，第三项正确；推断$P(h|v)$可以”traverse the arrows in the opposite direction”，但注意这只是一种近似方法。

#### 选择题 3

In which of the following cases is pretraining likely to help the most (compared to training a neural net from random initialization) ?

• A dataset of images is to be classified into 100 semantic classes. Fortunately, there are 100 million labelled training examples.

• A speech dataset with 10 billion labelled training examples.

• A dataset of binary pixel images which are to be classified based on parity, i.e., if the sum of pixels is even the image has label 0, otherwise it has label 1.

• A dataset of movie reviews is to be classified. There are only 1,000 labelled reviews but 1 million unlabelled ones can be extracted from crawling movie review web sites and discussion forums.

#### 选择题 4

Why does pretraining help more when the network is deep ?

• As nets get deeper, contrastive divergence objective used during pretraining gets closer to the classification objective.
• Backpropagation algorithm cannot give accurate gradients for very deep networks. So it is important to have good initializtions, especially, for the lower layers.
• Deeper nets have more parameters than shallow ones and they overfit easily. Therefore, initializing them sensibly is important.
• During backpropagation in very deep nets, the lower level layers get very small gradients, making it hard to learn good low-level features. Since pretraining starts those low-level features off at a good point, there is a big win.

#### 选择题 5

The energy function for binary RBMs goes by

When modeling real-valued data (i.e., when $\mathbf{v}$ is a real-valued vector not a binary one) we change it to

Why can’t we still use the same old one ?

• If we continue to use the same one, then in general, there will be infinitely many $\mathbf{v}$’s and $\mathbf{h}$’s such that, $E(\mathbf{v, h})$ will be infinitely small (close to $-\infty$). The probability distribution resulting from such an energy function is not useful for modeling real data.
• Probability distributions over real-valued data can only be modeled by having a conditional Gaussian distribution over them. So we have to use a quadratic term.
• If we use the old one, the real-valued vectors would end up being constrained to be binary.
• If the model assigns an energy $e_1$ to state $\mathbf{v_1,h}$, and $e_2$ to state $\mathbf{v_2,h}$, then it would assign energy $(e_1 + e_2)/2$ to state $\mathbf{(v_1+v_2)/2,h}$. This does not make sense for the kind of distributions we usually want to model.