Common Questions and Terminology
[list] []Sigmoid [] TanH [*] ReLU [/list] etc. -> Activation Functions
also see http://neuralnetworksanddeeplearning.com/chap3.html#other_models_of_artificial_neuron
aka error, loss or objective function
used to determine the error in output which is then used for Backpropagation. This boils down to calculating the gradient of the cost with respect to the weights (i.e the derivative of cost with respect to each weight for each node in each layer), and then using some optimization method to adjust the weights to reduce the cost.
- [Quadratic cost function] (http://neuralnetworksanddeeplearning.com/chap1.html#learning_with_gradient_descent)
- [Cross-entropy cost function] (http://neuralnetworksanddeeplearning.com/chap3.html#the_cross-entropy_cost_function)
Optimization methods include (source)
- Stochastic Gradient Descent
- Hessian-free optimization
- AdaGrad
- AdaDelta - An Adaptive Learning Rate Method
visual comparison:
aka L2 Regularization, weight decay, etc.
A network with large weights may change its behaviour quite much in response to small changes in the input. And so an unregularized network can use large weights to learn a complex model that carries a lot of information about the noise in the training data. In a nutshell, regularized networks are constrained to build relatively simple models based on patterns seen often in the training data, and are resistant to learning peculiarities of the noise in the training data. The hope is that this will force the networks to do real learning about the phenomenon at hand, and to generalize better from what they learn. But beware: simplicity is a guide that must be used with great caution! The true test of a model is not simplicity, but rather how well it does in predicting new phenomena. However, keeping the need for caution in mind, it's an empirical fact that regularized neural networks usually generalize better than unregularized networks.
http://neuralnetworksanddeeplearning.com/chap3.html#regularization
randomly (and temporarily) deleting half the hidden neurons in the network, while leaving the input and output neurons untouched. way of reducing overfitting. reduces complex co-adaptations of neurons ... forced to learn more robust features
Dropout has been especially useful in training large, deep networks, where the problem of overfitting is often acute.
http://neuralnetworksanddeeplearning.com/chap3.html#other_techniques_for_regularization
short: unsupervised pre-training is not necessary when the number of labeled training samples is large enough. Deep networks (beyond 3 layers or so) require unsupervised pre-training due to the diffusion of gradients problem.
Discussion of Answer (including remarks by Andrew Ng): http://www.quora.com/When-does-unsupervised-pre-training-improve-classification-accuracy-for-a-deep-neural-network-When-does-it-not