The Rectifying Linear Unit (ReLU) (or its younger siblings, the Leaky ReLU ³³Rectifier Nonlinearities Improve Neural Network Acoustic Models
Maas et al. ICML 2013​, GELU ⁴⁴Gaussian Error Linear Units
Hendrycks\, Gimpel Arxiv 2016, etc. etc. etc.,) is now the de-facto standard non-linearity in Deep Learning. The one thing that needs to be said about ReLU is that it doesn’t “saturate”⁵⁵“Saturating” is where the activation has an asymptotic maximum, and the output at some unit is very close to it. When this happens, gradient updates that need to reduce that unit’s output get “stuck” because when a non-linearity is already outputting near its asymptotic limit, it takes a massive gradient update to its inputs to decrease it. But, its suppressed magnitude also suppresses the gradient, meaning that a single gradient update won’t materially reduce its output. The Glorot paper below talks about this.. The paper does mention an earlier Hinton paper ⁶⁶Rectified Linear Units Improve Restricted Boltzmann Machines
Nair\, Hinton ICML 2010
To give an idea of how far back this is, the Hinton paper that introduced the ReLU was about fixing Restricted Boltzmann Machines. that originated ReLU units, but they were still new enough to warrant a few paragraphs to describe and motivate using them over the previously common sigmoid or tanh. Moreover, this paper rightly recognizes their importance – Section 3 (The Architecture) is ordered by the authors’ assessment of decreasing importance, and they put ReLU first.

One interesting observation is made in passing – when comparing with an earlier work that claimed tanh worked well, they point out that on the smaller Caltech-101 overfitting matters more than fast learning. So, it’s not so much that ReLU is always better than saturating functions, as it is more eager and regularizes less.

Their layer normalization is also highly customized, and imposes a kind of locality among channels. It’s a bit of a misnomer to call it “layer normalization” because each channel’s output is not normalized over the entire set of channels at the layer, but rather over a sliding window of channels. They found that a window size of 5 worked best. This is a particularly radical move because it disrupts the equivariance of almost all neural net units to reordering of the units.⁷⁷That is, you can scramble the order of hidden units in any Fully Connected layer and get the same DNN. The same goes for channels in a Conv unit, or an LSTM or Transformer. Instead, this normalization scheme imposes local interactions that implement a kind of lateral inhibition inspired by real neurons. They report a drop of 1.4 %-point in top-1 error with this trick.

One might think of this not as a “normalization” trick, but as a joint activation function.

This paper wasn’t the first ⁸⁸Improving neural networks by preventing co-adaptation of feature detectors
Hinton et al. arxiv 2012
This one may or may not be the first to do something similar, but it does have to define the term “dropout”. to use dropout, but it was close. Since Krizhevsky was in Hinton’s lab at the time, that explains where it came from. Since dropout was such a new thing at the time, the paper even provides an explanation for why they’re using it and what effect they expect it to have. The paper describes it as a kind of virtual ensemble method. Under this interpretation, the average output of the model, (under different realizations of dropout,) is effectively an ensemble of individually weaker classifiers, and see also the inference time voting scheme below. They report that using dropout roughly doubles the training time, but who’s counting when you’re winning contests?

For classification in DNNs, you almost always use the Categorical Cross Entropy loss without ever thinking about it. It’s yet another unsung hero of modern Deep learning. Prior to this, it wasn’t unheard of for there to be a whole paper about some bespoke loss function and optimization scheme for dealing with Multi-class classification. Now, it’s common to do classification tasks with tens of thousands of classes and no one bats an eye. This paper does not use that terminology, and instead describes it as,

Our network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution.

Compare this with the equation for Categorical Cross Entropy:
\[H(p, q) = -\sum_{x} p(x)\log q(x) \] where \(p(x)\) is the class probability of \(x\), (which is uniform by default in all implementations I know of,) and \(q(x)\) is the softmax output.

Data augmentation is yet another indispensable step in modern Deep Learning. The term “augmentation” doesn’t mean “adding features” so much as “multiplying the size of your training set by adding perturbed examples”, so long as the perturbation is believed to not affect the label. Their data augmentation scheme is to take a random 224x224 patch from within the 256x256 input, and reflect horizontally at random. Like everything else, this was clever then, standard now.

They also did a colorspace manipulation trick that makes a lot of sense, but that I haven’t seen or heard much of since then – compare with this list of torchvision methods. The idea is to do PCA of the data in RGB space. That is, treating every pixel as a sample point in RGB space, presumably for all 78.6 billion pixels, they mean-centered them and computed a 3x3 covariance matrix, and then extracted its eigenvectors. For each new image they draw 3 random scaling coefficients \(\alpha_{1..3} \sim \cal{N}(0, 0.1)\), scale each eigenvector \(v_i\) by the corresponding \(\alpha_i\) and add it to every RGB pixel value in the image. This is a crude, but effective, way of altering the lighting of each image without drastically altering its appearance. Oddly, the paper doesn’t show what the eigenvalues are, nor what the 3 eigen-colors are, but one must assume they are meaningful because by itself this trick got them yet another %-point of error reduction.

The paper uses the augmentation scheme in another way – at inference time, it takes 5 crops, with and without horizontal flipping, for 10 in total. Then it averages the outputs across all. Where an ensemble averages across multiple realizations of the training process, this data augmentation averages across multiple realizations of the feature vector. I’ve seen at least one other paper since then that re-invented this trick, and it is an effective one. But with all of the other novelties in this paper it seems to have been lost in the shuffle.

They used a clever-then / standard-now learning rate schedule wherein every time validation error leveled out, they dropped the learning rate by 10x. Nowadays pytorch has a host of learning rate schedulers, and keras has hooks for rolling your own, but at that time all of this was hand made.

At last the paper has one archaism – they used a 0-mean Gaussian distribution for random initialization, rather than the now-common Glorot scheme ⁹⁹Understanding the difficulty of training deep feedforward neural networks
Glorot & Bengio AISTATS 2010
The essence of the Glorot initialization is for the variance of the weights to be scaled by both the size of the current layer and the next one. That seems to be a sufficient condition for activations’ variance to be comparable between one layer and the next, which is a necessary condition for gradients to propagate well. Having bounded support doesn’t seem to be necessary, though most implementations I’m aware of have it. Tensorflow has a clipped Gaussian that they call Glorot Normal, whereas pytorch has both a plain uniform distribution like eq. (1) in the Glorot paper, and a normal distribution as well., which is a Uniform distribution.

For the biases, however, they had yet another novel scheme. Recalling that in the 2-GPU scheme, the 2nd, 4th and 5th convolutions only take input from the stream on the same GPU, while the 1st and 3rd takes input from both. The first set had their biases initialized to 1, while all others were initialized to 0. They claim this speeds up learning in the early iterations since effectively all neurons are getting gradient updates rather than being suppressed. They don’t say why this is preferable only in the layers that don’t take input from both GPUs.

They reference earlier sources regarding max-pooling, however they pool with overlapping fields, which gives a 0.4%-point bump in top-1 accuracy.

Model parallelization is of course alive and well today, but it’s much less common to see any bespokeness to how it’s implemented¹⁰¹⁰Or maybe it is and I’m just missing out???. However, the implications of having each branch only take inputs from one branch are interesting. The paper notes that in every run that they did, one GPU always learned (mostly) color invariant features, which resemble Haar wavelets, while the other would learn color sensitive patterns like blue-yellow or red-green gradients. It is a bit maddening that the paper notices this, and then just says “huh” and moves on.

They report yet another decrease of 1.7%-points in top-1 test error over a 1-GPU scheme with about half as many convolutional channels as for the 2-GPU scheme.