The paper sets itself the task of figuring out how to make Deep networks Deeper. The intro dismisses the old bugbear of Vanishing / Exploding gradients since the proper forms of normalization had been discovered. Instead, the paper makes an observation that’s been sitting in plain sight, but which goes against several common intuitions.

When deeper networks start converging, a degradation problem has been exposed: as the network depth increases, accuracy first gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error
[emphasis original]

In other words, a certain dogma says “More layers, more capacity, higher training accuracy, more overfit” and this paper says, “Actually no!”

Instead of attacking the problem in terms of managing model capacity, this paper looks at another problem altogether: difficulty in optimizing the model. That is, if your model isn’t even realizing the capacity it has in theory, then there’s no use in trying to manage or control it. This points to a real problem – how many papers from before 2016 can you think of that explored architectures with over 100 layers? ²²That is, from the current generation of DNNs. Apparently Jurgen Schmidhuber was experimenting with over 1000 layers in 1993.

The paper examines the declining training accuracy phenomenon in light of a simple thought experiment: what would happen if one were to set the later layers exactly equal to the Identity transform? Of course this would always have equal training and test accuracy. Yet, the above plot shows that for some reason the solvers are converging to something worse as models have more and more layers. The paper reasons that declining train accuracy is therefore not intrinsic to deeper nets, but rather an artifact of the optimization characteristics resulting from architecture choices. This thought experiment led to the core idea of the paper: adding skip connections, which the paper calls “shortcut connections”.

Let us consider \({\cal H}(x)\) as an underlying mapping to be fit…, with \(x\) denoting the inputs… If one hypothesizes that multiple nonlinear layers can asymptotically approximate complicated functions, then it is equivalent to hypothesize that they can asymptotically approximate the residual functions, i.e., \({\cal H}(x) − x\)… So rather than expect stacked layers to approximate \({\cal H}(x)\), we explicitly let these layers approximate a residual function \({\cal F}(x) := {\cal H}(x) − x\). The original function thus becomes \({\cal F}(x)+x\).

With skip connections, the output of each layer is a delta between the previous layer and whatever the expected output is, which is the definition of a residual. This way, if the model can’t find a no-op on its own, then at least we could try to make it easier to find one – by giving quiescent residuals. Hence, ResNet.

The central proposition of the paper is the residual unit, a combination of convolutional layers with a pass-through connection (the identity transform from the thought experiment), which they later refine into the bottleneck unit.

In practice, the residual units (left), tend to be about as common as bottleneck units (right, described below).

• It’s hard to overstate how important that initial insight is – that adding more layers to a network doesn’t just magically give it more capacity. Capacity isn’t just a function of the architecture, but also the preprocessing, and optimization too. For instance, by carefully normalizing the data or fine-tuning the learning rate schedule one can coax a DNN into getting near-perfect training error⁴⁴Training error decreases as model capacity increases. Once it hits 0 on one dataset, one would need a larger / harder dataset to observe further increases in capacity. on larger data sets. The paper does more than just provide an insight, however – it actually makes it useful.
• In this paper, skip connections were thought of as a way of fixing something broken about the models to see if they could make a thought experiment came out as expected. The fact that they might have actually improved training error and generalization was total serendipity! This is why it’s important to always build a toy model to try to fix or control something that doesn’t make sense.
• Interestingly, they point out that the 2015 Inception paper⁵⁵Going Deeper with Convolutions
  Szegedy et al. CVPR 2015 had skip layers leading directly to a separate classification / loss term that was discarded at inference time, that was explicitly intended as a fix for exploding and vanishing gradients. This is telling – it suggests you can’t climb the next hill as long as you’re struggling with the current one. Several other papers came close to making the same discovery, but they either made their use of skip connections too complicated with e.g. gating functions, or else they only added skip connections from the input layer, and not elsewhere throughout the network.
• The paper demonstrates in several ways that there are better and worse ways of investing a compute budget. Where it really shows is in the channel width of the convolutions, and they are judicious about raising the channels or not using them frivolously, e.g., in the \(1\times 1\) convolutions on the skip connections. The purpose of all of the frugality, though, is really to enable them to add more layers.
• One possible reason why the model can’t find the identity transform on its own could be because the identity is a very sparse pattern. In order for a model to settle on such a sparse pattern it would be necessary⁶⁶Technically, there is a 0-measure set of non-sparse gradient updates that would accomplish this, but it’s not clear whether any error term at the head could produce such an effect at more than one layer., but not sufficient, to have a sparse gradient, and that is clearly not happening on its own. The skip connection trick bypassed the problem altogether by changing variables such that the same no-op pattern is not sparse, but just extremely low mean / low variance, which SGD can find by itself. But what if there are other sparse patterns⁷⁷The Lottery Ticket Hypothesis: finding sparse, trainable neural networks
  Frankle, Carbin ICLR 2019 waiting to be discovered?
• The Gibbs-like effect in the activation variability per layer makes me wonder if there is some feedback happening. Consider that at each residual unit, every gradient step affects both the residual and the previous layer at the same time. This is bound to produce some double-correction, and where there is overcorrection there may be oscillation.
• To my eye looking back from 2022⁸⁸and now publishing in 2023…, Residual Units look a lot like a poor man’s partial concept classes (PCCs)⁹⁹A Theory of PAC Learnability of Partial Concept Classes
  Alon et al. FOCS 2021. That is, the earlier layers learn the broad concept, and later layers are, if working as intended, only learning the exceptions, and then the exceptions to the exceptions and so on. Like partial concepts, exceptions are only defined in certain regions of latent space. Given the sparsity inducing behavior of ReLU units it’s conceivable that something sort of like this may be happening, but on the other hand given the variability in the activations per layer in the 110 layer ResNet, maybe something else is going on.
• By now it’s pretty well known that the teacher-student distillation model¹⁰¹⁰Distilling the Knowledge in a Neural Network
  Hinton et al. arxiv 2015, wherein one first trains a big complicated model, and then trains a single, simpler model using just the input/output pairs from the teacher model, is a very effective trick. In a way, this may be similar to what the bottleneck units are doing. That is, they have a source and a target, and they try to learn to update, but they have to mimic it at a lower channel resolution.

It’s easy to think of this paper as a lab writeup on a new widget that wins contests easily, but I think a better way to read it is to see that, at least in this instance, you can win all of that fame and fortune by optimizing better. The widget isn’t for winning contests, or even improving generalization. It’s for improving the optimization. Who knows, maybe there are more ideas to be found by thinking of some obvious pattern that SGD ought to be able to find, but can’t, and then designing a widget to help it.

The Rich Sutton Bitter Lesson essay says, essentially, it’s a waste of time trying to encode beliefs directly into the loss function via some fancy regularizer, and that instead one should look for a framework that improves with more compute power, no matter how much one has. It uses the term “invariances” only once, but I think it’s the most critical word in the entire essay. Invariances are a kind of prior, but they aren’t facts or knowledge, they’re properties. They’re regularities. I think residual connections are a way of finding regularities in a much more compute-efficient way.

This paper¹¹¹¹Visualizing the Loss Landscape of Neural Nets
Li et al., NeurIPS 2018
See our outline of this paper., and others, show that the loss landscape is impacted greatly, and positively, by the residual connections. By making the loss landscape more regular they ensure that no small perturbation can radically alter the model’s training accuracy, and that is the kind of regularity that really matters.

I also can’t help but notice that when done right, the total compute burden goes down, which may take some of the bitterness out of the lesson.

Thanks for reading to the end!

Comments can be left on twitter, mastodon, as well as below, so have at it.

New post!

Paper review: The ResNet paperhttps://t.co/rSbpXk2cI6

Reply here if you have comments.

— The Weary Travelers blog (@wearyTravlrsBlg) July 15, 2023