How do we enforce such a property? The Hopfield network provides a weight update rule that ensures it. Concretely, the state space is described by \(N\) binary bits \(V_i\). The evolution function is described by a perceptron-like summation of a set of weights \(T \in \{0,1\}^{N\times N}\) connecting them, with a step-function nonlinearity. \[ \underset{V_i\to 0}{V_i\to 1} \qquad \mbox{if } \sum_{j\ne i} T_{ij}V_j \qquad \underset{< U_i}{> U_i } \] Note the very un-perceptron-like global recurrence implied here. So, in order to recover a pristine string \(V^* \in \{0,1\}^N\) that was actually stored, given some fragment or corrupted version of it \(\hat{V}\), we would set the state vector \(V_0 := \hat{V}\), and let the state evolve under this rule until it reaches a fixed point, which is hopefully \(V^*\).

That nonlinearity does something interesting - without it, the purely linear update rule at time \(t\) would be \[ V_{t+1} = \frac{T V_t}{\| T V_t \|} \] and the stable states would be exactly the eigenvectors of \(T\). If \(\tilde{V}\) isn’t an eigenvector, then it’ll asymptotically converge to the eigenvector \(v_i\) that maximizes \(\lambda_i v_i^T\tilde{V}\). It’s not very useful if one can only encode patterns that are strictly orthogonal to one another. With the nonlinearity, we can instead encode patterns with more flexibility - they simply have to be uncorrelated, which can be ensured with some pre-processing.

So we know how the Hopfield net retrieves stored patterns, but how does it store them? Suppose we have a dictionary of strings \(V^s \in \{0,1\}^N, s \in 1 \dots n\) that we would like to store. There just so happens to be an initialization rule that will enable this: \[ T_{i\ne j} = \sum_s(2V_i^s - 1)(2V_j^s - 1) \] The \((2V_i^s - 1)\) pattern is just converting \(\{0,1\}\) binaries into \(\{-1,1\}\) binaries⁴⁴Alternatively, we can think of this rule as setting \(T_{ij}\) to be the number of patterns that are the same at \(i\) and \(j\) minus the number of patterns that are different.. From here out we also assume that \(T_{ii} = 0\) since whatever value \(T_{ii}\) has can instead be put into the threshold \(U_i\).

With that we can initialize \(T\), but how do we update it? We can assemble the \(0-1\) binary patterns \(V^s\) into a matrix \(D\) so that \(D_{\cdot s} = 2V^s -1\), which are \(\pm 1\) binary patterns. The rule can also be written as a sum of rank-one updates⁵⁵A rank-one update is where we update a matrix \(T\) by adding to it another matrix \(M_1\) whose rank is one. Any rank-one matrix can be written as an outer product of two vectors, \(M_1 = vv^T\). Updating \(T\) by adding \(DD^T\) is equivalent to doing a series of rank-one updates. to \(T\), \[ T = \sum_s D_{\cdot s}D_{\cdot s}^T - n {\mathbf I} = DD^T - n {\mathbf I}, \] It follows that storing, or attempting to store, a single new pattern is also accomplished via a rank-one update.

With this update rule in hand, the recovery of \(V^s\) given some corruption \(\tilde{V^s}\) still requires \(V^s\) to be a stable state. How do we know that it is? When \(T\) is set as above then the evolution rule for a single element \(V_i\) is given as, \[ V_i \sim \sum_j T_{ij}V_j = \sum_sD_{i s} \left[ D_{\cdot s}^TV \right]. \] The term in square brackets is the inner product of pattern \(D_{\cdot s}\) with the current state \(V\). If \(V\) happens to be the same pattern as \(D_{\cdot,s}\), then the inner product of \(V\) with \(D_{\cdot, s}\) will be the number of ones in \(V^s\) – on the order of \(N/2\) – and all of the other inner products will be \(\sim 0\) – on average. If it is large, then the sign of \(D_{is}\) will determine the value of \(V_i\) at the next time step, which is to say, \(V_i\) will stay the same.

Unfortunately we don’t get a free lunch. All of this depends on that square-bracketed term being close to \(0\) for all of the other patterns that are not \(V^s\). So long as they are distributed more or less isotropically⁶⁶Isotropy is the property that a set or distribution is spherically symmetric. This is synonymous with the axis variables \(V_i\) being uncorrelated., then it’s safe to assume so. Equivalently, if all of the \(i\) and \(j\) elements have low correlations across the dictionary then this will work. The paper argues that this is ok as long as we assume that some sensory subsystem has found an embedding for the patterns that has this property.

With this learning rule we always get a symmetric weight matrix \(T\), but that seems wasteful – half of the weights are redundant. Being symmetric gives \(T\) the useful property that it will be stable like we saw above, but there are also ways of setting \(T\) that aren’t symmetric. The problem is, sometimes those have stable cycles rather than stable states, and that is a whole other topic.

The model proposed in the paper is indeed novel, but it didn’t come out of nowhere. There were already several models trying to solve the same kind of problem, but they didn’t quite have the nice properties of storage and retrieval that this one does. One of the things they do have in common is that they have a similar update rule, which is the Hebbs rule, which the paper gives this form: \[ \Delta T_{ij} = [V_i(t)V_j(t)]_\mbox{average} \] with the average being taken over some windowed time series indexed by \(t\). Clearly it’s a sum of rank-one updates, just as this paper’s own rule is. The difference is, they’re not given explicitly as specific patterns, but as potentially noisy ones. In the Hebbian learning model, the states are potentially noisy, and the averaging attenuates that. Presumably they’re perturbed by specific patterns that an external system wishes to store, or to retrieve them, and otherwise keep evolving and updating on their own.

Above we showed via plain linear algebra that when \(T\) is composed of rank-one outer products then those patterns are the stable points. But, there is another way to show this property, and it links Hopfield nets with statistical mechanics. It turns out that if we can define some “energy function” for each state, and if we can show that the evolution function never increases this energy function, then we know that any local minimum of the resulting energy “landscape”⁷⁷Here, “energy landscape” means something like a Gibbs Free Energy or Potential Energy, which can be expected to decrease over time towards a local minimum. Depending on which statistical ensemble one uses to model the system, energy may or may not be locally conserved as it moves in or out of the system. is a stable state with high probability as desired, and will have an approximately convex local neighborhood.

This is more than what we knew before – the only thing we showed above was that the patterns \(V^s\) are stable. Now, according to the analogy from Ising models, we should see that the patterns that converge to \(V^s\) form a locally convex neighborhood. To do this we just need to create such an energy function, and the paper shows there is indeed such a function: \[ E = -\frac 1 2 \sum_{i,j} T_{ij}V_iV_j = V^TTV, \] and indeed the evolution step reduces it: \[ \Delta E = - \Delta V_i \sum_jT_{ij}V_j. \] This means that stable states are local minima, as desired. The paper says that this form is isomorphic to Ising Spin Glass models, which allows the application of a result from that theory stating that if \(T\) is symmetric and “random” then there will be “many” stable local minima to the above energy function, implying that this network will be able to store “many” points of data. But how many, and under what conditions?

There were several aspects of the model that the paper examined experimentally. Since it was 1982, the paper only explored cases where \(N = 30\) and \(N = 100\), so there might be different results for higher parameter settings. Nevertheless, a biologically plausible model for simple ganglia in invertebrates would be for \(N\) in the range of \(100\) – \(10,000\), putting the experiments here on the lower end of that range.