Continuing the theme from our earlier post on additive loss functions, this week we’re again revisiting the foundational assumptions of modern Statistical Learning.

The main idea of Statistical Learning is that we can use sampling alone to discover homogeneous regions in input space. That is, given an example on a classification task, the inductive bias is that there must be some ball or other region around it where all other points in that region must have the same label. Given an embedding vector for a word token, all other points nearby must represent the same token. Given a latent embedding vector… you get the idea.

The sampling part is the idea that we can use statistics to discover those regions, which is a bit like using statistics to prove that a coin is fair. That is, you can’t. Proving a coin is unfair is the forte of statistics, but it simply does not afford the ability to prove that a coin truly has \(50-50\) odds, because its axioms don’t allow for that¹¹Statistics can prove that a given coin is probably fair enough, and this is related to the Probably approximately correct theory of learning.. Yet, flipping a coin often is used in high-stakes public events including Sporting Events and even some Political outcomes.

But notice that no one ever bothers to check the actual coin used in such a tossing process²²At lest, not by flipping it \(10,000\) times., because the reasoning for that decision never had anything to do with statistics in the first place. That reasoning relies on an altogether different set of metaphysical axioms³³What those might be will be a topic for the future. Today, we’re only interested in making a negative point. That said, something like the Scientific Method might be appropriate. That is, there has to be a way to entertain a model that makes homogeneous assertions, as long as it remains useful, and then reject it later. Decision Trees do something like that, but they rarely have a back-tracking mechanism for doing that. Secondly, they operate directly on the observed domain rather than a latent/derived domain. Thirdly, they probably aren’t sufficient for general intelligence because for that one does need statistical reasoning for some things..

Clearly, there have to be limits to the homogeneity principle, and furthermore there’s absolutely no reason to think that the homogeneity radius around every observation is the same, or isotropic, or even non-zero. But that’s why statistics is so useful, isn’t it? We already know that we won’t have local homogeneity around each point, so why not search for the regions that are as close to purity as possible?

The problem is that in interesting problems, the cost of being wrong can often be arbitrarily high in a neighborhood that’s arbitrarily small. In some cases, the cost of being wrong will be higher than the cost of achieving the desired statistical purity that would give one sufficient confidence to ignore it. In other words, Sense and Nonsense can be arbitrarily close⁴⁴This is a similar, but more specific claim than the one made by Black Swan theory – we are going a step further in saying that in some cases a “Black Swan” can be found arbitrarily close to a white one. OTOH, perhaps the rare, high cost, difficult to predict events we are thinking of don’t quite meet the technical definition of “Black Swans”..

This fact, and the awareness of it, is reflected in some human habits. For instance, when communicating with another person, even, or especially when they are hostile, the most useful example is often precisely the one that lies on a boundary, meaning its radius is \(0\)⁵⁵Hostile situations are interesting because those are the times when communicating costs is especially important.. More generally, when someone realizes they’re on a precipice, suddenly they start choosing their steps very carefully.

In practice there are many examples of tasks wherein the difference between sense and nonsense can be arbitrarily close to 0:

• In most programming languages, every correct program is never more than one character away from one that does not compile, or worse, from one that compiles, and crashes spectacularly.
• Contracts can have entirely different consequences depending on even a single word.
• Any math proof or formal reasoning is invalid if it has even a single error.
• If you’re building something out of parts, say a PC workstation, then if the supplier sends a part with a model number missing a single digit, you would probably want to file a complaint.

Ultimately, it depends on whether you can tolerate even a single speck of sand in your machine. Some are built for it, but others fail catastrophically. Machine learning is a bit weird in that for most real world tasks, if you want to be Probably Approximately Correct, then statistics is all you need – you can tolerate sand, and with what appears to be an exponentially lower bounded amount of work, you can shake most of the grit out of your model. But, you’ll never get a Swiss watch that way.

This idea can be formulated somewhat more precisely as the Last Inch Problem:

Using statistical / empirical risk minimization alone, directly on example-wise error, will result in a situation wherein each successive order of magnitude by which the error is reduced will tend to require at least the same order of effort, or more.

In other words, getting the error from \(0.01\) to \(0.001\) is often much easier than getting the error from \(0.001\) to \(0.0001\), and it only gets worse from there.