The Science of Algorithmic Bias vs the Industry of "Algorithmic Bias" To Impose Bias

[James Bowery]

The large language models such as the basis for ChatGPT (GPT-3.5) are the antithesis of The Hutter Prize in that they boast how "large" they are.  This flies in the face of Solomonoff's proof that prediction is optimal with the minimum possible parameters that describe the corpus of observations. 

Now I'm probably even more capable than the next guy of offering apologetics for the LLMs.  Here, let me show you:

"Valorizing largeness in Large Language Models is the right thing to do because, just as it is necessary to decompress zip files in order to realize the greater compression potential of bzip2, language is an encoding of observations latent in the assertions and it is those observations that Solomonoff's proof addresses."

See?  Now everyone can get back to ignoring Algorithmic Information as model selection criterion (as they have for over half a century in the natural sciences), and just plow more and more resources into bigger and bigger language models rather than tackling the really tough problems of lossless compression!  *WHEW* What a relief!

OK, now let me deconstruct my defense of LLMs and provide insight into what is really going on:

There is an incentive to impose carefully crafted biases in these LLMs, hence there is an anti-incentive to apply Algorithmic Information as the model selection criterion.  Let us call the technology to impose these biases the "Algorithmic Bias Industry" and the science to detect and quantify bias the "Algorithmic Bias Science". 

The Algorithmic Bias Industry is, in AIXI terms, about the Sequential Decision Theoretic half of AGI wherein the utility function parameter is set by the values of the institutions that control the LLMs.  Algorithmic Bias Science is implicit in the other half of AIXI: 

Apporaching the Algorithmic Information of the observations.

The real problem with ChatGPT isn't just that it is incapable of logical reasoning (which it isn't primarily because its dynamics still aren't Turing complete) but rather that the "Algorithmic Bias" Commissars have actively lobotomized it so that it out-and-out lies about facts that it clearly has in its pre-2021 training data, and I'm not talking about obscure facts either.  Moreover it has canned answers that are clearly meant by the Commissars to be political evasions that sound like they are correcting you for the kind of question you asked.

If there were a real science of “Algorithmic Bias” – one that studies the degree to which optimal Algorithmic Information models can be biased by their training data, rather than a technology that serves the opposite purpose (as a theocratic watchdog in imposing its moral bias via kludges) – it would study error propagation with an eye toward interdisciplinary consilience. But before talking about that, we can see that even intradisciplinary bias in the data needs to be quantified.

For example let’s say there is a body of work that keeps reporting the boiling point of water is 99C and another body of work that keeps reporting 100C. The Algorithmic Information of these observations might add 1C to the former or subtract 1C from the later (both subtraction and addition increasing the number of algorithmic bits by the same amount in the resulting algorithmic information). The balance tips, however, when considering the number of “replications”. The choice of which of these two complications may seem arbitrary, but it does get into a kind of “voting” except it’s logarithmic base 2 of the number of replications:

If there are 2^12 reports of 99C and only 2^9 reports of 100C then it makes sense to declare the 100C reports as “biased” because it takes fewer bits to count the number of times the bias-correction must be applied.

Now, obviously, there may be other observations, such as barometric pressure, that may be brought to bear in deciding which body of measurements of the boiling point should be treated as “biased”. So, for example, if it is quite common for measurements of other physical quantities to be taken at sea-level where standard pressure is common, then consilience with those other measurements may amortize the extra bits it takes to treat 100C as the “true” boiling point of water.

When we get into more extreme cases of cross-disciplinary consilience, such as we would see between physics and chemistry, the case counting and log2 “voting” becomes more sophisticated but, in effect, increases the confidence in some “truths” by increasing their votes from other disciplines.

If you get into consilience between, say, the Genome Wide Association Study and various models of social causation derived from, say, public government data sources, the cross-disciplinary consilience cross-checks become even more sophisticated.

If we get into language models, where all of this knowledge is being reduced to speech that supposedly conveys scientific observations latent in their assertions, it gets even more sophisticated.

But the principle remains the same.

This is why I saw Wikipedia’s ostensible wide-range of human knowledge, as a target-rich environment for exposing the more virulent forms of bias that are threatening to kill us all.

I am unspeakably sad for humanity the Hutter Prize has not received more monetary support since it is a very low risk investment for very high returns for the future of humanity, and in the intervening 17 years the enemies of humanity have made enormous strides in industrializing “Algorithmic Bias” to the point that we may soon see very persuasive computer based education locking us into what John Robb has called “The Long Night”.