Hi James and all, and I trust that all are well.
Both re the use of the acronym AIC and re the stated history of MDL and related work (e.g.,
algorithmic information theory and the work of R J Solomonoff), I feel now might be a good time
to offer some comments.
First, re the term Algorithmic Information Criterion and the acronym AIC that I believe was used
for this in recent days in this thread, a couple of comments. Akaike's Information Criterion is a
statistical approach that has been used in machine learning (e.g., neural networks) and has been
referred to as AIC for decades.
My readings of the writings of Ray Solomonoff (1926-2009) include mention of the term
"algorithmic probability", which Ray also abbreviated to ALP - indeed, I recall once discussing with
Ray that the (political party) Australian Labor Party uses this same acronym.
Given Ray's use of (his term) ALP (and I can give references, including to a survey I did of Ray's work)
and given that AIC has been used for decades to refer to Akaike's Information Criterion - given these
two precedents - I suggest taking care regarding acronyms, for the sake of clarity and/or disambiguation.
Next, while I can understand the mention of J Rissanen's minimum description length (MDL) work
from 1978 and subsequently, I note the seeming omission in this thread to date of the decade-earlier
minimum message length (MML) work of Chris Wallace (1933-2004) and colleagues, going back to
C S Wallace and D M Boulton (Computer J, 1968).
One place where one can see the relevant MML references prior to 1978 spelt out is p901 (1st page)
Here are some relevant works involving Chris Wallace and David Boulton from 1968 to 1975:
Christopher S Wallace and David M Boulton. An information measure for
classification. Computer Journal, 11(2):185–194, 1968.
David M Boulton and Christopher S Wallace. The information content of
a multistate distribution. Journal of Theoretical Biology, 23(2):269–278, 1969.
David M Boulton and Christopher S Wallace. A program for numerical
classification. The Computer Journal, 13(1):63–69, 1970.
David M Boulton. Numerical classification based on an information measure,
M.Sc. thesis, Basser Computing Dept., University of Sydney, Sydney, Australia, 1970.
David M Boulton and Christopher S Wallace. A comparison between information
measure classification. In Proc. of the Australian & New Zealand Association for
the Advancement of Science (ANZAAS) Congress, August 1973. Abstract.
David M Boulton and Christopher S Wallace. An information measure for
hierarchic classification. Computer Journal, 16(3):254–261, 1973.
David M Boulton and Christopher S Wallace. An information measure for
single link classification. Computer Journal, 18(3):236–238, 1975.
Christopher S Wallace and David M Boulton. An invariant Bayes method
for point estimation. Classification Society Bulletin, 3(3):11–34, 1975.
David M Boulton. The Information Measure Criterion for Intrinsic Classification,
PhD thesis, Dept. Computer Science, Monash University, Clayton, Australia, August 1975.
As above, Dowe (
2011, page 901) is one place where these MML references from
1968-1975, prior to 1978, are spelt out. I can find at least one other such place if
anyone is interested.
J J Rissanen's MDL and G Schwarz's Bayesian Information Criterion (BIC) were both
published in 1978, both advocating the same approach back then in 1978. (Akaike's AIC
pre-dates these, and MML pre-dates all of these.) In the time since 1978, MDL has evolved.
Whatever one's views on MML (going back to 1968) and MDL (going back to 1978), given the
discussion in this thread re Solomonoff-Kolmogorov complexity and algorithmic information
1999a, "Minimum Message Length and Kolmogorov complexity"). That issue of the Computer J
contributions by R J Solomonoff, J J Rissanen, (former PhD student of A N Kolmogorov) L A Levin,
and other relevant researchers.
C S Wallace's work - not just MML but also (e.g.) random number generation, fast multiplication, etc.
(G Barmpalias and D L Dowe, 2012) - is also due to Chris Wallace.
Keep well, thank you for your consideration, cheers and sincerely from David D