Language Universal Priors Not Exploited by Existing Learning Algorithms

[James Bowery]

While it is always dangerous to impose priors that are not universal in the sense of Universal Turing Machines, it is also a practical necessity.  This pragmatic is nowhere more urgent than in resource constrained benchmarks like The Hutter Prize.

Having said that, there are proposals for what might be called "Language Universal Priors"  that have not yet been tried in machine learning of language corpora -- at least not the LLMs:

The Final-over Final Condition

and

The (indexed) grammar of mammalian brain capacity

I inserted the "(indexed)" qualifier to make it clear that so-called "indexed grammars" may be a way of escaping the intractability of fully context sensitive grammars equivalent to UTMs, in a manner that provides greater learning algorithm data efficiency in language corpora.

Indexed grammars are equivalent to nested pushdown stack automata.  Despite their relevance to naturally evolved learning mechanisms, they are almost entirely ignored in machine learning.

[Matt Mahoney]

The first paper really says that in the 5 European languages they studied, that in those languages where the verb comes after the object, the auxiliary verb follows the verb. Then they generalize to the entire parse tree in all languages.

That is not so. Spanish adverbs modifying adjectives violate the final over final condition. For example, "rio grande" literally means "river big" (head final) but "very big river" is "rio muy grande" (head initial) not "rio grande muy".

The prior for natural language is the limited capacity for short term memory, about 7 words. This limits the complexity and depth of sentences to those where the head and modifier both fit into memory, with low frequency words persisting longer.

--
You received this message because you are subscribed to the Google Groups "Hutter Prize" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hutter-prize...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hutter-prize/151bf85d-6e64-47dd-8eaf-81ed8e44170an%40googlegroups.com.