Matt's "Questions on Intelligence" thread on NuPIC-theory mailing list

[Rob Freeman]

This forum has been dead for eight years. But the issues which motivated its creation eight years ago still apply. In software distributed models of meaning have begun to have a small literature, particularly "Neural Probabilistic Language Models" from Yoshua Bengio starting about 2004, and a small vector compositional literature starting about 2007, but a complex systems perspective is still missing, and the focus in the new Deep Learning community remains on learning, belying the unlearnable assertions of grammatical incompleteness.

Recently Matt Lind started a thread "Questions on Intelligence" on the NuPIC-theory mailing list. Since NuPIC-theory is philosophically opposed to too much top down theorizing, particularly top down theorizing motivated from an analysis of language, I propose to move some of the language motivated further discussion here.

Matt's original post on NuPIC-theory was as follows:

http://lists.numenta.org/pipermail/nupic-theory_lists.numenta.org/2015-June/002625.html

A continuation of that thread is as below:

On Thu, Jun 4, 2015 at 10:19 PM, Matt Lind via nupic-theory <nupic-...@lists.numenta.org> wrote:

---------- Forwarded message ----------
From: Matt Lind <...>
To: NuPIC theory mailing list <nupic-...@lists.numenta.org>
Cc: 
Date: Thu, 4 Jun 2015 23:16:46 +0800
Subject: Re:
Many thanks to all who replied! That's all great input and brain food to digest. :)

"Meaning" as the "organization of information" is definitely a good way to look at it. Then the meaning of a sentence is derived from the organization (i.e. sequence) of its words

Yes.

At least, the sequence is a forcing function for a reorganization of the meaningful bit. The meaningful bit is actually groupings "across" the sequence. So groupings of things which could be substituted in the sequence, or paradigms (with a nod to Thomas Kuhn, I don't think Kuhn's modern meaning for the same word is unconnected, but that is another story.)

For instance "brain food" in your sentence above. If we take our "meaning" for the word "brain" to be a set of things which could be substituted for it across the language (mind, head...), and our "meaning" for the word "food" to similarly be a set of things which can be substituted for that (sustenance, meal,...), then the combination of "brain food" should select subsets of these two sets for "brain" and "food" and by this selection create another set (e.g. selecting only sets which occur with things like "food" should eliminate uses of "brain" like the violent act synonym for "hit".)

That means the combination specifies a subset meaning for the two words.

That's one way to, if not create new meanings, then at least to more precisely specify existing meanings. Other people do this. You can actually think of it as the basis of modern indexed search engines.

But we can do more. Going beyond mere specification, which is non-creative, we might make some new sets. For instance we can estimate a set of things which could be substituted for the two words "brain food" together. E.g. there might be some sequence where "ideas" has been used interchangeably with "brain food" or at least used interchangeably with two words from the sets for "brain" and "food".

This estimation can form a new set: "brain food" = "ideas, thoughts, inspiration".

It's this last that I don't think anyone is doing at the moment, and which results in creative formation of patterns, and thus new meanings.

 

, and the meaning of a word would be derived form the organization of its letters.

Actually no, not words.

Words are exactly that where the organization of internal elements has ceased to be meaningful in language. Or where it starts to become conventionalized. Actually that is a good definition for what we mean by the "words" of a language, where the creative part of syntax dies. Formally, frequency of repetition gives a good model for the concept of "words" in a language (which is often much less well defined than in English.)

Most words come originally from new combinations of elements (older words) which were intrinsically meaningful, yes, but through long repetition these combinations come in time to just have some conventionalized meaning instead. That is the very point at which they have ceased to be syntax and become words. For instance the word "awful" presumably originally consisted of a meaningful syntactic combination of elements conveying "full of awe", but now it just has the conventionalized meaning of "bad", or even by association with extremes, simply "very".

So combinations of letters are not generally productively meaningful in a compositional sense. (Though there may be systems of analogy which tend to associate similar sounding words.) If we are dealing with the creation of new meaning we are dealing with syntax.

 

Yet, English, for example, has lots of homonyms where the exact same organization of information/letters may have multiple meanings. I guess here "organization of information" would need to be understood as the broader context in which the homonym appears.

Yes, several conventionalized meanings can be packed into the same sequences of letters (associatively, they are no longer creative) and only decoded by context.

Actually the existence of such homonyms is in itself another predictive success for the contradictory structures/complex pattern generator/chaotic language model. You remember I suggested contradictory structures can make a set of elements "bigger than itself". They can do this because the same elements can participate in multiple meaningful structures. This predicts individual elements will appear "ambiguous". So this model predicts word ambiguity. Instead of wondering why nature is so messy as to allow word ambiguity, wishing language would just unambiguously assign one clear meaning to each word, or just assuming nature is inefficient (never true), instead we see word ambiguity is a feature which allows us to pack more meaning into the same sets of elements, and will allow those sets of elements to generate more meaning, creatively.

I also like the idea of thinking of creativity as reorganizing previously accumulated elements of "meanings" in a new manner. Obviously, in the case of language, such reorganization would have to be guided, e.g. through prescribed rules of grammar, but that should be no problem.

In fact "guiding" reorganizing previously accumulated elements of language using rules turns out to be a BIG problem. "Rules" for such grammar cannot be found.

That's why computationally linguistics has largely failed to progress for 50+ years. Our inability to find such rules has completely floored linguists.

But now, from this new complex system point of view, that turns out to be a good thing, because if grammar could be abstracted as rules it would not be creative (it would be impossible to create meaning not already abstracted in the system from the start, i.e. exactly that finite number of rules. The "meaning" of the system would be limited to that meaning programmed into the system from the start, and all the conundrums which have bedeviled philosophers would apply.) This is the case with all contemporary software, where any "meaning" is limited to an expression of rules by design. Software motivated by the system of language suggests a more creative possibility.
 

Yet this then leads to the question of "why". Why would the system be "creative", ie. try to reorganize elements in a new manner? The 2 answers I'd come up with are: 1) in order to answer a question (=reactive behavior), or 2) in order to comply with some higher-level objective (like a human instinct) that would be coded into the system (=pro-active behavior).

The system just seeks to generalize in order to make better predictions. It doesn't plan that the generalizations will create new structure. It has a generalization mechanism, new structure/creativity comes out of it as a side effect. Generating new structure is just something sufficiently complex sets of elements do, see fractals, and chaos.

-Rob