Situational Awareness

[John Clark]

Leopold Aschenbrenner was fired in April from OpenAI for leaking information about the company to the press. He also wrote a paper called "Situational Awareness" about the state of artificial intelligence now and what we can expect from it during the next 3 to 5 years; it's the most brilliant analysis of the subject that I have ever seen.  

Situational Awareness

If Aschenbrenner is correct, and I'm almost certain he is, in about a year every single one of the political issues that today we think are SOOO important and will determine who will be the next president (illegal immigration, global warming, high gas prices, and even excessive wokeness) are going to seem pretty damn trivial in about a year compared with the overwhelming importance of AI.

Aschenbrenner also gave a 4 1/2 hour interview that I wish every politician and business leader in the country would watch. Aschenbrenner is only 21 and he looks like he's 15, but he's obviously smart as hell.

Leopold Aschenbrenner - 2027 AGI, China/US Super-Intelligence Race

 John K Clark    See what's on my new list at  Extropolis

sas

[John Clark]

Sabine Hossenfelder came out with a video attempting to discredit Leopold Aschenbrenner. She failed. 

Is the Intelligence-Explosion Near? A Reality Check

I wrote this in the comment section of the video: 

"You claim that AI development will slow because we will run out of data, but synthetic data is already being used to train AIs and it actually works! AlphaGo was able to go from knowing nothing about the most complicated board game in the world called "GO" to being able to play it at a superhuman level in just a few hours by using synthetic data, it played games against itself. As for power, during the last decade the total power generation of the US has remained flat, but during that same decade the power generation of China has not, in just that same decade China constructed enough new power stations to equal power generated by the entire US. So a radical increase in electrical generation capacity is possible, the only thing that's lacking is the will to do so. When it becomes obvious to everybody that the first country to develop a super intelligent computer will have the capability to rule the world there will be a will to build those power generating facilities as fast as humanly possible. Perhaps they will use natural gas, perhaps they will use nuclear fission."   

  John K Clark    See what's on my new list at  Extropolis

hid

[PGC]

A lot of the excitement around LLMs is due to confusing skill/competence (memory based) with the unsolved problem of intelligence, its most optimal/perfect test etc. There is a difference between completing strings of words/prompts relying on memorization, interpolation, pattern recognition based on training data and actually synthesizing novel generalization through reasoning or synthesizing the appropriate program on the fly. As there isn't a perfect test for intelligence, much less consensus on its definition, you can always brute force some LLM through huge compute and large, highly domain specific training data, to "solve" a set of problems; even highly complex ones. But as soon as there's novelty you'll have to keep doing that. Personally, that doesn't feel like intelligence yet. I'd want to see these abilities combined with the program synthesis ability; without the need for ever vaster, more specific databases etc. to be more convinced that we're genuinely on the threshold.

John, as you enjoyed that podcast with Aschenbrenner, you might find the following one with Chollet interesting. Imho you cannot scale past not having a more advanced approach to program synthesis (which nonetheless could be informed or guided by LLMs to deal with the combinatorial explosion of possible program synthesis).

https://www.youtube.com/watch?v=UakqL6Pj9xo

[John Clark]

On Sun, Jun 16, 2024 at 10:26 PM PGC <multipl...@gmail.com> wrote:

> A lot of the excitement around LLMs is due to confusing skill/competence (memory based) with the unsolved problem of intelligence,

Intelligence was an unsolved problem but not anymore, it was solved about 18 months ago. Certainly if we thought we were dealing with a human and not a machine you and I would say that person was quite intelligent. 

 > There is a difference between completing strings of words

Why? Basically all Einstein did was to complete this string of words "in general the way things behave when they move close to the speed of light and gravity becomes very strong is ...."  

> As there isn't a perfect test for intelligence, much less consensus on its definition,

That is true, nevertheless despite that lack of a definition it has not prevented us from judging that some of the human beings we have dealings with are quite intelligent while others are extremely stupid. How are we able to do that when we have no definition for intelligence? It's possible because we have something much better than a definition, we have examples of intelligent actions. After all, the definitions in a dictionary are made of words and those words are also in the dictionary and they're also made of words. The only thing that gets us out of that infinite Loop is examples, it is the way lexicographers got the knowledge to write their dictionary. 

 

> you can always brute force some LLM through huge compute and large, highly domain specific training data, to "solve" a set of problems;

I don't know what those quotation marks are supposed to mean but if you are able to "solve" a set of problems then the problems have been solved, the method of doing so is irrelevant. Are you sure you're not whistling past the graveyard?  

> you might find the following interview with Chollet interesting
Francois Chollet - LLMs won’t lead to AGI - $1,000,000 Prize to find true solution

I watched the video, thank you for recommending it. I gave credit to Chollet for devising the ARC AI benchmark, and for giving a prize of $500,000 to the first open source developer who makes an AI program that gets a 85% on that benchmark, the average human is supposed to be able to get an 80% on ARC, I'm a little skeptical that the average human could actually get a score that high but never mind. It's true that most large AI programs don't do very well on ARC, but Jack Cole wrote a very small AI program of only 240 million parameters that beat GPT4 on that benchmark despite the fact that GPT 4 has 1.76 trillion parameters, 7300 times larger. And Cole's program was running on  just one P100 processor that is about a 10th as powerful as a H100, and yet Cole's program achieved a score of 34% on ARC, not bad considering the fact that two years ago no program could do better than 0%.  Chollet  says he wouldn't be impressed no matter how high a program scored  on ARC unless he closely examined how the AI was trained and was certain the good results were not a result of mere memorization (whatever that means) and all the questions were 100% novel to it. But no situation is ever 100% novel, if nothing else they all involve the distribution of matter and energy in spacetime. In affect Chollet is saying that if an AI passes a benchmark then there must be something wrong with the benchmark. I think the man is more interested in making excuses than finding the truth in a desperate attempt to preserve the last vestiges of vitalism, the idea that humans and only humans have some sort of magical secret sauce that a mere  machine could never emulate.

Also, Chollet doesn't do a very good job explaining why a machine, which is supposed to be in capable of doing anything novel, nevertheless managed to translate between English and Kalamang because:

"learning to translate between English and Kalamang -- a language with less than 200 speakers and therefore virtually no presence on the web -- using several hundred pages of field linguistics reference materials. This task framing is novel in that it asks a model to learn a language from a single human-readable book of grammar explanations, rather than a large mined corpus of in-domain data"

A Benchmark for Learning to Translate a New Language from One Grammar Book

And I was dumbfounded when Chollet said  OpenAI has held back progress towards AI by 5 to 10 years because for the first time in about 50 years they made something that actually worked and other companies had a hunch that they may just be onto something!

 John K Clark    See what's on my new list at  Extropolis

jep

h

John, as you enjoyed that podcast with Aschenbrenner, you might find the following one with Chollet interesting. Imho you cannot scale past not having a more advanced approach to program synthesis (which nonetheless could be informed or guided by LLMs to deal with the combinatorial explosion of possible program synthesis).

 

[PGC]

On Monday, June 17, 2024 at 7:28:13 PM UTC+2 John Clark wrote:

On Sun, Jun 16, 2024 at 10:26 PM PGC <multipl...@gmail.com> wrote:

 

> you can always brute force some LLM through huge compute and large, highly domain specific training data, to "solve" a set of problems;

I don't know what those quotation marks are supposed to mean but if you are able to "solve" a set of problems then the problems have been solved, the method of doing so is irrelevant. Are you sure you're not whistling past the graveyard?

In discussing the distinction between memorization, which LLMs heavily rely on, and genuine reasoning, which involves building new mental models capable of broad generalizations and depth, consider the following:

Even if I have no prior knowledge of a specific domain, with a large enough library, memory, and pattern recognition of word sequences and probabilistic associations, I could "generate" a solution by merely matching patterns. The more my memory aligns with the problem and domain, the higher the likelihood of "solving" it.

To illustrate, imagine a student unfamiliar with an advanced topic who stumbles upon a book in a library that contains the exact problem and its solution. By copying the solution verbatim, they have effectively cheated. This is akin to undergraduates peeking at each other's exams: they are unable to model the problem and derive a solution themselves but can memorize and reproduce the solution by glancing at others' work. This differs from students who, through understanding the domain's fundamentals, experiment with various approaches and reason their way to a solution. These students might even discover novel solutions, unlike those who merely copy and paste from their peers. Hence the quotation marks; there is no "solving" going on by cheating through memory. 

This analogy extends to the internet, where some people fake expertise by parroting buzzwords and formulations from Wikipedia, in contrast to genuine experts who contribute original insights. As discussions progress and become more complex, these parrots often become lost, unable to keep up with the depth and specificity required. Higher education attempts to address this by rewarding original, effective problem-solving approaches over mere memorization and repetition.

 

[Terren Suydam]

This is a really well articulated distinction, thank you for that. 

I agree that LLMs are not AGI (yet), but it's hard to ignore they're (sometimes astonishingly) competent at answering multi-modal questions across most, if not all domains of human knowledge. I spent a couple hours this morning trying to use chatGPT to design a prompt that might demonstrate that it's not merely parroting, synthesizing, or rearranging existing human ideas, but actually generating novel ones. Here's probably the best result but I'm not sure there's anything actually novel there. Despite that, it's still quite impressive, and to John's point, it's clearly an intelligent response, even if there are aspects of "cheating off of humans" in it. 

It's clear that the line between the genuine reasoning & creativity that are implicit in whatever we think of as human intelligence, vs the permutative repackaging of existing ideas we might think of as inherent in the intelligence exhibited by LLMs, is blurry.  Human creativity and intelligence is probably a lot closer to what LLMs do than we'd like to think. But it's also clear to me that we're not going to get Einsteinian leaps forward in any given domain from LLMs. That may well be coming from AI in the future, but the way I see it, there's still some significant breakthrough(s) necessary to get there.

Terren

[John Clark]

On Tue, Jun 18, 2024 at 11:23 AM Terren Suydam <terren...@gmail.com> wrote:

 

LLMs are not AGI (yet), but it's hard to ignore they're (sometimes astonishingly) competent at answering multi-modal questions across most, if not all domains of human knowledge

I agree.  

 

>  Here's probably the best result but I'm not sure there's anything actually novel there. Despite that, it's still quite impressive, and to John's point, it's clearly an intelligent response, even if there are aspects of "cheating off of humans" in it. 

Concerning the cheating off humans question; Isaac Newton was probably the closest the human race ever got to producing a transcendental genius, and nobody ever accused him of being overly modest, but even Newton admitted that if he had seen further than others it was only because "he stood on the shoulders of giants". Human geniuses don't start from absolute zero, they expand on work done by others. Regardless of how brilliant an AI's answer is, if somebody is bound and determined to belittle the AI they can always find *something* in the training data that has some relationship to the answer, however tenuous. Even if the AI wrote a sonnet more beautiful than anything of Shakespeare's, they can still claim that the sonnet, like everything in literature, concerns objects (and people) and how they move, and there are certainly things in its training data about the arrangement of matter and energy in spacetime, in fact EVERYTHING in its training data is about the arrangement of matter and energy in spacetime. And therefore writing a beautiful sonnet was not a creative act but was just the result of "mere memorization".

John K Clark    See what's on my new list at  Extropolis

r08

 

[Jason Resch]

On Sun, Jun 16, 2024, 10:26 PM PGC <multipl...@gmail.com> wrote:

A lot of the excitement around LLMs is due to confusing skill/competence (memory based) with the unsolved problem of intelligence, its most optimal/perfect test etc. There is a difference between completing strings of words/prompts relying on memorization, interpolation, pattern recognition based on training data and actually synthesizing novel generalization through reasoning or synthesizing the appropriate program on the fly. As there isn't a perfect test for intelligence, much less consensus on its definition, you can always brute force some LLM through huge compute and large, highly domain specific training data, to "solve" a set of problems; even highly complex ones. But as soon as there's novelty you'll have to keep doing that. Personally, that doesn't feel like intelligence yet. I'd want to see these abilities combined with the program synthesis ability; without the need for ever vaster, more specific databases etc. to be more convinced that we're genuinely on the threshold.

I think there is no more to intelligence than patter recognition and extrapolation (essentially, the same techniques required for improving compression). It is also the same thing science is concerned with: compressing observations of the real world into a small set of laws (patterns) which enable predictions. And prediction is the essence of intelligent action, as all goal-centered action requires predicting probable outcomes that may result from any of a set of possible behaviors that may be taken, and then choosing the behavior with the highest expected reward.

I think this can explain why even a problem as seemingly basic as "word prediction" can (when mastered to a sufficient degree) break through into general intelligence. This is because any situation can be described in language, and being asked to predict next words requires understanding the underlying reality to a sufficient degree to accurately model the things those words describe. I confirmed this by describing an elaborate physical setup and asked GPT-4 to predict and explain what it thought would happen over the next hour. It did so perfectly, and also explained the consequences of various alterations I later proposed.

Since any of thousands, or perhaps millions, of patterns exist in the training corpus, language models can come to learn, recognize, and extrapolate all of those thousands or millions of patterns. This is what we think of as generality (a sufficiently large repertoire of pattern recognition that it appears general).

Jason

John, as you enjoyed that podcast with Aschenbrenner, you might find the following one with Chollet interesting. Imho you cannot scale past not having a more advanced approach to program synthesis (which nonetheless could be informed or guided by LLMs to deal with the combinatorial explosion of possible program synthesis).

https://www.youtube.com/watch?v=UakqL6Pj9xo

On Friday, June 14, 2024 at 7:28:50 PM UTC+2 John Clark wrote:

Sabine Hossenfelder came out with a video attempting to discredit Leopold Aschenbrenner. She failed. 

Is the Intelligence-Explosion Near? A Reality Check

I wrote this in the comment section of the video: 

"You claim that AI development will slow because we will run out of data, but synthetic data is already being used to train AIs and it actually works! AlphaGo was able to go from knowing nothing about the most complicated board game in the world called "GO" to being able to play it at a superhuman level in just a few hours by using synthetic data, it played games against itself. As for power, during the last decade the total power generation of the US has remained flat, but during that same decade the power generation of China has not, in just that same decade China constructed enough new power stations to equal power generated by the entire US. So a radical increase in electrical generation capacity is possible, the only thing that's lacking is the will to do so. When it becomes obvious to everybody that the first country to develop a super intelligent computer will have the capability to rule the world there will be a will to build those power generating facilities as fast as humanly possible. Perhaps they will use natural gas, perhaps they will use nuclear fission."   

  John K Clark    See what's on my new list at  Extropolis

hid

--
You received this message because you are subscribed to the Google Groups "Everything List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to everything-li...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/everything-list/1a991958-5828-4405-83b1-5c8a6671dad6n%40googlegroups.com.

[PGC]

I'm hypothesizing here, as the nature of intelligence is still a mystery. Thank you, Terren, for your thoughtful contribution. You aptly highlight the confusion between skill and intelligence. Jason and John could be right; intelligence might emerge from advanced LLMs. The recent achievements are impressive. The differences between models like Gemini and ChatGPT might stem from better data curation rather than compute power.

However, I see LLMs currently more as assistants that help us organize and structure our work more efficiently. Terence Tao isn't talking about replacing mathematicians but about enhancing collaboration and verification. If LLMs were truly intelligent, all jobs, including AI researchers', would soon vanish. But I don't foresee real engineers, AI researchers, or IT departments being replaced in the short to mid-term. There's too much novelty and practical knowledge involved in complex human work that LLMs can't replicate.

Take engineers, for example. Much of their work relies on practical experience and intuition developed over years. LLMs aren't producing groundbreaking results like Ramanujan's infinite series etc; they're more about aiding in tasks like automated theorem proving. Intelligence might just be memory and vast training data, but I believe there's an element of freedom in human reasoning that leads to novel ideas.

Consider Russell's best ideas coming while walking to the coffee machine. This unstructured thinking grants fresh perspectives. Creativity often involves discarding old approaches, a process that presupposes freedom. Machines would need to run long or even endlessly, reasoning in inscrutable code, which is neither practical nor desirable. Or somebody finds something that would bring inference to LLMs to effectively reduce the infinite space of all possible programs for effective synthesis of new programs. Fully deterministic and static programs are not enough to deal with the complex situations we face everyday. There's always some element of novelty that we have to deal with, combining reasoning and memory. 

Ultimately, while everyone appreciates a helpful assistant, few truly seek machines that challenge our understanding or autonomy. That's why I find the way we talk about LLMs and AGI a bit disingenuous. And no this is not a case of setting the bar higher and higher to preserve some kind of notion of human superiority. If all those jobs are replaced in short order, I'll just be wrong empirically speaking, and you can all make fun of these posts and yell "told you so". 

[Terren Suydam]

I would never claim that the works of trascendental geniuses like Newton & Einstein, or for that matter, Picasso & Dali, did not derive from earlier works. What I'm saying that they did, which I doubt very much current LLMs can do, is to break ground into novel territory in whatever territory. I'm not trying to belittle current LLMs, but it seems important to understand their limitations especially because nobody, not even their creators, seems to really understand why they're as good as they are. And just as importantly, why they're as bad as they are at some things given how smart they are in other ways.

Terren

[Terren Suydam]

On Tue, Jun 18, 2024 at 3:24 PM Jason Resch <jason...@gmail.com> wrote:

On Sun, Jun 16, 2024, 10:26 PM PGC <multipl...@gmail.com> wrote:

A lot of the excitement around LLMs is due to confusing skill/competence (memory based) with the unsolved problem of intelligence, its most optimal/perfect test etc. There is a difference between completing strings of words/prompts relying on memorization, interpolation, pattern recognition based on training data and actually synthesizing novel generalization through reasoning or synthesizing the appropriate program on the fly. As there isn't a perfect test for intelligence, much less consensus on its definition, you can always brute force some LLM through huge compute and large, highly domain specific training data, to "solve" a set of problems; even highly complex ones. But as soon as there's novelty you'll have to keep doing that. Personally, that doesn't feel like intelligence yet. I'd want to see these abilities combined with the program synthesis ability; without the need for ever vaster, more specific databases etc. to be more convinced that we're genuinely on the threshold.

I think there is no more to intelligence than patter recognition and extrapolation (essentially, the same techniques required for improving compression). It is also the same thing science is concerned with: compressing observations of the real world into a small set of laws (patterns) which enable predictions. And prediction is the essence of intelligent action, as all goal-centered action requires predicting probable outcomes that may result from any of a set of possible behaviors that may be taken, and then choosing the behavior with the highest expected reward.

I think this can explain why even a problem as seemingly basic as "word prediction" can (when mastered to a sufficient degree) break through into general intelligence. This is because any situation can be described in language, and being asked to predict next words requires understanding the underlying reality to a sufficient degree to accurately model the things those words describe. I confirmed this by describing an elaborate physical setup and asked GPT-4 to predict and explain what it thought would happen over the next hour. It did so perfectly, and also explained the consequences of various alterations I later proposed.

Since any of thousands, or perhaps millions, of patterns exist in the training corpus, language models can come to learn, recognize, and extrapolate all of those thousands or millions of patterns. This is what we think of as generality (a sufficiently large repertoire of pattern recognition that it appears general).

Jason

Hey Jason,

You've articulated this idea before, that the result of the training on such large amounts of data may result in the ability of LLMs to create models of reality and simulate minds and so forth, and it's an intriguing possibility.  However, one fact of how current LLMs operate is that they don't know when they're wrong. If what you're saying is true, shouldn't an LLM be able to model its own state of knowledge?

Terren

[John Clark]

On Wed, Jun 19, 2024 at 8:55 AM PGC <multipl...@gmail.com> wrote:

> I don't foresee real engineers, AI researchers, or IT departments being replaced in the short to mid-term.

I think they will be the first to be replaced, the last to be replaced will be nursing home orderlies.  

> Take engineers, for example. Much of their work relies on practical experience and intuition developed over years.

Before 1997 people said exactly the same thing when they explained why no computer could ever beat the best human chess grandmaster.  

 John K Clark    See what's on my new list at  Extropolis

wzy

[Jason Resch]

There is a very close relationship between compression and prediction. For any signal one tried to compress, the better you can predict it, the better you can compress it.

Take the example of predicting the next letter in text. If you had algorithm B which was 90% accurate, then you would only need to record the errors (which happens 1 character in 10). If another algorithm, algorithm A, was 99% accurate, then you would only need to record the errors that happen 1 time in 1,000.

So the better one can predict, the better one can compress.

But this relationship works both ways: if you can compress very well, you can predict very well.

Consider the ideal compression of a huge text corpus C, consisting of every published book.

There is some smallest program, program X, that when executed outputs this massive training corpus and halts.

In order for X to be the smallest representation possible, this ideal compression of C must take advantage of every pattern that exists in the text, and in every data set contained in C. If, for example, there was a book that included observations of the planetary motion, then the ideal compression in program X must contain the laws of planetary motion that describe (and compress) those measurements.

The ideal compression of C, would even include laws of physics that remain unknown to today's physicists and theorists, so long as the data necessary to account for observations are recorded in some book within the corpus. So, if for example, the corpus only had books up to 1904, before Einstein's relativity, but you had books describing the observations that could only best be accounted for by relativity, then the ideal compression of C must include Einstein's theory of relativity.

Now for the interesting part:

It is possible to discover program X by brute force with a simple program: execute all programs shorter than C, and find the shortest one that outputs C. That process would find a program that has ideal models of our physical reality, including laws not yet discovered.

True: it is not computationally feasible to do the brute force search in this way, but there are heuristics we can use for finding better ways of compressing datasets that we have. In fact this is what I see ourselves doing with training models to be able to more accurately predict text (that is making them better at compressing text) which is the same thing as making them better understand the processes (the universe and the human brain (that operates within and observes that universe) to writes those works) that underlying them.

As to the limitations of LLMs, they have a finite and fixed depth. This means they are only capable of computing functions they can complete within that fixed time (unless you argument them with a loop and memory). This is like considering the limits of a human brain that was.onky given, say, 10 seconds to solve any problem. This is why it fails at multiplying long numbers, which we might consider easy for a computer, but if you have a fixed-depth circuit, there are only so many times you can shift and add, and thus only so big a multiplicand you can handle.

Jason 

Terren

--
You received this message because you are subscribed to the Google Groups "Everything List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to everything-li...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/everything-list/CAMy3ZA99toTiJ9YahG-8BwJhWO%3DJ72u6oy2HFQuOa0ROo92VDg%40mail.gmail.com.

[Jason Resch]

Just the other day (on another list), I proposed that the problem "hallucination" is not really a bug, but rather, it is what we have designed LLMs to do (when we consider the training regime we subject them to). We train these models to produce the most probable extrapolations of text given some sample.

Now consider if you were placed in a box and rewarded or punished based on how accurately you guessed the next character in a sequence.

You are given the following sentence and asked to guess the next character:

"Albert Einstein was born on March, "

True, you could break the fourth wall and protest "But I don't know! Let me out of here!"

But that would only lead to your certain punishment. Or: you could take a guess, there's a decent chance the first digit is a 1 or 2. You might guess one of those and have at least a 1/3 chance of getting it right.

This is how we have trained the current crop of LLMs. We don't reward them for telling us they don't know, we reward them for having the highest accuracy possible in making educated guesses.

We can develop more elaborate training processes that punish wrong answers and reward statements that they are unsure, don't know and are making an educated guess, but that would be something other than a pure decoder model.

Another issue is a model doesn't know what it says until after it says it, so some secondary step may still be needed to have it checked it's own answer for consistency before outputting it.

Jason 

[John Clark]

On Wed, Jun 19, 2024 at 12:33 PM Jason Resch <jason...@gmail.com> wrote:

> Just the other day (on another list), I proposed that the problem "hallucination" is not really a bug, but rather, it is what we have designed LLMs to do (when we consider the training regime we subject them to). We train these models to produce the most probable extrapolations of text given some sample. Now consider if you were placed in a box and rewarded or punished based on how accurately you guessed the next character in a sequence.

You are given the following sentence and asked to guess the next character:

"Albert Einstein was born on March, "

True, you could break the fourth wall and protest "But I don't know! Let me out of here!"

But that would only lead to your certain punishment. Or: you could take a guess, there's a decent chance the first digit is a 1 or 2. You might guess one of those and have at least a 1/3 chance of getting it right.

This is how we have trained the current crop of LLMs. We don't reward them for telling us they don't know, we reward them for having the highest accuracy possible in making educated guesses.

Damn, I wish I'd said that! Very clever.   

 John K Clark    See what's on my new list at  Extropolis

mze

 

[Jason Resch]

Thank you! Feel welcome to use it. :-)

Jason 

 John K Clark    See what's on my new list at  Extropolis

mze

 

--

You received this message because you are subscribed to the Google Groups "Everything List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to everything-li...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/everything-list/CAJPayv0%2BMkb0y%2B1fY-AryeOC6C-FwuR%2B654Ua_EjMb_%3D6CGNCQ%40mail.gmail.com.

[John Clark]

On Wed, Jun 19, 2024 at 1:50 PM Jason Resch <jason...@gmail.com> wrote:

> Thank you! Feel welcome to use it. :-)

I certainly will!  

John K Clark    See what's on my new list at  Extropolis

oqq

[Brent Meeker]

You can always add some randomness to a computer program.  LLM's aren't deterministic now.  Human intelligence may very well be memory plus randomness, although I'd bet on the inclusion of some inference algorithms.  The randomness doesn't even have to be in the brain.  People interact with their environment which provides a lot of effective randomness plus some relevant prompts.

Brent

To view this discussion on the web visit https://groups.google.com/d/msgid/everything-list/a3ecb0a7-7a3f-417b-bb47-f449febb73e7n%40googlegroups.com.

[Jason Resch]

On Wed, Jun 19, 2024 at 6:05 PM Brent Meeker <meeke...@gmail.com> wrote:

You can always add some randomness to a computer program.  LLM's aren't deterministic now.  Human intelligence may very well be memory plus randomness, although I'd bet on the inclusion of some inference algorithms.  The randomness doesn't even have to be in the brain.  People interact with their environment which provides a lot of effective randomness plus some relevant prompts.

Yes, I think there is no great mystery to creativity. It requires only 1. random permutation/combination, and 2. an evaluation function: how much better is this new thing compared to the previous thing? This is the driver behind all the innovation in biology produced by natural selection. And this same mechanism is replicated in the technique of "genetic programming." Koza, who invented genetic programming, used it to create his "invention machine" which has created patent-worthy improvements across multiple domains of technology.

I use genetic programming to evolve bots, and in only a few generations, they move from stumbling around at random, to deriving unique, environment-specific strategies to maximize their ability to feed themselves while avoiding obstacles:

https://www.youtube.com/watch?v=InBsqlWQTts&list=PLq_mdJjNRPT11IF4NFyLcIWJ1C0Z3hTAX&index=2

There is no intelligence imparted to the design of the bots. They evolve purely based on random variation of traits of the top performers (as evaluated based on how much they ate during their life).

Jason

 

To view this discussion on the web visit https://groups.google.com/d/msgid/everything-list/04fffd28-1a61-48a3-8a5e-d1af5b901caa%40gmail.com.

[PGC]

On Thursday, June 20, 2024 at 4:13:25 AM UTC+2 Jason Resch wrote:

On Wed, Jun 19, 2024 at 6:05 PM Brent Meeker <meeke...@gmail.com> wrote:

You can always add some randomness to a computer program.  LLM's aren't deterministic now.  Human intelligence may very well be memory plus randomness, although I'd bet on the inclusion of some inference algorithms.  The randomness doesn't even have to be in the brain.  People interact with their environment which provides a lot of effective randomness plus some relevant prompts.

Yes, I think there is no great mystery to creativity. It requires only 1. random permutation/combination, and 2. an evaluation function: how much better is this new thing compared to the previous thing? This is the driver behind all the innovation in biology produced by natural selection. And this same mechanism is replicated in the technique of "genetic programming." Koza, who invented genetic programming, used it to create his "invention machine" which has created patent-worthy improvements across multiple domains of technology.

I use genetic programming to evolve bots, and in only a few generations, they move from stumbling around at random, to deriving unique, environment-specific strategies to maximize their ability to feed themselves while avoiding obstacles:

https://www.youtube.com/watch?v=InBsqlWQTts&list=PLq_mdJjNRPT11IF4NFyLcIWJ1C0Z3hTAX&index=2

There is no intelligence imparted to the design of the bots. They evolve purely based on random variation of traits of the top performers (as evaluated based on how much they ate during their life).

Your addition about randomness is interesting. It’s true that LLMs incorporate some degree of randomness, and human intelligence might also be influenced by randomness and inference algorithms. The interaction with our environment introduces effective randomness contributing to our decision-making processes. The notion that creativity stems from random permutation/combination and an evaluation function resonates with the principles of natural selection and genetic programming. The example of genetic programming evolving bots to optimize their behavior through random variation and evaluation showcases this mechanism effectively.

However, we should differentiate between speculation and facts in your statements. While randomness and evaluation are essential components of genetic programming, the assertion that there is "no great mystery to creativity" oversimplifies: what you're bringing up is a kind of creativity, which is constrained by its iterative limitations. A change here, a small new feature there... it's clear that this is creativity on a budget, making only the smallest adaptations necessary for survival instead of yielding radically new designs from the ground up. The kind that is found and most sought after in boundary-breaking science and/or art, even if everybody stands on shoulders: not every PhD has a Newtonian impact on the world.

Randomness + evaluation = creativity looks rhetorically simple and clear. However, there are two problems I see:

1. Who/What is Evaluating? Evaluation can be completely deterministic and mechanical, it can be effective on levels like natural selection, or it can result from a subject with intuition, experience, and a refined sense of taste or a more rudimentary one. It can involve a particular psychology, some world or even multiverse-based ontology to embed said subject, and more. The questions raised encompass our entire history and all qualia, if not more. Therefore, evaluation is not as simple or clear as that seemingly factual statement suggests. "Evaluation," as you sketch out rather unclearly, merely hides the problem of subject and reality for a rhetorical mirage of clarity.

2. Oversimplification of Creativity: By all means, build the creativity machine, order the randomness and evaluation in bottles from Amazon, and win every prize from science to the arts by cranking it up to 11. But this oversimplification doesn't capture the full depth of human creativity, which involves more than just random variations and evaluations. It involves cognitive processes we have difficulty describing, emotional influences, and the ability to synthesize disparate ideas into something more original on the novelty spectrum.

Ultimately, while LLMs and AI can significantly augment our capabilities, they remain, for now, advanced assistants rather than autonomous intelligences capable of independent breakthroughs. The future may bring further integration and enhancement, but the unique qualities of human intelligence—our ability to synthesize thought, exercise creativity, and approach problems from unstructured perspectives with imperfect information to name just a few aspects —are not yet replicable by anything people have built.

I'm sure Quentin, Telmo, and Russell are reading this and shaking their heads. But they have probably been fired and replaced by LLMs much smarter than them. This should provide additional motivation to build that machine though, Jason. They need our support. Then again, the way we/people behave in the world... it's best we don't develop that, IF it is possible in the first place.  

I'm not saying we won't see fascinating developments. The threshold for me is overwhelming evidence that something can independently formulate and learn to solve problems effectively with a notable degree of originality in unspecified environments on problems it hasn't been trained on. Synthetic data or not. Superintelligence is more like the thing that can spit out 3000 years worth of mathematical/scientific discoveries in a second. The problem with this, presupposing optimistically and irrationally that it is possible, is that I'm not sure we would understand it. 

 

[Jason Resch]

On Fri, Jun 21, 2024, 8:48 AM PGC <multipl...@gmail.com> wrote:

On Thursday, June 20, 2024 at 4:13:25 AM UTC+2 Jason Resch wrote:

On Wed, Jun 19, 2024 at 6:05 PM Brent Meeker <meeke...@gmail.com> wrote:

You can always add some randomness to a computer program.  LLM's aren't deterministic now.  Human intelligence may very well be memory plus randomness, although I'd bet on the inclusion of some inference algorithms.  The randomness doesn't even have to be in the brain.  People interact with their environment which provides a lot of effective randomness plus some relevant prompts.

Yes, I think there is no great mystery to creativity. It requires only 1. random permutation/combination, and 2. an evaluation function: how much better is this new thing compared to the previous thing? This is the driver behind all the innovation in biology produced by natural selection. And this same mechanism is replicated in the technique of "genetic programming." Koza, who invented genetic programming, used it to create his "invention machine" which has created patent-worthy improvements across multiple domains of technology.

I use genetic programming to evolve bots, and in only a few generations, they move from stumbling around at random, to deriving unique, environment-specific strategies to maximize their ability to feed themselves while avoiding obstacles:

https://www.youtube.com/watch?v=InBsqlWQTts&list=PLq_mdJjNRPT11IF4NFyLcIWJ1C0Z3hTAX&index=2

There is no intelligence imparted to the design of the bots. They evolve purely based on random variation of traits of the top performers (as evaluated based on how much they ate during their life).

Your addition about randomness is interesting. It’s true that LLMs incorporate some degree of randomness, and human intelligence might also be influenced by randomness and inference algorithms. The interaction with our environment introduces effective randomness contributing to our decision-making processes. The notion that creativity stems from random permutation/combination and an evaluation function resonates with the principles of natural selection and genetic programming. The example of genetic programming evolving bots to optimize their behavior through random variation and evaluation showcases this mechanism effectively.

However, we should differentiate between speculation and facts in your statements. While randomness and evaluation are essential components of genetic programming, the assertion that there is "no great mystery to creativity" oversimplifies: what you're bringing up is a kind of creativity, which is constrained by its iterative limitations. A change here, a small new feature there... it's clear that this is creativity on a budget, making only the smallest adaptations necessary for survival instead of yielding radically new designs from the ground up. The kind that is found and most sought after in boundary-breaking science and/or art, even if everybody stands on shoulders: not every PhD has a Newtonian impact on the world.

Randomness + evaluation = creativity looks rhetorically simple and clear. However, there are two problems I see:

1. Who/What is Evaluating? Evaluation can be completely deterministic and mechanical, it can be effective on levels like natural selection, or it can result from a subject with intuition, experience, and a refined sense of taste or a more rudimentary one. It can involve a particular psychology, some world or even multiverse-based ontology to embed said subject, and more. The questions raised encompass our entire history and all qualia, if not more. Therefore, evaluation is not as simple or clear as that seemingly factual statement suggests. "Evaluation," as you sketch out rather unclearly, merely hides the problem of subject and reality for a rhetorical mirage of clarity.

Evaluation functions can be arbitrarily complex. It could be the aesthetic sense of an artist, or a a mathematical function devised by an engineer to evaluate a jet engine's weight and efficiency.

People have studied creativity in humans and found that it consists of two parts, as it does in genetic programming:

There is the open ended ideation where the brain comes up with as many ideas as possible, without concern for their practicality or feasibility. This part is called "divergent thinking" human children are often rated at genius levels compared to adults in this domain ( https://twentyonetoys.com/blogs/teaching-21st-century-skills/creative-genius-divergent-thinking ).

Then there is a phase of"convergent thinking" where the set of ideas is evaluated and narrowed down based on concerns of practicality, cost, efficiency, marketability, aesthetic properties, etc.

So while it may seem reductive to say creativity is merely permutation and evaluation, the studies of creativity in humans suggest it is essentially the same (divergent thinking) + (convergent thinking) -- which is generating a large number of possibilities, followed by evaluating those possibilities to select the best one(s).

If anything else is required, I don't know what else ot would be. This seems sufficient to me.

2. Oversimplification of Creativity: By all means, build the creativity machine, order the randomness and evaluation in bottles from Amazon, and win every prize from science to the arts by cranking it up to 11. But this oversimplification doesn't capture the full depth of human creativity, which involves more than just random variations and evaluations. It involves cognitive processes we have difficulty describing, emotional influences, and the ability to synthesize disparate ideas into something more original on the novelty spectrum.

The functions of random permutation and combination is more sophisticated than a die role. It often involves combining aspects of different ideas one has been exposed to  (evolution stumbled on this with sexual reproduction). Or eliminating constraints (which allows an increase of flexibility in divergent thinking).

Ultimately, while LLMs and AI can significantly augment our capabilities, they remain, for now, advanced assistants rather than autonomous intelligences capable of independent breakthroughs. The future may bring further integration and enhancement, but the unique qualities of human intelligence—our ability to synthesize thought, exercise creativity, and approach problems from unstructured perspectives with imperfect information to name just a few aspects —are not yet replicable by anything people have built.

The test question used in evaluating human creativity is often something of the form: "come up with as many possible uses of a paperclip that you can think of". LLMs score in the top  percentile compared to humans in this domain.

Existing LLMs have come up with novel mathematical proofs, which I would consider a breakthrough of a certain kind.

LLMs may be at the level of high schoolers today, but they were preschoolers a few years ago. If so n the next few years we build models  at the level of PhD engineers, perhaps it will be common for them to have breakthroughs of the kind you describe.

I'm sure Quentin, Telmo, and Russell are reading this and shaking their heads. But they have probably been fired and replaced by LLMs much smarter than them. This should provide additional motivation to build that machine though, Jason. They need our support. Then again, the way we/people behave in the world... it's best we don't develop that, IF it is possible in the first place.  

I'm not saying we won't see fascinating developments. The threshold for me is overwhelming evidence that something can independently formulate and learn to solve problems effectively with a notable degree of originality in unspecified environments on problems it hasn't been trained on. Synthetic data or not. Superintelligence is more like the thing that can spit out 3000 years worth of mathematical/scientific discoveries in a second. The problem with this, presupposing optimistically and irrationally that it is possible, is that I'm not sure we would understand it. 

Quite true.

What really scared and impressed me was this paper evaluating the abilities of an unreleased GPT-4:

https://arxiv.org/abs/2303.12712

Jason