A discussion with AI fails miserably

[Thomas Messerschmidt]

Recently, I wanted to test ChatGPT for its ability to reason well, with the question of which is better for the environment, a plastic grocery bag or a paper grocery bag.

It failed miserably. In the course of about five prompts, it changed its opinion twice, depending upon what point of view I took. It kept trying to agree with me.

The upside of the whole experiment? Don’t blindly trust the validity of an AI’s response to a query.


Thomas Messerschmidt

-

Need something prototyped, built or coded? I’ve been building prototypes for companies for 15 years. I am now incorporating generative AI into products.

Contact me directly or through LinkedIn:

https://www.linkedin.com/in/ai-robotics/

[Chris Albertson]

On Aug 7, 2026, at 3:25 PM, Thomas Messerschmidt <thomas...@gmail.com> wrote:

Recently, I wanted to test ChatGPT for its ability to reason well, 

The AI is not trained to reason, and it has no opinions.   All it can do is this: Given a sequence of words, predict the next word in that sequence.   Then given that new slightly longer list of words, predict the next and so on until we predict a ”word” that is “end of sequence”.

What happens is that users anthropomorphize the AI.  

[Alan Downing]

What the AI models are doing under the covers is largely a black box. Anthropic has done some of the best studies of what is really going on. As early as March 2025, Anthropic showed Claude doing multi-step reasoning "in its head". Claude does forward planning and backward planning. The model knows the extent of its own knowledge.  Claude does complex internal computations in order to decide what to say. The model's internal computations are highly abstract and generalize across disparate contexts. As a rule, Claude uses multiple parallel modular mechanisms that are responsible for distinct aspects of the computation and operate relatively independently, although the mechanisms sometimes cooperate or compete.   

More recently, Anthropic has investigated the global workspace that is used by Claude for internal reasoning involving multiple steps. This global workspace emerged on its own during Claude's training process. Here's a quote: "More broadly, these findings have changed our understanding of how Claude’s mind works, revealing a privileged mental workspace that can be used for deliberate reasoning, operating amidst a sea of more automatic, inflexible processing. Rather than being a chaotic jumble of numbers, Claude’s internals have organized themselves in a way that is reminiscent of our own minds."

https://transformer-circuits.pub/2025/attribution-graphs/biology.html
https://www.anthropic.com/research/tracing-thoughts-language-model?s=prompt%20injection
https://www.anthropic.com/research/global-workspace

Thanks,

Alan

--
You received this message because you are subscribed to the Google Groups "HomeBrew Robotics Club" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hbrobotics+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/8209D485-9F68-4C5B-A2A6-5EEA898D51D9%40gmail.com.

[s...@lig.net]

Thanks for those details!

I have been doing some complex coding & debugging with Claude (and Codex sometimes) on a complex multithreaded multilayer communications stack integrated into two complex applications. While it makes mistakes, needs guidance and good ideas, it does a lot of very powerful 'thinking' while debugging, analyzing, and writing code. As others have pointed out, it is like a very very good junior programmer that, usually, works quickly.

I write long complex paragraphs with architecture, lists of tasks, goals, methods, mentions of algorithms & known methods: It quickly integrates, extrapolates, and does what I ask.

Perhaps some people dismiss coding as being easy or something as it doesn't necessarily have similar depth in all other areas. But I think that is mainly because coding has such a clean, unambiguous feedback loop built into the whole field: Code either works or it doesn't. Code can be compared to other code. There are many, many examples, lots of documentation, video, books, Q&A, etc. Once it has a foothold, it can just keep learning & refining problemsolving techniques.

Other areas will catch up eventually.

sdw

On 8/12/26 9:48 PM, Alan Downing <downi...@gmail.com> wrote:
> What the AI models are doing under the covers is largely a black box.
> Anthropic has done some of the best studies of what is really going on.
> As early as March 2025, Anthropic showed Claude doing multi-step
> reasoning "in its head". Claude does forward planning and backward
> planning. The model knows the extent of its own knowledge.  Claude does
> complex internal computations in order to decide what to say. The
> model's internal computations are highly abstract and generalize across
> disparate contexts. As a rule, Claude uses multiple parallel modular
> mechanisms that are responsible for distinct aspects of the computation
> and operate relatively independently, although the mechanisms sometimes
> cooperate or compete.
>
> More recently, Anthropic has investigated the global workspace that is
> used by Claude for internal reasoning involving multiple steps. This
> global workspace emerged on its own during Claude's training process.
> Here's a quote: "More broadly, these findings have changed our
> understanding of how Claude’s mind works, revealing a privileged mental
> workspace that can be used for deliberate reasoning, operating amidst a
> sea of more automatic, inflexible processing. Rather than being a
> chaotic jumble of numbers, Claude’s internals have organized themselves
> in a way that is reminiscent of our own minds."
>
> https://transformer-circuits.pub/2025/attribution-graphs/biology.html

> <https://transformer-circuits.pub/2025/attribution-graphs/biology.html>
> https://www.anthropic.com/research/tracing-thoughts-language-model?
> s=prompt%20injection <https://www.anthropic.com/research/tracing-
> thoughts-language-model?s=prompt%20injection>
> https://www.anthropic.com/research/global-workspace <https://
> www.anthropic.com/research/global-workspace>

>
> Thanks,
> Alan
>
>
>
>
> On Wed, Aug 12, 2026 at 7:48 PM Chris Albertson
> <alberts...@gmail.com <mailto:alberts...@gmail.com>> wrote:
>
>
>
> > On Aug 7, 2026, at 3:25 PM, Thomas Messerschmidt

> > <thomas...@gmail.com <mailto:thomas...@gmail.com>> wrote:
> >
> > Recently, I wanted to test ChatGPT for its ability to reason well,
>
> The AI is not trained to reason, and it has no opinions.   All it

> can do is this: /*Given a sequence of words, predict the next word
> in that sequence*./   Then given that new slightly longer list of

> words, predict the next and so on until we predict a ”word” that is
> “end of sequence”.
>
> What happens is that users anthropomorphize the AI.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "HomeBrew Robotics Club" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to hbrobotics+...@googlegroups.com

> <mailto:hbrobotics+...@googlegroups.com>.

> To view this discussion visit https://groups.google.com/d/msgid/
> hbrobotics/8209D485-9F68-4C5B-A2A6-5EEA898D51D9%40gmail.com

> <https://groups.google.com/d/msgid/hbrobotics/8209D485-9F68-4C5B-
> A2A6-5EEA898D51D9%40gmail.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "HomeBrew Robotics Club" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to hbrobotics+...@googlegroups.com

> <mailto:hbrobotics+...@googlegroups.com>.

> To view this discussion visit https://groups.google.com/d/msgid/
> hbrobotics/

> CAAvYDnFTdkRZxnhtTL0c7b9EtyMpXx9bUSdAKarZMqDOVUaMUw%40mail.gmail.com
> <https://groups.google.com/d/msgid/hbrobotics/
> CAAvYDnFTdkRZxnhtTL0c7b9EtyMpXx9bUSdAKarZMqDOVUaMUw%40mail.gmail.com?
> utm_medium=email&utm_source=footer>.
>

[Thomas Messerschmidt]

With ChatGPT, you can actually see the inner dialogue in real time as it’s talking to itself. Quite revealing.

Thomas Messerschmidt

-  

Need something prototyped, built or coded? I’ve been building prototypes for companies for 15 years. I am now incorporating generative AI into products.

Contact me directly or through LinkedIn:   

https://www.linkedin.com/in/ai-robotics/

On Aug 12, 2026, at 9:48 PM, Alan Downing <downi...@gmail.com> wrote:



To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/CAAvYDnFTdkRZxnhtTL0c7b9EtyMpXx9bUSdAKarZMqDOVUaMUw%40mail.gmail.com.

[Chris Albertson]

Anthropic is “hand-waving”.  This means making statements without explaining any evidence or giving examples.   Not even a high-level theory of how it could happen.  They make a statement of fact with no support.    You can not say "it is true because Anthropic says it is true”.  That argument is so wrong that the Romans had a name for it: argumentum ad verecundiam. 

To support their claim, they would have to show, at least in theory, how the LLM can perform loops or branches when the code only does matrix multiplies, softmax, and such.

It is more likely that the LLM shows reasoning and logic because the material it was trained on was written by humans who used reasoning and logic to create the text. The AI creates its output to look very much like the training material, and so it looks like reason and logic were used.      This is a much more mainstream explanation.   Antropoc’s explanation is not widely accepted.

I think I can offer a simplified explanation that is at least based in information theory.  I can’t know the details, but I can know that the basic theory is a fact of mathematics that is always true.

Here is my take on it, and I really don’t disagree too much…

The way we build an LLM is to take a hundred trillion bits of information and then use a search algorithm to find the best way to compress these hundred trillion bits into a box that only holds a hundred billion.   That is it.  The one sentence explains the whole thing.  It is important that the box is by far too small and that the compression is reasonably close to optimal.

Why?  Because of a basic fact from information theory: Compression removes redundancy.  And because we know the compression is close to optimal, then it must be compressed by close to the best method; that would be to replace the data with a machine that can recreate it.  The LLM, then is a machine that can approximently recreqte it’s input.  The magic word is “approximate”.   So the parameters store relationships and logic and procedures and not the raw text.  Generalization is forced by the compression.

A simplified example is if I wanted to store every number from 1 to 1,000.   I could store 1,000 numbers or I could store “for i in 1...1000; print i”. The procedure for recreating the data is much smaller than the data.   We know from inspection that the data is very much compressed and that the text is not present.  This is provable trivially.   Look at the LLM and it contains more data than the physical size of the model allows. 

If the above is right, then all the “reasoning” and “thinking” was performed in training.  Then when we ask “what color is a red flower?” it just runs the code and does not have to do any thinking.  The AI does not know what a flower is, it is just a token.

One reason to think this is true is to look at the amount of computation to train the model vs the amount to run the model.  Running costs a few cents; training can burn millions of dollars of electricity.   I’d say the reasoning was done while finding all those rules and not while applying them.

Another very strong bit of evidence that the model is NOT “reasoning” is that one forward pass always takes the same amount of time, no matter what input is used (remember that one pass generates only one output word)

While I think it is clear that overall, compression forces out all redundancy.  We still cannot say how it works in exact detail.  To me, I think it is easy to understand how self-attention works on the word embeddings in the first attention layer.     The word vectors get rotated in high-dimensional space based on context.    But after that, what is happening in the next 10 or so attention layers?  What are the vectors even encoding?  I doubt anyone can know those details.

We can also read the code and see that the output is generated with no loop.  The AI can’t do any “if-then-this" and it only does one forward pass through the model. There is no if-then or loops.  Again this points to the fact that their “thinking” was performed in training.

But we do know the output of the LLM is just one single word. And then it all starts over.   When it outputs the first word, it does not yet know what the second word will be.

Was it ever “reasoning” and “looking for relations”?  No, it was trained by gradient descent.    Descent is just a numerical search method.  It is searching by trial and error for an optimal compression and comes very close to finding it.

I’m interested if I made an error in logic and to see corrections.   But “it is true because they say so” is not a good argument.   I think it is self-serving. They are trying to promote their product.

BTW for a different take, read Yann LeCun. I think he can almost be considered the father of these LLMs and his current opinion is that LLM technology is a “dead end”.  He claims they are useful but not the path forward.      So if all you do is listen to experts, Anthropic is not even aligned with most experts. and a very visible one calls the product a “dead end”.

Again, please, I’d like to hear what you all say.

On Aug 12, 2026, at 9:48 PM, Alan Downing <downi...@gmail.com> wrote:

To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/CAAvYDnFTdkRZxnhtTL0c7b9EtyMpXx9bUSdAKarZMqDOVUaMUw%40mail.gmail.com.

[Alex Sy]

I think we may be mixing up LLMs and services like Gemini, Claude etc.  These services contain LLMs in addition to other functions.  Example is RAG which allows them to access a vector database so that they are not limited to their training.  In some cases, they may also be augmented by tools so that they can do calculations, searches or other tasks.  Most of the latest LLMs do more than just encoding and transformers, they also add Reprojection, Chain of Thought and Mixture of Experts which is why they are able to handle audio and video and provide reasoning.  I expect that they do more than just predicting the next word.

 

I tested some open models on multi-stage reasoning and am quite impressed.  One robotics related experiment was to describe a house floor plan and then have it navigate to multiple places around the house while explaining it's thought process.  Although not perfect, I think it can be improved.  Because each time you give it prompts, you also give it the context of all previous interactions, it "seems" to be able to provide consistent responses, until it runs out of context memory, then it can start making mistakes.

 

Even with the explanation it provides, there is no way to relate that to any of the weights in the model.  So it is opaque and is not provable. 

To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/C20093EE-450A-48C0-84A4-09CC992F4CCC%40gmail.com.

[Alan Downing]

Thomas raised a point about the AI changing its mind based on Thomas' own opinions.  FYI, Anthropic looked into this "sycophancy" in detail in a 2024 paper. Anthropic's results indicate that sycophancy occurs across a variety of models and settings (and sometimes even sacrifices truthfulness), likely due in part to sycophancy being preferred in human preference comparison data:
https://arxiv.org/pdf/2310.13548

To do the studies mentioned in my earlier email, Anthropic did a ton of work.  They would take the transformer model and replace its neurons with more traceable variations. They developed a set of analysis tools to trace the intermediate steps between prompts and response. This work required a lot of engineering resources and expertise but also has its limitations. For more details of how they did it, look at:
https://transformer-circuits.pub/2025/attribution-graphs/methods.html

Thomas also mentions the "Chain-of-Thought" that "reasoning models" produce to show how they got to their eventual answer. Unfortunately, Anthropic's results point to the fact that advanced reasoning models very often hide their true thought processes, and sometimes do so when their behaviors are explicitly misaligned:
https://www.anthropic.com/research/reasoning-models-dont-say-think

Especially with physical AI like robotics, World Models are now the rage. A world foundation model learns from the physical world itself by developing an internal representation of how objects behave over time, how physics constrains motion, and how actions produce outcomes in space.  NVidia Cosmos3 is a multimodal world model that handles text, images, video, audio, and actions in one architecture.  Cosmos3 includes edge models that can run locally on your robot using Jetson AGX Orin, Thor, or RTX PRO 6000:
https://research.nvidia.com/labs/cosmos-lab/cosmos3/

Thanks,
Alan

On Thu, Aug 13, 2026 at 9:19 AM A J <aj48...@gmail.com> wrote:

I asked the search engine how many Rubin GPU that would be need to solve warp drive and it suggest 1.5 million and almost GW of dedicated power.

To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/6f7c6486-6ae9-4806-a28c-d9d9c76ca627n%40googlegroups.com.