LLM and robotics

[Alex Shkotin]

Colleagues,

Today we can be calm about the application of LLM in the field of robotics:

“Although difficult to deploy to real-world systems today due to latency, context size limitations, and computational cost, the approach of using LLM to drive low-level control can provide exciting insight into how patterns between words can be transferred to actions.” [1]

But there are attempts.

Alex

[1] Large Language Models as General Pattern Machines https://arxiv.org/abs/2307.04721

[John F Sowa]

Alex,

I am not attacking LLMs for what they do best:  relating languages to languages, formal or informal.

But I keep telling people that natural languages are not precision instruments.  You can use ordinary language to tell a robot to do something vague, such as "Please bring me a sandwich."

Since you can't use ordinary language to specify spatial positions and actions with high precision and detail, you have to be very careful to distinguish what you mean by controlling a robot.

For example, somebody may have used precise methods to specify a sandwich-making procedure.   In that case, the LLMs would just match your vague word 'sandwich' to a predefined subroutine that does not use LLMs.

That article you cited describes a useful method for learning patterns.  But the patterns it learns don't deal with the issues above.  It doesn't go from a vague command in ordinary language to the methods that are necessary to carry out the procedures at the detailed level.

John

 

---

From: "Alex Shkotin" <alex.s...@gmail.com>

--

All contributions to this forum are covered by an open-source license.

For information about the wiki, the license, and how to subscribe or

unsubscribe to the forum, see http://ontologforum.org/info/

---

You received this message because you are subscribed to the Google Groups "ontolog-forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ontolog-foru...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/CAFxxRORsG46OyLgt7RNUPKekZQFhqZ46w-Go1_c_bKi7QhYsWA%40mail.gmail.com.

[Alex Shkotin]

John,

It will be interesting to see what types of GenAI or any other AI are used in robotics.

Alex

пн, 27 нояб. 2023 г. в 23:18, John F Sowa <so...@bestweb.net>:

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/47a1c6f5f2cc4f02bc6e7ba8b2edd7a1%40bestweb.net.

[Alex Shkotin]

IN ADDITION: but this topic is overwhelming https://www.v7labs.com/blog/ai-in-robotics

вт, 28 нояб. 2023 г. в 10:35, Alex Shkotin <alex.s...@gmail.com>:

[John F Sowa]

Alex,

I agree that robotics includes a very important range of applications.  The most important applications of LLMs include the ability to talk (or type)  in natural languages to control and communicate with robots and other kinds of systems.  And the same kinds of communications will be used for any and every kind of machinery that is stationary, moving, or flying -- as part of any kind of device used on earth or in space. 

I'm saying that in order to show that I am an enthusiastic supporter of well designed and safely controlled applications of  LLMs.  But I also want to emphasize that the second L in LLM is "Language".  That is good for the user interface.  But it implies a limitation to what can be expressed in ordinary natural languages.

Alex> this topic is overwhelming https://www.v7labs.com/blog/ai-in-robotics

That is an impressive range of valuable applications.  But note that NONE of them use LLMs.  Please reread the article to see what they do and how they do it.  LLMs may be useful to simplify the user interface, but they won't replace the AI methods that are currently being used.

I would also like to cite another very impressive truly graphic application for which language-based LLM would be totally useless (except perhaps for the user interface):  "Towards Garment Sewing Pattern Reconstruction from a Single Image",  https://arxiv.org/pdf/2311.04218v1.pdf

 

Note that the operations are multidimensional spatial transformations.  Sewing patterns are two-dimensional DIAGRAMS that humans or machines use to cut cloth that is used in constructing a garment.  And they are mapped to and from three dimensional structures (a human body and the clothing on it).  This is an extremely time-consuming process that humans perform, and words (by humans or LLMs) are useless for specifying how to perform the graphic transformations.

Sewing patterns are just one of an immense range of applications in every branch of construction from houses to bridges to airplanes to space travel and operations by robots along the way.  LLMs are hopelessly limited by their dependence on what can be expressed in language.  They won't replace the AI in the article you cited, and they would be useless for the AI used to derive sewing patterns -- or many, many other kinds of graphic transformations, stationary or moving.

These issues, by the way, are the topic of the article I'm writing about diagrammatic reasoning by people.  The most complicated part is the step from action and perception to diagrams.  That article about sewing patterns is  an example of the kinds of transformations that the human brain does every second.  Those transformations, which Peirce called phaneroscopy, are a prerequisite for language.  Most of them are performed in the cerebellum, which is the high performance graphic processing unit (GPU) of the human brain.

Some people claim that they are never consciously aware of thinking in images.  That is true because everything in the human GPU  (cerebellum) is outside of the cerebral cortex.  When people are walking and talking on their cell phones, the cerebellum is in total control -- until they step off the curb and get hit by a bus.

John

PS:  It's true that anything expressed in mathematics or computer systems can be translated to a natural language.  But the result of writing out what each machine instruction does would be overwhelming.   Nobody would do that.

---

From: "Alex Shkotin" <alex.s...@gmail.com>

[Nadin, Mihai]

Dear and respected colleagues,

Dear and respected John Sowa,

Dear and respected Alex Shkotin,

The focus on LLM is misleading. LLM means nothing else but processing data encoded in language. Language is still the most impressive means of representation humans developed. BUT: not the only one. What made LLM possible is the TRANSFORMER technology—i.e. a self-attention procedure based on neural networks. After LLM followed the larger set of mixed media, and the generative approach.

Regarding the spectacular progress in using LANGUAGE as an interface for the transformer type of processing—it all goes back to A. A. Markov (and his counting words in Puskin). But this is a different subject.

Best wishes.

 

Mihai Nadin

--

All contributions to this forum are covered by an open-source license.
For information about the wiki, the license, and how to subscribe or
unsubscribe to the forum, see http://ontologforum.org/info/
---
You received this message because you are subscribed to the Google Groups "ontolog-forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ontolog-foru...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/7f9842329c8f4f02a8bf7a2603602601%40bestweb.net.

[Alex Shkotin]

Colleagues,

I'm glad you supported my humble initial observation in this thread: " Today we can be calm about the application of LLM in the field of robotics."

Alex

ср, 29 нояб. 2023 г. в 01:44, Nadin, Mihai <na...@utdallas.edu>:

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/BL3PR01MB68975A5FCC49BF0DDC10090EDABCA%40BL3PR01MB6897.prod.exchangelabs.com.