AI revisited (popular survey)
Alexander Shen
20 october 17.30 Paris time (18.30 Moscow time)
Zoom:
https://u-bordeaux-fr.zoom.us/j/88402787361?pwd=WktCdEhBT3pXN0pLUGg4Z3RuMlpsQT09In this talk Maxim Ushakov (who has some experience in using AI tools and developing them at a small scale) will share his experiences and answer our questions. Here are the questions I tried to collect.
Few years ago we heard that LLM are advanced token predictors: the dialog stream is split into tokens and then the "most probable" next token is predicted (not just using good old statistical counting but some more advanced "machine learning" tools). Obviously what we see now goes far beyond that, and something obviously impossible happens. But what is under the hood? More specific questions: - Are there separate stages while preparing the answer? (ChatGPT even produces some reasonably looking names for these stages) What happens during this stages?
- How the engine interacts with other programs like programming language interpreters, formal provers etc.? If I ask it to give an approximate solution of some equation (or just ask to multiply two 10-digit integers), does it run some math software? If I ask you to debug some code, does it tries to run it in the debugger? If I ask it when will be the next concert of Angela Hewitt, does it perform internet search? How all this is organized?
- How the model uses the token-prediction engine (probably several times)?
- What is the entire cycle of developing and using the model? It is trained (as predictor of tokens or as a whole), then fine-tuned, then specialized using a prompt, and then applied to some request? What are the costs and resources needed for each step? What can be done inside a medium company like Yandex, in a small startup company like your friends try to create, locally at good home computer, or in a mobile phone?
- What is preserved during the dialog with AI model and between the dialogs? - What amount of information needs to be learned to get some new "skill"? For example, can it learn ancient Greek or some other language where we have only a limited amount of texts in the language like people do (studying grammar textbooks, making graded exercises etc.)? Is is possible to learn some new mathematical theory just by reading the only textbook on this theory (and may be having some dialogs with experts)? - Is the technology more or less known to all the players (companies -- from Yandex to OpenAI) and the difference is just the resources and few years of development, or there are some "key secrets"? Related question: these companies hire key people for millions, presumably those people have some much needed knowledge/abilities. What are those abilities and where they come from? What kind of topics and experiences should be included in the curriculum to prepare to the New World? - Linguistics and logic seem to be in big trouble (even if they don't see it now like this): it turns out that the ``rigid'' structures like grammars and formal theories they tried to discover and develop are not something fundamental but appear from the sea of fuzzy parameters in quite mechanical (though probably not understood) way, and what looked like syntactical sugar and user-friendly interface could be much more fundamental -In general, what are the current frontiers between what is easy / difficult / very difficult and expensive / hardly possible / out of reach? What is now possible locally (on a reasonable desktop) in terms of education/reeducation/dialog?
= What is happening in specialized topics like audio/video/photo analysis and generation? Can the models write down the score of a symphony looking at the video (with sound), or find an error in the performance of a Beethoven's sonata? How do they do it (what is replacing the tokens etc.)?
If all goes well, we can have some live experiments and comments (сеанс магии и её разоблачения)
-------
Reminder (was sent earlier)
The possible application of AI in math are obviously a hot topic. The Mathematical Intelligencer is trying to collect people's observations about their personal experiences (the quotes may be published); just a few lines about these questions (some of them) would be great.
Questions:
1) What was yours most impressive ``success story'' of using AI, LLM and related tools for mathematics research/teaching?
2) What was yours most disappointing experience of this kind?
3) What would you expect to happen, say, 5 years from now in this regard?
Please send your answers to me (sasha...@gmail.com), I will forward them to M.I.
--