[vim/vim] vim9script llms.txt (Issue #18576)
Enno
Konfekt created an issue (vim/vim#18576)To assist an LLM writing Vim9script in view of the yet limited training set an llms.txt could be provided (similar to what Svelte offers in view of more established competing frameworks). For starters, a variant of @kennypete 's LearnXinY could do. This could be offered on the homepage as well, or mentioned in :help.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Peter Kenny
kennypete left a comment (vim/vim#18576)Putting aside the matter of whether it will ever have sufficient adoption making it worthwhile - and the jury seems to be out on that - I guess one key question is whether there is a relatively easy way to build and maintain it/them. A basic llms.txt for the toc-level content can be generated, as attached at the end of this comment, by an LLM (Claude) from a couple of the vimhelp.org files. That would be low effort to update.
Creating Vim9 script llms_vim9.txt in a similar depth and breadth as Svelte docs would be more challenging, especially if 'full', 'medium', and 'small' outputs were wanted. Maybe an LLM could take vim9.txt and vim9class.txt and/or LearnXinY content, though those are more complex than the toc-level content. It points to the challenge, i.e., there is no single-source editing environment/source for the docs like DocBook, TEI, or even a highly featured, structured format like AsciiDoc, which would better enable single-source-to-several-outputs. [ A tangent... Coming from a single-source SGML, and later XML, editing work environment (in the 1990s-2000s), the value converting from a single source grows markedly as more outputs are sought, this being a case in point. Vim help, in its bespoke .txt format is a challenge, notwithstanding it is miraculous that vimhelp.org generates such brilliant, reliable output with vimhelp ]
Back to llms.txt. If it was limited to only Vim9 script content (plus, perhaps, a few toc-level docs), narrowing the scope, it would require careful design consideration regarding how to make generation of it/them repeatable and reliable without manual reformatting as Vim9 script develops, more examples are added, etc.
Incidentally, @Konfekt, thanks for the prod; it reminds me I need to do that PR for LearnXinY (which will get some content updates too because things have moved ahead even since it was released). I'll revisit that after pushing the vim9.txt Section 4 PR, which should be coming in the next day or two.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
chrisbra left a comment (vim/vim#18576)To assist an LLM writing Vim9script in view of the yet limited training set an llms.txt could be provided
That blog article you mention, does not even recommend to create such a document and apparently no LLM does yet support it. So it seems a bit unnecessary ?
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Enno
Konfekt left a comment (vim/vim#18576)I am sorry, that's a misplaced article, as it talks about websites for which it is unnecessary, agreed. Just as a hastily pasted reference for a by now common file name. I should have omitted it.
Its use is rather to provide LLMs vim9script context as in https://svelte.dev/docs/llms as explained in the opening post, it's this link that inspired this issue.
There's more legacy than vim9script out there, so adding additional vim9script context could help an LLM in forming correct syntax. Either learnxiny or doc/vim9.txt could serve as starting points.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
chrisbra left a comment (vim/vim#18576)Well, the question remains: if no LLM supports this yet, why should we take the additional effort to provide something that is not yet supported and for dubious value and basically duplicates the existing documentation? Remember this doc also needs to be maintained and updated as we progress with Vim development.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Enno
Konfekt left a comment (vim/vim#18576)not yet supported
Every LLM supports ingesting text. It's rather a question of the LLM interface ingesting files (most do).
dubious value
Svelte deemed it worthwhile as a measure to compensate better LLM support by more popular competing frameworks. A frequent argument being that any language without sufficient tooling, in particular LLMs, is unlikely to get any traction. (Of course, there's more missing, like language servers, but at least the lack of LLM support can be somewhat mitigated.)
duplicates the existing documentation
Fair point, though in view of legacy vimscript far outweighing the vim9script training data, this additional context keeps the LLMs on vim9script track. The smaller llms.txt provided at above link are in fact human-readable documentation, so one could maybe indeed reuse the existing documentation.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Enno
Konfekt left a comment (vim/vim#18576)Looking again at https://svelte.dev/docs/svelte/llms-small.txt, it describes how the syntax of the new major version 5 differs from the previous major version 4. https://learnxinyminutes.com/vim9script/ is of a similar format. I assumed that if it helps them to make LLMs understand the new Svelte syntax, so will a similar file for the new vimscript syntax.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Peter Kenny
kennypete left a comment (vim/vim#18576)duplicates the existing documentation
Fair point, though in view of legacy vimscript far outweighing the vim9script training data, this additional context keeps the LLMs on vim9script track. The smaller llms.txt provided at above link are in fact human-readable documentation, so one could maybe indeed reuse the existing documentation.
A couple more thoughts on this, particularly in relation to reusing existing documentation.
- Once
vim9.txt(and maybevim9class.txttoo) have been updated with more examples, there will be many more complete, sourceable/working examples of Vim9 script. Yesterday I PR-ed vim9.txt Section 4, which has several. You're right, though, there is much more legacy Vim script out there than Vim9 script (and most LLMs today seem to default their responses to legacy Vim script). Instructing an LLM to use the applicable help files (either providing links to vim9.txt or attaching the.txtfiles could drive better responses so could be recommended as an alternative to producing/using an llms file(s) if it's proven that underpins better responses). - Regarding the learnxinyminutes content potentially being a source for building an llms file: Sure, that's a possibility - i.e., [semi-]automate its creation from its Markdown. It's not very long, so wouldn't be too hard to do in parallel when it is updated, which you'd not expect will happen frequently. I'm most of the way through updating it now, which, in addition to addressing the errors I called out a while ago, makes changes that should make it easier to use. Examples: 1. Splitting the current massive script into sections, and 2. Fixing examples that are Linux-dependent.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Enno
Konfekt left a comment (vim/vim#18576)Thank you very much @kennypete for your valuable vim9script work
providing links to vim9.txt or attaching the .txt files
That sounds like an elegant solution; just hint in the help, say at vim9.txt at feeding the LLM with doc/legcacy-vs-vim9.txt as additional input which is, similar to Svelte's llms-small.txt useful for humans as a concise overview of the principal differences as well.
if it's proven that underpins better responses
LLMs are black working magic. Hard to prove, but sensible and worth a try
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Enno
Konfekt left a comment (vim/vim#18576)Thank you; I am always wondering to some extent URLs sent to LLMs are actually scraped. Grok is explicit on it in the web interface. That's why providing the /usr/share/vim/doc/vim9diffs.txt file seems more reliable. If output is fine on the first try, an LLM usually only adds noise. That's maybe their biggest issue, endless adding with little pruning, the difficult part.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
D. Ben Knoble
benknoble left a comment (vim/vim#18576)Well, the question remains: if no LLM supports this yet, why should we take the additional effort to provide something that is not yet supported and for dubious value and basically duplicates the existing documentation? Remember this doc also needs to be maintained and updated as we progress with Vim development.
Indeed. I would consider a bit of a misstep by Vim for the core project to become too tightly tied to any technology that isn't fundamental for text editing, LLMs especially. I have my own opinions, but factually it's clear that there is a lot of polarity around them and that even including "AI" features can turn away users.
If folks want to maintain Vim-related things for use with an LLM, the core project need not be where that happens: you can create and maintain your own stuff for that. If it proves valuable enough to integrate, that can always come later?
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Enno
Konfekt left a comment (vim/vim#18576)I can relate to that prudence. A doc/legcacy-vs-vim9.txt is of independent stand-alone interest for human readers; hinting add injecting it whenever LLMs keep spitting out legacy vimscript is the least intrusive. So maybe this issue should move to compiling such a easily (human and machine) parsable file
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
D. Ben Knoble
benknoble left a comment (vim/vim#18576)Great. I find the current vim9script doc hard to use as a reference, so I think what we might really need is a clearer overall picture of the language (I think the current doc gets muddied by spending time on the differences, but I might be remembering wrong).
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
chrisbra left a comment (vim/vim#18576)Then we should perhaps concentrate on make the documentation more clear where it isn't. @kennypete and @yegappan have spent quite some time on the existing documentation, so I think we are improving here already.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
Closed #18576 as not planned.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
chrisbra left a comment (vim/vim#18576)not planning to do this right now.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.