AI technical writers are here!

[Martin Eden]

Hello guys,

This night I've discovered AI site that provides structured
documentation generation for guthub repos. I've tested it a
bit on three repositories and quite impressed in results.


1. Lua's repo

https://deepwiki.com/lua/lua

I have general understanding how Lua internally done but
generated docs and graphs provide nice high-scope modules
overview with links to code.


2. Lua code formatter

https://deepwiki.com/martin-eden/lua_code_formatter

I wrote it like 9 years ago and not maintaining it.
Code organization is not orthodoxal, personal grammar parser,
AST transformer, formatter and text printer. I did my best
when was writing it but documented that times a lot less.

Generated documentation is structured sufficiently complete.


3. Lua code melder

https://deepwiki.com/martin-eden/lua_code_melder

Again, it's mine code I wrote relatively recently. Lot more simple.

Generated documentation is unnecessary complex. Mainly because
code contains parts from my personal framework which are not
central part of program.


Conclusion

Choosing meaningful names, writing comments and documenting
parts even in small files pays off. Generated documentation
is in parts are more useful than "stock" documentation.

-- Martin

[Родион Горковенко]

The subtle issue with AI-generated documentation is that it is not reliable.

Unless the tool has explicit control allowing user to prevent it from going delirious when data are insufficient it may

sometimes lead to unpleasant consequences.

--
You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/lua-l/449a9fb7-8275-4f1c-a1d7-9e0b563a360f%40disroot.org.

[Sainan]

I'm so fucking tired of this. AI still either just copies text that was already there (maybe rephrashing it a little) or it just straight-up hallcuinates. It had the same problems when it was first introduced over 2 years ago. Why is no one learning this?

-- Sainan

[Martin Eden]

On 2025-11-29 10:17, 'Sainan' via lua-l wrote:
> I'm so fucking tired of this. AI still either just copies text that was already there (maybe rephrashing it a little) or it just straight-up hallcuinates. It had the same problems when it was first introduced over 2 years ago. Why is no one learning this?
>
> -- Sainan
>

Well maybe.

I expect some people here have sufficient expertise in Lua internals
(looking at you Roberto!)  to compare computer-generated text with
their vision. And make their opinions.

Also that's why I used two repos in test that are completely mine
and mostly unknown.

Scope of documentation is biased based on available code files and
related text. So yeah, not like we humans really like to perceive it.

Also we can test what will happen when documentation contradicts
code. But I'm too lazy for this.

And even so, just text snippets, AI text stitcher, callgraph generator
in Python and `gv` called from some Bash script produce result parts of
which I would like to borrow. (Mostly because English is not my native
language.) And that's mass media, not some script at half-dead
university's site.

-- Martin

[Augusto Goulart]

No LLMs please. I’ve got an “Awesome No-LLM” repo and would be very sad to add Lua to the “Do not trust” section.

Regarding AI-generated documentation, I think that’s worse than generated code because you can test code, with documentation you’ll have to proofread it, and even then — like you said — the code used to generate it might not reflect its intended purpose from a system-wide view.

Just a thought,

Augusto

--
You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/lua-l/a1f6e6bb-e6fe-4e73-8a2a-60c088cf8750%40disroot.org.

[bil til]

I think it might be nice to complete "small tasks", which are not
really new but somehow near to some standard, so more adaptions to a
special application.

But not the planning ... and the planning is most important for
programming. There must be some people who set define the "golden
thread" in a software project, this could never be AI, as this is
typically a very unique work.

If in such a project a mass of programmers is adapting software to
this application, and that application, and expecially if there are
lots of such "more boring subtasks", then AI might be useful.. .
Typically you might see this in very large software projects (which I
myself am NOT expert for - I prefer to write all critcial software
parts by myself...).

Real testing of a NEW software system you can only do if you clearly
know the "golden thread" of an idea / of a project / of the
motivation.. .

E. g. for documentation: AI is quite perfect meanwhile for transation,
also in very different languages as Chinese, see deepl... the progress
here in the last 10 years is really extremely impressive. For any
"automation tasks" AI clearly will get better and better in
programming and also documention, I am quite sure.

Am Sa., 29. Nov. 2025 um 18:02 Uhr schrieb Augusto Goulart
<joseaugust...@gmail.com>:
>

[Sean Conner]

It was thus said that the Great 'Martin Eden' via lua-l once stated:

>
> On 2025-11-29 10:17, 'Sainan' via lua-l wrote:
> >I'm so fucking tired of this. AI still either just copies text that was
> >already there (maybe rephrashing it a little) or it just straight-up
> >hallcuinates. It had the same problems when it was first introduced over 2
> >years ago. Why is no one learning this?
>

> Well maybe.
>
> I expect some people here have sufficient expertise in Lua internals
> (looking at you Roberto!)  to compare computer-generated text with
> their vision. And make their opinions.
>
> Also that's why I used two repos in test that are completely mine
> and mostly unknown.
>
> Scope of documentation is biased based on available code files and
> related text. So yeah, not like we humans really like to perceive it.
>
> Also we can test what will happen when documentation contradicts
> code. But I'm too lazy for this.

^^^^^^^^^^^^^^^^^^^^^^^^^

This does not surprise me. My second experience with LLMs involved
someone too lazy to do a good job with LLMs in an attempt to disprove my
stance that LLMs are not worth the effort. [1] They didn't put in the
effort and toally misunderstood the problem. Funny that.

But I too, decided to try for myself this site. And I submitted two repos
of my own [2]. I'm in the process of wring up my experience with it, but
over all, I'm still not convinced. The first repo is a code base I've been
working on for 25 years; the second one for just a few years.

Overall, not a great showing. Plenty of errors in the "documentation",
some quite subtle unless you know the code well. Some that are just so
egregiously bad that it borders on maliciousness. I also noticed that while
both repos are about the same size lines of code wise (7,400 on one, 9,500
in the other; both in the same order of magnitude) that it was a bit less
wrong on the smaller repo. Two reasons: the smaller is the one I've been
using for 25 years and I've been removing features I don't use for the past
year so it's easier to follow, and the larger has a bit more logic
complexity due to the nature (and I've been working on it for far fewer
years). I suspect that the larger one is just closer to the LLMs token
limit to work effectivly. I'd try it on a larger repo, 155,000 lines of
code written in the early 90s, but I'm not versed enough with the code base
to find errors in the generated "documentation" unfortunately.

I'll leave it to you, the reader, to find the egregiously bad errors in
the repo "documentation" [2]. My hypothosis: no one will (even with the
hint that it's in the second repo) , which to me says this is not a tool
worth using.

-spc (and I suspect this thread will be closed as "off topic" rather
quickly now ... )

[1] https://boston.conman.org/2025/06/05.1

[2] https://deepwiki.com/spc476/mod_blog
https://deepwiki.com/spc476/a09

[Sainan]

To add my own experience, I did indeed also test the tool on a repo I know quite well[1], it's a fork of Lua, so we're staying on-topic. :^)

And it's pretty clear it just headed over to our official documentation website instead of the actual codebase which is how you very quickly reach hallucination land, e.g. here[2] where it correctly says that Pluto adds crypto libraries, but points to a non-existent "crypto/ directory" when this implementation in reality lives in "src/lcryptolib.cpp", etc.

I do not see the value-add here if it just butchers the documentation and if you want to contribute, you're gonna need to read the codebase anyway to understand where things actually are.

[1] https://github.com/PlutoLang/Pluto/
[2] https://deepwiki.com/PlutoLang/Pluto#extended-library-ecosystem

-- Sainan

[Martin Eden]

On 2025-12-03 00:25, Sean Conner wrote:
> I'll leave it to you, the reader, to find the egregiously bad errors in
> the repo "documentation"

Hello Sean,

This statement is Russell's Teapot.

Despite it's out of maillist scope, for the sake of history and
scientific approach can you please point to "so
egregiously bad that it borders on maliciousness" error in generated
documentation and source of truth in code repo?

P.S. You have nice old-school code style and nice documentation.
P.P.S. I'm not using Motorola 6809 and your assembler to discover it
occasionally.

-- Martin

[Sean Conner]

It was thus said that the Great 'Martin Eden' via lua-l once stated:
>

> On 2025-12-03 00:25, Sean Conner wrote:
> >I'll leave it to you, the reader, to find the egregiously bad errors in
> >the repo "documentation"
>
> Hello Sean,

Hello.

> Despite it's out of maillist scope, for the sake of history and
> scientific approach can you please point to "so
> egregiously bad that it borders on maliciousness" error in generated
> documentation and source of truth in code repo?

I cover the mistakes in this blog post:

https://boston.conman.org/2025/12/02.1

but the "so egregiously bad" one is this:

7. Oh my God! I can't say how bad this backend matrix table is. It's
all sorts of wrong. It's not that it got the supported/non-supported
markers backwards, it appears to have just made up the results! And
the same information on another page is bad as well. Not as bad as
the first, but that's like saying bronchitus is not as bad as
pneumonia. Both are bad. And it uses a different format for both
tables.

-spc