ltokenp feedback

[Martin Eden]

Hello list,

I've just tried lhf's ltokenp. It's amazingly simple end effective tool.

Basically it's lexer pipe. It provides strings with types to your
handler function.

Here is script written in 10 minutes to dump Lua script code
and print list of used names and strings.

https://gist.github.com/martin-eden/9608fe89b1af672ec7932a2308787934

I guess ltokenp is useful if someone writes syntax highlighting
or vars renaming.

But well it skips comments. I think lexer can return typed strings
for comments (for line comments and block ones). Despite they are
not used in code generation.

Are you using it already in your projects?

-- Martin

[Luiz Henrique de Figueiredo]

> I've just tried lhf's ltokenp. It's amazingly simple end effective tool.

I'm glad you find it useful. Thanks for the feedback.

You may want to avoid the numerical token values; they may change in
different versions of Lua.
Here is a patch that does that:

% diff Dump.lua new.lua
23,24c23,24
< local NameTypeId = 291
< local StringTypeId = 292
---
> local NameTypeId = "<name>"
> local StringTypeId = "<string>"
75c75
< Process(token, value)
---
> Process(text, value)

> I guess ltokenp is useful if someone writes syntax highlighting
> or vars renaming.

ltokenp can also be useful for experimenting with simple syntax
changes without having to delve into the Lua parser.
Enjoy.
--lhf

[Martin Eden]

On 2025-11-13 16:29, Luiz Henrique de Figueiredo wrote:
> You may want to avoid the numerical token values; they may change in
> different versions of Lua.
> Here is a patch that does that:
>
> % diff Dump.lua new.lua
> 23,24c23,24
> < local NameTypeId = 291
> < local StringTypeId = 292
> ---
> > local NameTypeId = "<name>"
> > local StringTypeId = "<string>"
> 75c75
> < Process(token, value)
> ---
> > Process(text, value)

Thanks for feedback, lhf!

I've spent another 30 minutes reviewing code and updating gist's snippet.

Now there are no integer constants and somewhat better names.

So we can rely that third argument FILTER() receives is string,
and if it is in angle brackets it can be only "<file>", "<string>",
"<name>", "<number>" or "<eof>"?

-- Martin

[Luiz Henrique de Figueiredo]

> So we can rely that third argument FILTER() receives is string,

Yes.

> and if it is in angle brackets it can be only "<file>", "<string>",
> "<name>", "<number>" or "<eof>"?

There is also "<integer>", as listed in
https://www.lua.org/source/5.4/llex.c.html#luaX_tokens, although
"<file>" is not there but only in proxy.c.