Lua 5.4.7-rc3: Apparent impedance mismatch with LuaRocks 3.11.1...

[sur-behoffski]

G'day,

I've tried to add the latest Lua Release Candidate (lua-5.4.7-rc3) to
my lglicua test rig, and things break, due to an impedance/assumption
change.

Lua itself compiles and install correctly:

- Using C, default (static) library;
- Using C++, default (static) library;
(then, after patching the Lua sources for dynamic building...)
- Using C, dynamic objects; and
- Using C++, dynamic objects.


So far, so good.

------------

However, lglicua next tries to install LuaRocks-3.11.1, and
the "./configure" script breaks.

I've narrowed the problem down to the configure script looking for
and "authentic" lua.h in a suitable place (e.g.
"/usr/include/lua/5.4/lua.h").

The problem is that, by "authentic", the configure script uses grep
to look for "LUA_VERSION_NUM.*$LUA_VERSION", i.e.

grep "LUA_VERSION_NUM.*501"
#define LUA_VERSION_NUM 501
grep "LUA_VERSION_NUM.*502"
#define LUA_VERSION_NUM 502
grep "LUA_VERSION_NUM.*503"
#define LUA_VERSION_NUM 503

And, up to Lua 5.4.6:

#define LUA_VERSION_NUM 504

------------

Looking at lua-5.4.7-rc3, the definition of LUA_VERSION_NUM has
changed, in order to reduce redundancy. This has had the side-effect
of breaking the simple-minded version check in LuaRocks. I'm not
sure exactly where to change things, but the current situation is
broken: The LuaRocks-3.11.1 "./configure" script fails, because it
can't find "504", as lua.h now has:

#define LUA_VERSION_NUM (LUA_VERSION_MAJOR_N * 100 + LUA_VERSION_MINOR_N)

------------

Best wishes,

sur-behoffski (Brenton Hoff)
programmer, Grouse Software

--

[...and the 0.1-beta2 lglicua release is now DEFINITELY on hold...]

[Frank Dana (FeRD)]

On Saturday, June 8, 2024 at 1:16:41 AM UTC-4 sur-behoffski wrote:

However, lglicua next tries to install LuaRocks-3.11.1, and
the "./configure" script breaks.

I've narrowed the problem down to the configure script looking for
and "authentic" lua.h in a suitable place (e.g.
"/usr/include/lua/5.4/lua.h").

 

Looking at lua-5.4.7-rc3, the definition of LUA_VERSION_NUM has
changed, in order to reduce redundancy. This has had the side-effect
of breaking the simple-minded version check in LuaRocks. I'm not
sure exactly where to change things, but the current situation is
broken: The LuaRocks-3.11.1 "./configure" script fails, because it
can't find "504", as lua.h now has:

#define LUA_VERSION_NUM (LUA_VERSION_MAJOR_N * 100 + LUA_VERSION_MINOR_N)

I've reported this in the luarocks issue tracker: https://github.com/luarocks/luarocks/issues/1685 

[Thijs Schreijer]

On Sat, 8 Jun 2024, at 16:40, Frank Dana (FeRD) wrote:

On Saturday, June 8, 2024 at 1:16:41 AM UTC-4 sur-behoffski wrote:

However, lglicua next tries to install LuaRocks-3.11.1, and

the "./configure" script breaks.

...

I've reported this in the luarocks issue tracker: https://github.com/luarocks/luarocks/issues/1685 

Running into the same problem with luawinmake. Shouldn't be a hard to fix issue (though would have been easier to get this in 5.5)

Thijs

[sur-behoffski]

On 2024-06-10 17:00, 'Thijs Schreijer' via lua-l wrote:
> [...]

>> However, lglicua next tries to install LuaRocks-3.11.1, and
>> the "./configure" script breaks.
>> [...]
>> I've reported this in the luarocks issue tracker: https://github.com/luarocks/luarocks/issues/1685 
>
> Running into the same problem with luawinmake. Shouldn't be a hard to fix issue (though would have been easier to get this in 5.5)
>

G'day Thijs,

Just had a look at your fix to LuaWinMake (e39a848728402a4a9102dad20efb64031b0c8b97),
and your "Check on a Lua 5.4.7+ version" paragraph seems to label anything
with a Lua version MAJOR_N of 5 as LUA_VER=5.4. There is no test of Lua's
MINOR_N. [See new code near line 192].

I'm no expert at Windows batch files (thankfully), but I believe that this
commit labels far too many Lua versions as 5.4 (e.g. 5.2.0 to 5.99.99).

Humble apologies to you if I'm wrong (and also, apologies to the list for
any line noise),

b

[Sainan]

Nah, the list is just starving for any bite of juicy Lua drama that we can get our hands on.

[Thijs Schreijer]

On Mon, 10 Jun 2024, at 11:45, sur-behoffski wrote:
> On 2024-06-10 17:00, 'Thijs Schreijer' via lua-l wrote:
>> [...]
>>> However, lglicua next tries to install LuaRocks-3.11.1, and
>>> the "./configure" script breaks.
>>> [...]
>>> I've reported this in the luarocks issue tracker: https://github.com/luarocks/luarocks/issues/1685 
>>
>> Running into the same problem with luawinmake. Shouldn't be a hard to fix issue (though would have been easier to get this in 5.5)
>>
>
> G'day Thijs,
>
> Just had a look at your fix to LuaWinMake
> (e39a848728402a4a9102dad20efb64031b0c8b97),
> and your "Check on a Lua 5.4.7+ version" paragraph seems to label
> anything
> with a Lua version MAJOR_N of 5 as LUA_VER=5.4. There is no test of
> Lua's
> MINOR_N. [See new code near line 192].
>
> I'm no expert at Windows batch files (thankfully),

Congrats to you for that :)

> but I believe that this
> commit labels far too many Lua versions as 5.4 (e.g. 5.2.0 to 5.99.99).
>
> Humble apologies to you if I'm wrong (and also, apologies to the list for
> any line noise),
>
> b

The first check is for

#define[tab(s)]LUA_VERSION[tab]

Which is only present in Lua 5.1

Second check is for

#define[tab(s)]LUA_VERSION_MAJOR_N[tab]

The "_N" postfix is new, and points to 5.4.7
The in between versions do not have the "_N" postfix, and hence the code falls through and parses

#define[tab(s)LUA_VERSION_MAJOR[tab]"x" -- x being 5
#define[tab(s)LUA_VERSION_MINOR[tab]"x" -- x being 2/3/4

So that works afaict.

Though if your comment is about future-prove-ness, then yes, I'll have to revisit the 2nd
check once 5.5 is released.

Thx for your checks, much appreciated. I love being proven wrong when I'm wrong :)

Thijs

[Thijs Schreijer]

Nevermind, fixed it anyway, to save future-self some work.
Thx for the nudge.

[Frank Dana (FeRD)]

On Monday, June 10, 2024 at 3:31:09 AM UTC-4 Thijs Schreijer wrote:

Running into the same problem with luawinmake. Shouldn't be a hard to fix issue (though would have been easier to get this in 5.5)

A PR has also been submitted to fix luarocks (https://github.com/luarocks/luarocks/pull/1686), but I kind of have to agree with this. Doesn't this seem like a BIG, disruptive change to be making in a point release?

The change to lua.h doesn't just affect LUA_VERSION_NUM. It also *ELIMINATES* LUA_VERSION_{MAJOR,MINOR,RELEASE} (which were string #defines), replacing them with the new LUA_VERSION_{MAJOR,MINOR,RELEASE}_N integer #defines. That's a *lot* of disruption that will affect a lot of projects, to be suddenly appearing in the update from Lua 5.4.6 to Lua 5.4.7.

[Michael Lenaghan]

Indeed.

FindLua.cmake, which is the module CMake uses to find Lua and determine which version it found, looks for LUA_VERSION_{MAJOR,MINOR,RELEASE}:

https://github.com/Kitware/CMake/blob/master/Modules/FindLua.cmake

[Frank Dana (FeRD)]

On Monday, June 10, 2024 at 9:02:05 PM UTC-4 Frank Dana (FeRD) wrote:

The change to lua.h doesn't just affect LUA_VERSION_NUM. It also *ELIMINATES* LUA_VERSION_{MAJOR,MINOR,RELEASE} (which were string #defines)

Sorry, I'm wrong about that. it actually moves them waaaay down to the end of the file, and now defines them as:

 #define LUA_VERSION_MAJOR       LUAI_TOSTR(LUA_VERSION_MAJOR_N)

#define LUA_VERSION_MINOR       LUAI_TOSTR(LUA_VERSION_MINOR_N)
#define LUA_VERSION_RELEASE     LUAI_TOSTR(LUA_VERSION_RELEASE_N)

(Which is really the same problem, they're not going to work with any sort of grep-based version checks. Like in CMake, like in luarocks, like in...)

[Roberto Ierusalimschy]

If we understood correctly, some of these tools would break even if we
changed tabs for spaces, or changed other minimum details in the format of
some '#define'. None of that is part of the official API. If tools need
that kind of thing, they could have asked since the beginning some
official way to statically recognize a Lua version by inspecting its
header file.

-- Roberto

[Frank Dana (FeRD)]

On Tuesday, June 11, 2024 at 1:51:57 PM UTC-4 Roberto Ierusalimschy wrote:

If we understood correctly, some of these tools would break even if we
changed tabs for spaces, or changed other minimum details in the format of
some '#define'. None of that is part of the official API.

True.

 

If tools need
that kind of thing,

Wait, this comes as a SURPRISE to you?!?

 

they could have asked since the beginning some
official way to statically recognize a Lua version by inspecting its
header file.

That's... certainly one way to deflect the issue, I suppose. 

[Sean Conner]

It was thus said that the Great Roberto Ierusalimschy once stated:

I'm reminded of this XKCD: https://xkcd.com/1172/

Also, I'm reminded of Hyrum's Law: With a sufficient number of users of
an API, it does not matter what you promise in the contract: all observable
behaviors of your system will be depended on by somebody.

-spc

[Pierre Chapuis]

If you want my opinion, those tools should be fixed, not the Lua headers. Headers are headers, you are not supposed to parse them otherwise.

There are many ways to deal with the situation, one being to write a simple code sample in C that loads the header and outputs the version.

-- 

Pierre Chapuis

--

You received this message because you are subscribed to the Google Groups "lua-l" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lua-l/01127cb6-8261-42aa-8739-608c2cf70889n%40googlegroups.com.

[Spar]

I'm surprised why they didn't use _VERSION variable.
Not a fan of luarocks, but sadly nothing else exists as far

To view this discussion on the web visit https://groups.google.com/d/msgid/lua-l/982a5d75-ec2d-4241-a63e-6cb9ed629d29%40app.fastmail.com.

[Martin Eden]

"Weinberg's Second Law: If builders built buildings the way programmers
wrote programs, then the first woodpecker that came along would destroy
civilization."

Maybe not grep #define's?

-- Martin

[Warner Losh]

As someone burned by this for FreeBSD's sys/param.h, I sympathize with the sentiment. But I'd also hold off until 5.5 now that there is at least one known example. You'll be happier in the long run. Just my opinion,  forged by a little of the same fire.. it's an unexpected problem that likely runs deeper than we now know.

Warner

-- Martin

--
You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lua-l/8940b770-2549-4b84-a2a2-ecde8bfc89e9%40disroot.org.

[Marc Balmer]

> Am 12.06.2024 um 00:02 schrieb Spar <develo...@gmail.com>:
>
> I'm surprised why they didn't use _VERSION variable.
> Not a fan of luarocks, but sadly nothing else exists as far

Which is not true. You can use rpm, dnf, yum, zypper, apt etc. to distribute Lua.

We do that all the time, for all of our products…

[sur-behoffski]

G'day,

Checking the version in the header file is a symptom of a
different need. (If people simply wanted the current Lua
version, they could simply execute `lua -v`, and parse the
one-line output, e.g. in POSIX/Bash:

$ LUA_VERSION_WITH_PATCHLEVEL="$(lua -v 2>&1 | cut -d ' ' -f 2)"
$ LUA_VERSION_NUM="$(echo "$LUA_VERSION_WITH_PATCHLEVEL" | cut -d '.' -f 1-2)"

This means that looking at _VERSION will not be sufficient, i.e.

$ lua -e "print(_VERSION)"
Lua 5.2
$ which lua
/usr/bin/lua

.)

I saw a comment by Hisham that laid out the nub of the problem:
There may be multiple lua.h and associated headers, for different
versions of Lua, in different paths. Therefore, a user
(especially an automated script) needs to somehow work through
candidate paths, and evaluate the Lua version API represented
by the header files in each path, until a suitable version+path
is found, or the script runs out of candidates.

If a mismatched header/library version combination is used, this
can lead to runtime defects, such as require() failing due to
incorrect symbols. Hisham found that a significant number of
the reports he received were due to (spurious, if handled
correctly) mismatches, and so he added the include-path search
code to cut down the chances of a mismatch occurring.

The way the candidate list is built is informed by an
inspection of the way different OSes handle versioning; two
examples are given here:

$ locate lua.h | grep -v hpp # Gentoo (edited)
/usr/include/lua5.1/lua.h
/usr/include/lua5.3/lua.h
/usr/include/lua5.4/lua.h
/usr/include/luajit-2.1/lua.h
/usr/include/texlua53/lua.h
/usr/include/texluajit/lua.h
/usr/local/include/lua.h

$ locate lua.h | grep -v hpp # Ubuntu 23.10 (lglicua)
/usr/include/lua/5.4/lua.h

Also, note that, in the Lua sources, there are variables in
the top-level Makefile that can tailor a Lua build to a
client's needs:

$ cat lua-5.4.7/Makefile

# == CHANGE THE SETTINGS BELOW TO SUIT YOUR ENVIRONMENT =======================
# [...]
# Where to install. The installation starts in the src and doc directories,
# so take care if INSTALL_TOP is not an absolute path. See the local target.
# You may want to make INSTALL_LMOD and INSTALL_CMOD consistent with
# LUA_ROOT, LUA_LDIR, and LUA_CDIR in luaconf.h.
INSTALL_TOP= /usr/local
INSTALL_BIN= $(INSTALL_TOP)/bin
INSTALL_INC= $(INSTALL_TOP)/include
INSTALL_LIB= $(INSTALL_TOP)/lib
INSTALL_MAN= $(INSTALL_TOP)/man/man1
INSTALL_LMOD= $(INSTALL_TOP)/share/lua/$V
INSTALL_CMOD= $(INSTALL_TOP)/lib/lua/$V

My gut feeling is that an extra file could be added to the
headers, with a simple NAME=Value one-liner syntax, in a format
similar to, e.g. "/etc/os-release" ("lua-release"?):

$ cat /etc/os-release # on Gentoo
NAME=Gentoo
ID=gentoo
PRETTY_NAME="Gentoo Linux"
ANSI_COLOR="1;32"
HOME_URL="https://www.gentoo.org/"
SUPPORT_URL="https://www.gentoo.org/support/"
BUG_REPORT_URL="https://bugs.gentoo.org/"
VERSION_ID="2.15"

$ cat /etc/os-release # on Ubuntu 23.10
PRETTY_NAME="Ubuntu 23.10"
NAME="Ubuntu"
VERSION_ID="23.10"
VERSION="23.10 (Mantic Minotaur)"
VERSION_CODENAME=mantic
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=mantic
LOGO=ubuntu-logo

I defer to people with more knowledge of the wider affected
software ecosystems as to how to proceed.

Hope this helps,

b

[Thijs Schreijer]

On Tue, 11 Jun 2024, at 19:51, Roberto Ierusalimschy wrote:
> If we understood correctly, some of these tools would break even if we
> changed tabs for spaces, or changed other minimum details in the format of
> some '#define'.

Yes, that's what I'm seeing, and yes it is very brittle. At the same time this stuff is very static. I have code for 5.1 and then 5.2+, and only now the new RC breaks it again. So that is many years in between.

> None of that is part of the official API. If tools need
> that kind of thing, they could have asked since the beginning some
> official way to statically recognize a Lua version by inspecting its
> header file.
>
> -- Roberto

In theory it would be great to have that. But in practice, it's been many years, and to me that is good enough (others will have other opinions probably). Though if this is going to change again, then static markers of some kind would be very nice to have.

In general, I think the responses in this thread assume posix like systems and simple installs (eg. 1 Lua version). For me (author of luawinmake) and I assume likewise for LuaRocks, that doesn't work, since systems can have multiple versions and it has to work anywhere, with all supported source trees, no matter how many in parallel.

On Windows, which uses binary distributions, doing source-distributions is just painful, and writing grep-like statements in Windows Batch files is even more painful (I wish I had grep available, sigh).

Thijs

[Roberto Ierusalimschy]

Would something like this make tools happy?

#define LUA_VERSION_MINOR_N 4
#define LUA_VERSION_RELEASE_N 7

-#define LUA_VERSION_NUM (LUA_VERSION_MAJOR_N * 100 + LUA_VERSION_MINOR_N)
+#define LUA_VERSION_NUM 504
#define LUA_VERSION_RELEASE_NUM (LUA_VERSION_NUM * 100 + LUA_VERSION_RELEASE_N)

-- Roberto

[Michael Lenaghan]

Why make the change now, in a patch release? Why not wait until Lua 5.5?

Here’s a Github code search for “ LUA_VERSION_PATCH”:

https://github.com/search?utf8=%E2%9C%93&q=%22LUA_VERSION_PATCH%22+&type=code

A lot of matches are for customized versions of CMake’s FindLua.cmake module. There’s no question that the change will have a huge impact.

So why make the change now, in a patch release?

[Thijs Schreijer]

The code in the RC definitely suffices for my goals. So a #define for each of the 3 identifiers, each with a "_N" suffix. Especially the suffix makes proper matching (with whatever tool) easy and pretty fool proof imho.

So I like the changes made. That said, I agree with Michael Lenaghan that it is an unfortunate change in a patch release. So if it were up to me, then I'd postpone the change to 5.5.

my 2cts
Thijs

[Luiz Henrique de Figueiredo]

> Looking at lua-5.4.7-rc3, the definition of LUA_VERSION_NUM has
> changed, in order to reduce redundancy. This has had the side-effect
> of breaking the simple-minded version check in LuaRocks.

We've reverted to the previous scheme in Lua 5.4.7-rc4.
We'll reconsider that change in Lua 5.5.
--lhf