configuring multiple versions/different OS version of LUA over HPC shared directory

[Robert Kudyba]

Within a HPC cluster we're attempting to migrate from CetOS 7.9 to RHEL 9. We took one compute node and upgraded it to RHEL 8 and this of course breaks the modules. On login or running 'ml' we get the following error:

 ml avail
/usr/bin/lua: /usr/share/lua/5.1/posix/init.lua:29: module 'posix.ctype' not found:
no field package.preload['posix.ctype']
no file '/usr/share/lua/5.1/posix/ctype.lua'
no file '/usr/share/lua/5.1/posix/ctype/init.lua'
no file '/usr/lib64/lua/5.1/posix/ctype.lua'
no file '/usr/lib64/lua/5.1/posix/ctype/init.lua'
no file '/usr/lib64/lua/5.1/posix/ctype.so'
no file '/usr/lib64/lua/5.1/loadall.so'
no file '/usr/lib64/lua/5.1/posix.so'
no file '/usr/lib64/lua/5.1/loadall.so'
stack traceback:
[C]: in function 'require'
/usr/share/lua/5.1/posix/init.lua:29: in main chunk
[C]: in function 'require'
/share/lmod/lmod/libexec/ml_cmd:62: in main chunk
[C]: in ?

This makes sense as the RHEL 8 LUA is version 5.3. I tried to create a sym link:

ln -s /usr/share/lua/5.3 /usr/share/lua/5.1

But that returns different errors:

/usr/bin/lua: /usr/share/lua/5.1/posix/init.lua:29: module 'posix.ctype' not found:
no field package.preload['posix.ctype']
no file '/usr/share/lua/5.1/posix/ctype.lua'
no file '/usr/share/lua/5.1/posix/ctype/init.lua'
no file '/usr/lib64/lua/5.1/posix/ctype.lua'
no file '/usr/lib64/lua/5.1/posix/ctype/init.lua'
no file '/usr/lib64/lua/5.1/posix/ctype.so'
no file '/usr/lib64/lua/5.1/loadall.so'
no file '/usr/lib64/lua/5.1/posix.so'
no file '/usr/lib64/lua/5.1/loadall.so'
stack traceback:
[C]: in function 'require'
/usr/share/lua/5.1/posix/init.lua:29: in main chunk
[C]: in function 'require'
/share/lmod/lmod/libexec/addto:65: in main chunk
[C]: in ?
/usr/bin/lua: /usr/share/lua/5.1/posix/init.lua:29: module 'posix.ctype' not found:
no field package.preload['posix.ctype']
no file '/usr/share/lua/5.1/posix/ctype.lua'
no file '/usr/share/lua/5.1/posix/ctype/init.lua'
no file '/usr/lib64/lua/5.1/posix/ctype.lua'
no file '/usr/lib64/lua/5.1/posix/ctype/init.lua'
no file '/usr/lib64/lua/5.1/posix/ctype.so'
no file '/usr/lib64/lua/5.1/loadall.so'
no file '/usr/lib64/lua/5.1/posix.so'
no file '/usr/lib64/lua/5.1/loadall.so'
stack traceback:
[C]: in function 'require'
/usr/share/lua/5.1/posix/init.lua:29: in main chunk
[C]: in function 'require'
/share/lmod/lmod/libexec/addto:65: in main chunk
[C]: in ?

Is there a quick fix for this to get us working so we can continue to test the upgrade to RHEL 8 then 9? I've seen there are ways to have multiple versions of LUA I just don't want to break the older version.

Thanks!

[Denis Dos Santos Silva]

It is necessary to know the correct version of lua.
different versions have no guarantee of compatibility ("even more so when it comes to external modules made in c/c++)

It may be necessary to recompile in case the version used by the distribution dont matchi; or use docker.

[Martin Eden]

On 2024-09-18 16:47, Robert Kudyba wrote:
> /usr/bin/lua: /usr/share/lua/5.1/posix/init.lua:29: module 'posix.ctype'
> not found:

Looks like this issue to me:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891541#15

-- Martin

[sur-behoffski]

On 2024-09-19 00:17, Robert Kudyba wrote:
> Within a HPC cluster we're attempting to migrate from CetOS 7.9 to RHEL 9. We took one compute node and upgraded it to RHEL 8 and this of course breaks the modules. On login or running 'ml' we get the following error:

> [...]

G'day,

My SourceForge project "lglicua": "Lua"+"GNU/Linux"+"IM/CD/IUP"+Assistant
does part, but perhaps not all, of what you want.

https://sourceforge.net/projects/lglicua/files/

Incidentally, you can use this Assistant merely for its
GNU/Linux-based Lua and LuaRocks capabilities, without having to
delve into IM/CD/IUP project fetch/build/run.

The latest release, 0.1-beta2, has been (briefly) tried and run
on many Debian/Ubuntu and Red Hat distributions. The Ubuntu
support came first, because I was aiming to get Tecgraf's
CD/IM/IUP sci-tech toolkit to compile and run, and the GNU/Linux
documentation was given only for Ubuntu; Red Hat-based system
support came later:

--------

* Debian/Ubuntu:

- Debian 12.5;

- Kali 2024.2;

- Linux Mint 19.x, 20.x, 21.x and 22;

- MX 23.3_ahs_64; and

- Ubuntu 22.04.4, 23.10 and 24.04.

--------

* Red Hat family:

- CentOS Core 7;

- CentOS Stream 8 and Stream 9; and

- Rocky Linux 8.10, 9.3 and 9.4.

--------

On all of these platforms, the Assistant supports the latest version
Lua (X.Y), along with default or selectable release (.Z):

* Lua 5.1.5;

* Lua 5.2.4;

* Lua 5.3.6; and

* Lua 5.4.7.

Sadly, Lua 5.0 or earlier, and LuaJIT, are not supported (I use
explicit whitelisting in the installer).

--------

Lua is always built from (possibly-slightly-patched) sources, on
all Distributions, including:

* Static linking, C and C++; and

* Dynamic linking, C and C++.

--------

Given "INSTALL" as "/path/to/project/install", and "$" showing a
Bash shell, the installer ("./i") gives useful sub-commands:

INSTALL$ ./i lua-install 5.4
(...sudo request, build environment, build output...)

INSTALL$ lua -v
Lua 5.4.7 Copyright (C) 1994-2024 Lua.org, PUC-Rio

INSTALL$ ./i lua-install 5.1
(...sudo request if needed, build environment if incomplete, build output...)

INSTALL$ lua -v
Lua 5.4.7 Copyright (C) 1994-2024 Lua.org, PUC-Rio

INSTALL$ ./i lua-select 5.1

INSTALL$ lua -v
Lua 5.1.5 Copyright (C) 1994-2012 Lua.org, PUC-Rio

INSTALL$ ./i luarocks-lua-install
(...download, build and install LuaRocks 3.11.1 for Lua 5.1)

INSTALL$ ./i lua-select 5.4

INSTALL$ ./i luarocks-lua-install
(...download, build and install LuaRocks 3.11.1 for Lua 5.4)

The "luarocks-install" step also brings in some lua-version-specific
rocks, e.g.:

- luasec

- luasocket

- std.normalize

- std.strict

- sha1

In addition, in a "cringeworthy" fashion, my PosixExec.lua script is
wedged in as if it was a Rock. I rely heavily on this module, as it
gives me the ability to run commands in a separate process, but with
the command line (parameters etc) under very, very strict control of
a Lua table (array of strings) -- No need to worry about escaping
special characters; a few features which would otherwise be
excluded (I/O redirection, limited filename globbing) are provided
via the "posix" rock.

"lglicua" uses the "update-alternatives" mechanism on Debian-based
distributions to switch between Lua and LuaRocks versions, and does
a credible imitation of this facility on the Red Hat-based
distributions.


HOWEVER: LuaRocks "knows" which version of Lua it is associated
with via having environment variables LUA_PATH and LUA_CPATH
tailored to include the Lua version when pointing to the base of
the Rocks tree.

The LuaRocks documentation notes that switching versions is a tad
tricky; see the documentation "Rocks Trees and the Lua Libraries
Path", about half-way down the page, at:

https://github.com/luarocks/luarocks/wiki/Using-LuaRocks

Here's an extended excerpt from that section:

****

Most LuaRocks installations will feature two rocks trees:

"system" rock tree (used by default)
"user" rock tree

To be able to use the module, we need to make sure that
Lua can find that dkjson.lua file when we run
require("dkjson"). You can check your Lua paths from the
Lua environment, using

print(package.path)
print(package.cpath)

These variables can be pre-configured from outside Lua,
using the LUA_PATH and LUA_CPATH environment variables.

If you installed both Lua and LuaRocks in their default
directories (/usr/local on Linux and Mac OSX), then the
"system" tree is /usr/local and it will work by default.
However, the "user" tree (for installing rocks without
admin privileges) is not detected by Lua by default.
For that we'll need to configure these environment
variables.

LuaRocks offers a semi-automated way to do this. If
you type the following command:

luarocks path --bin

...it will print commands suitable for your platform
for setting up your environment. On typical Unix
terminal environments, you can type this:

eval "$(luarocks path --bin)"

and it apply the changes, temporarily, to your shell.
To have these variables set permanently, you have to
configure the environment variables to your shell
configuration (for example, by adding the above line to
your .bashrc file if your shell is Bash).

****

My installer, "lglicua", adopts an approach similar to the
"permanent environment change", by using system privileges to
add a file to /etc/profile.d/ (3.11.1 was the latest version
of LuaRocks, as of the 0.1-beta2 release):

/etc/profile.d/17-LuaRocks-3.11.1-LUA_PATH-and-LUA_CPATH.sh

and places the output of "$(luarocks path --no-bin)" in it.

I decided that running eval-perhaps-on-multiple-shells, as in
the documentation quoted above, would be a potential source of
confusion for unwary users, and so I would instead strongly
recommend to the user that they reboot immediately for the new
permanent paths to take effect:

INSTALL$ ./i reboot-now

*******************************

ONE BIG ELEPHANT IN THE ROOM: Lua and LuaRocks are installed in
system directories, instead of being neatly placed in the
(user or project?) development tree alongside IM/CD/IUP etc.

I've chosen to live with this for now, and am coping by having
each installation run in its own virtual machine. A choice
such as this is also suggested by the deliberate liberal
sprinkling of "sudo"-enabled commands during the installation
process; the major upside of this approach is that the project
can be worked on with NO system privileges once installation is
finished and development is ready to start (we momentarily
extend LD_LIBRARY_PATH in the workspace play/ and/or support/
areas to use dynamic libraries).

-------------------------------


Whew! Lots of stuff here. Hope this helps,


sur-behoffski (Brenton Hoff)
programmer, Grouse Software

[Rob Kudyba]

Thanks to all who replied. To get this to work I took a suggestion from the linked bug report (albeit we're on RHEL) and created 2 symbolic links:

ln -s /usr/share/lua/5.3 /usr/share/lua/5.1
ln -s /usr/lib64/lua/5.3 /usr/lib64/lua/5.1

Then installed luarocks, and used that to install:

 luarocks install luaposix
 luarocks install luafilesystem

There are probably other ways to get this done. But until we upgrade the head node and respective Ansible playbook this should work for us.

--
You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lua-l/cf9c38d5-2c60-4ce0-89fa-a0c8f2c12b0e%40grouse.com.au.