LuaDist Binary Distributions

[Paul K]

Hi Peter,

 

I'd like to continue this discussion on LuaDist binaries (http://lua-users.org/lists/lua-l/2013-09/msg00767.html). I've tried to integrate it with ZBS and it seems like it may be possible to ship some of the dependencies (luagit, zlib, and lfs) with ZBS to allow for seamless install of other modules. It would still significantly simplify my work if you could link against lua51.dll and lua52.dll (and liblua52.dylib on OSX; I also use liblua.dylib, but can rename it to liblua51.dylib). It shouldn't change much as you can keep the existing (separate) directory structure.

 

I understand the convenience of keeping different versions in different folders and I'll check if I can migrate to the same structure, but this is complicated by several factors: ZBS itself needs to use one set of libraries for its own launch and the structure needs to work on all three platforms as a package and also from a repository clone (which somewhat limits my options). I'm also using luajit, which doesn't fit nicely into 51 or 52 categories (although it maintains 51 ABI compatibility).

 

I'd appreciate if you could consider renaming the DLL as it would provide LuaDist users with more flexibility in terms of directory structure and would eliminate the need for proxy DLLs. Thank you.

 

Paul.

[Peter Drahoš]

Hello Paul,

On Mon, Oct 7, 2013 at 6:31 PM, Paul K <paulc...@gmail.com> wrote:
> Hi Peter,
>
> I'd like to continue this discussion on LuaDist binaries
> (http://lua-users.org/lists/lua-l/2013-09/msg00767.html). I've tried to
> integrate it with ZBS and it seems like it may be possible to ship some of
> the dependencies (luagit, zlib, and lfs) with ZBS to allow for seamless
> install of other modules. It would still significantly simplify my work if
> you could link against lua51.dll and lua52.dll (and liblua52.dylib on OSX; I
> also use liblua.dylib, but can rename it to liblua51.dylib). It shouldn't
> change much as you can keep the existing (separate) directory structure.

From the build system point of view this requires no changes but it
has some negative implications. If you use LuaDist modules directly
(without "luadist" tool) you can set up the LUA_LIBRARY CMake variable
to point to any lua library you wish against from the command-line.
Additionally the FindLua.cmake module included in most modules is
aware of the usual naming conventions in the Lua ecosystem and will
select the first matching lua library it finds. The automated finding
of the library may (and probably will) be affected by the change if
there are multiple lua versions in locations CMake searches for the
library.

Manual setting of the library will also work with the "luadist" tool
also but it would require a slightly modified lua variant that
produces Lua module with the library name change. After that it
_SHOULD_ work out of the box but testing first is recommended. Also
note that the installations generated with the "luadist" commandline
use RPATH on most unix systems to avoid overriding
(DY)LD_LIBRARY_PATH.

> I understand the convenience of keeping different versions in different
> folders and I'll check if I can migrate to the same structure, but this is
> complicated by several factors: ZBS itself needs to use one set of libraries
> for its own launch and the structure needs to work on all three platforms as
> a package and also from a repository clone (which somewhat limits my
> options). I'm also using luajit, which doesn't fit nicely into 51 or 52
> categories (although it maintains 51 ABI compatibility).

The best solution would be to keep everything ZBS related in a
separate directory distributed using LuaDist itself. Since that is
unrealistic the next best thing would be to build the modules LuaDist
needs against the Lua version ZBS uses dirctly using CMake (should be
a farily simple script). Once LuaDist is built it can
install/uninstall/manage lua interpreters in separate directory as
needed, possibly on a per-project basis.

> I'd appreciate if you could consider renaming the DLL as it would provide
> LuaDist users with more flexibility in terms of directory structure and
> would eliminate the need for proxy DLLs. Thank you.

The support is there, its just the default behavior.

pd

[Paul K]

Hi Peter,

 

> Manual setting of the library will also work with the "luadist" tool
> also but it would require a slightly modified lua variant that
> produces Lua module with the library name change. After that it
> _SHOULD_ work out of the box but testing first is recommended.

 

> the next best thing would be to build the modules LuaDist
> needs against the Lua version ZBS uses dirctly using CMake (should be
> a farily simple script).

 

As far as I understand, this would require me to either run the build process on the client (which I wanted to avoid) or to maintain my own set of compiled libraries remotely (which I wanted to avoid as well, as I'd prefere to piggyback on the infrastructure you're building).

 

The only other option I see would be to use the current versions of modules, but to put "liblua.dll" proxy dll into version specific folders. This would require playing with PATH and possibly other settings, but may be doable.

 

Just to confirm, you don't plan to change the current naming for the interpreter libraries to add versions?

 

Another related question. How do you deal with shared dependencies between modules. For example, both luacrypto and luasec depend on openssl (which usually generates two libraries). Do you include them separately or do you statically link them into luasec and luacrypto libraries (possibly duplicating openssl code, which is quite large)?

 

Paul.

[Peter Drahoš]

Hi Paul,

On Wed, Oct 9, 2013 at 12:28 AM, Paul K <paulc...@gmail.com> wrote:
> Hi Peter,
>

>> Manual setting of the library will also work with the "luadist" tool
>> also but it would require a slightly modified lua variant that
>> produces Lua module with the library name change. After that it
>> _SHOULD_ work out of the box but testing first is recommended.
>
>> the next best thing would be to build the modules LuaDist
>> needs against the Lua version ZBS uses dirctly using CMake (should be
>> a farily simple script).
>
> As far as I understand, this would require me to either run the build
> process on the client (which I wanted to avoid) or to maintain my own set of
> compiled libraries remotely (which I wanted to avoid as well, as I'd prefere
> to piggyback on the infrastructure you're building).
>

Another option I mentioned is to use LuaDist to build everything
needed for ZBS as is done in current binaries on luadist.org where
"zbstudio" is available for windows only at the moment. That basically
does what you want to achieve but for now only for Windows. To make it
a cross-platform solution we just need to update wxWidgets and wxLua
to the latest needed versions. Most of the issues can be solved out
fairly quickly.

For example I would do the following:
1. Create a ZBS specific LuaDist deployment with only ZBS dependencies
and LuaDist installed in it, this will only run ZBS itself
2. Using LuaDist install and manage (probably with a GUI later) Lua
interpreters in separate subdirectories which will be used for
development
3.

> The only other option I see would be to use the current versions of modules,
> but to put "liblua.dll" proxy dll into version specific folders. This would
> require playing with PATH and possibly other settings, but may be doable.
>

This is doable and will work. I tried the same approach on different
project but I would not recommend it for this case.

> Just to confirm, you don't plan to change the current naming for the
> interpreter libraries to add versions?
>

No, but I can release different set of binaries that can have
versioned libraries if there is no alternative solution. This is
doable even now, the binaries would be marked as different
"Architecture" that LuaDist marks binaries with. Eg. Windows-x86
binaries would be as they are now but I could add WindowsV-x86
binaries that would be versioned. This would be completely transparent
to users.

> Another related question. How do you deal with shared dependencies between
> modules. For example, both luacrypto and luasec depend on openssl (which
> usually generates two libraries). Do you include them separately or do you
> statically link them into luasec and luacrypto libraries (possibly
> duplicating openssl code, which is quite large)?

In LuaDist all libraries are built as shared this means that luasec
and luascryto would share the openssl library. Both depend on it so
the dependency manager will pull in "openssl" when you install the
first module.

pd

[Andrew Starks]

--
You received this message because you are subscribed to the Google Groups "LuaDist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luadist+u...@googlegroups.com.
To post to this group, send email to lua...@googlegroups.com.
Visit this group at http://groups.google.com/group/luadist.
For more options, visit https://groups.google.com/groups/opt_out.

Hopefully not off topic:

I use ZBS and it plays nice with my compatible-with-nothing installation style. Importantly, it does not use "LUA_PATH" etc. and it somehow is able to work it all out, using my environment libraries as needed. 

Can LuaDist work this way? (Or does it already?)

If so, it strikes me that the architectures have some similarities, in that they both create a kind of virtual application machine, along the lines  of".app" folders on the Mac, etc. 

So l'll observe that, with a lot of work that I'm most likely failing to appreciate, you could imagine LuaDist spawning into a sort if application framework, where ZBS could be a good use case to see how that could would work.  LuaDist configures and installs dependancies, creates the dir structure, maybe even does a little registering / installing...

They are both great projects and it seems like there is some good interface points. 

-Andrew

[Peter Drahoš]

Hi Andrew,

On Wed, Oct 9, 2013 at 3:12 AM, Andrew Starks <andrew...@trms.com> wrote:
> Hopefully not off topic:
>
> I use ZBS and it plays nice with my compatible-with-nothing installation
> style. Importantly, it does not use "LUA_PATH" etc. and it somehow is able
> to work it all out, using my environment libraries as needed.
>
> Can LuaDist work this way? (Or does it already?)

Yes, it does already, and it works by extending the "!" notation used
on Windows. The "!" sign in LUA_PATH/LUA_CPATH is exchanges on runtime
with lua.exe location. LuaDist has a non-intrusive patch in Lua that
extends this behavior to all unix systems. Additionally it uses the
already mentioned RPATH functionality to link libraries and generates
executables with srlua. This ensures that you do not have to set any
PATH, LUA_PATH, LUA_CPATH or LD_LIBARY_PATH on any of the supported
systems.

> If so, it strikes me that the architectures have some similarities, in that
> they both create a kind of virtual application machine, along the lines
> of".app" folders on the Mac, etc.

That is one of the reasons why luadist works as it does. I like the
.app foldes in OSX and it really is a great way to distribute binary
software. This was intentional and it translates well to other
architectures.

> So l'll observe that, with a lot of work that I'm most likely failing to
> appreciate, you could imagine LuaDist spawning into a sort if application
> framework, where ZBS could be a good use case to see how that could would
> work. LuaDist configures and installs dependancies, creates the dir
> structure, maybe even does a little registering / installing...

LuaDist was intended to be a re-usable distribution for Lua based apps
from the start. So you are correct in your observation. That is why I
suggested and even Paul mentioned that ZBS would be able to pack the
application for the user based on what the script requires. We already
had a proof of concept that Michal Kottman did with this approach and
we know it works well.

> They are both great projects and it seems like there is some good interface
> points.

Indeed, ZBS changed a lot of things, it is mature and productive IDE
that Lua REALLY needed for development purposes. Now, with integrated
LuaDist it may become a complete Lua app development and distribution
framework. I do not think any Ruby/Python/Perl distributions are
offering anything similar to this. Once the integration and bugs are
polished out this might be game changing for Lua as standalone
language.

pd

[Andrew Starks]

On Tuesday, October 8, 2013, Peter Drahoš wrote:

> They are both great projects and it seems like there is some good interface
> points.

Indeed, ZBS changed a lot of things, it is mature and productive IDE
that Lua REALLY needed for development purposes. Now, with integrated
LuaDist it may become a complete Lua app development and distribution
framework. I do not think any Ruby/Python/Perl distributions are
offering anything similar to this. Once the integration and bugs are
polished out this might be game changing for Lua as standalone
language.

pd

I suspected that there was a lot of existing thinking that I hadn't paid enough attention to. Your last claim is a very bold one and I can't see anything inflated about it, at all. 

-Andrew 

[Paul K]

Hi Peter,

> For example I would do the following:
> 1. Create a ZBS specific LuaDist deployment with only ZBS dependencies and LuaDist installed in it, this will only run ZBS itself
> 2. Using LuaDist install and manage (probably with a GUI later) Lua interpreters in separate subdirectories which will be used for development
> 3.

Was there a third option that didn't go through?

These are good options, but I'd prefer to not make a parallel process
as I'd like the libraries that are installed through ZBS to work when
running without it as well. This means that I would prefer not to do
anything ZBS specific with the libs or the installation process.

>> The only other option I see would be to use the current versions of modules,
>> but to put "liblua.dll" proxy dll into version specific folders. This would
>> require playing with PATH and possibly other settings, but may be doable.

> This is doable and will work. I tried the same approach on different
> project but I would not recommend it for this case.

I still prefer this option as it would be the least invasive. What I'm
thinking about is a plugin that integrates with luadist and allows to
install its modules in two ways: either into ZBS folders (default) or
to the project folders. The project then can use those modules when
running from ZBS and the second option can be used for deployment and
packaging (and can be guided by the script that figures out module
dependencies, but it's a separate issue).

It would be useful if I could (at least for now) to avoid packaging
all the bootstrap dependencies and would just get and install them
when it's requested for the first time (I may need to have zlib if we
want to distribute dependencies as one archive). I need to play with
these options little bit to figure out what the most convenient and
least error prone option is.

> In LuaDist all libraries are built as shared this means that luasec
> and luascryto would share the openssl library. Both depend on it so
> the dependency manager will pull in "openssl" when you install the
> first module.

This is great; thank you.

Paul.

> --
> You received this message because you are subscribed to a topic in the Google Groups "LuaDist" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/luadist/iWh4VXT2QoI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to luadist+u...@googlegroups.com.

[Peter Drahoš]

Hello Paul,

On 9 Oct, 2013, at 7:02 , Paul K <paulc...@gmail.com> wrote:

> Hi Peter,
>

>> For example I would do the following:
>> 1. Create a ZBS specific LuaDist deployment with only ZBS dependencies and LuaDist installed in it, this will only run ZBS itself
>> 2. Using LuaDist install and manage (probably with a GUI later) Lua interpreters in separate subdirectories which will be used for development
>> 3.
>
> Was there a third option that didn't go through?
>
> These are good options, but I'd prefer to not make a parallel process
> as I'd like the libraries that are installed through ZBS to work when
> running without it as well. This means that I would prefer not to do
> anything ZBS specific with the libs or the installation process.

Was writing late at night, that was reserved for project specific installations of luadist. Eg. you start a new project and select an interpreter, once you write a require in your script it would automatically download the required modules for you. The end effect would be that you end up with a complete project that can be packed and distributed with only the needed dependencies. I did mention this to Andrew in the previous message. Its the same idea you mention later in your message.

>>> The only other option I see would be to use the current versions of modules,
>>> but to put "liblua.dll" proxy dll into version specific folders. This would
>>> require playing with PATH and possibly other settings, but may be doable.
>
>> This is doable and will work. I tried the same approach on different
>> project but I would not recommend it for this case.
>
> I still prefer this option as it would be the least invasive. What I'm
> thinking about is a plugin that integrates with luadist and allows to
> install its modules in two ways: either into ZBS folders (default) or
> to the project folders. The project then can use those modules when
> running from ZBS and the second option can be used for deployment and
> packaging (and can be guided by the script that figures out module
> dependencies, but it's a separate issue).
>
> It would be useful if I could (at least for now) to avoid packaging
> all the bootstrap dependencies and would just get and install them
> when it's requested for the first time (I may need to have zlib if we
> want to distribute dependencies as one archive). I need to play with
> these options little bit to figure out what the most convenient and
> least error prone option is.

This is easy. The only needed dependencies are lua, lua-git, zlib, lzlib, lfs and luasocket. Everything else can be pulled in from the net on use.

>> In LuaDist all libraries are built as shared this means that luasec
>> and luascryto would share the openssl library. Both depend on it so
>> the dependency manager will pull in "openssl" when you install the
>> first module.
>
> This is great; thank you.

pd

[Paul K]

Hi Peter,

> This is easy. The only needed dependencies are lua, lua-git, zlib, lzlib, lfs and luasocket. Everything else can be pulled in from the net on use.

A quick update on this. I tried to test the ability to run "install"
from ZBS, but ran into several issues I need your help with.

First of all, you are using "module", which doesn't work well with
sandboxed environments; I think this is because of 'module ("dist",
package.seeall)', but can't be sure. You can see the effect quite
easily; just run bin/lua.exe from ZBS (it's LuaJIT built with Lua52
compatibility option) and after running dist = require "dist" you will
see that dist.install value is nil.

Second, I still can't make LuaDist to download binaries:

LuaJIT 2.0.2 -- Copyright (C) 2005-2013 Mike Pall. http://luajit.org/
JIT: ON CMOV SSE2 SSE3 fold cse dce fwd dse narrow loop abc sink fuse
> package.path = package.path .. ";../lualibs/?/init.lua;../lualibs/?.lua"
> package.cpath = package.cpath .. ";clibs/?.dll"
> require "dist"
> conf = require "dist.config"
> conf.source = false
> dist.install("lpeg")
Finding out available versions of lpeg...
'lpeg-0.11' already in cache, skipping downloading (use '-cache=false' to force
download).
Finding out available versions of lua...
'lua-5.2.2' already in cache, skipping downloading (use '-cache=false' to force
download).
Error: command 'cmake' not found on system. See installation instructions at
https://github.com/LuaDist/Repository/wiki/Installation-of-System-Dependencies

I think this is because, as you said earlier "using an interpreter
that was not installed using LuaDist itself thus it does have
information about itself in share/luadist-git/dists directory that
stores "dist.info" files". The question is, how do I "fake" that
"dist.info" file to make it work with Lua interpreter I have? Also,
why does it try to load Lua 5.2? I'm running it from LuaJIT, but it's
Lua 5.1 binary compatible, so LuaDist should be using Lua 5.1 modules.
Maybe it detects the version incorrectly?

Paul.

[Paul K]

Hi Peter,

> This is easy. The only needed dependencies are lua, lua-git, zlib, lzlib, lfs and luasocket. Everything else can be pulled in from the net on use.

One more thing; I noticed the following in dist.git:

local command = "git clone " .. repository_url
....
local ok, err = sys.exec(command)

This particular command would require local "git" to be installed and
I can't expect that. Is this command being used in binary
installations?

Paul.

[Paul K]

Hi Peter,

 

> A quick update on this. I tried to test the ability to run "install" from ZBS, but ran into several issues I need your help with.

Do you have any suggestions on what I should try to do differently? Essentially, I'd like to be able to install LuaDist binary modules to the directories I specify by running from a non-LuaDist aware interpreter (I have all bootstrap modules and can configure it any way needed, but need help with dist.info files).

 

Paul.

 

[Peter Drahoš]

Hello Paul,
sorry I missed your previous message. LuaDist basically looks for ../share/luadist-git/dists/* folder that stores dist.info files for information about installed modules. It does not analyze the interpreter itself so it is not aware of anything not defined with dist.info. You basically need to manually enter dist.info files for each installed module into the /share/luadist-git/dists directory.

For luadist-git you can extract the info files from attachment into
[executable]../share/luadist-git/dists

In case your architecture is different from arch=“Darwin” type=“x86_64” (it needs to match settings in dist/config.lua) you may need to edit the info files accordingly.

This should provide luadist with all the info it needs and additional binary modules will simply be added without installation of the already present dependencies.

Hopefully this will be helpful. I still would prefer if LuaDist was installed into separate folder outside ZBS as it would make things much easier, however this should work.

pd

[mnicky]

Hi! Developer of the luadist command line utility here.

Since the time we started to use lua-git, the 'clone()' function of 'dist.git' is no longer used. So you don't have to worry :)

The only place where we still call the 'git' command line utility is when uploading the binary modules (built by us) to the LuaDist's main Github Repository, which you can see here: https://github.com/LuaDist/luadist-git/blob/master/dist/init.lua#L269 . But this 'luadist upload' functionality is intended only for luadist developers themselves...

There is no need to have git installed on the system for any developer using luadist as a build tool, not even for binary installations.

Marek Šrank

[Paul K]

Hi Marek,

> Since the time we started to use lua-git, the 'clone()' function of 'dist.git' is no longer used. So you don't have to worry :)

> There is no need to have git installed on the system for any developer using luadist as a build tool, not even for binary installations.

This is good; thank you for the clarification!

Paul.

[Paul K]

Hi Peter,

> Hopefully this will be helpful. I still would prefer if LuaDist was installed into separate folder outside ZBS as it would make things much easier, however this should work.

Thank you for the detailed suggestion. I did make some progress, even
though I'm not completely out of the woods yet.

I managed to come up with some configuration that allowed me to run
"dist.install('lpeg')", get lpeg DLL and be able to load it (after
copying manually where it needs to be). I tried with and without
dist_info files you sent and it worked in both cases, so, I'm not
going to use them for now. There are several things that can be
streamlined:

1. Is it possible to extract only files.Runtime files and nothing
else? Do I specify conf.components = {"Runtime"}? I tried that, but
still got all the files installed.

2. Is it possible to get the directory the module was installed into
(in this case it was "lpeg-0.10.2")? Even better, it is possible to
specify that the Runtime for the module should be installed into
directory "X" instead of "lpeg-0.10.2/lib/lua" (or at least skip
lpeg-0.10.2 part)?

3. It would be great to avoid using "module" and to allow local
variables to be used (as I described earlier, dist = require "dist"
doesn't work from a sandboxed environment).

4. How do I specify if I want Lua 5.1 or Lua 5.2 version of the module?

Thank you!

Paul.

[Peter Drahoš]

Hello Paul,

On 12 Oct, 2013, at 7:37 , Paul K <paulc...@gmail.com> wrote:

> Hi Peter,
>

>> Hopefully this will be helpful. I still would prefer if LuaDist was installed into separate folder outside ZBS as it would make things much easier, however this should work.
>
> Thank you for the detailed suggestion. I did make some progress, even
> though I'm not completely out of the woods yet.
>
> I managed to come up with some configuration that allowed me to run
> "dist.install('lpeg')", get lpeg DLL and be able to load it (after
> copying manually where it needs to be). I tried with and without
> dist_info files you sent and it worked in both cases, so, I'm not
> going to use them for now. There are several things that can be
> streamlined:
>
> 1. Is it possible to extract only files.Runtime files and nothing
> else? Do I specify conf.components = {"Runtime"}? I tried that, but
> still got all the files installed.

LuaDist will always download all the files of the package to a temporary folder. From there it will install files as specified by conf.components and entries in dist_info files table.

> 2. Is it possible to get the directory the module was installed into
> (in this case it was "lpeg-0.10.2")? Even better, it is possible to
> specify that the Runtime for the module should be installed into
> directory "X" instead of "lpeg-0.10.2/lib/lua" (or at least skip
> lpeg-0.10.2 part)?

Not for binary packages, for source packages you can specify INSTALL_[LIB|INCLUDE|BIN|LUA_LUAC] destinations. As for the lpeg-0.10.2 prefix, this is probably the temporary directory used by the luadist cli tool before it tries to install the files to a destination.

You can however specify the top level directory for the module to be deployed into using the install command e.g.:

dist = require 'dist'
conf = require 'dist.config'
conf.binary = true
conf.components = { 'Runtime' }
dist.install( ‘lpeg', '/Users/drahosp/Desktop/lpeg' )

>
> 3. It would be great to avoid using "module" and to allow local
> variables to be used (as I described earlier, dist = require "dist"
> doesn't work from a sandboxed environment).

I am already working on a updated version of the utility that will also support installation of “builtin” type rocks. It should be ready for my talk on LuaDist for Lua Workshop. It will be lua-5.1 5.2 and LuaJIT compatible without module.

> 4. How do I specify if I want Lua 5.1 or Lua 5.2 version of the module?

The dependency handler determines this based on the installed lua version or in case where the install destination does not contain any lua you can simply install the lua version you want first (or fake it using the dist_info files). For example this will install lua-5.1.5 first (or skip it if its installed) and progress to install lpeg that is lua-5.1.5 compatible.

dist.install( { 'lua-5.1.5', 'lpeg' }, '/Users/drahosp/Desktop/test' )

pd

> Thank you!
>
> Paul.
>
> On Fri, Oct 11, 2013 at 3:23 PM, Peter Drahoš <ra...@necroraiders.net> wrote:
>> On 11 Oct, 2013, at 19:59 , Paul K <paulc...@gmail.com> wrote:
>>
>>> Hi Peter,
>>>
>>>> A quick update on this. I tried to test the ability to run "install" from ZBS, but ran into several issues I need your help with.
>>> Do you have any suggestions on what I should try to do differently? Essentially, I'd like to be able to install LuaDist binary modules to the directories I specify by running from a non-LuaDist aware interpreter (I have all bootstrap modules and can configure it any way needed, but need help with dist.info files).
>>
>> Hello Paul,
>> sorry I missed your previous message. LuaDist basically looks for ../share/luadist-git/dists/* folder that stores dist.info files for information about installed modules. It does not analyze the interpreter itself so it is not aware of anything not defined with dist.info. You basically need to manually enter dist.info files for each installed module into the /share/luadist-git/dists directory.
>>
>> For luadist-git you can extract the info files from attachment into
>> [executable]../share/luadist-git/dists
>>
>> In case your architecture is different from arch=“Darwin” type=“x86_64” (it needs to match settings in dist/config.lua) you may need to edit the info files accordingly.
>>
>> This should provide luadist with all the info it needs and additional binary modules will simply be added without installation of the already present dependencies.
>>
>> Hopefully this will be helpful. I still would prefer if LuaDist was installed into separate folder outside ZBS as it would make things much easier, however this should work.
>>
>> pd
>>
>> --
>> You received this message because you are subscribed to a topic in the Google Groups "LuaDist" group.

>> To unsubscribe from this topic, visithttps://groups.google.com/d/topic/luadist/iWh4VXT2QoI/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email toluadist+...@googlegroups.com.

>> To post to this group, send email to lua...@googlegroups.com.
>> Visit this group at http://groups.google.com/group/luadist.
>> For more options, visit https://groups.google.com/groups/opt_out.
>

> --
> You received this message because you are subscribed to the Google Groups "LuaDist" group.
> To unsubscribe from this group and stop receiving emails from it, send an email toluadist+...@googlegroups.com.

[Paul K]

Hi Peter,

> You can however specify the top level directory for the module to be deployed into using the install command e.g.:

dist = require 'dist'
conf = require 'dist.config'
conf.binary = true
conf.components = { 'Runtime' }

dist.install( ‘lpeg', [[d:\deploy]] )

I tried this exact sequence and still get all the components (and not
just Runtime as I expected). I also get more than just lpeg; this is
the structure I see under "d:\deploy":

bin
tmp
tmp/lpeg-0.10.2
tmp/lpeg-0.10.2/lib
tmp/lpeg-0.10.2/lib/lua
tmp/lpeg-0.10.2/share
tmp/lpeg-0.10.2/share/lpeg
tmp/lpeg-0.10.2/share/lpeg/doc
tmp/lpeg-0.10.2/share/lpeg/test

I expected to only see "lib/lua". Is there a way to configure it to
get only lib/lua folder and nothing else (without tmp/lpeg-0.10.2)?

> I am already working on a updated version of the utility that will also support installation of
> “builtin” type rocks. It should be ready for my talk on LuaDist for Lua Workshop. It will be
> lua-5.1 5.2 and LuaJIT compatible without module.

This sounds good; thank you.

>> 4. How do I specify if I want Lua 5.1 or Lua 5.2 version of the module?
> The dependency handler determines this based on the installed lua version or in case
> where the install destination does not contain any lua you can simply install the lua
> version you want first (or fake it using the dist_info files). For example this will install lua-
> 5.1.5 first (or skip it if its installed) and progress to install lpeg that is lua-5.1.5 compatible.
> dist.install( { 'lua-5.1.5', 'lpeg' }, '/Users/drahosp/Desktop/test' )

What is the right way to "fake it using the dist_info files"? I'd
prefer not to install lua binaries for this if possible. Also, I tried
this, but I got wlua.exe installed and no liblua.dll. I also tried
lua-5.1 (without .5) and didn't get anything.

It would be great if there was a configuration option that would allow
me to specify Lua version and get a binary module for that version
without getting the Lua interpreter itself. Is it possible? Thank you!

Paul.

[Peter Drahoš]

Hello Paul,

On 14 Oct, 2013, at 6:39 , Paul K <paulc...@gmail.com> wrote:

> Hi Peter,
>
>> You can however specify the top level directory for the module to be deployed into using the install command e.g.:
>
> dist = require 'dist'
> conf = require 'dist.config'
> conf.binary = true
> conf.components = { 'Runtime' }
> dist.install( ‘lpeg', [[d:\deploy]] )
>
> I tried this exact sequence and still get all the components (and not
> just Runtime as I expected). I also get more than just lpeg; this is

> the structure I see under "d:\deploy”:

Seems to me there is an error when copying the files from the temporary directory to the destination. Try enclosing the dist.install function with a print function.

print( dist.install( ‘lpeg', [[d:\deploy]] ) )

It should return true or false, error message.

> bin
> tmp
> tmp/lpeg-0.10.2
> tmp/lpeg-0.10.2/lib
> tmp/lpeg-0.10.2/lib/lua
> tmp/lpeg-0.10.2/share
> tmp/lpeg-0.10.2/share/lpeg
> tmp/lpeg-0.10.2/share/lpeg/doc
> tmp/lpeg-0.10.2/share/lpeg/test
>
> I expected to only see "lib/lua". Is there a way to configure it to
> get only lib/lua folder and nothing else (without tmp/lpeg-0.10.2)?

Probably a bug as stated above. It should just install lib/lua but luadist will also add share/luadist-git/dists where it saves the info file for the installed module.

>> I am already working on a updated version of the utility that will also support installation of
>> “builtin” type rocks. It should be ready for my talk on LuaDist for Lua Workshop. It will be
>> lua-5.1 5.2 and LuaJIT compatible without module.
>
> This sounds good; thank you.
>
>>> 4. How do I specify if I want Lua 5.1 or Lua 5.2 version of the module?
>> The dependency handler determines this based on the installed lua version or in case
>> where the install destination does not contain any lua you can simply install the lua
>> version you want first (or fake it using the dist_info files). For example this will install lua-
>> 5.1.5 first (or skip it if its installed) and progress to install lpeg that is lua-5.1.5 compatible.
>> dist.install( { 'lua-5.1.5', 'lpeg' }, '/Users/drahosp/Desktop/test' )
>
> What is the right way to "fake it using the dist_info files"? I'd
> prefer not to install lua binaries for this if possible. Also, I tried
> this, but I got wlua.exe installed and no liblua.dll. I also tried
> lua-5.1 (without .5) and didn't get anything.

The luadist resolver handles lua as any other dependency. It is not specifically coded to handle Lua 5.2/5.1 or LuaJIT differently as LuaRocks is. This means that you can force it to install specific lua version as before. Or you can simply add its dist.info file info share/luadist-git/dists/lua-5.1.5/dist.info (which I shared in previous mail). This will fool luadist into thinking that lua is already present.

> It would be great if there was a configuration option that would allow
> me to specify Lua version and get a binary module for that version
> without getting the Lua interpreter itself. Is it possible? Thank you!

Due to the above reason this is not relevant for luadist. However an older version of the utility had a concept of “provides” table. Basically you could do.

...
conf.provides = { “lua-5.1.5” }

dist.install( ‘lpeg', [[d:\deploy]] )

The provides table would then be inserted into the dependency resolver as list of already installed modules. I am not sure if luadist-git already supports this but I can add it fairly quickly.

pd

[Paul K]

Hi Peter,

> Seems to me there is an error when copying the files from the temporary directory to the
> destination. Try enclosing the dist.install function with a print function.
> print( dist.install( ‘lpeg', [[d:\deploy]] ) )
> It should return true or false, error message.

I got it to work; it was returning this:

nil Error when deploying package 'lua-5.2.1': cannot copy file
'bin\lua.exe' to the directory 'd:\deploy\bin': Error when running the
command: copy /Y "d:\\deploy\\tmp\\lua-5.2.1\\bin\\lua.exe"
"d:\\deploy\\bin"

It turned out that this is caused by os.execute change; I'm running in
luajit with lua5.2 compatibility, which changed the result from
os.execute. You have a check for ok ~= 0, but the result can now be
"true", so it should probably be changed to this (which works in Lua
5.1 and Lua 5.2):

if ok ~= nil then
return true, "Sucessfully executed the command: " .. command
else
return nil, "Error when running the command: " .. command
end

> install specific lua version as before. Or you can simply add its dist.info file info
> share/luadist-git/dists/lua-5.1.5/dist.info (which I shared in previous mail). This will fool
> luadist into thinking that lua is already present.

I understand, it's an option, but I'm be looking for something similar
to your "conf.provides" functionality (to avoid anything in the
"share" folder).

Also, if I need to support both lua-5.1 and lua-5.2, I'll have
dist.info files for both, but how do I specify that I need to download
lpeg for Lua 5.1 or Lua 5.2?

> conf.provides = { “lua-5.1.5” }
> dist.install( ‘lpeg', [[d:\deploy]] )
> The provides table would then be inserted into the dependency resolver as list of already
> installed modules. I am not sure if luadist-git already supports this but I can add it fairly
> quickly.

This would be great unless you see a simpler way that doesn't involve
managing dist.info files. Thank you!

Paul.

[Paul K]

Hi Peter,

> how do I specify that I need to download lpeg for Lua 5.1 or Lua 5.2?

My apologies, I missed this from your earlier email:

> For example this will install lua-5.1.5 first (or skip it if its installed) and progress to install lpeg that is lua-5.1.5 compatible.

> dist.install( { 'lua-5.1.5', 'lpeg' }, '/Users/drahosp/Desktop/test' )

So, assuming I have both Lua interpreters, if I want a library for a
particular Lua version, I just need to specify Lua version (that
matches one of dist.info files) and luadist will download the
appropriate version; right?

Paul.

[Peter Drahoš]

Exactly. The proposed config.provides enhancement will also suffice, the dependency handler will download the correct binary if available.

pd

[Paul K]

Hi Peter,

 

> Exactly. The proposed config.provides enhancement will also suffice, the dependency handler will download the correct binary if available.

 

I got it all working down to a command I can run from ZBS, but it's still a bit unclear to me how to install modules for 5.1 and 5.2 into different folders. Also, I tried to specify 5.2.1 as you suggested and am getting this error:

 

> print(dist.install({"lua-5.2.1", "lpeg"}, [[d:\deploy]]))

 

Cannot install package 'lua-5.2.1': Package 'lua 5.2.1' needed, but installed at version '5.1.5'.

 

I think it's probably because I have both lua-5.1.5 and lua-5.2.1 in my share\luadist-git\dists\ (both include dist.info file from the archive you sent) and they are keyed with the same key ("lua"), so only one can be handled at any given time.

 

I tried to keep only one there, but event when I specify {"lua-5.2.1", "lpeg"}, I still get exactly the same version (as I got for Lua 5.1), and I expected one for Lua 5.2. What am I doing wrong? Thank you.

 

Paul.

[Paul K]

Hi Peter,

>> print(dist.install({"lua-5.2.1", "lpeg"}, [[d:\deploy]]))
> Cannot install package 'lua-5.2.1': Package 'lua 5.2.1' needed, but installed at version '5.1.5'.

I still can't figure out how to request a download for Lua 5.2 module.
I always get the same module as I get for Lua 5.1. I'm sure my
dist.info file is correct. Is it possible I'm doing something wrong?

I've made it all work on Windows for Lua 5.1. I got a similar set of
commands working to the one you have in luadist CLI.

I also tried to run this on OSX, but can't find any binary modules
there. Do you have anything binary available on OSX? Do you plan to?
It would be great if you could compile the modules with the following
settings:

-arch i386 -arch x86_64 -mmacosx-version-min=10.6 -dynamiclib
-undefined dynamic_lookup

This would allow using the same libraries with 32bit and 64bit apps
and would eliminate the need for liblua.dylib on OSX. I also
re-compiled git/core.dylib to avoid linking against libz.dylib (it's
now linked statically).

Paul.

[Paul K]

Hi Peter,

 

I think I mostly got the results I wanted on Windows and Linux, but I can't quite get the things working on OSX.

 

I need to be able to add "-undefined dynamic_lookup", which I can probably do using CMAKE_SHARED_MODULE_CREATE_C_FLAGS, but I also need to be able to disable adding -framework lua as it fails in combination with -isysroot and I don't want it anyway as I want to use dynamic lookup. Should this be done by setting CMAKE_FIND_FRAMEWORK to "NEVER"?

 

I only have access to "variables" in dist/config.lua, so I plan to use variables.CMAKE_FIND_FRAMEWORK = "NEVER". Is this sufficient, or should I use some other way to disable finding/adding frameworks? I still need to be able to add Lua libraires, but they are already installed by LuaDist, and this is exactly what I want to reference (just not frameworks). Thank you.

 

Paul.

[Peter Drahoš]

Hello Paul,
sorry for the late reply. The current binaries are not generated for
OSX are not generated with the C flags you need but I was planing to
re-generate those with similar settings for binary compatibility with
LuaJIT.

Your plan to use config.variables will work, its exactly there for
this purpose. Setting CMAKE_FIND_FRAMEWORK="NEVER" should work for
most modules unless they explicitly search for frameworks such as
OpenGL, for these module this setting will result in failed build.
Alternatively you can avoid that and point CMake directly to the Lua
you want by setting LUA_LIBRARIES and LUA_INCLUDE_DIRECTORIES which
when set will skip the whole FindLua.cmake fuinctionality.

pd

> --
> You received this message because you are subscribed to the Google Groups
> "LuaDist" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to luadist+u...@googlegroups.com.

[Paul K]

Hi Peter,

 

> sorry for the late reply. The current binaries are not generated for
> OSX are not generated with the C flags you need but I was planing to
> re-generate those with similar settings for binary compatibility with LuaJIT.

 

Sounds good; thank you. Are you also going to use the latest module versions? For example, compiled lpeg is at 0.10.2, but the latest is 0.12.

> Your plan to use config.variables will work, its exactly there for
> this purpose. Setting CMAKE_FIND_FRAMEWORK="NEVER" should work for
> most modules unless they explicitly search for frameworks such as
> OpenGL, for these module this setting will result in failed build.

 

Good point. I got it to work with "LAST" value, so this is how I'll keep it for now. Note that I had to upgrade CMake from 2.8.8 to 2.8.12 to get it to work. There was an issue with CMAKE_FIND_FRAMEWORK not being used from command line, which was fixed in 2.8.9: http://public.kitware.com/Bug/view.php?id=13261

 

I also set CMAKE_OSX_ARCHITECTURES to "i386 -arch x86_64" to generate universal binaries. This is a bit of a hack, but works for my case.

 

I still can't get CMAKE_SHARED_MODULE_CREATE_C_FLAGS to work. I set it to "-bundle -undefined dynamic_lookup", but it doesn't show up in the linker command. Should I use a different variable?

 

The last thing that doesn't work for me is binary download for lua 5.2. I specify {'lua-5.2.2', 'lpeg'}, but still get 'lpeg' for lua 5.1. Maybe there is no compiled module for lua 5.2, but it should then give some indication and not get the wrong module installed.

 

Paul.

 

[Paul K]

Hi Peter,

 

One more thing; any library I build links against lualib.dll and the dependency is shown (otool -L) as @executable_path/../lib/liblua.dylib.

 

I need to remove that dependency (as I want to build with -undefined dynamic_lookup). What is the way for me to make the linker NOT TO INCLUDE lualib as a dependency? I tried setting LUA_FOUND to "FALSE", but that obviously stops the process.

 

If that is not possible, how can I change the dependency itself? I tried setting CMAKE_INSTALL_RPATH to "@loader_rpath/..", but that didn't work. I also tried CMAKE_INSTALL_RPATH_USE_LINK_PATH="FALSE" and CMAKE_INSTALL_NAME_DIR="@loader_rpath/..", but it always sets it to the original @executable_path value.

 

How can I remove the dependency or change it? Thank you.

 

Paul.

[Marek Šrank]

Hi.

On Wednesday, October 30, 2013 7:24:02 PM UTC+1, Paul K wrote:

The last thing that doesn't work for me is binary download for lua 5.2. I specify {'lua-5.2.2', 'lpeg'}, but still get 'lpeg' for lua 5.1. Maybe there is no compiled module for lua 5.2, but it should then give some indication and not get the wrong module installed.

 

If the binary isn't available it shouldn't install a wrong module. E.g.:

$ bin/luadist _test install lua-5.2.2 lpeg -source=false

Finding out available versions of lua...

Cannot install package 'lua-5.2.2': Error when resolving dependencies: Error when retrieving the info about 'lua': Error fetching package 'lua-5.2.2' from 'git://github.com/LuaDist/lua.git' to '/home/mnicky/tmp/luadist/LuaDist/_test2/tmp': Binary version of module not available and using source modules disabled.

(-I tried this on Debian-x86_64 and the binaries of lua-5.2.2 are built only for Darwin-x86_64 and Debian-i686, hence the message)

Or do you mean some other use case?

Btw, to see for which platform have the binaries been built, you can use:

$ git ls-remote --tags git://github.com/LuaDist/lua.git
068edf316ea60ba0d32c4e50af3b9e1711876aa2 refs/tags/5.1.4
0937bc974bbf0c062a222ffa49fa20c7cc6ab357 refs/tags/5.1.5
c695b35a93242602fc01919f06cc82fd1e518953 refs/tags/5.1.5-Darwin-x86_64
5fb6db8e98dfd811e7f70feed3cd812f8b564141 refs/tags/5.1.5-Debian-i686
34b94e8354266f71e68b65d854d9acc5eeb3e67d refs/tags/5.1.5-Ubuntu-x86_64
580afc45474aedbd07e394514a2daafec132b456 refs/tags/5.1.5-Windows-x86
0c085822a7055ec20aefe7582abe875a7af59f9e refs/tags/5.2
59b3d7cabfe51c965a22bb6eb1c58c19613a27b3 refs/tags/5.2.1
5ca33d748d4dcc21b1a26e67912ea60f4b041240 refs/tags/5.2.1-Darwin-x86_64
2f85d2a037aa938e5b93e9dd8e0f76295a4c887f refs/tags/5.2.1-Ubuntu-x86_64
565334ce9991e43ae1a11c6b618875499d98ac40 refs/tags/5.2.1-Windows-x86
ceb92f3d955faa317ce0703e788dcbdaeef81407 refs/tags/5.2.2
2bb95aaa40935f7d3830a8d50ef5cde8ec2481f2 refs/tags/5.2.2-Darwin-x86_64
60f9dd9951a0588ba8d9deec05a4260585019703 refs/tags/5.2.2-Debian-i686

(-the tags without platform specification are of course the source ones)

...or from lua, by using luadist:
tbl, err = dist.git.get_remote_tags("git://github.com/LuaDist/lua.git")

[Paul K]

Hi Marek,

> If the binary isn't available it shouldn't install a wrong module. E.g.:
> $ bin/luadist _test install lua-5.2.2 lpeg -source=false

It's a slightly different case; it didn't install the module because
it's not available as binary for that arch/type. Try to find a module
that is available for Lua 5.1, but is not available for Lua 5.2 as
binary; one example is lpeg on Windows as it's available for Lua 5.1
and not Lua 5.2. Then try to install it with "'lua-5.2.2', 'lpeg',
-source=false". I get this:

Finding out available versions of lpeg...
- trying another candidate due to: Error when resolving dependencies:
Error when retrieving the info about 'lpeg': Error fetching package
'lpeg-0.12' from 'git://github.com/LuaDist/lpeg.git' to
'D:\users\paul\luadist\5.2\tmp': Binary version of module not

available and using source modules disabled.

- trying another candidate due to: Error when resolving dependencies:
Error when retrieving the info about 'lpeg': Error fetching package
'lpeg-0.11' from 'git://github.com/LuaDist/lpeg.git' to
'D:\users\paul\luadist\5.2\tmp': Binary version of module not

available and using source modules disabled.

Getting lpeg-0.10.2 (binary)...

But this installs the same (Lua 5.1) module for me.

> Btw, to see for which platform have the binaries been built, you can use:
> $ git ls-remote --tags git://github.com/LuaDist/lua.git

This is very useful; thank you. BTW, it would be good to have it in
the documentation somewhere as I spent quite a bit of time searching
for supported platforms on Linux.

Why did you decide to have Debian/Ubuntu/whatever instead of Linux? Do
the modules tagged with those platforms only work on those platforms?
I compile ZBS on Ubuntu 10.4 and get it to run on various flavors of
Linux (including all those you listed).

One more question. Any idea how I can disable linking against
liblua.dylib on OSX and add C flags as I need "-bundle -undefined
dynamic_lookup"? Thanks!

Paul.

[Paul K]

Hi Peter, Marek,

 

Do you have any suggestions on what I may be doing wrong with binary installs for Lua 5.2 (as described in my email below) and with disabling lualib.dylib dependency on OSX? Do you have any way to disable the dependency on OSX and to set -undefined dynamic_lookup? I tried to do it using CMAKE_SHARED_MODULE_CREATE_C_FLAGS, but the value I provide is not being used.

 

Paul.