Current short term plans on LuaDist side

[Peter Drahoš]

Since the discussion has stalled I decided to write up my current plans to move things forward. Hopefully Hisham will bring some news into the project name thread in the mean time.

There are two areas I plan to focus my attention. The first and most important task is the setup and management of the CI server which Andrew S. donated last week. Andrew S. has been helping me set up the CI server so that we can now directly test all modules on multiple platforms. So far I have done some initial burn tests to ensure the setup is stable and works as intended, after some crashes and lockups I'm confident that the machine will be reliable. While there is still some work to be done on the integration with Jenkins and GitHub the current list of build targets will be as follows:

present:

MinGW 4.8 32bit on Windows XP

MinGW 4.8 64bit on Windows 7

Clang 32bit + MinGW 4.7.1 on Windows 7 (experimental)

MSVC 2012 Decktop Edition on Windows 7

XCode Clang on OSX 10.8

XCode Clang on OSX 10.7

GCC 64bit on Ubuntu 12.10 64bit

Clang 64bit on Ubuntu 12.10 64bit

planned:

GCC 32bit on Mint 14 32bit

Clang 32bit on Mint 14 32bit

GCC 64bit on FreeBSD

GCC on ARM Linux (cross compile probably)

The CI process will be based on running 'luadist make' on the checked out repositories. So the primary challenge is to make sure LuaDist is running on all platforms. Currently only the MSVC target needs attention and all other targets can build LuaDist just fine. CI builds on the core modules will run daily, all other modules will only run based on git activity.

The second step is to focus on integration between LuaDist and LuaRocks. Here I plan to adopt the rockspec format once it is extended to support features such as maintainers, provides and conflicts which is essential for LuaDist. Once a unified format is agreed on LuaDist will install all information about modules into package.path, probably into the package.info.[name] namespace. This will allow LuaRocks and LuaDist to look for each others packages even if require is overloaded into LuaRocks specific paths. After this it should be possible to cleanly provide CI services for LuaRocks too.

We still need to agree on how to store tests and documentation and how to create and deliver reports from the CI process.

pd

[Ryan Pusztai]

Hi Peter,

Thanks for this update. It seems like your priorities are correct and in the right direction. You and your contributions are very appreciated. Thanks.

--

Regards,
Ryan

[Andrew Starks]

Very much agreed.

Also, I do regret my last email regarding my opinions on naming.

I while I do disagree with the general consensus on naming, in doing
so I brought my emotions along with that disagreement, which led to me
being disagreeable. Please accept my apologies and I hope that I won't
need to lean on everyone's patience in this way, in the future. :)

So, I'd like to suggest that name and branding decisions need to be
settled, as well. Once that's done, then much work can begin on CSS
and web site magic.

-Andrew

[marbux]

Peter, on a unified format for LuaDist and LuaRocks, do you think that
might need further metadata for purposes of classifying modules in the
consolidated documentation?

I ask because you (IIRC) mentioned at some point that the CI server
could automatically update the module documentation stored on the web
site and because a separate download of all documentation is
contemplated.

My thought being that an alphanumeric sorted list of module names,
each with a short one-sentence or so description, would not seem to be
an optimal organization of consolidated documentation.

Best regards,

Paul

[Peter Drahoš]

On Wed, Mar 27, 2013 at 3:48 PM, marbux <mar...@gmail.com> wrote:

Peter, on a unified format for LuaDist and LuaRocks, do you think that
might need further metadata for purposes of classifying modules in the
consolidated documentation?

LuaDist uses short string for description of the package. LuaRocks uses two, a short and long version. None provide keywords which would be useful. Thats all the info you can gather from the info/rockspec files.

Additionally LuaRocks does not install any documentation at all, it usually installs just the lua/c modules. LuaDist by default installs to share/[module]/doc and keeps track of the files when they are installed in the info file. That information can be used but the files vary in type and qulity, generally we would be better off if the documentation was in lua module form.

I ask because you (IIRC) mentioned at some point that the CI server
could automatically update the module documentation stored on the web
site and because a separate download of all documentation is
contemplated.

Yes, it could do this. I can make the CI process to run ldoc/whatever and push the output to git/githubpages on any change. So the website could just link to it and we would only need to maintain documentation per module. Everything else could be pushed out and re-generated automatically.

My thought being that an alphanumeric sorted list of module names,
each with a short one-sentence or so description, would not seem to be
an optimal organization of consolidated documentation.

This can be done now. It is related to the package manifest issue. The short descriptions are also available on every git repository in LuaDist which can be extracted through github api with a bit of bash-foo and curl. The better solution is to simply have a generated manifest of all packages that the CI server keeps up to date*.

pd

* I will try to generate it as soon as the CI server is set up properly.

[Paul K]

Hi Peter,

 

> There are two areas I plan to focus my attention. The first and most important task is the setup and management of the CI > server which Andrew S. donated last week. Andrew S. has been helping me set up the CI server so that we can now

 

This looks good. I cover a variety of platforms and it seems like I may have a similar setup to build ZeroBrane Studio binaries. ZBS requires wxlua, luasocket, lua, and winapi (on Windows); everything is built from sources using MinGW.

- For Windows I only build 32bit binaries, but they run on anything starting from XP up to Windows 8.

- For OSX I build 32bit binaries on OSX 10.7.4, but I use 10.6 SDK, so the binaries run on 10.6+

- For Linux I build on Ubuntu 10.04 (both 32bit and 64bit) and has tested on Ubuntu 12.04, Fedora 18, ArchLinux (in various configurations) without isues. I'm not sure if these binaries would run on Mint (probably) or FreeBSD.

 

I run Linux in VMWare Fusion on OSX; the build process is automated, but the VM management is not. I plan to fully automate the build process, so that I could start a VM, run a build, save the results, and bring a VM down. Do you have script that do something similar?

 

Paul.

[Peter Drahoš]

On Thu, Mar 28, 2013 at 5:50 AM, Paul K <paulc...@gmail.com> wrote:

Hi Peter,

 

> There are two areas I plan to focus my attention. The first and most important task is the setup and management of the CI > server which Andrew S. donated last week. Andrew S. has been helping me set up the CI server so that we can now

 

This looks good. I cover a variety of platforms and it seems like I may have a similar setup to build ZeroBrane Studio binaries. ZBS requires wxlua, luasocket, lua, and winapi (on Windows); everything is built from sources using MinGW.

May I ask what specific version are you using ?

 

- For Windows I only build 32bit binaries, but they run on anything starting from XP up to Windows 8.

Windows has excellent binary compatibility so if it runs on Windows XP it will probably run on newer versions. I plan to use the 2012 Desktop Edition MSVC also to test packages for people that are stuck using it but I will probably not release any binaries for it.

 

- For OSX I build 32bit binaries on OSX 10.7.4, but I use 10.6 SDK, so the binaries run on 10.6+

That is reasonable, I plan to support 10.6+ also.

 

- For Linux I build on Ubuntu 10.04 (both 32bit and 64bit) and has tested on Ubuntu 12.04, Fedora 18, ArchLinux (in various configurations) without isues. I'm not sure if these binaries would run on Mint (probably) or FreeBSD.

There are small linking issues on FreeBSD and I also need to fix RPath on it. Usually everything runs out of the box ... surprisingly.

 

 

I run Linux in VMWare Fusion on OSX; the build process is automated, but the VM management is not. I plan to fully automate the build process, so that I could start a VM, run a build, save the results, and bring a VM down. Do you have script that do something similar?

Not yet, I plan to idle the VMs for the time being (CPU and Memory utilization is OK for now). There are plugins for Jenkins that will start the VMs on demand [1][2].

Since the ZeroBraneStudio is central to the project I can provide the CI service for it if you want me to. However I will do this anyway for the fork in LuaDist so alternatively you can use that for distribution later.

pd

[1] https://wiki.jenkins-ci.org/display/JENKINS/VMware+plugin

[2] https://wiki.jenkins-ci.org/display/JENKINS/VirtualBox+Plugin

[marbux]

On Wed, Mar 27, 2013 at 9:50 PM, Paul K <paulc...@gmail.com> wrote:

> - For Linux I build on Ubuntu 10.04 (both 32bit and 64bit) and has tested on
> Ubuntu 12.04, Fedora 18, ArchLinux (in various configurations) without
> isues. I'm not sure if these binaries would run on Mint (probably) or
> FreeBSD.

Likely there would be no problems on Mint. We've stopped doing
separate builds of NoteCase Pro for Mint because extensive user
testing using the Ubuntu 12.04 build produced nary a single problem on
Mint 12-14. Mint is an Ubuntu derivative.

Best regards,

Paul

[Paul K]

Hi Paul,

> Likely there would be no problems on Mint. We've stopped doing
> separate builds of NoteCase Pro for Mint because extensive user
> testing using the Ubuntu 12.04 build produced nary a single problem on
> Mint 12-14. Mint is an Ubuntu derivative.

Thank you for confirming this. It makes it easier for me: one more
platform I don't need to run a VM for.

Paul.

> --
> You received this message because you are subscribed to a topic in the Google Groups "luaforwindows" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/luaforwindows/h2DmPKFjbxs/unsubscribe?hl=en.
> To unsubscribe from this group and all its topics, send an email to luaforwindow...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Paul K]

Hi Peter,

>> This looks good. I cover a variety of platforms and it seems like I may
>> have a similar setup to build ZeroBrane Studio binaries. ZBS requires wxlua,
>> luasocket, lua, and winapi (on Windows); everything is built from sources
>> using MinGW.
>
> May I ask what specific version are you using ?

I'm not sure what version you are asking about. wxlua: 2.8.13 (svn
trunk), wxwidgets: 2.9.5 (svn trunk), lua: 5.1.5, luasocket: 2.0.2,
MinGW-make: 3.82, gcc: 4.6.2.

> Not yet, I plan to idle the VMs for the time being (CPU and Memory
> utilization is OK for now). There are plugins for Jenkins that will start
> the VMs on demand [1][2].

Thank you; I'll take a look at it.

> Since the ZeroBraneStudio is central to the project I can provide the CI
> service for it if you want me to. However I will do this anyway for the fork
> in LuaDist so alternatively you can use that for distribution later.

I'll continue to build it myself for now for ZBS as I'm still making
regular changes to the binaries (because wxwidgets is under active
development and I track several issues that need to be fixed). I'm not
sure what version of wxwidgets and wxlua you want to bundle, but I
plan to stay with the latest one as much as possible and will probably
keep updating it every release until we get to more stable versions.

Just one more thing to clarify: I'm using a "normal" version of wxlua,
but there is still one patch that I need to apply to support live
coding. John L hasn't integrated it yet; here is the last discussion
on this: http://sourceforge.net/mailarchive/forum.php?thread_name=CADUw5q1a7nu4mepgAD%3Dp%2BVG6_OoCs8h-rE4eNBTbFu_6cmQ24g%40mail.gmail.com&forum_name=wxlua-users

Paul.

[Hisham]

On 27 March 2013 08:30, Peter Drahoš <dra...@gmail.com> wrote:
> The second step is to focus on integration between LuaDist and LuaRocks.
> Here I plan to adopt the rockspec format once it is extended to support
> features such as maintainers, provides and conflicts which is essential for
> LuaDist.

Good to know. Do you have a specification/documentation of the
dist.info format?

> Once a unified format is agreed on LuaDist will install all
> information about modules into package.path, probably into the
> package.info.[name] namespace.

I advise against using package.* because this namespace is used by the
standard Lua libraries. Many people frown upon modules that touch on
the standard namespaces (eg, extending table.*, etc.)

The idea of installing rockspec information inside the Lua modules
tree is intriguing. It strikes me as a clever idea... however, at the
same time it makes me wary that it might be an abuse of the modules
tree, especially since those files do not qualify as proper modules
(and my initial reaction is that making every rockspec file into a
`return { ... }` just so it doubles as a module wouldn't be the
cleanest idea). So, mixed feelings about that, at this point.

In any case, a new rockspec format is in the plans for the next major
version, we should continue discussing it at the luarocks mailing
list.

-- Hisham
http://hisham.hm/

[Peter Drahoš]

On Fri, Mar 29, 2013 at 10:16 PM, Hisham <h...@hisham.hm> wrote:

On 27 March 2013 08:30, Peter Drahoš <dra...@gmail.com> wrote:
> The second step is to focus on integration between LuaDist and LuaRocks.
> Here I plan to adopt the rockspec format once it is extended to support
> features such as maintainers, provides and conflicts which is essential for
> LuaDist.

Good to know. Do you have a specification/documentation of the
dist.info format?

Sure, it is available on the wiki[1].

  

> Once a unified format is agreed on LuaDist will install all
> information about modules into package.path, probably into the
> package.info.[name] namespace.

I advise against using package.* because this namespace is used by the
standard Lua libraries. Many people frown upon modules that touch on
the standard namespaces (eg, extending table.*, etc.)

I agree, hopefully we can agree on a different namespace.

 

The idea of installing rockspec information inside the Lua modules
tree is intriguing. It strikes me as a clever idea... however, at the
same time it makes me wary that it might be an abuse of the modules
tree, especially since those files do not qualify as proper modules
(and my initial reaction is that making every rockspec file into a
`return { ... }` just so it doubles as a module wouldn't be the
cleanest idea). So, mixed feelings about that, at this point.

There are some drawbacks but it offers some significant benefits in return. I think this would solve some of the issues we have, for example packages installed by the host package manager could be integrated seamlessly as they would provide their own modules with information that LuaRocks/LuaDist can use. That feature alone would be worth it.

  

In any case, a new rockspec format is in the plans for the next major
version, we should continue discussing it at the luarocks mailing
list.

 Sure, I'm watching for any relevant discussions already.

pd

 

[1] https://github.com/LuaDist/Repository/wiki/LuaDist:-Package-Structure#the-distinfo-file