Premake based package manager

[Paul Visscher]

We have been very busy to create a premake based package manager solution. For complete documentation go to http://zpm.zefiros.eu/ and the repository is at https://github.com/Zefiros-Software/ZPM . Please note that the project is still in beta, and is in active development. However we feel it is ready enough to be used.

In short our requirements were:

• Easy, cross-platform package manager for modules, libraries and assets
  
• Easy updating of premake5
  
• Integrates with premake5
• Both for using and publishing packages.
• All Git repositories supported, even private repositories.
• For packages, premake5 modules, and assets.
• Assets may be hosted using Git LFS, and from urls.
• Optionally separating the ZPM package and build files.
• Git tags for versioning.
  

ZPM uses json files to present library and build definitions. However all the work is done in premake5, and projects consuming libraries can fully define how libraries are used. The json commands are extensible.

We would appreciate feedback, pull-requests and bug reports!

Cheers,

Paul Visscher

[starkos]

Hello Paul,

Just getting back from traveling and beginning to get caught up. Based on the website, etc. this looks pretty sweet. Would be curious to hear what the Blizzard folks think?

-st.

[Tom van Dijck]

I only took a brief look at it... It's exciting to see a community open source effort for sure...

That said, while I have not tried things out, I did have a few thoughts:

1. The first thing that I was looking for was an index of available packages, so I could write a premake script to try them out.

I could not find that anywhere on the website.. maybe I misunderstand the idea and an index just isn't available, but that makes it fairly difficult to 'find' available packages.

Basically without an index, we're going to have 5 different forks of zlib with ZPM scripts, all mentioning 1.2.8 as their version number. That is in my opinion just wrong, and single authoritative index like nuget.org is preferable in my opinion.

For example:

https://www.nuget.org/packages

2. the _package.json and _build.json... I like the idea of json... 

I very much dislike the _ filenames.... a '.' (dot), no prefix, or support both those two options would be preferable in my opinion.

same kind of goes for the _registries.json, etc... basically there is a 'standard' for 'hidden' files, I would just stick to that.

3. I don't understand what the _build.json is for.. in my opinion a premake5.lua script is what describes how the package is build, so I don't understand why a _build.json is needed?

The documentation almost looks like the _build.json is a json format of a premake5.lua script. I really don't want to learn a secondary scripting api to define premake projects.

The example here:

http://zpm.zefiros.eu/packages/commands/premake5/#example

could and probably should just be written as a premake5.lua script

    project 'example'

       architecture 'x86_64'

       endian 'little'

       links { 'mpi', 'pthreads' }

4. The requirement to have a particular folder structure... /extern, /assets etc..

I would like to see that not be a requirement, but rather something defined in the package configuration.

For example, in our system we allow you to setup a local package cache somewhere in a shared location...

our buildfarm builds 50 branches, but they all share the same location for external packages...

In the ZPM system, it looks like all external packages would get downloaded into the 'source tree', which would be expensive in our case (2GB+ of packages per branch).

something as simple as 

{

    "package_root" : "c:\package_cache",

    "asset_root"  : "c:\asset_cache",

}

would be really great...

I think most of these things are probably easily 'fixed' if so desired. And I think the rest looks really promising, although fairly similar to 

https://github.com/mversluys/premake-ghp

http://www.premake-ghp.com/

In all honesty, I love efforts like this, but I don't think it be trivial for us to use them as none of these systems provide what we need.

I wish we could all just talk about what to make, and how to make it... before we have 20 competing products... 

[starkos]

Still looking, but agree with Tom's points so far, especially about the use of JSON over Lua.

> I wish we could all just talk about what to make, and how to make it... before we have 20 competing products... 

I agree, but open source is chaos. Best bet is probably for you guys (Blizzard) to throw your weight behind one of the current solutions and lean on the author to get what you need. 

[Paul Visscher]

Thanks for the feedback! Please keep in mind that this was a solution designed for our needs in the first place, and we hope to extend the system in the following months
so it can be used in general.

That said:
1) https://github.com/Zefiros-Software/Manifest servers an index of available packages. In august we will launch a site around it so it is searchable etc. I think this will solve the issue.
2) I agree, and we will update it as soon as possible to use "." instead.
3) The _build.json (will probably be renamed to .export.json) is used to define how other projects should use the project. We choose json over lua to prevent
execution of random lua scripts on build servers as a security issue. However I agree that this is rather tedious to write, and we will work on a parser that
converts the normal lua format premake uses, to the json format. This way we can write a .build.lua, that serves the same purpose, and is executable in a local premake5.lua script to avoid duplication.
4) These settings are already overwritable in a local configuration file. However I still need to add support for absolute filepaths, and will update ZPM accordingly as soon as possible. Currently ZPM clones repositories in the temporary folder, which is not configurable at the moment. We will also allow ZPM to install the cache some place else.

I hope you can share some more information with us, so we can work to a system that can be applied generaly (just like @starkos suggests)! :)
I am currently on vacation until the first week of august, so I won't be responding as much as I would like during this period.

Keep the suggestions comming!

Cheers,

Paul Visscher

[starkos]

> We choose json over lua to prevent execution of random lua scripts on build servers as a security issue

But that's how Premake works: by running Lua scripts. In this regard, I much prefer the Premake-GHP approach.

[Tom van Dijck]

Yeah, I think the concern about security is commendable, but can probably be solved otherwise... Lua already runs in a sandboxed environment, as in, it is already a VM, and hence every function is captured before it goes into actual executing code.

While I'm sure there is all kinds of options for exploits, most of those exploits can probably be fixed, or at least caught before execution.

For example, obviously the os.execute is something you probably want to prevent if security is a concern.

If you are going to write a 'parser' to parse the lua to make a json to then create a lua script for premake... you might as well just stop with the parser and say 'hey, this script contains code we don't allow on this server'.

Other the os.execute there isn't much in lua you can do to completely hack the system, by default we don't load any binary lua bytecode, so the known bytecode exploits don't apply..

You can read and write to files... those could easily be sandboxed to only allow writes to files within the source tree.

Furthermore, nobody should be running premake in a root/admin environment...

Anyway, back to the functional part however.

1. I look forward to the index website...

3. I guess that is discussed above.

4. cool...

> I hope you can share some more information with us, so we can work to a system that can be applied generaly (just like @starkos suggests)! :)

All our code except for our server is on github. 

https://github.com/blizzard/premake-blizzard

I can probably post a few examples as well, although without a server that wouldn't really work..

Our server implementation is really nothing special though... just an http server that responds to a few api calls that our premake module makes.

http://packageserver/archives?name=zlib&version=1.2.8

which response with a json table 

for example:

["noarch.zip", "win32-i386-vc140-debug.zip"]

[Tom van Dijck]

grr... wasn't done yet... ok...

    http://packageserver/archives?name=zlib&version=1.2.8

response:

    ["noarch.zip", "win32-i386-vc140-debug.zip"]

based on that we download the 

    http://packageserver/zlib/1.2.8/noarch.zip

unzip it, and execute the "premake5-package.lua" script that is contained within if a package has it.

there is a few other API's that are optional.

    http://packageserver/link?name=zlib&version=1.2.8&variant=noarch

response:

       link to where to download the zip file.

    http://packageserver/aliases?name=zlib

response:

      ["Z Library"]

a list of aliases 'zlib' can be called as well, this was mostly to support a legacy system we had where some packages where called "Blizzard Standard Template Library", but was later named "blz"... so to not break dependencies, we added support for aliases. Not strictly a requirement for a public open source service.

Then obviously the website just shows what packages are available...

[starkos]

> Yeah, I think the concern about security is commendable, but can probably be solved otherwise

Tom's suggestion to create a "safe mode" for Premake is a good idea, and probably would not be too hard to implement. 

For me, having to support a second JSON format feels like a non-starter.

> I look forward to the index website...

Likewise. Premake-GHP has a nice one, even if there isn't much on it yet…my favorite part of that effort.

> The _build.json…is used to define how other projects should use the project

Ugh…it would not be hard to write a decent usage/uses system but I just can't find the time…drives me nuts…

> All our code except for our server is on github. 

I like that this works without a server and can hit a bare repository, though I understand that doesn't do anything for discoverability.

I personally think the official "server" should just be a Lua table of known packages fetched from a well-known HTTPS URL (i.e. GitHub), with all of the query/processing logic done on the client side. Changes can be done with PRs like anything else (which should be trivial to approve, so fast). I believe this would be the lowest maintenance (no server to upkeep) and most flexible (people can innovate their own approaches to using/presenting the data, including a Blizzard or Premake-GHP style website).