Snabb Switch packaging and distribution, or, Torch'ification?

[Luke Gorrie]

Cross-posting to snabb-devel and torch7 for reasons that will become apparent:

On 28 January 2015 at 11:17, Alexander Gall <alexand...@gmail.com> wrote:

Don't bother about that one. I have changed the VPN modules substantially over the past months but I will probably not push to the Snabb core due to the size constraint.

Our little project is growing up! :-)

 

In fact, I'll request the apps/vpn directory to be removed entirely at some point. The packaging of non-core modules is still an open question (and I still owe Luke feedback on it).

There is exciting new information for us to consider: Torch7 (http://torch.ch).

Torch7 is basically Snabb Switch (http://github.com/SnabbCo/snabbswitch/) but:

- Has already been developed for many years

- Targets scientific computing and machine learning (not networking)

- Has grown a vibrant and decentralised software ecosystem

- Heavily used by high-profile industry groups (Facebook, Google, etc)

So: what lessons can we learn from their experience? Should we adopt their packaging and distribution model, for example?

Here is their main community ecosystem page:

https://github.com/torch/torch7/wiki/Cheatsheet

Their build/packaging/deployment is based on Lua Rocks:

http://luarocks.org

And here is the mailing list thread from when they adopted their current build system (~ 18 months ago):

https://groups.google.com/forum/#!topic/torch7/_b7sX3u6X0c

Facebook recently recruited the main author and Facebook Engineering are now maintaining a bunch of open source LuaJIT tools (e.g. a source level debugger):

https://www.facebook.com/Engineering/posts/10152495827632200

(Have we found some kindred spirits? what does everybody think? any Torch7 hackers want to give us some tips?)

Cheers!

-Luke

[Daurnimator]

On 28 January 2015 at 07:49, Luke Gorrie <lu...@snabb.co> wrote:
> Their build/packaging/deployment is based on Lua Rocks:
> http://luarocks.org
>
> And here is the mailing list thread from when they adopted their current
> build system (~ 18 months ago):
> https://groups.google.com/forum/#!topic/torch7/_b7sX3u6X0c

You don't even need to do this any more!

You can create your own manifest up on moonrocks:
https://rocks.moonscript.org/new-manifest
and just use normal luarocks :)

[smth chntla]

Luke,

We've chosen luarocks because it was the least bad apple ;)

Since the moonscript guys took over the maintenance of luarocks it's gotten a lot better, but still has it's slight quirks. https://rocks.moonscript.org/

We love the decentralized distribution, but we have started packaging a core set of packages together, calling them the "distro" https://github.com/torch/distro . The distro is pretty much installed via luarocks, so nothing special there.

Similar to Snabbswitch, torch has a presence in the embedded space, with torch-android and torch-iOS, and people routinely run it on ubuntu-ARM to hook it up with FPGA co-processors.

Looking forward to having some cross-collaboration between projects.

Cheers,

S

[smth chntla]

We also do not do any special version of luarocks anymore. We just ship standard luarocks, but we add our rocks-server by default to the moonscript rocks server.

[Luke Gorrie]

Hi Smth,

On 31 January 2015 at 01:26, smth chntla via torch7 <torch7+APn2wQeWkU_KsmAv8TIk9zu8L...@googlegroups.com> wrote:

We've chosen luarocks because it was the least bad apple ;)

Interesting :).

I see a bunch of potential ways we could use luarocks in Snabb Switch:

1. To pull in development tools (debuggers, code analyzers, etc).

2. To pull in libraries for use in applications (ljsyscall, luafilesystem, etc).

3. To publish and share Snabb Switch extensions (on moonrocks)

4. To publish Snabb Switch itself as a rock.

5. To deploy software to end-users.

The first one sounds great: making it easy to use luarocks in the privacy of your own home.

I am more cautious about the others though. Freely adding dependencies on extra libraries, moving away from a pure Github workflow, developing our code as a library rather than an application, having more moving parts in our deployment process... I wonder if it is more beneficial for us to stay in full control even at the cost of some extra work and dependency frugality.

For example, this week we needed to pull in a fix to LuaJIT. That's easy for us because we reference an exact git version of LuaJIT as a submodule and we statically link that with the application. In a luarocks universe it seems harder to express that Snabb Switch requires an unreleased version of LuaJIT that includes commit 6d0654 (?).

On reflection I find myself leaning in a somewhat radically different direction at the moment:

https://groups.google.com/d/msg/snabb-devel/_ZkJTUNyj6I/oNiUipfW0XIJ

but we'll see if manually importing ljsyscall casts things in a different light.. :-)

Make any sense? (Words of caution? :-))

Cheers!

-Luke