RFH: Cross-platform binary modules

[Isaac Schlueter]

There is a lot of interest in Node, and a lot of work to do. Several
of you have asked me lately how you can help in IRC and in person.

In order to try to help match up the interest with the work, I'm going
to start occasionally making requests for help in specific areas where
I think improvements need to be made. In the past, these kinds of
calls have been very productive.

You don't need permission. Just play around with the problem, and if
you make some headway, share it with the rest of the group. If you
find it's something you really can get excited about, then maybe it
can be your thing :)

A very problematic area right now is binary module compilation across
platforms, especially on Windows.

In the past, we've deployed a node-waf program, which is a
lightly-customized fork of the "waf" build tool. This program has
worked ok, but it is fundamentally non-portable, and my hope is that
it will not have to be in 0.8.

In the node source tree is a file called tools/gyp_addon. You can use
this to generate either a Makefile, or the appropriate MS Visual C
stuff to build an addon. There are some examples in test/addons/, but
the whole process is not very well documented, and requires a lot of
manual tweaking.

To make matters even more hairy, there are a bunch of node programs
floating around in the wild that are using node-waf. We need to make
it as easy as possible to port these to a better future.

The goal:

There should be a standalone `npm install addon-gyp-toolchain -g` or
something (probably ought to pick a better name) that sets up some
scripts. If you have this toolchain installed, we'll assume that you
also have "make" on unix, and Visual Studio on Windows. If you then
run the appropriate command in a project directory, it'll build an
addon using whichever method is appropriate for the current system.

Once such a thing is in place, and it works reliably on Windows and
Unix, and is pretty easy and generalized, start making pull requests
to everybody to replace their wscripts with a gyp file.

The most important thing is that it makes the process easier than it
is today. There's probably more API design, debugging, and picking
through bits, than actually writing new code.

And, of course, it could be that my vision of this is way off. Your
task should be to make it easier to compile a bunch of C code into a
.node addon file in a way that works on Windows and Unix, and doesn't
completely suck. Don't be afraid to get creative.

Why not just deploy gyp_addon with node? Because that would be
recreating the node-waf experience, which is terrible. This is a step
towards binary deployments (at least for operating systems and
platforms that can do such things reliably). It would not be terrible
to install it when node is configured with --debug or something, but
by default the msi and pkg installers should never include gyp. Users
who *want* install-time compilation will always be able to do this by
installing the toolchain and then opting out of binary builds.

Thanks for your help.

[Nathan Rajlich]

I believe strongly in getting this right as well, and have been dabbling with the gyp_addon system so far. With the removal of node-waf, and the new reliance of the developer to compile their modules, rather than the end-user, I see 3 new problems that open up for native module devs:

1. Devs have to now compile for each supported platform and architecture before publishing
2. Devs now have to figure out where to store these precompiled binaries (in the cloud? in the repo? where?)
3. Devs now have to figure out which binary to load at runtime, given the current arch and platform

I've written a small module that attempts to solve #2 and #3, called node-bindings: https://github.com/TooTallNate/node-bindings

It's a nice convention in my opinion that doesn't rely on npm's `bin-publish` feature.

So that leaves problem #1. I like the "addon-gyp-toolchain" module idea. I hadn't considered that before, and that should help, though I'm not exactly sure how it would work (currently Windows addons require the node source tree, so that would need to be fixed I guess).

I'm still envisioning some kind of Travis-CI-like cloud-based build system, that would receive a git-hook and recompile your module on the various platforms/archs and commit them back to your repo (or something), whenever a change happened in the `src` dir. If that was done right, it would make it painless for both the devs and the uses of the module.

[Isaac Schlueter]

On Wed, Feb 1, 2012 at 16:15, Nathan Rajlich <nat...@tootallnate.net> wrote:
> Devs have to now compile for each supported platform and architecture before
> publishing

Problem 1 is indeed a problem, but it can be solved in 2 ways.

1. If you have the toolchain installed, you'll be able to build at
install time, just like you do today with node-waf, so there's a
fallback.
2. We'll build some cloudy buildbot CI stuff to compile all the things.

Node-bindings could definitely be handy.

> So that leaves problem #1. I like the "addon-gyp-toolchain" module idea. I
> hadn't considered that before, and that should help, though I'm not exactly
> sure how it would work (currently Windows addons require the node source
> tree, so that would need to be fixed I guess).

Any addon build would require the node source tree, or at least the headers.

Like I said, lots of fiddly little bits to get all in a row, probably
some core changes, etc. But, this should happen mostly independent of
any changes in node itself.

> I'm still envisioning some kind of Travis-CI-like cloud-based build system,
> that would receive a git-hook and recompile your module on the various
> platforms/archs and commit them back to your repo (or something), whenever a
> change happened in the `src` dir. If that was done right, it would make it
> painless for both the devs and the uses of the module.

Yes, but it should be based on publishes, and attach the tarball to
the package in the registry tagged with the appropriate platform
indicators.

[Paddy Byers]

Hi,

I've written a small module that attempts to solve #2 and #3, called node-bindings: https://github.com/TooTallNate/node-bindings

I think this is a very good start. Apart from anything else, it defines a convention for a specific location of the bindings that's independent of the build tool. Already we have lots of modules that are looking in a series of locations (./out/Release/, ./out/default/ etc) and this will only get worse when we have gyp as well which builds by default to some other place like ./build/Default/.

One thing it does do is impose the requirement to wrap every native addon with a js script, even if the API is completely implemented in the native library; there's no portable way to just drop the .node into the node_modules directory and have it located by module.js. I don't know how big an issue this is, but it needlessly creates a v8::Context. (There's a simple enough, bug ugly, way to deal with this by adding a variant-specific extension, eg

Module._extensions['.win32.ia32.node'] = ...

But I agree we should look first for solutions that don't hard-code these kinds of things into the core.)

There is no stable C++ ABI, so you will also have to add the compiler version as another argument to that list of targets. Some compilers do try to do versioning with mapfiles, but it is rarely used. The worst thing that can happen is having a version of node built with one version of a compiler and an addon with another.

The build tool would need to use the ABI that was used to build the node that's installed; not just guess based on the host platform, and that ABI (with all of the toolchain parameters that implies) would need to be fixed from the time of the first release of a new minor version for each platform.  I think that's a realistic aim for the stable (ie even minor version) releases but is liable to break for the unstable ones. Although I like the idea of the addon SDK being separate (eg as with node and node-dev packages), you do guarantee they are aligned if they are the same installable entity. It's all too easy on a debian system to install a new xxx package, only to find that you then can't install the corresponding xxx-dev since some of its dependencies are not stable, or upgrading those breaks something else, or whatever.

Thanks - Paddy

[Isaac Schlueter]

Let's start with:

"I already wrote a little binding C++ thing for node, and now I want
to compile a .node file." Start with the example in node's
test/addons/hello-world.

Right now, that is unnecessarily difficult, and I believe it's a good
sized problem to tackle. How would you make it trivial to do that?

On Thu, Feb 2, 2012 at 00:58, Nathan Rajlich <nat...@tootallnate.net> wrote:
> The compiler used *may* be another paramater to consider, but I haven't ran
> into any problems in use with node-bindings yet. One issue getting the
> compiler name is that it's not available in node anywhere, so getting that
> value at runtime would be hard/impossible.
>
>
> On Wed, Feb 1, 2012 at 6:38 PM, Robert Mustacchi <r...@fingolfin.org> wrote:

>>
>> On 02/01/2012 04:52 PM, Isaac Schlueter wrote:
>>>
>>> On Wed, Feb 1, 2012 at 16:15, Nathan Rajlich<nat...@tootallnate.net>
>>>  wrote:
>>>>
>>>> Devs have to now compile for each supported platform and architecture
>>>> before
>>>> publishing
>>>
>>>
>>> Problem 1 is indeed a problem, but it can be solved in 2 ways.
>>>
>>> 1. If you have the toolchain installed, you'll be able to build at
>>> install time, just like you do today with node-waf, so there's a
>>> fallback.
>>> 2. We'll build some cloudy buildbot CI stuff to compile all the things.
>>>
>>> Node-bindings could definitely be handy.
>>
>>

>> There is no stable C++ ABI, so you will also have to add the compiler
>> version as another argument to that list of targets. Some compilers do try
>> to do versioning with mapfiles, but it is rarely used. The worst thing that
>> can happen is having a version of node built with one version of a compiler
>> and an addon with another.
>>

>> Robert
>
>

[Nathan Rajlich]

Well something like:

  $ npm install -g node-gyp

  $ 

  $ cd my-addon

  $ node-gyp configure    (generate the Makefile/VC project in cwd from gyp)

  $ node-gyp build     (do the actual build, either with Make or msbuild, depending on the platform)

Then you would have either "out/Debug/bindings.node" or "out/Release/bindings.node" depending on the build type. We could have `node-gyp configure --debug` make a Debug build and then default to a Release build, kinda like how Node does it in its configure/Makefile.

That would normalize the commands needed for building on the various platforms, which is a big step. node-gyp should have the ability to select/install header files from different versions of Node, as well as download precompiled node.lib files for Windows (`node-waf configure --target=0.6.x` maybe).

I'm not sure what else it would need off the top of my head. Thoughts?

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Isaac Schlueter]

Yes! That'd be awesome. Go build that :)

[Paddy Byers]

Hi,

I'm not sure what else it would need off the top of my head. Thoughts?

I think you missed the bit where you write the .gyp file :)

But .. this, for most modules, should be nothing more than listing the source files, include dirs, defines, and library dependencies, by adding to a template, with all other node-specific things (-DBUILDING_NODE_EXTENSION etc) handled by the addon.gypi.

Paddy

[C. Mundi]

As has been mentioned, the module build system needs the node headers.  (That much seems unavoidable for native modules.). But which *version* of the headers?  I would not expect the build system to figure this out, but I would like to be able to specify which version(s) of node to build against, limited only by the module metadata specifying which node version a module promises to support.

[Bradley Meck]

Currently we have several modules that have to be built and coexist as available for use from the same directory from multiple Node versions (daemon and node-fork). Right now we have a install script and runtime detection handling this, but it would be nice if package.json or NODE_PATH were a little cleaner and allowed us to specify what to pull at specific versions in instead of sharing the same compiled module filename for multiple ABIs and forcing runtime detection.

[Nathan Rajlich]

Just so that people aren't duplicating efforts, I have begun creating a `node-gyp` project as described earlier in this thread. I'll be working on getting an initial version out by the end of this weekend that will hopefully be usable by everyone.

I'll notify this thread as soon as I have something worthwhile to show. Cheers!

On Fri, Feb 3, 2012 at 1:23 PM, Bradley Meck <bradle...@gmail.com> wrote:

Currently we have several modules that have to be built and coexist as available for use from the same directory from multiple Node versions (daemon and node-fork). Right now we have a install script and runtime detection handling this, but it would be nice if package.json or NODE_PATH were a little cleaner and allowed us to specify what to pull at specific versions in instead of sharing the same compiled module filename for multiple ABIs and forcing runtime detection.

--

[Isaac Schlueter]

Thanks, Nathan, this is very exciting :)

[rhdoenges]

I will contribute!

[Nathan Rajlich]

Ok I just tagged and published v0.0.1 of node-gyp on npm.

Here is the repo: https://github.com/TooTallNate/node-gyp

Native addon devs, please try it out and begin migrating your module to using gyp. Let me know about your results!!

[Christopher Jeffrey]

Works well for me, very painless. I hope others start supporting it
soon.

[Isaac Schlueter]

Wow, that looks really cool.

The readme doesn't mentioning actually creating a gyp file. You still
have to do that, right? It'd be nice to document what it should look
like.

[Nathan Rajlich]

On Sun, Feb 5, 2012 at 9:39 PM, Isaac Schlueter <i...@izs.me> wrote:

Wow, that looks really cool.

The readme doesn't mentioning actually creating a gyp file.  You still
have to do that, right?  It'd be nice to document what it should look
like.

README updated! Thanks for the shout-out on nodeup today too!

[Samori Gorse]

+1 on the general idea.

I've been waiting for the node-waf -> node_gyp switch for a while now. 

I'm working on a project to help node.js module authors get started with the whole C++ thing:

 https://github.com/shinuza/node-cc-module-boilerplate

The biggest problem with node-waf is that the version bundled with node is super outdated and the documentation is hard to find. Same thing goes with the automated doc for the V8 API.

> 2. We'll build some cloudy buildbot CI stuff to compile all the things.

I thought of this while reading the original post, although it might require some special setup to register a particular module for compiling.

[Nathan Rajlich]

Samori, you should definitely update your example module to use node-gyp instead of node-waf! We want new module authors to use the new stuff; forward compatibility and all that.

--

[Roman Shtylman]

I think you guys are reinventing the wheel here with respect to building addons. Gyp and CMake were create specifically so you don't need to roll your own system. Maybe it is just me but I liked the fact that node-waf came bundled with the node install. This meant that I didn't have to go fetch any additional packages or items to build the addon.

I will also add that I am against shipping binary addons. The number of "parameters" you could be pivoting on is too great imho. If someone has a system in place to deploy binaries (deb, rpm, etc) I would think they should use that. Otherwise compiling these small addons for deployment is not that big a deal is it? I would be hesitant on a binary solution until someone can prove to me anyone would actually care to use it in a meaningful way. Right now, I just always build when I deploy and that works fine. The benefit here is that build time failure is much preferred to run time failure.

[Nathan Rajlich]

On Fri, Feb 10, 2012 at 12:35 PM, Roman Shtylman <shty...@gmail.com> wrote:

I think you guys are reinventing the wheel here with respect to building addons. Gyp and CMake were create specifically so you don't need to roll your own system. Maybe it is just me but I liked the fact that node-waf came bundled with the node install. This meant that I didn't have to go fetch any additional packages or items to build the addon.

I don't really see your concern here. We're attempting to do the same thing with gyp as we did with waf, by providing a light wrapper around its basic usage to help simplify the necessary gyp file module devs need to create (same as the wscript file before). The only difference so far is that instead of being bundled with node, you have to `npm install -g node-gyp` to get it. As isaacs said at the beginning of this thread, we only want more advanced/comfortable users compiling, so once new users are ready, they can install node-gyp. While they're still new, they can rely on precompiled binaries (same situation with the precompiled binaries officially offered for node: I don't use them since I'm "advanced", but they're wonderful for new user adoption).

I will also add that I am against shipping binary addons. The number of "parameters" you could be pivoting on is too great imho. If someone has a system in place to deploy binaries (deb, rpm, etc) I would think they should use that. Otherwise compiling these small addons for deployment is not that big a deal is it? I would be hesitant on a binary solution until someone can prove to me anyone would actually care to use it in a meaningful way. Right now, I just always build when I deploy and that works fine. The benefit here is that build time failure is much preferred to run time failure.

The way I see it, nobody is forcing you to use any precompiled binaries. There is always still the source code and installing node-gyp is a one-line command, so just like node itself, more advanced users are probably going to stick with compiling their native addons themselves, which is perfectly fine in my opinion. npm could even offer a flag where it would compile locally on the 'install' phase instead of downloading a precompiled binary, much like the node-waf situation now.

As said in the last paragraph, these precompiled binaries will mostly be for the benefit of new users (especially Windows users) where they may not even have a compilation toolchain installed (also especially true for OS X users who rely on the installer, and don't have XCode installed). But for a more advanced user like yourself, the only change in your workflow will probably be invoking node-gyp instead of node-waf at build-time.

[billywhizz]

i tend to agree with Roman's sentiments. Whatever is used to build
addons should be included with the core node package and not be an
external module. node-waf worked really nice as far as i was
concerned.

also, node-gyp has a ton of dependencies which means if i don't want
to use npm (which i really, really don't) then i have to install all
these dependencies by hand just to be able to build an addon. that's a
giant PITA from where i am standing.

On Feb 10, 10:26 pm, Nathan Rajlich <nat...@tootallnate.net> wrote:

[billywhizz]

as an alternative to having gyp as an npm module, how about having a
node-sdk build of core with everything included to allow building of c+
+ modules and no need to install/use npm or any other external
modules? this is the way .Net and Java do things and it seems to
work...

[Nathan Rajlich]

So Billy your argument is that you don't want to use node-gyp since it has dependencies, and you for some reason are against npm? Well that's a first... What possible argument could you have at this point for not wanting to use npm? It comes with node!!!

Additionally, limiting yourself to just modules that have no dependencies, just for the purpose of being able to git-clone the repo and have it work, seems really disadvantageous to me. *shrug*

As for your thought on the separation between a regular node and a dev node, well... this the same thing. "normal" users install just node, while native module devs install node+node-gyp. The levels of separation are the same, but since you're against npm you are hoping for something different. I'm not gonna speak for the core team but I personally don't see 2 different tiers of node ever happening.

[Nathan Rajlich]

And on top of that, I could just as easily just start commiting the node_modules dir so that git-clone works properly. Then what argument is there for not using node-gyp?

[Isaac Schlueter]

On Fri, Feb 10, 2012 at 18:34, Nathan Rajlich <nat...@tootallnate.net> wrote:
> And on top of that, I could just as easily just start commiting the
> node_modules dir so that git-clone works properly.

That's a good idea regardless, actually. Commit the node_modules
folder, and list all the deps as bundledDependencies.

node-gyp is a command line tool, not something you'd expect to
require(). Like npm itself, it should be a single thing that works
out of the box.

[Nathan Rajlich]

Ok thanks for the confirmation Isaac. The latest version of node-gyp (v0.0.5) now has its dependencies bundled.

[billywhizz]

Nate - i'm not "against" npm so to speak. it just doesn't suit my
requirements as i like to do my own builds on a build machine and roll
everything out to the production machines i am using and have full
control over where everything goes. npm takes too much control away
from me and i am a control freak, that is all. ;) having the modules
bundled solves the problem for me - thanks for doing that.

[Roman Shtylman]

On Friday, February 10, 2012 5:26:34 PM UTC-5, TooTallNate wrote:

On Fri, Feb 10, 2012 at 12:35 PM, Roman Shtylman <shty...@gmail.com> wrote:

I think you guys are reinventing the wheel here with respect to building addons. Gyp and CMake were create specifically so you don't need to roll your own system. Maybe it is just me but I liked the fact that node-waf came bundled with the node install. This meant that I didn't have to go fetch any additional packages or items to build the addon.

I don't really see your concern here. We're attempting to do the same thing with gyp as we did with waf, by providing a light wrapper around its basic usage to help simplify the necessary gyp file module devs need to create (same as the wscript file before). The only difference so far is that instead of being bundled with node, you have to `npm install -g node-gyp` to get it. As isaacs said at the beginning of this thread, we only want more advanced/comfortable users compiling, so once new users are ready, they can install node-gyp. While they're still new, they can rely on precompiled binaries (same situation with the precompiled binaries officially offered for node: I don't use them since I'm "advanced", but they're wonderful for new user adoption).

My main point was that by shipping it you don't make it harder for new people to use node, but by not shipping it you do make it harder for module creators to get started or tweak things. I always found that making modules is more of a "built in" process that is highly coupled with the release of node (especially given how all the tools are node specific). I also like the idea that if I don't want to use npm, I don't have to. Now if you make the tool downloadable through npm you end up having to use npm to start developing modules. I like lower barrier to entry :)

 

I will also add that I am against shipping binary addons. The number of "parameters" you could be pivoting on is too great imho. If someone has a system in place to deploy binaries (deb, rpm, etc) I would think they should use that. Otherwise compiling these small addons for deployment is not that big a deal is it? I would be hesitant on a binary solution until someone can prove to me anyone would actually care to use it in a meaningful way. Right now, I just always build when I deploy and that works fine. The benefit here is that build time failure is much preferred to run time failure.

The way I see it, nobody is forcing you to use any precompiled binaries. There is always still the source code and installing node-gyp is a one-line command, so just like node itself, more advanced users are probably going to stick with compiling their native addons themselves, which is perfectly fine in my opinion. npm could even offer a flag where it would compile locally on the 'install' phase instead of downloading a precompiled binary, much like the node-waf situation now.

As said in the last paragraph, these precompiled binaries will mostly be for the benefit of new users (especially Windows users) where they may not even have a compilation toolchain installed (also especially true for OS X users who rely on the installer, and don't have XCode installed). But for a more advanced user like yourself, the only change in your workflow will probably be invoking node-gyp instead of node-waf at build-time.

I suppose statically linking all the binaries will get around many of such concerns. Do consider that npm does not interface with the system's package manager (if it has one) and so the static linking may be the only sane approach.

 

[Juraj Vitko]

> Yes, but it should be based on publishes, and attach the tarball to
> the package in the registry tagged with the appropriate platform
> indicators.

Will it not be too late to catch build errors after the package is
already published? OTOH, building upon each git commit is too often.
Seems like a step in between would be best, e.g. `npm pre-publish`.

Great stuff.

J.


On Feb 2, 2:52 am, Isaac Schlueter <i...@izs.me> wrote:

> On Wed, Feb 1, 2012 at 16:15, Nathan Rajlich <nat...@tootallnate.net> wrote:
> > Devs have to now compile for each supported platform and architecture before
> > publishing
>
> Problem 1 is indeed a problem, but it can be solved in 2 ways.
>
> 1. If you have the toolchain installed, you'll be able to build at
> install time, just like you do today with node-waf, so there's a
> fallback.

> 2. We'll build some cloudy buildbot CI stuff to compile all the things.
>

> Node-bindings could definitely be handy.
>

> > So that leaves problem #1. I like the "addon-gyp-toolchain" module idea. I
> > hadn't considered that before, and that should help, though I'm not exactly
> > sure how it would work (currently Windows addons require the node source
> > tree, so that would need to be fixed I guess).
>
> Any addon build would require the node source tree, or at least the headers.
>
> Like I said, lots of fiddly little bits to get all in a row, probably
> some core changes, etc.  But, this should happen mostly independent of
> any changes in node itself.
>
> > I'm still envisioning some kind of Travis-CI-like cloud-based build system,
> > that would receive a git-hook and recompile your module on the various
> > platforms/archs and commit them back to your repo (or something), whenever a
> > change happened in the `src` dir. If that was done right, it would make it
> > painless for both the devs and the uses of the module.
>
> Yes, but it should be based on publishes, and attach the tarball to
> the package in the registry tagged with the appropriate platform
> indicators.