There is a lot of interest in Node, and a lot of work to do. Several of you have asked me lately how you can help in IRC and in person.
In order to try to help match up the interest with the work, I'm going to start occasionally making requests for help in specific areas where I think improvements need to be made. In the past, these kinds of calls have been very productive.
You don't need permission. Just play around with the problem, and if you make some headway, share it with the rest of the group. If you find it's something you really can get excited about, then maybe it can be your thing :)
A very problematic area right now is binary module compilation across platforms, especially on Windows.
In the past, we've deployed a node-waf program, which is a lightly-customized fork of the "waf" build tool. This program has worked ok, but it is fundamentally non-portable, and my hope is that it will not have to be in 0.8.
In the node source tree is a file called tools/gyp_addon. You can use this to generate either a Makefile, or the appropriate MS Visual C stuff to build an addon. There are some examples in test/addons/, but the whole process is not very well documented, and requires a lot of manual tweaking.
To make matters even more hairy, there are a bunch of node programs floating around in the wild that are using node-waf. We need to make it as easy as possible to port these to a better future.
The goal:
There should be a standalone `npm install addon-gyp-toolchain -g` or something (probably ought to pick a better name) that sets up some scripts. If you have this toolchain installed, we'll assume that you also have "make" on unix, and Visual Studio on Windows. If you then run the appropriate command in a project directory, it'll build an addon using whichever method is appropriate for the current system.
Once such a thing is in place, and it works reliably on Windows and Unix, and is pretty easy and generalized, start making pull requests to everybody to replace their wscripts with a gyp file.
The most important thing is that it makes the process easier than it is today. There's probably more API design, debugging, and picking through bits, than actually writing new code.
And, of course, it could be that my vision of this is way off. Your task should be to make it easier to compile a bunch of C code into a .node addon file in a way that works on Windows and Unix, and doesn't completely suck. Don't be afraid to get creative.
Why not just deploy gyp_addon with node? Because that would be recreating the node-waf experience, which is terrible. This is a step towards binary deployments (at least for operating systems and platforms that can do such things reliably). It would not be terrible to install it when node is configured with --debug or something, but by default the msi and pkg installers should never include gyp. Users who *want* install-time compilation will always be able to do this by installing the toolchain and then opting out of binary builds.
I believe strongly in getting this right as well, and have been dabbling with the gyp_addon system so far. With the removal of node-waf, and the new reliance of the developer to compile their modules, rather than the end-user, I see 3 new problems that open up for native module devs:
1. Devs have to now compile for each supported platform and architecture before publishing 2. Devs now have to figure out where to store these precompiled binaries (in the cloud? in the repo? where?) 3. Devs now have to figure out which binary to load at runtime, given the current arch and platform
It's a nice convention in my opinion that doesn't rely on npm's `bin-publish` feature.
So that leaves problem #1. I like the "addon-gyp-toolchain" module idea. I hadn't considered that before, and that should help, though I'm not exactly sure how it would work (currently Windows addons require the node source tree, so that would need to be fixed I guess).
I'm still envisioning some kind of Travis-CI-like cloud-based build system, that would receive a git-hook and recompile your module on the various platforms/archs and commit them back to your repo (or something), whenever a change happened in the `src` dir. If that was done right, it would make it painless for both the devs and the uses of the module.
On Wed, Feb 1, 2012 at 3:14 PM, Isaac Schlueter <i...@izs.me> wrote: > There is a lot of interest in Node, and a lot of work to do. Several > of you have asked me lately how you can help in IRC and in person.
> In order to try to help match up the interest with the work, I'm going > to start occasionally making requests for help in specific areas where > I think improvements need to be made. In the past, these kinds of > calls have been very productive.
> You don't need permission. Just play around with the problem, and if > you make some headway, share it with the rest of the group. If you > find it's something you really can get excited about, then maybe it > can be your thing :)
> A very problematic area right now is binary module compilation across > platforms, especially on Windows.
> In the past, we've deployed a node-waf program, which is a > lightly-customized fork of the "waf" build tool. This program has > worked ok, but it is fundamentally non-portable, and my hope is that > it will not have to be in 0.8.
> In the node source tree is a file called tools/gyp_addon. You can use > this to generate either a Makefile, or the appropriate MS Visual C > stuff to build an addon. There are some examples in test/addons/, but > the whole process is not very well documented, and requires a lot of > manual tweaking.
> To make matters even more hairy, there are a bunch of node programs > floating around in the wild that are using node-waf. We need to make > it as easy as possible to port these to a better future.
> The goal:
> There should be a standalone `npm install addon-gyp-toolchain -g` or > something (probably ought to pick a better name) that sets up some > scripts. If you have this toolchain installed, we'll assume that you > also have "make" on unix, and Visual Studio on Windows. If you then > run the appropriate command in a project directory, it'll build an > addon using whichever method is appropriate for the current system.
> Once such a thing is in place, and it works reliably on Windows and > Unix, and is pretty easy and generalized, start making pull requests > to everybody to replace their wscripts with a gyp file.
> The most important thing is that it makes the process easier than it > is today. There's probably more API design, debugging, and picking > through bits, than actually writing new code.
> And, of course, it could be that my vision of this is way off. Your > task should be to make it easier to compile a bunch of C code into a > .node addon file in a way that works on Windows and Unix, and doesn't > completely suck. Don't be afraid to get creative.
> Why not just deploy gyp_addon with node? Because that would be > recreating the node-waf experience, which is terrible. This is a step > towards binary deployments (at least for operating systems and > platforms that can do such things reliably). It would not be terrible > to install it when node is configured with --debug or something, but > by default the msi and pkg installers should never include gyp. Users > who *want* install-time compilation will always be able to do this by > installing the toolchain and then opting out of binary builds.
On Wed, Feb 1, 2012 at 16:15, Nathan Rajlich <nat...@tootallnate.net> wrote: > Devs have to now compile for each supported platform and architecture before > publishing
Problem 1 is indeed a problem, but it can be solved in 2 ways.
1. If you have the toolchain installed, you'll be able to build at install time, just like you do today with node-waf, so there's a fallback. 2. We'll build some cloudy buildbot CI stuff to compile all the things.
Node-bindings could definitely be handy.
> So that leaves problem #1. I like the "addon-gyp-toolchain" module idea. I > hadn't considered that before, and that should help, though I'm not exactly > sure how it would work (currently Windows addons require the node source > tree, so that would need to be fixed I guess).
Any addon build would require the node source tree, or at least the headers.
Like I said, lots of fiddly little bits to get all in a row, probably some core changes, etc. But, this should happen mostly independent of any changes in node itself.
> I'm still envisioning some kind of Travis-CI-like cloud-based build system, > that would receive a git-hook and recompile your module on the various > platforms/archs and commit them back to your repo (or something), whenever a > change happened in the `src` dir. If that was done right, it would make it > painless for both the devs and the uses of the module.
Yes, but it should be based on publishes, and attach the tarball to the package in the registry tagged with the appropriate platform indicators.
> On Wed, Feb 1, 2012 at 16:15, Nathan Rajlich<nat...@tootallnate.net> wrote: >> Devs have to now compile for each supported platform and architecture before >> publishing
> Problem 1 is indeed a problem, but it can be solved in 2 ways.
> 1. If you have the toolchain installed, you'll be able to build at > install time, just like you do today with node-waf, so there's a > fallback. > 2. We'll build some cloudy buildbot CI stuff to compile all the things.
> Node-bindings could definitely be handy.
There is no stable C++ ABI, so you will also have to add the compiler version as another argument to that list of targets. Some compilers do try to do versioning with mapfiles, but it is rarely used. The worst thing that can happen is having a version of node built with one version of a compiler and an addon with another.
The compiler used *may* be another paramater to consider, but I haven't ran into any problems in use with node-bindings yet. One issue getting the compiler name is that it's not available in node anywhere, so getting that value at runtime would be hard/impossible.
On Wed, Feb 1, 2012 at 6:38 PM, Robert Mustacchi <r...@fingolfin.org> wrote: > On 02/01/2012 04:52 PM, Isaac Schlueter wrote:
>> On Wed, Feb 1, 2012 at 16:15, Nathan Rajlich<nat...@tootallnate.net**> >> wrote:
>>> Devs have to now compile for each supported platform and architecture >>> before >>> publishing
>> Problem 1 is indeed a problem, but it can be solved in 2 ways.
>> 1. If you have the toolchain installed, you'll be able to build at >> install time, just like you do today with node-waf, so there's a >> fallback. >> 2. We'll build some cloudy buildbot CI stuff to compile all the things.
>> Node-bindings could definitely be handy.
> There is no stable C++ ABI, so you will also have to add the compiler > version as another argument to that list of targets. Some compilers do try > to do versioning with mapfiles, but it is rarely used. The worst thing that > can happen is having a version of node built with one version of a compiler > and an addon with another.
On Fri, Feb 3, 2012 at 11:13 PM, rhdoenges <rhdoen...@gmail.com> wrote: > I will contribute!
> On Feb 3, 2:53 pm, Nathan Rajlich <nat...@tootallnate.net> wrote: > > Just so that people aren't duplicating efforts, I have begun creating a > > `node-gyp` project as described earlier in this thread. I'll be working > on > > getting an initial version out by the end of this weekend that will > > hopefully be usable by everyone.
> > I'll notify this thread as soon as I have something worthwhile to show. > > Cheers!
> > On Fri, Feb 3, 2012 at 1:23 PM, Bradley Meck <bradley.m...@gmail.com> > wrote: > > > Currently we have several modules that have to be built and coexist as > > > available for use from the same directory from multiple Node versions > > > (daemon and node-fork). Right now we have a install script and runtime > > > detection handling this, but it would be nice if package.json or > NODE_PATH > > > were a little cleaner and allowed us to specify what to pull at > specific > > > versions in instead of sharing the same compiled module filename for > > > multiple ABIs and forcing runtime detection.
I've been waiting for the node-waf -> node_gyp switch for a while now. I'm working on a project to help node.js module authors get started with the whole C++ thing:
The biggest problem with node-waf is that the version bundled with node is super outdated and the documentation is hard to find. Same thing goes with the automated doc for the V8 API.
> 2. We'll build some cloudy buildbot CI stuff to compile all the things.
I thought of this while reading the original post, although it might require some special setup to register a particular module for compiling.
Samori, you should definitely update your example module to use node-gyp instead of node-waf! We want new module authors to use the new stuff; forward compatibility and all that.
On Tue, Feb 7, 2012 at 8:14 AM, Samori Gorse <samorigo...@gmail.com> wrote: > +1 on the general idea.
> I've been waiting for the node-waf -> node_gyp switch for a while now. > I'm working on a project to help node.js module authors get started with > the whole C++ thing:
> The biggest problem with node-waf is that the version bundled with node is > super outdated and the documentation is hard to find. Same thing goes with > the automated doc for the V8 API.
> > 2. We'll build some cloudy buildbot CI stuff to compile all the things.
> I thought of this while reading the original post, although it might > require some special setup to register a particular module for compiling.
On Feb 1, 3:14 pm, Isaac Schlueter <i...@izs.me> wrote:
> The goal:
> There should be a standalone `npm install addon-gyp-toolchain -g` or
> something (probably ought to pick a better name) that sets up some
> scripts. If you have this toolchain installed, we'll assume that you
> also have "make" on unix, and Visual Studio on Windows. If you then
> run the appropriate command in a project directory, it'll build an
> addon using whichever method is appropriate for the current system.
A LOT of Node deployments will end up tweaking/fixing native modules,
or writing new ones to access other APIs/libs, so most serious users
will want the toolchain to do this. Also modules in development may
not ship with binaries, so all users testing them will need it as
well.
At the very least the default install procedure ought to end with a
message giving the toolchain-install command.
I think you guys are reinventing the wheel here with respect to building addons. Gyp and CMake were create specifically so you don't need to roll your own system. Maybe it is just me but I liked the fact that node-waf came bundled with the node install. This meant that I didn't have to go fetch any additional packages or items to build the addon.
I will also add that I am against shipping binary addons. The number of "parameters" you could be pivoting on is too great imho. If someone has a system in place to deploy binaries (deb, rpm, etc) I would think they should use that. Otherwise compiling these small addons for deployment is not that big a deal is it? I would be hesitant on a binary solution until someone can prove to me anyone would actually care to use it in a meaningful way. Right now, I just always build when I deploy and that works fine. The benefit here is that build time failure is much preferred to run time failure.
On Fri, Feb 10, 2012 at 12:35 PM, Roman Shtylman <shtyl...@gmail.com> wrote: > I think you guys are reinventing the wheel here with respect to building > addons. Gyp and CMake were create specifically so you don't need to roll > your own system. Maybe it is just me but I liked the fact that node-waf > came bundled with the node install. This meant that I didn't have to go > fetch any additional packages or items to build the addon.
I don't really see your concern here. We're attempting to do the same thing with gyp as we did with waf, by providing a light wrapper around its basic usage to help simplify the necessary gyp file module devs need to create (same as the wscript file before). The only difference so far is that instead of being bundled with node, you have to `npm install -g node-gyp` to get it. As isaacs said at the beginning of this thread, we only want more advanced/comfortable users compiling, so once new users are ready, they can install node-gyp. While they're still new, they can rely on precompiled binaries (same situation with the precompiled binaries officially offered for node: I don't use them since I'm "advanced", but they're wonderful for new user adoption).
> I will also add that I am against shipping binary addons. The number of > "parameters" you could be pivoting on is too great imho. If someone has a > system in place to deploy binaries (deb, rpm, etc) I would think they > should use that. Otherwise compiling these small addons for deployment is > not that big a deal is it? I would be hesitant on a binary solution until > someone can prove to me anyone would actually care to use it in a > meaningful way. Right now, I just always build when I deploy and that works > fine. The benefit here is that build time failure is much preferred to run > time failure.
The way I see it, nobody is forcing you to use any precompiled binaries. There is always still the source code and installing node-gyp is a one-line command, so just like node itself, more advanced users are probably going to stick with compiling their native addons themselves, which is perfectly fine in my opinion. npm could even offer a flag where it would compile locally on the 'install' phase instead of downloading a precompiled binary, much like the node-waf situation now.
As said in the last paragraph, these precompiled binaries will mostly be for the benefit of new users (especially Windows users) where they may not even have a compilation toolchain installed (also especially true for OS X users who rely on the installer, and don't have XCode installed). But for a more advanced user like yourself, the only change in your workflow will probably be invoking node-gyp instead of node-waf at build-time.
On Friday, February 10, 2012 5:26:34 PM UTC-5, TooTallNate wrote:
> On Fri, Feb 10, 2012 at 12:35 PM, Roman Shtylman <shtyl...@gmail.com>wrote:
>> I think you guys are reinventing the wheel here with respect to building >> addons. Gyp and CMake were create specifically so you don't need to roll >> your own system. Maybe it is just me but I liked the fact that node-waf >> came bundled with the node install. This meant that I didn't have to go >> fetch any additional packages or items to build the addon.
> I don't really see your concern here. We're attempting to do the same > thing with gyp as we did with waf, by providing a light wrapper around its > basic usage to help simplify the necessary gyp file module devs need to > create (same as the wscript file before). The only difference so far is > that instead of being bundled with node, you have to `npm install -g > node-gyp` to get it. As isaacs said at the beginning of this thread, we > only want more advanced/comfortable users compiling, so once new users are > ready, they can install node-gyp. While they're still new, they can rely on > precompiled binaries (same situation with the precompiled binaries > officially offered for node: I don't use them since I'm "advanced", but > they're wonderful for new user adoption).
My main point was that by shipping it you don't make it harder for new people to use node, but by not shipping it you do make it harder for module creators to get started or tweak things. I always found that making modules is more of a "built in" process that is highly coupled with the release of node (especially given how all the tools are node specific). I also like the idea that if I don't want to use npm, I don't have to. Now if you make the tool downloadable through npm you end up having to use npm to start developing modules. I like lower barrier to entry :)
>> I will also add that I am against shipping binary addons. The number of >> "parameters" you could be pivoting on is too great imho. If someone has a >> system in place to deploy binaries (deb, rpm, etc) I would think they >> should use that. Otherwise compiling these small addons for deployment is >> not that big a deal is it? I would be hesitant on a binary solution until >> someone can prove to me anyone would actually care to use it in a >> meaningful way. Right now, I just always build when I deploy and that works >> fine. The benefit here is that build time failure is much preferred to run >> time failure.
> The way I see it, nobody is forcing you to use any precompiled binaries. > There is always still the source code and installing node-gyp is a one-line > command, so just like node itself, more advanced users are probably going > to stick with compiling their native addons themselves, which is perfectly > fine in my opinion. npm could even offer a flag where it would compile > locally on the 'install' phase instead of downloading a precompiled binary, > much like the node-waf situation now.
> As said in the last paragraph, these precompiled binaries will mostly be > for the benefit of new users (especially Windows users) where they may not > even have a compilation toolchain installed (also especially true for OS X > users who rely on the installer, and don't have XCode installed). But for a > more advanced user like yourself, the only change in your workflow will > probably be invoking node-gyp instead of node-waf at build-time.
I suppose statically linking all the binaries will get around many of such concerns. Do consider that npm does not interface with the system's package manager (if it has one) and so the static linking may be the only sane approach.
