Native Abstractions for Node.js (NAN) - a utility to ease the pain of C++ addon development across 0.8, 0.10 and 0.11/0.12

[Rod Vagg]

Related to the "You're going to have to rewrite it anyway" discussion here: https://groups.google.com/forum/#!topic/nodejs/VlUJ68n6QBg (and here: http://atxconsulting.com/2013/07/06/rewrite-it-anyway/).

https://github.com/rvagg/nan is an evolution of code I've been gathering for a while now to make addon compatibility easier. The 0.11.4 V8 changes introduce major trauma and the last thing we all need are conditionals all through our code just to make it work across versions so the aim of NAN is to provide basic utilities that completely remove the need for conditionals as 0.11 evolves while maintaining compatibility all the way back to at least 0.8.

I know that TJ and others are interested in working on a more complete abstraction that serves us into the future, beyond 1.0 and NAN may become obsolete, depending on the scope of this new work. I personally hate having to sit on breaking changes and am not comfortable waiting for much later in the 0.11 cycle for something that can let me test my addons against the latest code, so I was working on this anyway.

As a simple example, consider this code:

• Node 0.8/0.10 (breaks in 0.11): https://github.com/rvagg/node-addon-examples/blob/master/9_async_work/async.cc
• Node 0.11 (incompatible with pre-0.11): https://github.com/kkoopa/node-addon-examples/blob/5c01f58fc993377a567812597e54a83af69686d7/9_async_work/async.cc
• NAN (0.8/0.10/0.11): https://github.com/rvagg/nan/blob/master/examples/async_pi_estimate/async.cc

And NAN in actual addon code:

• Here's how NAN is used to adapt LevelDOWN to the 0.11 changes: https://github.com/rvagg/node-leveldown/pull/48/files
• Here it is against ssh (libssh binding): https://github.com/rvagg/node-libssh/commit/a983d055a6c4e32f248e6ec026cb128828a8e35d
• And here it is against node-canvas (may or may not be merged but it was a good challenge): https://github.com/LearnBoost/node-canvas/pull/305/files

The plan is to try and keep up with breaking changes so new features may be added as required. For now it's fairly minimal with some extra utilities that I just find useful for Node/V8 C++ development, not strictly related to compatibility.

Feedback welcome of course and I'd appreciate any additional challenges that your addons may need tackling that I haven't addressed yet.

Cheers,
 -- Rod

[Nathan Rajlich]

This is pretty cool Rod! I'm sure TJ will have some pointers. Also, related, but I know you're aiming for something a little simpler: have you seen: https://github.com/tjfontaine/node-addon-layer?

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Rod Vagg]

On Wednesday, 24 July 2013 02:32:52 UTC+10, Nathan Rajlich wrote:

This is pretty cool Rod! I'm sure TJ will have some pointers. Also, related, but I know you're aiming for something a little simpler: have you seen: https://github.com/tjfontaine/node-addon-layer?

Nope, I hadn't seen that, good to know that there's work in progress here.

On this topic, there's probably some blockers to distributing something like this in npm, particularly given that there's a compiled component. I've played a little with having native plugins to leveldown, where you `npm install X` and X depends on leveldown existing so it can compile against it. Although in my case the dependency works the opposite way to what TJ's addon-layer will work in that you have one leveldown and multiple addons compiling against it. Here's some hackery I had to do to make it work: https://github.com/rvagg/node-downer-rangedel/blob/master/common.gypi - basically executing a node process to use require.resolve in a .gypi to work out where the installed version is.
The real problem for me though is that you can't determine compile order with an `npm install`, so if you `npm install leveldown downer-rangedel` then it'll likely fail because a leveldown compile will take longer than a downer-rangedel compile and they happen in parallel, so you have to `npm install leveldown; npm install downer-rangedel`. Which kind of sucks.

Ideally we'd be able to do these things with npm and node-gyp:

• Easily determine the path to a dependency/peerDependency or some other dependency that you don't want to list in package.json (e.g. in my case I don't want leveldown as a dependency of downer-rangedel because I only want one instance of leveldown installed & compiled per application).  It'd be nice to be able to do something like "<@(node_dependency_path:addon_layer)" in your .gyp and have node-gyp take care of it for you.
  
• Specify build order in some way, perhaps a package.json property that says that this package shouldn't be compiled until some other package has been compiled.
  

[Timothy J Fontaine]

On Tue, Jul 23, 2013 at 4:37 PM, Rod Vagg <r...@vagg.org> wrote:

On Wednesday, 24 July 2013 02:32:52 UTC+10, Nathan Rajlich wrote:

This is pretty cool Rod! I'm sure TJ will have some pointers. Also, related, but I know you're aiming for something a little simpler: have you seen: https://github.com/tjfontaine/node-addon-layer?

Nope, I hadn't seen that, good to know that there's work in progress here.

Ya, I'm glad it's not duplication of work, just a different approach. I was considering this approach as a fallback in case my effort fails in a burning fire of misery. So I'm glad someone else has already started it :)

 

On this topic, there's probably some blockers to distributing something like this in npm, particularly given that there's a compiled component. I've played a little with having native plugins to leveldown, where you `npm install X` and X depends on leveldown existing so it can compile against it. Although in my case the dependency works the opposite way to what TJ's addon-layer will work in that you have one leveldown and multiple addons compiling against it. Here's some hackery I had to do to make it work: https://github.com/rvagg/node-downer-rangedel/blob/master/common.gypi - basically executing a node process to use require.resolve in a .gypi to work out where the installed version is.
The real problem for me though is that you can't determine compile order with an `npm install`, so if you `npm install leveldown downer-rangedel` then it'll likely fail because a leveldown compile will take longer than a downer-rangedel compile and they happen in parallel, so you have to `npm install leveldown; npm install downer-rangedel`. Which kind of sucks.

I discussed this briefly with Isaac to confirm my expectations, and I think I'm relatively safe in my assumptions.

This is partly why I wanted it to be a dependency in both the package.json and the binding.gyp. As it's relatively safe to assume that your dependency will be extracted before your build will start.

In truth though, addon-layer is defining a binding.gyp, which it certainly should *not*. Instead it should be addon.gyp such that it's only compiled when a module author adds the dependency in their binding.gyp.

The other reason I wanted it as a dependency is so I can explicitly state I want it to be a static library, and prevent some weird symbol clashing happening because of addon-layer itself.

 

Ideally we'd be able to do these things with npm and node-gyp:

• Easily determine the path to a dependency/peerDependency or some other dependency that you don't want to list in package.json (e.g. in my case I don't want leveldown as a dependency of downer-rangedel because I only want one instance of leveldown installed & compiled per application).  It'd be nice to be able to do something like "<@(node_dependency_path:addon_layer)" in your .gyp and have node-gyp take care of it for you.
  
• Specify build order in some way, perhaps a package.json property that says that this package shouldn't be compiled until some other package has been compiled.

Nathan and I have been bemoaning the need for node-gyp to define some helper macros for this scenario, as using `require.resolve` for this feels very fragile. But if we're going to do it a lot we should first class it.

Also, I hope we don't get too fancy with interdependency among binary modules, it's not quite as easy to hand wave over as it is in the js layer.

Keep up the great work!