Re: NPM Downloading Binaries

[mscdex]

On Apr 16, 4:58 am, Richard Astbury <richard.astb...@gmail.com> wrote:
> Could the same approach be used to install other native modules?

My general issue with this approach is that you either have to bundle
compiled bits for every platform and configuration (ballooning the
tarball size due to less compression), or you have a custom npm script
that executes on install and downloads the appropriate binary files
for the current particular platform (this means an internet connection
is required wherever you want to install) from some place.

[Nathan Rajlich]

I do agree that this is an important piece of the native addon puzzle that is still missing. This would be extremely helpful for Windows users especially. The reality of making it happen though is not trivial, as we're talking about a build farm of real machines which cost real $ to buy, run and maintain.

Plus there's also the problem of "trustworthiness" of a binary. Who created the binary? Do I trust them? This isn't an insolvable problem, but it's just one of the many things we'll need to consider about this.

Also, perhaps TJ Fontaine would have some additional information here, since he set up the nodejs Jenkins builds which perhaps we could leverage somehow.

On Tue, Apr 16, 2013 at 1:58 AM, Richard Astbury <richard...@gmail.com> wrote:

I recently installed the 'phantomjs' module, and was surprised to see the installation script downloaded the phantomjs binary for the appropriate platform (i.e. it did this on linux and windows). This makes complete sense, phantom is probably both time consuming and complicated to compile, but it made the module installation quite slick.

Could the same approach be used to install other native modules? 

I know the node approach is to compile the source for native modules, but on Windows this is painful (sometimes very painful).

Perhaps the missing part of the equation is an automated build system for native modules, which would provide the built binaries for certain platforms. If you're on a compatible platform then NPM gets the binary from there, otherwise you're back to building it on your own.

Rich.

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Floby]

Wouldn't it be possible to publish a npm package with binaries which is arch-specific ?

like if you're installing node-fibers, the compiled sourced for my machine would be published at node-fibers-linux-x86_64 or something like this. if the package doesn't exist for my architecture, then just build it.

I realise that this would mean a lot of "unusable out of the box" packages, but it seems that a lot of package managers do things like that. apt ?

[Richard Astbury]

Here's a crazy idea for you.

Why not extend NPM to connect to a registry (maybe this is an additional registry, maybe the NPM one) to look for a compiled version of the native module for your OS. Perhaps it only does this if you supply a specific command line argument (hopefully this satisfies @mscdex's point).

The bigger problem is: how do you get the compiled modules into the repository?

My suggestion is to crowd source this, again using NPM. If you successfully compile a module which doesn't exist in the repository, NPM could then offer to upload it, thus making it available for others to download.

No need for an automated build system, no additional demands on the module author, only change required is to NPM.

Nate is absolutely right, there are some trust issues here. Perhaps there is a way to sign the module in some way to demonstrate authenticity?

Rich.

[Diogo Resende]

On Wednesday, April 17, 2013 at 9:28 , Richard Astbury wrote:

Here's a crazy idea for you.

Why not extend NPM to connect to a registry (maybe this is an additional registry, maybe the NPM one) to look for a compiled version of the native module for your OS. Perhaps it only does this if you supply a specific command line argument (hopefully this satisfies @mscdex's point).

 If going for the flag option, I think people will probably just ignore and compile locally. Perhaps this could be asked by npm when fetching a module? ("Module X needs compilation. There's one ready online, do you want to download it? - Author: ....")

The bigger problem is: how do you get the compiled modules into the repository?

My suggestion is to crowd source this, again using NPM. If you successfully compile a module which doesn't exist in the repository, NPM could then offer to upload it, thus making it available for others to download.

+1 

No need for an automated build system, no additional demands on the module author, only change required is to NPM.

Nate is absolutely right, there are some trust issues here. Perhaps there is a way to sign the module in some way to demonstrate authenticity?

This had to be signed by the user and the trust issue is still there. Whether you know the author and trust him or you don't. 

[Bradley Meck]

Arch specific registries using multiple registry support is much simpler.

npm --registries="ubuntu.x64.prebuilt.local registry.npmjs.org"

when requesting against "ubuntu.x64.prebuilt.local" a proxy would prebuild the module if it 404s. end. Can be done as an internal proxy as well.

[Isaac Schlueter]

Yes there are plans to do something like this. Package signing has to come first. Crowd sourcing isn't a great approach, partly for the trust issues, but also because no one will built on !$my_platform, and cross compiling is far from trivial. 

It's a bit trickier than it seems, but not rocket surgery. If you're interested in taking this on, let me know. If not, it'll come eventually, but not until some other higher-priority things get done first. 

--

[Richard Astbury]

Isaac,

I'm not rocket surgeon but I'm very willing to help where I can.

How can I get started?

Rich.

On Wednesday, April 17, 2013 7:36:34 PM UTC+1, Isaac Schlueter wrote:

Yes there are plans to do something like this. Package signing has to come first. Crowd sourcing isn't a great approach, partly for the trust issues, but also because no one will built on !$my_platform, and cross compiling is far from trivial. 

It's a bit trickier than it seems, but not rocket surgery. If you're interested in taking this on, let me know. If not, it'll come eventually, but not until some other higher-priority things get done first. 

On Wednesday, April 17, 2013, Bradley Meck wrote:

Arch specific registries using multiple registry support is much simpler.

npm --registries="ubuntu.x64.prebuilt.local registry.npmjs.org"

when requesting against "ubuntu.x64.prebuilt.local" a proxy would prebuild the module if it 404s. end. Can be done as an internal proxy as well.

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to

nodejs+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.