Is it legit to distribute binaries via npm?

[Marcel Laverdet]

Title says it all.

It seems that modules with a compile stage can sometimes be difficult for users to get installed. Ubuntu users may not have their node headers installed, Windows users may not have any compiler at all, Linux users may have broken build environments, there's just lots of issues that come up and often require manual intervention.

@japj right now is doing some awesome work getting fibers working on Windows (as far as I can tell it's pretty much done) and it's got me thinking about distribution for Windows users. It would be simple enough to build a version of each tag for Windows 32/64 & Linux 32/64/arm & OS X 32/64. Of course source would also be distributed so you could build it on your own if you're feeling adventurous or if you're on a different platform.

But is someone going to beat me up if I stick a bunch of compiled binaries into npm?

[mscdex]

On May 4, 6:58 pm, Marcel Laverdet <mar...@laverdet.com> wrote:
> But is someone going to beat me up if I stick a bunch of compiled binaries
> into npm?

Probably not, but it might depend on how easy it is to compile the
binary on Windows.

For example, my mmmagic module depends on libmagic which has a lot of
*nix-specific stuff and would take more time than I have to create a
floating patch for it to build easily under msvc. So what I ended up
doing as a workaround was to compile it under mingw (as a static lib)
and then link that to my addon. Since this requires much more than
just "install vc2010 express" (e.g. also requiring a proper mingw
setup, etc), IMHO it's well worth making a pre-compiled version of the
addon for Windows users.

Also, I think it'd be nice to let non-Windows users compile from
scratch (if possible), mostly because of the varying architectures/
platforms that node supports.

That's my 2 cents.

[Stewart Mckinney]

I'm pretty sure its legit as long as the module's build is organized in such a way as to provide the binaries/link the binaries as a fallback and not necessarily as a first option. i.e, if the module can compile natively, it should do so, but otherwise if it cannot, see if there is a supported binary for the system, and then include/link that.

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Marco Rogers]

I'd be fine if people distributed binaries as long as they're not huge. Npm doesn't try to build anything by default. The only real trick is having some way to tell npm which bin to use when doing install -g. It may be something Isaac is willing to help out with. I would ping him on this thread.

:Marco

[Morteza Milani]

Create tarballs for each platform that contains all binaries you need. serve them somewhere.

Create a downloader script that downloads tarballs based on users' platform.

Then in your package.json:

    "scripts": {

        "preinstall": "./bin/downloader || true; node-gyp rebuild"

    }

This is what I do here: https://github.com/milani/appjs/tree/master/bin

[Marcel Laverdet]

Why do that? Then you're just adding another point of failure.

--

[Glenn Block]

Side note: it is great to hear that fibers is getting ported :-)

For folks on Windows particularly it would be much more convenient to have binaries and would generally help promote much more node adoption.


Sent from my Windows Phone

---

From: Marcel Laverdet
Sent: 5/4/2012 3:58 PM
To: nodejs
Subject: [nodejs] Is it legit to distribute binaries via npm?

[Brandon Benvie]

Because the module has Chromium Embedded Frame as a dependency. It takes like an hour to compile on my pretty fast computer, and the Windows binaries are like 40mb. I think the linux binaries are 300mb.

On Saturday, May 5, 2012 12:51:48 PM UTC-4, Marcel wrote:

Why do that? Then you're just adding another point of failure.

[Marco Rogers]

Haha, no offense but it sounds like this isn't the best use case for distributing through npm :) I don't think people want to download a huge binary and certainly not multiple. I would suggest you provide links to downloadable binaries separately. The trouble is it's not just multiple platforms, but multiple node versions that you may want to support. I'm assuming your binary is a node extension. Is that true? If not there's another option. You could distribute your module without the binary and tell people to grab the binary themselves in whatever way they feel is best.

:Marco

[Morteza Milani]

> The trouble is it's not just multiple platforms, but multiple node versions that you may want to support. I'm assuming your binary is a node extension. Is that true?

No its not node extension.

>  You could distribute your module without the binary and tell people to grab the binary themselves in whatever way they feel is best.

Do you feel ok if there was no npm to install modules and you had to install them manually yourself? Of Course not! its just to ease the process. So my downloader is! This is what node-gyp does even.

I don't think its another point of failure. Its even a positive point:

1) Users don't need to download a huge node module that contains unnecessary files (those for other platforms). 

2) They don't need to choose among packages in a download page and put it in the proper place in directory structure!

[Isaac Schlueter]

The correct approach is to set up your module to compile using
node-gyp, such that it can be compiled on windows or on Unix.

Relying on a preinstall script to fetch stuff from a different site is
weird. In many ways, that's even worse than relying on install-time
compilation, because it's yet another service to trust and depend on.
What's worse, I'm guessing that the bin/downloader thing assumes bash
or sh or something.

Bundling multiple pre-compiled binaries is also awful.

Use a binding.gyp file. Use node-gyp. This is what we're going to be
optimizing for in the future.

[Marcel Laverdet]

For several hundred meg libraries the answer is much less obvious.. But even with a sophisticated or integrated build system, you still can't make assumptions about their environment? I think Node developers on Windows without a compiler even installed are quite common. Does node-gyp solve that?

[Brandon Benvie]

I don't think this is a reasonable requirement for a module that takes an hour to compile.

[Isaac Schlueter]

What modules take an hour to compile?

[Morteza Milani]

> What modules take an hour to compile? 

Binary files I use took 2 hours to compile. its a 12Gb repository. Should I pass it to users to compile?! No one likes that.

There are bugs in what I implemented I know. but It doesn't mean the approach is weird:)

I use node-gyp to compile node extension but It requires some extra libraries which are distributed in tarballs and installer script only downloads them as preinstall hook.

[Shancat]

If I had run an npm install and I came back to find it trying to download a 12GB repository I would be extremely 
pissed and left with a huge bill. In my area in Australia we can't even get ADSL, I have to resort to mobile broadband.

The situation is the same for many other users in Australia and, I'm sure, other places too.

[Morteza Milani]

> If I had run an npm install and I came back to find it trying to download a 12GB repository I would be extremely 
pissed and left with a huge bill.

Source code is 12GB, when compiled it would be much less like 300MB.

[Brandon Benvie]

Modules that have WebKit/Chromium in them.

[Isaac Schlueter]

!!?!?

What is this thing we're actually talking about?

If your module is 12GB, takes 2 hours to compile, or includes a copy
of WebKit or Chromium in it, then perhaps it's bigger than what you
should be installing using npm. Also, 12GB of source is certainly
nowhere near a 12GB gzipped tarball. In fact, your 300MB binaries are
very likely harder to compress. (There are exceptions, of course.
Maybe the repository contains a file with 12GB of /dev/random which
the compiler ignores.)

But anyway, it could be that's a large project, not a node module.
Deploy it the old fashioned way, like node does, by putting tarballs
on a website and building binaries for architectures where it is
reasonable to do so.

It's suboptimal, but it's sometimes quite reasonable to say, "You have
to download and install xyz to use this module". Maybe you could just
have people install webkit themselves, and write your little binding
as a thing that talks to it. The 'redis' module doesn't install redis
for you. The libmysql module doesn't install mysql. You wouldn't
expect a rubygems or cpan module to install Chrome.

Think of your users' experience. It's not just the install-time
experience. They're actually going to *use* this thing, right? A
module that ships as a binary is virtually impossible to debug. A
module that ships as 12GB of code is probably just as bad, if not
worse. That's an application. Treat it like one.

[Isaac Schlueter]

Spoke with Morteza in IRC. So... appjs *might* be an exception for
now. But once we have binary deployments working, it's going to have
to be changed around significantly, because as a matter of policy and
practical reality, bundled-binaries and custom install scripts will
not be effective. Gyp-based install scripts will be optimized and
strongly preferred.

[Brandon Benvie]

Yeah this is a specific issue with the WebKit stuff, although it does have an effect on a handful of modules that rely on that. node-webkit and appjs. I'm not sure how zombie handles it, though it's a bit different when using QT-webkit. Essentially boils down to distributing binaries, which mostly solves this problem and pushes it upstream to being able to distribute binaries for every platform.