Re: NPM on windows

221 views
Skip to first unread message

Isaac Schlueter

unread,
Jul 25, 2011, 9:17:21 PM7/25/11
to Luke Schafer, nodejs, np...@googlegroups.com
(cc'ed to nodejs and npm mailing lists, since people keep asking this)

On Mon, Jul 25, 2011 at 18:06, Luke Schafer <luke.s...@gmail.com> wrote:
> Perhaps a
> .Net app shelling calls to the node exe is the quickest solution?

Oh, please no! Porting npm to .Net would be a tremendous amount of
work. If you really want do this, that'd be totally fine, just don't
call it "npm" please, so that people don't expect me to support it.
Don't do it unless you think it'd be fun. If you DO think it would be
fun, then you have my blessing :)


> I'm keen to help, I just don't know where to start.

Ryan is right now working intently on making child_process.spawn work
on windows. That is a blocker for npm. npm runs scripts to install
packages, and shells out to execute tar and gzip. It cannot function
properly without child_process.spawn.

Next, npm depends on having gzip and tar in the OS already. That's a
bit painful for Windows users, but they'll probably just have to
suffer through installing gzip and tar themselves. (They'll have to
install gcc to make any binary addons work anyway, which is way more
painful.)

We've talked about getting gzip encoding/decoding streams into core,
but it's highly non-trivial. I have half of a fully-functional tar
implementation, but again, it's non-trivial, and requires a lot of
work yet.

Lastly, npm will have to write out .bat file shims instead of symlinks
for bin files. I haven't bothered with this just yet, since npm
requires child_process.spawn anyway, and can't run without it. But
very soon, I'll investigate that. It's been a long time since I did
any serious DOS scripting, but I know that it's possible to
programmatically generate a little proxy shim that runs a specific
program with the same set of arguments.

And, of course, there will be bugs. Several of them, I'm sure.

I hope to have all this done by the end of the year. It is a
several-month process.

John Hurliman

unread,
Jul 26, 2011, 12:36:49 AM7/26/11
to nod...@googlegroups.com, np...@googlegroups.com
On Mon, Jul 25, 2011 at 6:17 PM, Isaac Schlueter <i...@izs.me> wrote:
(cc'ed to nodejs and npm mailing lists, since people keep asking this)

...


Next, npm depends on having gzip and tar in the OS already.  That's a
bit painful for Windows users, but they'll probably just have to
suffer through installing gzip and tar themselves.  (They'll have to
install gcc to make any binary addons work anyway, which is way more
painful.)


It might be possible to just ship gzip.exe and tar.exe binaries with the Windows npm download. Also, earlier on this list I remember talk of supporting native code through npm where a developer could compile .dll files for whatever target Windows architectures they wanted to support. "npm install" would look for a binary package first before falling back to a local compilation toolchain. Is that path still being considered? It would align better with the typical Windows developer model where library maintainers are also expected to release binaries.

John

Jeroen Janssen

unread,
Jul 26, 2011, 2:08:51 AM7/26/11
to nodejs, np...@googlegroups.com
On Jul 26, 3:17 am, Isaac Schlueter <i...@izs.me> wrote:
> (cc'ed to nodejs and npm mailing lists, since people keep asking this)
>
> On Mon, Jul 25, 2011 at 18:06, Luke Schafer <luke.scha...@gmail.com> wrote:
> > Perhaps a
> > .Net app shelling calls to the node exe is the quickest solution?

Actually, the quickest solution is to just go to https://github.com/japj/ryppi
and fetch the python script.

ryppi is a "really super scaled down featureless node package manager
for node 0.4 and higher."
Since it is implemented in python it will also run on windows.

If you find a bug or need a certain missing feature in ryppi then feel
free to fork it, fix it and if you really want to then send me a pull
request.

> Ryan is right now working intently on making child_process.spawn work
> on windows.  That is a blocker for npm.  npm runs scripts to install
> packages, and shells out to execute tar and gzip.  It cannot function
> properly without child_process.spawn.

ryppi does not run scripts.

> Next, npm depends on having gzip and tar in the OS already.  That's a
> bit painful for Windows users, but they'll probably just have to
> suffer through installing gzip and tar themselves.  (They'll have to
> install gcc to make any binary addons work anyway, which is way more
> painful.)

ryppi uses pythons buildin tar/gz support, so no need to install
additional executables on windows.

> Lastly, npm will have to write out .bat file shims instead of symlinks
> for bin files.  I haven't bothered with this just yet, since npm
> requires child_process.spawn anyway, and can't run without it.  But
> very soon, I'll investigate that.  It's been a long time since I did
> any serious DOS scripting, but I know that it's possible to
> programmatically generate a little proxy shim that runs a specific
> program with the same set of arguments.

ryppi does not do symlinks to bin files or writing out .bat files

> And, of course, there will be bugs.  Several of them, I'm sure.

ryppi (of course) also has bugs.

> I hope to have all this done by the end of the year.  It is a
> several-month process.

ryppi is in a usable state *at this very moment*

If you are really desperate to have something like npm on windows, go
use ryppi for now.

Best regards,

Jeroen

Dh

unread,
Jul 26, 2011, 8:08:02 AM7/26/11
to nodejs
I have tried ryppi on windows at the office behind a proxy and it
works very well.

Donald


> Actually, the quickest solution is to just go tohttps://github.com/japj/ryppi

anand punitha

unread,
Jul 26, 2011, 2:26:17 AM7/26/11
to nodejs

beriberikix

unread,
Jul 28, 2011, 2:30:16 PM7/28/11
to nod...@googlegroups.com, Luke Schafer, np...@googlegroups.com
FWIW, I came across a node TAR module that might be helpful: tar-sync.

beriberikix

unread,
Jul 28, 2011, 2:47:43 PM7/28/11
to nod...@googlegroups.com, Luke Schafer
And here's a simple gist for GZIPing: https://gist.github.com/417483

Isaac Schlueter

unread,
Jul 28, 2011, 6:55:47 PM7/28/11
to nod...@googlegroups.com, Luke Schafer
The gzip gist looks kind of awesome. However, I am still somewhat
convinced that should be done in C. I suppose we could do something
like that in JS, and ship it with npm, and use that until gzip is
incorporated into node-core.

All but one of the tar implementations in JS that I've seen do not
properly handle extended pax headers, or global header files. The
only one I've seen that does (my node-tar repo) is pretty badly broken
in other ways. Here's the spec for the files that a conformant tar
program needs to be able to consume and generate:
http://cdrecord.berlios.de/private/man/star/star.4.html

(Note that many versions of gnutar and solaris tar do not satisfy this
requirement, either.)

On Thu, Jul 28, 2011 at 11:47, beriberikix <jmb...@gmail.com> wrote:
> And here's a simple gist for GZIPing: https://gist.github.com/417483
>

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

factor

unread,
Sep 2, 2011, 5:22:53 PM9/2/11
to nod...@googlegroups.com, Luke Schafer, np...@googlegroups.com
What's the hold up regarding symlinks? What is the limitation of symlinks on Windows that prevents them from being useful?

Jeroen Janssen

unread,
Sep 3, 2011, 3:39:01 AM9/3/11
to nodejs
For one thing, normal users (non admins) are not allowed to make
symlinks on windows.

factor

unread,
Sep 3, 2011, 11:13:47 AM9/3/11
to nodejs
Well I meant technical limitations wrt node/npm internals. If all it
was, was that npm needed to be run in an administrative context
(trivial and not the least bit of a hurdle for windows devs) then I
doubt it would even be a bullet point.

Marco Rogers

unread,
Sep 3, 2011, 6:01:12 PM9/3/11
to nod...@googlegroups.com
Weren't true symlinks introduced with Vista? Older versions have some clunky thing that only works with directories. If npm wants to support XP and Server 2003, that's a pretty big problem.

:Marco <- so not a windows guy

On Saturday, September 3, 2011 8:13:47 AM UTC-7, factor wrote:
Well I meant technical limitations wrt node/npm internals. If all it
was, was that npm needed to be run in an administrative context
(trivial and not the least bit of a hurdle for windows devs) then I
doubt it would even be a bullet point.

Isaac Schlueter

unread,
Sep 3, 2011, 9:02:17 PM9/3/11
to nod...@googlegroups.com
All that really needed to happen to get npm running on windows is for
someone familiar with the npm codebase to have access to a Windows
virtual machine, and time to dedicate to debugging it.

I've got Windows running, and this is my priority for the time being.
So far, I've run into about 10 random unexpected cross-platform
failures in the process of getting the `npm -h` usage banner to print.
It's just a grind that's gonna take some time. No rockets need
sciencing, no brains to surger.

Jeroen Janssen

unread,
Sep 4, 2011, 6:49:00 AM9/4/11
to nodejs
which reminds me... does "node debug" work yet on windows?

factor

unread,
Sep 5, 2011, 2:09:20 PM9/5/11
to nodejs
Well that's seems to be a bit of an exaggeration, npm can print the
help text without any code changes, once the initial submodules are
cloned. mkdir-p.js needed directory separation characters changed, and
you can get a bit farther. tar.js is where I got stalled, since
windows builds of tar seems to not accept absolute paths for -C
meaning you have to get clever to get it to work.

On Sep 3, 9:02 pm, Isaac Schlueter <i...@izs.me> wrote:
> All that really needed to happen to getnpmrunning onwindowsis for
> someone familiar with thenpmcodebase to have access to aWindows
> virtual machine, and time to dedicate to debugging it.
>
> I've gotWindowsrunning, and this is my priority for the time being.
> So far, I've run into about 10 random unexpected cross-platform
> failures in the process of getting the `npm-h` usage banner to print.
>  It's just a grind that's gonna take some time.  No rockets need
> sciencing, no brains to surger.
>
>
>
> On Sat, Sep 3, 2011 at 15:01, Marco Rogers <marco.rog...@gmail.com> wrote:
> > Weren't true symlinks introduced with Vista? Older versions have some clunky
> > thing that only works with directories. Ifnpmwants to support XP and
> > Server 2003, that's a pretty big problem.
> > :Marco <- so not awindowsguy
> > On Saturday, September 3, 2011 8:13:47 AM UTC-7, factor wrote:
>
> >> Well I meant technical limitations wrt node/npminternals. If all it
> >> was, was thatnpmneeded to be run in an administrative context
> >> (trivial and not the least bit of a hurdle forwindowsdevs) then I
> >> doubt it would even be a bullet point.
>
> >> On Sep 3, 3:39 am, Jeroen Janssen <jeroen....@gmail.com> wrote:
> >> > For one thing, normal users (non admins) are not allowed to make
> >> > symlinks onwindows.
>
> >> > On Sep 2, 11:22 pm, factor <factor...@gmail.com> wrote:
>
> >> > > What's the hold up regarding symlinks? What is the limitation of
> >> > > symlinks on
> >> > >Windowsthat prevents them from being useful?

Isaac Schlueter

unread,
Sep 8, 2011, 4:54:20 AM9/8/11
to nod...@googlegroups.com
On Mon, Sep 5, 2011 at 11:09, factor <factor...@gmail.com> wrote:
> Well that's seems to be a bit of an exaggeration, npm can print the
> help text without any code changes,

It took a bunch of code changes to be able to do that. Check the git
commit history ;)

> mkdir-p.js needed directory separation characters changed, and
> you can get a bit farther. tar.js is where I got stalled, since
> windows builds of tar seems to not accept absolute paths for -C
> meaning you have to get clever to get it to work.

Yep. We can't rely on there being a system tar. Need to do this in
JS. Bundling a tar.exe may be used as a stopgap.

C:\Documents and Settings\Administrator>tar
'tar' is not recognized as an internal or external command,
operable program or batch file.

Dean Landolt

unread,
Sep 8, 2011, 11:51:47 AM9/8/11
to nod...@googlegroups.com
On Thu, Sep 8, 2011 at 4:54 AM, Isaac Schlueter <i...@izs.me> wrote:
On Mon, Sep 5, 2011 at 11:09, factor <factor...@gmail.com> wrote:
> Well that's seems to be a bit of an exaggeration, npm can print the
> help text without any code changes,

It took a bunch of code changes to be able to do that.  Check the git
commit history ;)

> mkdir-p.js needed directory separation characters changed, and
> you can get a bit farther. tar.js is where I got stalled, since
> windows builds of tar seems to not accept absolute paths for -C
> meaning you have to get clever to get it to work.


It looked that way to my at first too, but it works just fine -- it just requires backslashes in paths to be escaped or turned into forward slashes. Goofy, sure, but to be expected. Perhaps node's path module could grow a function for normalizing windows paths for this use case. I'd even go as far as to say maybe it should always toString paths with a forward slash -- they work just fine.

 
Yep.  We can't rely on there being a system tar.  Need to do this in
JS.  Bundling a tar.exe may be used as a stopgap.

   C:\Documents and Settings\Administrator>tar
   'tar' is not recognized as an internal or external command,
   operable program or batch file.

Meh. I say msysgit+userland utils FTW :)

Isaac Schlueter

unread,
Sep 8, 2011, 12:55:11 PM9/8/11
to nod...@googlegroups.com
On Thu, Sep 8, 2011 at 08:51, Dean Landolt <de...@deanlandolt.com> wrote:
> I'd even go as far
> as to say maybe it should always toString paths with a forward slash -- they
> work just fine.

That's the approach PHP and a few other platforms have taken. You can
specify a path using c:\foo but when you get it back, it'll be c:/foo.
It's a little weird for newcomers who are familiar with Windows, but
it's easy to get used to, and does work fine.

Reply all
Reply to author
Forward
0 new messages