remove "engines" from package.json?

[Isaac Schlueter]

Do people actually rely on the "engines" hash being respected in npm
installs any more? It was super essential in the early days when the
API was changing constantly, but now, it seems like it just makes it
unnecessarily difficult to upgrade stuff.

If no one is relying on this right now, I'm going to reduce it to a
warning. If in time, people start complaining that the warning is to
warny, we can remove it, and just let "engines" be a thing of the
past. (Note that `npm init` already doesn't bother with it, and for a
while has just defaulted to {"node":"*"}.)

Please let me know what you think. Thanks :)

[Bradley Meck]

At least some PaaS use it. I think it it were tied to tests in some way it would be more meaningful for the average user, but for now it most commonly used for PaaS deploys.

[Joshua Holbrook]

We use the engines field for application deployment to nodejitsu,
since we need to know what version of node the user prefers their
application to run on. That said, this is for applications and not for
libraries, and I'm not sure it makes much sense for libraries
excepting for special cases where a deprecated version of a library is
known to only work with certain versions (in which case you set the
versions deliberately), which should be much less common now that the
0.4/0.6 jump is complete.

I suppose an engines field could also be useful for binary package
builds, maybe. Not sure.

--Josh

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en



--
Joshua Holbrook
Engineer
Nodejitsu Inc.
jo...@nodejitsu.com

[Arnout Kazemier]

The only reason that I'm currently using the engines field is that I want to support the latest and greatest version of Node.js, I don't want to waste any of my time trying to support or back port stuff that is missing in older version of V8 or Node.js. What I do like to see in return / addition of this field is a way that could specify which version of our packages work with what node.js version.

[Vitaly Puzrin]

Good practice is to limit minimal engine version, but set max version unlimited.

That's tested for ages in multiple package managers. Please, don't experiment
with such functionality. There is nothing genious in cutting mature concepts.

среда, 27 июня 2012 г., 9:06:52 UTC+4 пользователь Isaac Schlueter написал:

[Tim Caswell]

I use it (minimum versions) for libraries that require the user to upgrade their node version.  For example, vfs-local requires node >0.6.16 because of the uid/gid fix that went into that version is essential or there are huge security holes.

--

[Glenn Block]

We were just about to take a dependency on it for Azure. Is there an alternative recommended approach? I believe Heroku uses this as well.

Sent from my Windows Phone

---

From: Bradley Meck
Sent: 6/26/2012 10:09 PM
To: nod...@googlegroups.com
Subject: [nodejs] Re: remove "engines" from package.json?

At least some PaaS use it. I think it it were tied to tests in some way it would be more meaningful for the average user, but for now it most commonly used for PaaS deploys.

On Wednesday, June 27, 2012 12:06:52 AM UTC-5, Isaac Schlueter wrote:

[Isaac Schlueter]

Thanks for the feedback, everyone.

It seems like we can achieve the optimum balance like this:

1. add an "engines-strict" config, default false
2. If "engines-strict" == true, then the current behavior.
3. if "engines-strict" == false, then it's a warning

Then Heroku and Azure and the like can set the flag in their envs if
they want to (or just look at the engines value and do whatever they
want with it), and users will be alerted about the need to upgrade
(but without being forced to, or as is more often the case, forced NOT
to.)

Adding checking for "or higher" would seem to be too subtle a change
to me. Ie, if your package says `"engines":"0.4.x"`, and I have node
0.8, do you *really* mean >=0.6.0? Or do you mean that you rely on
listenFD, which is gone in 0.6, and reimplemented in 0.8, but somewhat
differently?

It seems like the better option is just to warn you that it might
break, and then let it break.

Another option might be to let authors add an "enginesStrict" boolean
flag in their package.json which would say, "No, seriously, this WILL
NOT WORK except with the specified versions, so don't even try to use
it."

What we're seeing in practice is just weird and broken behavior,
usually because an old version of something (say, a package from 2
years ago, which doesn't work on modern node) might have no engines
specified, but a newer version of the same package (which works fine
on 0.8) has "engines":{"node":"0.6.x"} specified. So, npm tries to
find the most recent version that it thinks will work, and it gets
0.1.2 (which is broken and old) instead of 4.8.12 (which is new, and
works). That's the opposite of the intent. A nag would be more
appropriate in that case, I think.

[Matt]

On Wed, Jun 27, 2012 at 1:45 PM, Isaac Schlueter <i...@izs.me> wrote:

What we're seeing in practice is just weird and broken behavior,
usually because an old version of something (say, a package from 2
years ago, which doesn't work on modern node) might have no engines
specified, but a newer version of the same package (which works fine
on 0.8) has "engines":{"node":"0.6.x"} specified.  So, npm tries to
find the most recent version that it thinks will work, and it gets
0.1.2 (which is broken and old) instead of 4.8.12 (which is new, and
works).  That's the opposite of the intent.  A nag would be more
appropriate in that case, I think.

What CPAN does is always get the most recent, and if it fails because of a reason like this (in Perl terms, the Makefile.PL has a "use <version>" tag), it just fails. To get an older version you have to ask for that version specifically. Would this not work here?

[Glenn Block]

In our case we are not using this for our own modules. We only want to
use this to allow someone to specify what version of node they would
like us to run their code on when deployed.

Sent from my Windows Phone

From: Isaac Schlueter
Sent: 6/27/2012 10:46 AM
To: nod...@googlegroups.com
Cc: Bradley Meck
Subject: Re: [nodejs] Re: remove "engines" from package.json?

Thanks for the feedback, everyone.

It seems like we can achieve the optimum balance like this:

1. add an "engines-strict" config, default false
2. If "engines-strict" == true, then the current behavior.
3. if "engines-strict" == false, then it's a warning

Then Heroku and Azure and the like can set the flag in their envs if
they want to (or just look at the engines value and do whatever they
want with it), and users will be alerted about the need to upgrade
(but without being forced to, or as is more often the case, forced NOT
to.)

Adding checking for "or higher" would seem to be too subtle a change
to me. Ie, if your package says `"engines":"0.4.x"`, and I have node
0.8, do you *really* mean >=0.6.0? Or do you mean that you rely on
listenFD, which is gone in 0.6, and reimplemented in 0.8, but somewhat
differently?

It seems like the better option is just to warn you that it might
break, and then let it break.

Another option might be to let authors add an "enginesStrict" boolean
flag in their package.json which would say, "No, seriously, this WILL
NOT WORK except with the specified versions, so don't even try to use
it."

What we're seeing in practice is just weird and broken behavior,
usually because an old version of something (say, a package from 2
years ago, which doesn't work on modern node) might have no engines
specified, but a newer version of the same package (which works fine
on 0.8) has "engines":{"node":"0.6.x"} specified. So, npm tries to
find the most recent version that it thinks will work, and it gets
0.1.2 (which is broken and old) instead of 4.8.12 (which is new, and
works). That's the opposite of the intent. A nag would be more
appropriate in that case, I think.

[Glenn Block]

Totally get the concern. I think one way we could prevent that is not
supporting ranges but requiring you to choose a specific version.

[P. Douglas Reeder]

It sounds like the problem is that metadata is not correct, rather than npm's algorithm is wrong.

That suggests a twofold approach:
1) bug authors to fix their metadata
2) allow user intervention on a package-by-package basis to override.

Globally ignoring metadata sounds like a recipe for DLL-hell

[Dan Shaw]

I'm +1 for reducing it to a warning and letting infrastructure
companies build, adapt and extend it to their needs in userland.

Daniel Shaw
@dshaw

On Wed, Jun 27, 2012 at 10:45 AM, Isaac Schlueter <i...@izs.me> wrote:
> Thanks for the feedback, everyone.
>
> It seems like we can achieve the optimum balance like this:
>
> 1. add an "engines-strict" config, default false
>

[Matt]

I'm -1 for reducing it to a warning. By doing that you're taking the opposite assumption, that the person who wrote the package doesn't know what he/she's doing. What if the package uses domains and puts in engines: ">=0.8.0" ? By reducing it to a warning you're letting people's code fail at runtime instead of at install time.

[Ian Young]

I share Matt's concern. I'd be bummed if I wanted to release a new version of a module that depended on 0.8 features, but doing so would mean most of my users on 0.6 got a broken version installed by default with only a warning to dissuade them. Especially if there were perfectly good older versions sitting in npm that worked on 0.6.

Maybe npm could issue a warning on maximum version failures, but keep the old behavior on minimum version failures. That's getting kinda complex, though.

I wonder if this isn't better solved by guiding the community towards removing upper limits from engines in all but the most specific cases?

[Matt]

How about this fix for npm: 

if (node version > package's "engines" tag), then don't track back to look for older versions.

And specifically, when tracking back to look for older versions, only look at older versions that HAVE the "engines" tag present, not ones lacking it.

[Isaac Schlueter]

Matt, Ian,

It was once said, in the Long Ago Days of the Time Before,

> Another option might be to let authors add an "enginesStrict" boolean
> flag in their package.json which would say, "No, seriously, this WILL
> NOT WORK except with the specified versions, so don't even try to use
> it."

How does that strike you? Then it would be a warning, unless the
author says, "No, srsly, I really mean it," in which case, it'd use
the current behavior. (And please don't do that for 0.8.x, because
it's probably just going to be annoying in a few months, like how
"0.6.x" is now.)

[Ryan Schmidt]

On Jun 27, 2012, at 15:33, Ian Young wrote:

> I share Matt's concern. I'd be bummed if I wanted to release a new version of a module that depended on 0.8 features, but doing so would mean most of my users on 0.6 got a broken version installed by default with only a warning to dissuade them. Especially if there were perfectly good older versions sitting in npm that worked on 0.6.
>
> Maybe npm could issue a warning on maximum version failures, but keep the old behavior on minimum version failures. That's getting kinda complex, though.
>
> I wonder if this isn't better solved by guiding the community towards removing upper limits from engines in all but the most specific cases?

To encourage this, npm could print a warning if the engines line has an upper limit. But that might be annoying for packages that actually do have an upper limit of supported node versions.

[Vitaly Puzrin]

You try to make magic in place, where it is scrictly unwelcome.

Broken metadata is just broken metadata, but not the problem, that should be solved in npm.
If package is not updated for ages and has wrong dependencies - just let him die.

среда, 27 июня 2012 г., 21:45:51 UTC+4 пользователь Isaac Schlueter написал:

> nodejs+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to

> nodejs+unsubscribe@googlegroups.com

[Isaac Schlueter]

Vitaly,

In practice, what happens is that a new version of Node comes out, and
no one can use it until everyone else uses it, which is not great for
me or for this community. We need to be moving forward. In a lot of
cases, the metadata was reasonable at the time, and added by `npm
init`. We encourage the creation of many hundreds of small reusable
modules every day. We can't then turn around and penalize people for
using the tools, or expect that they spend lots of time doing boring
hoop-jumping maintenance.

It's done: https://github.com/isaacs/npm/commit/ee1d168d7a4798b67bb9a7667b5ec93a8be3d953
Engines is advisory. It prints a warning. Set `engineStrict` in your
package.json, or `--engine-strict` config, to make it strict. (If the
package.json field is abused, it will be removed eventually. I don't
suspect this is going to be an issue.)

>> > nodejs+un...@googlegroups.com

>> > For more options, visit this group at
>> > http://groups.google.com/group/nodejs?hl=en?hl=en
>> >
>> > --
>> > Job Board: http://jobs.nodejs.org/
>> > Posting guidelines:
>> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> > You received this message because you are subscribed to the Google
>> > Groups "nodejs" group.
>> > To post to this group, send email to nod...@googlegroups.com
>> > To unsubscribe from this group, send email to

>> > nodejs+un...@googlegroups.com

>> > For more options, visit this group at
>> > http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to

> nodejs+un...@googlegroups.com

[Tim Caswell]

Isaac, so, in strict mode, do we still have the funky fallback logic where it tries to get the next best version of the package to satisfy the current environment?  If so, I think we should change it to only attempt the preferred version and either warn or throw depending on the strict flag.

For example, I published a version of vfs-local that allowed any node version >=0.6.0, but then later found out that node itself had a nasty security hole that was fixed in 0.6.16.  I don't want anyone running my code in an older node, so I add the strict flag to my package.json flag and leave the open-ended >=0.6.16 range for engine.  So what happens then when someone on 0.6.15 tried to install my library. Will it throw an error, or will it load the previously published version that had the >=0.6.0 constraint and the nasty security hole?

[Marcel Laverdet]

"Broken metadata is just broken metadata"

This hits the nail on the head. This fix to me feels very short-sighted and PHP-y to me. If this is a bad problem I feel the correct solution would be to add a flag to npm install such that you could skip the engines check: `npm install --force some-legacy-package`.

Regarding engines, I'm using it in node-fibers.. "node":">=0.5.2". There are older versions of fibers that work with older versions of node, and those are also marked correctly in their metadata.

nodejs+un...@googlegroups.com

[Vitaly Puzrin]

Advisory mode is artificial kludge. Adding new key to config breaks compatibility & KISS principle.
If someone needs to override conditions, it's common to add option "--force" or similar to CLI.

IMHO, in practice, you just fuckuped all packages, that really needed strong node version dependencies.
I don't know package managers from other languages, who support such logic. And i don't understand,
why node should resolve situation in another way.


четверг, 28 июня 2012 г., 5:16:20 UTC+4 пользователь Isaac Schlueter написал:

>> > nodejs+unsubscribe@googlegroups.com

>> > For more options, visit this group at
>> > http://groups.google.com/group/nodejs?hl=en?hl=en
>> >
>> > --
>> > Job Board: http://jobs.nodejs.org/
>> > Posting guidelines:
>> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> > You received this message because you are subscribed to the Google
>> > Groups "nodejs" group.
>> > To post to this group, send email to nod...@googlegroups.com
>> > To unsubscribe from this group, send email to

>> > nodejs+unsubscribe@googlegroups.com

>> > For more options, visit this group at
>> > http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to

> nodejs+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

четверг, 28 июня 2012 г., 5:16:20 UTC+4 пользователь Isaac Schlueter написал:

>> > nodejs+unsubscribe@googlegroups.com

>> > For more options, visit this group at
>> > http://groups.google.com/group/nodejs?hl=en?hl=en
>> >
>> > --
>> > Job Board: http://jobs.nodejs.org/
>> > Posting guidelines:
>> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> > You received this message because you are subscribed to the Google
>> > Groups "nodejs" group.
>> > To post to this group, send email to nod...@googlegroups.com
>> > To unsubscribe from this group, send email to

>> > nodejs+unsubscribe@googlegroups.com

>> > For more options, visit this group at
>> > http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to

> nodejs+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

четверг, 28 июня 2012 г., 5:16:20 UTC+4 пользователь Isaac Schlueter написал:

>> > nodejs+unsubscribe@googlegroups.com

>> > For more options, visit this group at
>> > http://groups.google.com/group/nodejs?hl=en?hl=en
>> >
>> > --
>> > Job Board: http://jobs.nodejs.org/
>> > Posting guidelines:
>> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> > You received this message because you are subscribed to the Google
>> > Groups "nodejs" group.
>> > To post to this group, send email to nod...@googlegroups.com
>> > To unsubscribe from this group, send email to

>> > nodejs+unsubscribe@googlegroups.com

>> > For more options, visit this group at
>> > http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to

> nodejs+unsubscribe@googlegroups.com

[tjholowaychuk]

Capping the low-end is reasonable but yeah I've really given myself
more work by using this

On Jun 27, 7:29 pm, Vitaly Puzrin <vit...@rcdesign.ru> wrote:
> Advisory mode is artificial kludge. Adding new key to config breaks
> compatibility & KISS principle.
> If someone needs to override conditions, it's common to add option
> "--force" or similar to CLI.
>
> IMHO, in practice, you just fuckuped all packages, that really needed
> strong node version dependencies.
> I don't know package managers from other languages, who support such logic.
> And i don't understand,
> why node should resolve situation in another way.
>
> четверг, 28 июня 2012 г., 5:16:20 UTC+4 пользователь Isaac Schlueter
> написал:
>
>
>
>
>
>
>
>
>
>
>
> > Vitaly,
>
> > In practice, what happens is that a new version of Node comes out, and
> > no one can use it until everyone else uses it, which is not great for
> > me or for this community.  We need to be moving forward.  In a lot of
> > cases, the metadata was reasonable at the time, and added by `npm
> > init`.  We encourage the creation of many hundreds of small reusable
> > modules every day.  We can't then turn around and penalize people for
> > using the tools, or expect that they spend lots of time doing boring
> > hoop-jumping maintenance.
>
> > It's done:

> >https://github.com/isaacs/npm/commit/ee1d168d7a4798b67bb9a7667b5ec93a...

> > >> On Wed, Jun 27, 2012 at 8:50 AM, Glenn Block <glenn.bl...@gmail.com>

> > >> > nodejs+un...@googlegroups.com

> > >> > For more options, visit this group at
> > >> >http://groups.google.com/group/nodejs?hl=en?hl=en
>
> > >> > --
> > >> > Job Board:http://jobs.nodejs.org/
> > >> > Posting guidelines:
> > >> >https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> > >> > You received this message because you are subscribed to the Google
> > >> > Groups "nodejs" group.
> > >> > To post to this group, send email to nod...@googlegroups.com
> > >> > To unsubscribe from this group, send email to

> > >> > nodejs+un...@googlegroups.com

> > >> > For more options, visit this group at
> > >> >http://groups.google.com/group/nodejs?hl=en?hl=en
>
> > > --
> > > Job Board:http://jobs.nodejs.org/
> > > Posting guidelines:
> > >https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> > > You received this message because you are subscribed to the Google
> > > Groups "nodejs" group.
> > > To post to this group, send email to nod...@googlegroups.com
> > > To unsubscribe from this group, send email to

> > > nodejs+un...@googlegroups.com

> > > For more options, visit this group at
> > >http://groups.google.com/group/nodejs?hl=en?hl=en
>
> четверг, 28 июня 2012 г., 5:16:20 UTC+4 пользователь Isaac Schlueter
> написал:
>
>
>
>
>
>
>
>
>
> > Vitaly,
>
> > In practice, what happens is that a new version of Node comes out, and
> > no one can use it until everyone else uses it, which is not great for
> > me or for this community.  We need to be moving forward.  In a lot of
> > cases, the metadata was reasonable at the time, and added by `npm
> > init`.  We encourage the creation of many hundreds of small reusable
> > modules every day.  We can't then turn around and penalize people for
> > using the tools, or expect that they spend lots of time doing boring
> > hoop-jumping maintenance.
>
> > It's done:

> >https://github.com/isaacs/npm/commit/ee1d168d7a4798b67bb9a7667b5ec93a...

> > >> NOT WORK...
>
> read more »

[Vitaly Puzrin]

In your case problem was with upper limit. It's a very rarely needed option.

From the other hand, users never read warnings. If i've marked my module as >= 0.6,
then i'll get complains now, that it produces strange bugs under 0.4.

четверг, 28 июня 2012 г., 7:26:06 UTC+4 пользователь tjholowaychuk написал:

[Matt]

On Wed, Jun 27, 2012 at 5:36 PM, Isaac Schlueter <i...@izs.me> wrote:

> Another option might be to let authors add an "enginesStrict" boolean
> flag in their package.json which would say, "No, seriously, this WILL
> NOT WORK except with the specified versions, so don't even try to use
> it."

How does that strike you?  Then it would be a warning, unless the
author says, "No, srsly, I really mean it," in which case, it'd use
the current behavior.  (And please don't do that for 0.8.x, because
it's probably just going to be annoying in a few months, like how
"0.6.x" is now.)

I think it's a dumb idea, sorry - why even have the "engines" flag now? Are we next going to have dependenciesStrict?

What was wrong with the fix I suggested?

[Ian Young]

On Wednesday, June 27, 2012 2:36:57 PM UTC-7, Isaac Schlueter wrote:

Matt, Ian,

It was once said, in the Long Ago Days of the Time Before,

> Another option might be to let authors add an "enginesStrict" boolean
> flag in their package.json which would say, "No, seriously, this WILL
> NOT WORK except with the specified versions, so don't even try to use
> it."

How does that strike you?  Then it would be a warning, unless the
author says, "No, srsly, I really mean it," in which case, it'd use
the current behavior.  (And please don't do that for 0.8.x, because
it's probably just going to be annoying in a few months, like how
"0.6.x" is now.)

That sounds decent to me. It's a bit more complexity and a bit of a band-aid, but if you're okay with it, I'm okay with it. 

[Isaac Schlueter]

On Thu, Jun 28, 2012 at 6:10 AM, Matt <hel...@gmail.com> wrote:
> I think it's a dumb idea, sorry - why even have the "engines" flag now?

Indeed. Removing it entirely was my first suggestion :)

> Are we next going to have dependenciesStrict?

No, dependencies are a fundamentally different sort of thing. You can
have multiple different copies of a js lib in memory, but you can't be
running on multiple different versions of node at once. The engines
restriction is a way for author A to prevent author B from upgrading,
which is bad.

> What was wrong with the fix I suggested?

I see two things you suggested. (Were there more that I'm missing?)

> What CPAN does is always get the most recent, and if it fails because
> of a reason like this (in Perl terms, the Makefile.PL has a "use <version>"
> tag), it just fails. To get an older version you have to ask for that version
> specifically. Would this not work here?

That's basically what I'm proposing as the default. Install the
latest available, and if it breaks, oh well. The "engines"
restriction is thus an advisory warning.

npm != CPAN. In many ways, CPAN is better (mostly owing to many more
years), but in others, npm is better (mostly owing to being invented
in an era when CPAN already existed.) "Just try the latest and then
fail" is not exactly the npm way, and making npm behave exactly like
CPAN would be a bit silly.

> if (node version > package's "engines" tag), then don't track back to look for older versions.

The problem is that it's a range, not a single value. (Though a
single semver value IS a valid range, just a very small one.)

What does it mean to be "greater than" a range like "0.5.4 || 0.5.6 ||
0.8 || <=0.9.4"?

To make that a meaningful question, we'd have to loop over all the
||-separated comparators, parse each into a "simple range" (ie, a
collection of space-separated /^((?:[<>])=?)({semver})/ tokens), and
then find the max of those that is a < or <= or =.

If the max is <, then the version would have to be >={semver}, and if
it is <= or =, then it'd have to be >{semver}.

So, then you'd have a set of maximums, and find the max of those? So,
for the range above, >{range} would mean >0.9.4.

In other words, version ranges are not necessarily contiguous or
differentiable. They have gaps. We can decide what "greater than a
range" means, but it's not quite as trivial as just skipping the
question entirely, and I'm not sure that the benefit is a big enough
deal to matter.

On Wed, Jun 27, 2012 at 11:04 PM, Vitaly Puzrin <vit...@rcdesign.ru> wrote:
> From the other hand, users never read warnings. If i've marked my module as
>>= 0.6,
> then i'll get complains now, that it produces strange bugs under 0.4.

Vitaly, I think some of your foreboding is a bit extreme here. I get
more node bugs landing in my inbox than probably anyone else on this
list. 0.4 is dead. Virtually no one is using it, and the people that
ARE using it, are using it specifically because they're not upgrading
everything or doing new work. This is not a real concern, and if it
ever becomes one, it won't be all that hard to address.

On Wed, Jun 27, 2012 at 7:26 PM, Marcel Laverdet <mar...@laverdet.com> wrote:
> If this is a bad problem I feel the correct solution would be
> to add a flag to npm install such that you could skip the engines check:
> `npm install --force some-legacy-package`.

Marcel, the fact that you suggest we "add" this is proof to me that it
is insufficient. --force has behaved this way for many months now.

> Regarding engines, I'm using it in node-fibers.. "node":">=0.5.2". There are
> older versions of fibers that work with older versions of node, and those
> are also marked correctly in their metadata.

The current version of npm requires node 0.6, and breaks super badly
on node 0.4. So really, this is a question about how node will change
moving forward. (New thread coming soon, regarding our stance on
future api breakage, stay tuned.)


If it causes massive problems, we can just revert it. It's not THAT
big of a deal. The main issue is that `npm init` was for several
versions too restrictive about the defualt "engines" value, and people
didn't even realize that they were making work for themselves in the
future. (And in nodes <=0.4, it was a much greater need to specify
the working engine precisely.) Since I removed a default engine from
`npm init`, most people don't even use it, and it's usually just a
silly reason to not upgrade.

[Glenn Block]

It does raise the question of if hosters should rely on engines at
all...

What if there's was a way to say in package.json this an 'app' vs
a 'library'. Then I could somehow indicate in package.json what version
of node my app requires, but do it in a way that is agnostic to host?

Sent from my Windows Phone

From: Isaac Schlueter
Sent: 6/28/2012 12:33 PM
To: nod...@googlegroups.com
Subject: Re: [nodejs] Re: remove "engines" from package.json?

[Ryan Schmidt]

I didn't know that was supported and I wonder how many module authors know that. Can we find out how many modules actually use those kinds of complicated engines specifications?

[Mikeal Rogers]

without having done any research I'd say between 0 and 4.

[Dan Shaw]

Express used to be one of the notable ones. TJ's removed that now, though.

Daniel Shaw
@dshaw

[Matt]

On Thu, Jun 28, 2012 at 3:33 PM, Isaac Schlueter <i...@izs.me> wrote:

> What CPAN does is always get the most recent, and if it fails because
> of a reason like this (in Perl terms, the Makefile.PL has a "use <version>"
> tag), it just fails. To get an older version you have to ask for that version
> specifically. Would this not work here?

That's basically what I'm proposing as the default.  Install the
latest available, and if it breaks, oh well.  The "engines"
restriction is thus an advisory warning.

Well I'm suggesting it checks if it breaks and requires you to do "npm install --force" or something like that if you want to ignore the "engines" flag.

That way you don't let things break at run time. You let the admin make a decision. And you break at npm install time, which is how it should be.

 

npm != CPAN.  In many ways, CPAN is better (mostly owing to many more
years), but in others, npm is better (mostly owing to being invented
in an era when CPAN already existed.)  "Just try the latest and then
fail" is not exactly the npm way, and making npm behave exactly like
CPAN would be a bit silly.

Of course I'm just suggesting things that work elsewhere in these scenarios - not saying you should be the same as CPAN (though I do dream of one day having something like the CPAN testers system).

> if (node version > package's "engines" tag), then don't track back to look for older versions.

The problem is that it's a range, not a single value.

Ah, yeah didn't think that one through - thanks for the thorough explanation - I was thinking of a single value.

Then what's wrong with npm just failing if "engines" doesn't match on the latest version of the module? The user can then choose to install an earlier version, or use --force.

Matt.

[Marcel Laverdet]

> Marcel, the fact that you suggest we "add" this is proof to me that it
> is insufficient.  --force has behaved this way for many months now.

If the flag already existed, why not add a tip to the ENOTSUP failure? If the only thing that is broken is user education, then why not add education right when the user needs it?

--- a/lib/utils/error-handler.js

+++ b/lib/utils/error-handler.js

@@ -202,7 +202,8 @@ function errorHandler (er) {

                 ,"Required: "+JSON.stringify(er.required)

                 ,"Actual:   "

                 +JSON.stringify({npm:npm.version

-                                ,node:npm.config.get("node-version")})

+                                ,node:npm.config.get("node-version")}),

+                                "If you want to try anyway, please use the --force flag to skip this check."

                 ].join("\n"))

       break

     } // else passthrough

> If it causes massive problems, we can just revert it.  It's not THAT
> big of a deal.  The main issue is that `npm init` was for several
> versions too restrictive about the defualt "engines" value, and people
> didn't even realize that they were making work for themselves in the
> future.  (And in nodes <=0.4, it was a much greater need to specify
> the working engine precisely.)  Since I removed a default engine from
> `npm init`, most people don't even use it, and it's usually just a
> silly reason to not upgrade.

You're right, it isn't that big of a deal on it's own.. My concern here is that this is a permanent solution to a temporary problem and is a culture not really healthy to long-term software development. It seems Node has always been of the mindset "maintain your shit or it may stop working in the future", at least until the platform is more mature. This is a new flag everyone has to be aware of now. Reading the documentation for "engines" now everyone will have to read and understand "enginesStrict" for the rest of time.

It reminds me of how in PHP, E_ALL doesn't actually mean "all errors". You have to do `E_ALL | E_STRICT` if you *really* want all errors. Or how after they created `mysql_escape_string` they realized it didn't work correctly non-latin1 charsets and had to add `mysql_real_escape_string`.

I know there are concerns of adoption of new versions of node, but cruft is never a good thing to add.

[Joshua Holbrook]

I think part of the issue is that node is maturing faster than people
thought it would.

--Josh

--
Joshua Holbrook
Engineer
Nodejitsu Inc.
jo...@nodejitsu.com

[Isaac Schlueter]

So, the real question here is about what our policies should be based on.

Is it more important to never upgrade for fear of breaking changes, or
to upgrade as soon as possible?

We need to reduce the barriers to upgrading node. In this case, it's
pretty easy to change the default behavior, see how it goes, and maybe
change it back. It's unlikely to ruin anyone's day in a significant
way (unlike, say, require("sys") throwing. See that other thread.)


Tim, Sorry, I missed this very good response of yours in my last reply:

On Wed, Jun 27, 2012 at 7:03 PM, Tim Caswell <t...@creationix.com> wrote:
> For example, I published a version of vfs-local that allowed any node
> version >=0.6.0, but then later found out that node itself had a
> nasty security hole that was fixed in 0.6.16.  I don't want anyone running
> my code in an older node, so I add the strict flag to my package.json flag
> and leave the open-ended >=0.6.16 range for engine.

Did you unpublish the versions that allowed >=0.6.0? (You probably should.)

> So what happens then
> when someone on 0.6.15 tried to install my library. Will it throw an error,
> or will it load the previously published version that had the >=0.6.0
> constraint and the nasty security hole?

Today, if you didn't remove the older version, then they'll get the
old version of vfs-local. With this change, they'll get the new
version of vfs-local, and see a warning that their node version needs
to be upgraded.

If you did unpublish, then before this change, they'd get an ENOTSUP
(and probably run with --force, and get no warning or error.) With
the change, they'd get the warning.

Seems to me like actually a slightly better outcome in both cases.

[Matt]

The problem is clearly that npm tries too hard to install older versions than the most recent one. That's the bug here. Just try and install the most recent one and fail hard if it doesn't match the requirements and if --force isn't set.

[Matt]

On Thu, Jun 28, 2012 at 9:12 PM, Isaac Schlueter <i...@izs.me> wrote:

> So what happens then
> when someone on 0.6.15 tried to install my library. Will it throw an error,
> or will it load the previously published version that had the >=0.6.0
> constraint and the nasty security hole?

Today, if you didn't remove the older version, then they'll get the
old version of vfs-local.  With this change, they'll get the new
version of vfs-local, and see a warning that their node version needs
to be upgraded.

If you did unpublish, then before this change, they'd get an ENOTSUP
(and probably run with --force, and get no warning or error.)  With
the change, they'd get the warning.

Seems to me like actually a slightly better outcome in both cases.

Seems to me like a bloody awful outcome. So you're saying now that if we find a security issue in an earlier version of node or our libraries, that we have to unpublish all older versions, or just hope that upon install (which npm can often have a lot of output) we have to watch for some warning message, or we get some massive security hole?

I ask again, what's wrong with: try the latest version. If engines doesn't match, FAIL HARD. Unless they use --force. Stop trying to backtrack to an earlier version.

Matt.

[Isaac Schlueter]

Matt,

What you describe is rare. Engines causing unnecessary pain is
*extremely* common.

As it stands, like I said, the prior behavior didn't really solve your
problem. In fact, it made it worse. (Silently installs the older
unsafe version.)

What you need to do in that case is with the new behavior is:

1. Remove the old copies of the module.
2. add to package.json: "engines": {"node": ">=0.6.16"}, "engineStrict": true

What you had to do with the prior behavior was:

1. Remove old copies of the module.
2. add to package.json: "engines": {"node": ">=0.6.16"}

So, for this fairly rare exception, you have to add a few extra bytes
to your package.json file (and people will be more likely to actually
*notice* the failure, because it won't be "Oh, yeah, that means i have
to -f it", it'll be "Wait, wtf?!"). Even if you don't add the strict
flag, users will get a warning, whereas before, they'd get the wrong
thing silently.

What we get in return for this overhead is *everyone else* gets to
upgrade with way less pain in the much more common case.

You are making this discussion about something it's not. The
"engines" field was never intended to be used as a way to guard
against security problems, and it has never been an effective way to
do so.

If you have a security problem in a package, unpublish it. If you
want to make a package "FAIL HARD", then add
"scripts":{"prepublish":"exit 1"} and it will fail as hard as
possible.

[Matt]

On Thu, Jun 28, 2012 at 11:41 PM, Isaac Schlueter <i...@izs.me> wrote:

What you need to do in that case is with the new behavior is:

1. Remove the old copies of the module.
2. add to package.json: "engines": {"node": ">=0.6.16"}, "engineStrict": true

What you had to do with the prior behavior was:

1. Remove old copies of the module.
2. add to package.json: "engines": {"node": ">=0.6.16"}

The problem with your solution is you're still allowing backtracking. A *much* more common scenario would be someone writes a first version of a package, without knowing much about npm or package.json contents, and publishes it. Maybe a few versions. Then he gets bug reports that it does require('domain') and so only works on 0.8+, so adds in the engines: tag to "fix" this. After all, that's what the docs indicate it does. What the docs don't indicate is that he now also has to delete all old versions of his module, because otherwise npm will just install it anyway on 0.6.

I still didn't see a valid argument as to why we can't error out (instead of warning) if the engines doesn't match. It seems like even with enginesStrict: true you haven't actually fixed anything, because node will still go back in history trying to find a version that will work.

Why is trying to go back in history to find a version that matches so important? If users want a specific version they can do that in dependencies - if they don't specify that, why would you *ever* want to give them anything other than the latest version?

Matt.

[Isaac Schlueter]

Matt,

> Why is trying to go back in history to find a version that matches so
> important?

It's important because there is a much more common case than the one
you're describing: You write version 1.0 with node 0.6 in mind. Make
changes, updates, yadda yadda. Then 0.8 comes out, and you add
support for that. You release 2.0 with the engines setting on it.

For a while, you may even be releasing updates to both the 1.0 and the
2.0 versions! So, you'd prefer to have 0.6.x users get the 1.0.latest
and 0.8.x users get the 2.0.latest.

I'm not saying that the backtracking behavior we've had is perfect,
but it's what's been there for a few years. The change you're
suggesting is significantly more subtle and has more side effects than
simply making engines advisory, with the option of opting into the old
behavior.

When in doubt, make changes that are minimal and easily reverted.

On Fri, Jun 29, 2012 at 6:32 AM, Matt <hel...@gmail.com> wrote:
> I still didn't see a valid argument as to why we can't error out (instead of
> warning) if the engines doesn't match. It seems like even with
> enginesStrict: true you haven't actually fixed anything, because node will
> still go back in history trying to find a version that will work.

My goal was to fix a specific scenario: Authors putting
"engines":{"node":"0.6.x"} in their package.json files, with the best
intentions (or no specific intentions), and then ending up with dozens
of little annoying barbs preventing their users from upgrading.

The problem you're talking about is different from that. This doesn't
make it any worse (though, admittedly, it doesn't make it much better,
as you've pointed out). If we address that, it'll be a separate
thing.

In the meantime, you DO have the capacity to add arbitrary commands to
your package.json file that can be run at various points in the
install process. Check out `npm help scripts`. Nothing's stopping
you from adding a script that checks the node version, the price of
tea, or the phase of the moon, and aborts if things are not good.

[Matt]

On Fri, Jun 29, 2012 at 12:46 PM, Isaac Schlueter <i...@izs.me> wrote:

> Why is trying to go back in history to find a version that matches so
> important?

It's important because there is a much more common case than the one
you're describing:  You write version 1.0 with node 0.6 in mind.  Make
changes, updates, yadda yadda.  Then 0.8 comes out, and you add
support for that.  You release 2.0 with the engines setting on it.

For a while, you may even be releasing updates to both the 1.0 and the
2.0 versions!  So, you'd prefer to have 0.6.x users get the 1.0.latest
and 0.8.x users get the 2.0.latest.

And this is why dependencies can specify "matts-module": "1.x".

Sure, under this scheme they have to go and add that to their package.json, but at least they wouldn't get a broken version installed with a warning they are likely to ignore in the meantime. They would actually get npm erroring out and telling them about the problem.

It seems to me that there's no perfect solution to this problem, but backtracking through versions is creating more problems than it solves. Are there *any* other packaging systems that do that? I can't think of one.

Matt.

[Brandon Benvie]

This is an honest curious question because I don't know the answer or have a preconceived notion. Is it possible to just try a change and then reactively revert it if it breaks enough stuff? What I do see here and in recent compat issues is a bike load of buttshedding and only a handful of people citing actual incidents. On either side all it amounts to are opinions and many, perhaps most, are based on speculation of "if this happens". I would guess many or most of those opinions have little relation to the actual outcome and are mostly based on whatever opinion the person has on the behavior being discussed.

If actual breakage is the concern then maybe things need to be broken when it seems plausible, just long enough to determine whether to reverse the break or not. For certain, when it IS broken, the feedback will be immediate and sincere.

[Isaac Schlueter]

Brandon,

That's a good question. I think the answer is, "It depends."

In some cases, you CAN just take whatever the risk is, see what
happens, then revert the change. Historically, I've done that a lot
in the npm client. However, when the problem ends up in lots of
persistent places, that can turn malignant (qv,
"engines":{"node":"0.6.x"} or require('sys').)

Then fixing it involves either (a) use a time machine to change the
past, (b) get the whole world to update their stuff, or (c) accept
that we have made a mistake we can't fully correct. To my knowledge,
(a) was never an option. (b) is only an option if you're takling
about a small userbase, and we no longer are. So you either have to
decide to be ok with it, or come up with an approach that doesn't make
things worse.

When `npm install` breaks, we can just fix it in the next version. A
few people will get annoyed, but the problem will be isolated. But if
`npm publish` or (in this case) `npm init` do something wrong, then it
can sometimes cause issues much later and affect many more users.

So, if a temporary problem in npm init or publish causes a bunch of
garbage metadata to end up in the registry, and we can fix this by
changing the behavior of `npm install` slightly, that can be a pretty
good approach.

[Marcel Laverdet]

In that case wouldn't it also be possible to just change all instances of "0.6.x" to ">=0.6.0" within npm itself and then throw an error if you try to publish with "0.6.x"? The problem here never was a bad feature, it was bad metadata. All the metadata is in one central repository for the most part..

[Bruno Jouhier]

If you publish your module with 0.6.x you tacitly engage yourself to maintaining it and publishing a newer version shortly after 0.8 is released, don't you?

The rule should be to publish with >= x.y.z unless you know that some upcoming or already released change will break your code.
If you don't know what the future will be, you should bet on the fact that things won't break. Does not mean that they won't ever break but you'll take action when they do. Be optimistic!.

So I don't see anything wrong with the "engines" feature itself. The problem is that it has been misused.

Bruno

[Matt]

On Sat, Jun 30, 2012 at 6:11 AM, Bruno Jouhier <bjou...@gmail.com> wrote:

If you publish your module with 0.6.x you tacitly engage yourself to maintaining it and publishing a newer version shortly after 0.8 is released, don't you?

Yes, IMHO.

 

The rule should be to publish with >= x.y.z unless you know that some upcoming or already released change will break your code.
If you don't know what the future will be, you should bet on the fact that things won't break. Does not mean that they won't ever break but you'll take action when they do. Be optimistic!.

So I don't see anything wrong with the "engines" feature itself. The problem is that it has been misused.

There's still a bug in that if you've published an older version without the "engines" tag, then npm will backtrack through history to find that version and install that instead. It's this backtracking that is a bug IMHO.

Matt.