npm package mutability

[Richard Marr]

This is probably a stupid/tired question, but why does npm allow mutable packages?

I'm working on an app where security is an issue, and among the (many) things that I'm frothingly paranoid about is the possibility of malicious (or more likely just untested) code somehow getting into our app, even though we're using shrink-wrapped versions. It means we'll have to be much more careful with the way we proxy the npm registry.

As a secondary point, I would have thought immutable packages would allow for much better caching behaviour, so reduce load on the registry itself and speed up npm for everybody.

--
Richard Marr

[Brian Lalor]

On Dec 18, 2013, at 7:23 AM, Richard Marr <richar...@gmail.com> wrote:

I'm working on an app where security is an issue, and among the (many) things that I'm frothingly paranoid about is the possibility of malicious (or more likely just untested) code somehow getting into our app, even though we're using shrink-wrapped versions. It means we'll have to be much more careful with the way we proxy the npm registry. 

I’d like to know this, as well.  One of the guarantees made by the Maven central repository is that artifacts (packages) can check in, but they can never check out.  I frankly don’t think NPM provides this type of assurance, but it should.  Otherwise the only way an organization can trust packages is to run their own repository.

--

Brian Lalor

bla...@bravo5.org

[Tim Caswell]

If you want this level of static dependencies you can check in your deps into node_modules in your git tree or use git submodules in there.  Git does guarantee that the thing you point to can't be changed because the hash *is* the hash of the content.  If anything changes, the hash changes.

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Forrest L Norvell]

I agree that packages should rarely be changed, but in practice if there's a major bug or the packaging gets totally botched (which has happened to me a few ties), it's good to have the ability to fix the problem in-place. I'm less enamored on the possibility of removing packages once they've been published. That seems like it's almost always a bad idea, and I would be in favor of altering the registry to disallow it.

F

[Alex Kocharin]

 

No idea why. I think it would be reasonable for npm registry to allow unpublishing, but deny republishing an exactly the same version number afterwards. So if something needs to be republished, maintainer will be forced to change version number (or add a build number although it's now ignored by npm).

 

You can try to use caching npm registry, for example this one:

https://github.com/rlidwka/sinopia

 

It wasn't specifically created for ensuring immutability, but it's a nice side effect. Once certain package (tarball) is cached there, all subsequent changes in npm registry will simply be ignored. This way I was able to detect changes in bson v0.2.3 deep inside our dependency tree, although nothing harmful was there.

 

 

Anyway, if you're thinking about security, I'd suggest to bring up another issue. Signed packages. Without them it's hard to rely even on your own published packages. Although I'm not quite sure how to implement this properly, I feel it really needs some attention.

 

 

18.12.2013, 16:23, "Richard Marr" <richar...@gmail.com>:

--

[Nuno Job]

Google "nodejs lockdown lloyd"

That should be easy enough to get started

Ideally modules should be signed by author and a couple more people to validate origin.

Nodejitsu sells a private npm product, with it comes full control over all things

You are right to be concerned: anyone that has a module you use can effectively attack you. All it takes is for their credentials to be compromised, credentials which I believe are just a plain text file on your home. I've raised this issues more than 2 years ago and it seems be a recurring one right now. So understanding this is how npm works and creating security validation processes for your releases is a good idea

You can also check the node security project

Hope this is helpful

Nuno

[Alex Kocharin]

 

19.12.2013, 07:18, "Nuno Job" <nunojo...@gmail.com>:

Google "nodejs lockdown lloyd"

 

That should be easy enough to get started

 

You're right, that one should probably be merged with npm shrinkwrap.

 

 

 

Ideally modules should be signed by author and a couple more people to validate origin.

 

Author only. Public keys are leaked much less frequently than passwords, so bothering other people don't make sense.

 

 

 

Nodejitsu sells a private npm product, with it comes full control over all things

 

PaaS are evil.

 

 

 

You are right to be concerned: anyone that has a module you use can effectively attack you. All it takes is for their credentials to be compromised, credentials which I believe are just a plain text file on your home. I've raised this issues more than 2 years ago and it seems be a recurring one right now. So understanding this is how npm works and creating security validation processes for your releases is a good idea

 

... and credentials that gets passed over a network very frequently. I wonder, is there a way around add-auth=true for private registries?

 

Anyway, I submitted a pull request 2 months ago, but it didn't bring much attention:

https://github.com/isaacs/npm/pull/4016

 

And it doesn't solve all of the issues. I wonder if it makes sense to start a discussion about it again, especially considering the fact that there are public replicas like npmjs.eu, and although I very much want to use them, lack of package verification makes me sad.

 

 

 

You can also check the node security project

 

Is it still alive? There is some activity on twitter, but otherwise the site is pretty much empty.

[Sean McArthur]

Yea, we use lockdown for Persona. It keeps a checksum of the package and verifies it each time we reinstall. https://npmjs.org/package/lockdown

[Richard Marr]

Hi Forrest,

I agree there should be a way to recover from a botched release, but mutable packages cause problems that are hard or impossible to solve in other ways. If the reason for mutable packages is just to allow recovery from botched releases then why not just increment the version? Anyone auto-upgrading would be fine, anyone manually installing would be no worse off.

It's a lot easier to say "0.5.10 doesn't work, you need 0.5.11" than "0.5.10 doesn't work if you installed it before 10.30am PST, unless you were looking at a mirror or proxy in which case compare the source code to this commit to see which version you have"

I don't want to pre-judge as there may be use cases I don't see yet, but so far I haven't heard anything that's worth the cost of allowing mutable packages.

Cheers,

Rich

[Richard Marr]

Thanks Tim, that's definitely a strong contender in terms of keeping the infrastructure simple in a security-conscious environment.

--
Richard Marr

[Richard Marr]

Thanks Alex.

Signed packages would be a real win too. Aside from some coding, I guess the challenge there is how to encourage everyone to jump through the additional hoops required. The signing could be happen by default (I'm guessing) but publishers would need to upload a public key. I guess we'd just need to make it easy, build some community support. Maybe node-security could hand out status badges like Travis does to show that a package and all its dependencies are signed.

--
Richard Marr

[Richard Marr]

Thanks Nuno, helpful as always.

Richard Marr

[Mikeal Rogers]

There have also been security issues where old packages were shipped with sensitive information that needed to be ripped out.

[Dean Landolt]

Even in that case it would still seem better to allow unpublish and bump the version number, right?

Immutability (plus unpublish) would make the npm registery an even better place.

[T.C. Hollingsworth]

On Thu, Dec 19, 2013 at 2:25 AM, Richard Marr <richar...@gmail.com> wrote:
> I agree there should be a way to recover from a botched release, but mutable
> packages cause problems that are hard or impossible to solve in other ways.
> If the reason for mutable packages is just to allow recovery from botched
> releases then why not just increment the version? Anyone auto-upgrading
> would be fine, anyone manually installing would be no worse off.

Yeah, speaking with my distro packager hat on, silently changing
tarballs for stuff like this is really awful. And it's by no means
just an npm problem, lots of open source projects like to do this, so
all manner of potential problems are silently fixed with no indication
to us whatsoever.

Somebody runs a cronjob to check our cached copies of tarballs against
whatever is upstream at the moment, but by that time you notice that
some weeks have passed...

-T.C.

[Sam Roberts]

On Wed, Dec 18, 2013 at 5:42 PM, Alex Kocharin <al...@kocharin.ru> wrote:
> So if something needs to be republished, maintainer will be
> forced to change version number (or add a build number although it's now
> ignored by npm).

I've found force republish invaluable in the last weeks when publishes
to npmjs.org silently half-work, leaving the tarball missing from
cloudfare. I'd prefer a build number, but like you say, its not
respected by npm, and I'm not really happy with having to use patch
number as a publish-retry number.

Sam

[Richard Marr]

That sucks. Any idea what's causing it?

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Richard Marr

[Mikeal Rogers]

That leaves all of the other packaged already published and relying on a specific version broken.

[Mikeal Rogers]

+1

[Brian Lalor]

That’s a terrible idea.  The simplest reason it’s a terrible idea is that binaries built for one platform will be checked in.  ie developing on OS X and deploying on Linux.

--

Brian Lalor

bla...@bravo5.org

[Mikeal Rogers]

Those are easy to exclude and so long as you `npm rebuild` on deploy they'll be overwritten anyway.

This is used as a best practice by quite a few people, people still tweet this article I wrote out every month and I wrote it's more than 2 years old. http://www.futurealoof.com/posts/nodemodules-in-git.html

-Mikeal

[Richard Marr]

Nobody wants to break all of the things.

Unpublish need only be used in extreme cases, where sensitive data is exposed, copyright problems, malware, etc... i.e. cases where, sadly, dependent projects should (or must) be allowed to break.

Even when a package is unpublished, dependent projects would only have the opportunity to break at the moment they (re)install dependencies... at which point they either they find a republished version or a version-bumped version depending on whether package versions are mutable. The only projects that would notice the difference between the two approaches would be those that were locked to a specific patch version.

Richard Marr

[Alex Kocharin]

 

Why is it a bad thing?

 

If someone wants to receive your update, he'll specify a version range and an update will be installed automatically.

 

If someone want to rely specifically on broken version and locked it, they kinda asked for it.

 

Well yeah, unpublishing is bad except for very few cases where public data is leaked. But when you do that, don't republish the same version ever again. It'll mess up things. I like to see how npm v1.3.19 is missing from registry for example.

 

 

20.12.2013, 02:27, "Mikeal Rogers" <mikeal...@gmail.com>:

[Mikeal Rogers]

In the real world it's just better to have your dependents not fail to install, you can't actually rely on the maintainers doing another publish after you break them. In the end, the less packages that fail to install from npm the better and whatever solution results in the least number of un-installable packages sitting in the registry is the one we should go with.

-Mikeal

[Richard Marr]

I completely agree that checking in dependencies is currently a sensible and pragmatic option for those concerned with consistency or security, but I disagree with anyone that thinks that we shouldn't be setting our sights a bit higher.

I don't want anyone to take this as a criticism, I've got immense respect for anyone who's contributed to node, npm, or any of the other packages I have the privilege to use... all I'm saying is that this one feature causes deep problems, and that I think there are other ways to address the problems it was designed to address.

Richard Marr

[Alex Kocharin]

 

Suppose somebody made a conscious choice to rely on your package version 1.2.3. If they wanted to receive updates, they would've specify ~1.2.3, and receive your 1.2.4 with a bugfix. But they don't even though npm specify "~" syntax by default. Why do you suggest to replace 1.2.3 (if it's broken or something) with a different version dependent explicitly didn't want to see installed?

 

 

20.12.2013, 03:03, "Mikeal Rogers" <mikeal...@gmail.com>:

[Alex Kocharin]

 

I was dealing with checked in dependencies in some private project with a few people who don't seem to be familiar with node very much.

 

It resulted in huge node_modules folder checked in (with binary deps because nobody cared), they weren't updated at all (and were outdated for like a year). And the worst thing is that when people find bugs in modules, they actually commit their change directly to node_modules!

 

After that we force pushed all dependencies out of git and swore never to repeat that again.

 

I'm not saying it's a bad idea, but in that case I strongly suggest to keep two separate git repositories. One for development (without node_modules and any generated files at all) and one for deployment.

 

 

20.12.2013, 03:07, "Richard Marr" <richar...@gmail.com>:

[Mikeal Rogers]

First off, if someone version locks they are already doing the wrong thing. Saying "make my package ignore bugfix releases" is almost always the wrong thing. In this case it's better to keep their package working for them since they clearly don't know what they're doing.

If they **really** didn't want any changes coming in that they didn't know about then they had two other options that would still work: 1) check the module in to git if they are deploying at app or shrinkwrap publish if it's not something being deployed 2) stick the md5 in package json to ensure nobody can give them another tarball for the same version.

We have to deal with what people are doing in practice when they don't necessarily understand the best practice and it's especially important when you maintain a common dependency to do what you can to keep everyone who relies on you working even when they don't do things correctly.

-Mikeal

[Alex Kocharin]

 

Well okay, just one silly argument. How will these people know that they're doing the wrong thing if nothing will ever break? :)

 

 

20.12.2013, 03:42, "Mikeal Rogers" <mikeal...@gmail.com>:

[Mikeal Rogers]

On Dec 19, 2013, at 3:21PM, Alex Kocharin <al...@kocharin.ru> wrote:

It resulted in huge node_modules folder checked in (with binary deps because nobody cared), they weren't updated at all (and were outdated for like a year).

this just requires some diligence and using dedupe often.

And the worst thing is that when people find bugs in modules, they actually commit their change directly to node_modules!

this is actually why we checkin our deps. we'll fix a module and deploy that fix before the maintainer has accepted it. we don't really see fixes that don't go back in to projects, working with open source maintainers is part of the culture.

[Mikeal Rogers]

They won't :)

Oh well :)

It's better for things to work than for everyone to agree.

-Mikeal

[Richard Marr]

Mikeal,

To sum up the way I currently see it, switching to immutable packages would:

 * solve a whole class of subtle problems caused by unrequested code changes

 * allow aggressive caching, reducing the cost of the npm registry and making npm use faster for most use cases

 * change behaviour only for version-locked dependencies when that (and only that) specific patch is unpublished

That's two HUGE wins, and one downside. The downside can be reduced into obscurity being very rare by making it hard to unpublish, so that authors only bother to do it if they genuinely need to... i.e. serious legal or security reasons... cases where it's actually the right thing to do to break dependent apps.

From what I've heard so far, force republish is mainly used to keep patch numbers slow and sequential, which seems like a much lower priority requirement than introducing behaviour changes silently and unexpectedly into dependent apps.

Please do contribute more if you have more time... I do want to understand the root cause of your replies, and please set me straight if I've missed or misunderstood any of your comments

Rich

[Dean Landolt]

I completely agree with this sentiment, but just because you can't change the code for a specific version number doesn't mean you can't add code with the same effective semver range...

If I publish v0.5.4 and then realize it's got a serious flaw and I need to unpublish, I can still republish, say, v0.5.4+fix1 or some such. I'm not sure how npm handles sorting on these suffixes but I suspect it's alphanumeric, and takes priority over the old v0.5.4. From the docs:

1.2.3 A specific version. When nothing else will do. Note that build metadata is still ignored, so 1.2.3+build2012 will satisfy this range.

It looks like that's at least one way to do it.

[Alex Kocharin]

 

Just found an interesting branch in npm repository:

https://github.com/isaacs/npm/commits/shrinkwrap-shasums

 

I wonder why it didn't make it to master.

 

 

--
// alex

 

 

19.12.2013, 07:18, "Nuno Job" <nunojo...@gmail.com>:

Google "nodejs lockdown lloyd"

 

That should be easy enough to get started

 

Ideally modules should be signed by author and a couple more people to validate origin.

 

Nodejitsu sells a private npm product, with it comes full control over all things

 

You are right to be concerned: anyone that has a module you use can effectively attack you. All it takes is for their credentials to be compromised, credentials which I believe are just a plain text file on your home. I've raised this issues more than 2 years ago and it seems be a recurring one right now. So understanding this is how npm works and creating security validation processes for your releases is a good idea

 

You can also check the node security project

 

Hope this is helpful

Nuno

On Thursday, December 19, 2013, Alex Kocharin wrote:

 

No idea why. I think it would be reasonable for npm registry to allow unpublishing, but deny republishing an exactly the same version number afterwards. So if something needs to be republished, maintainer will be forced to change version number (or add a build number although it's now ignored by npm).

 

You can try to use caching npm registry, for example this one:

https://github.com/rlidwka/sinopia

 

It wasn't specifically created for ensuring immutability, but it's a nice side effect. Once certain package (tarball) is cached there, all subsequent changes in npm registry will simply be ignored. This way I was able to detect changes in bson v0.2.3 deep inside our dependency tree, although nothing harmful was there.

 

 

Anyway, if you're thinking about security, I'd suggest to bring up another issue. Signed packages. Without them it's hard to rely even on your own published packages. Although I'm not quite sure how to implement this properly, I feel it really needs some attention.

 

 

18.12.2013, 16:23, "Richard Marr" <richar...@gmail.com>:

This is probably a stupid/tired question, but why does npm allow mutable packages?

I'm working on an app where security is an issue, and among the (many) things that I'm frothingly paranoid about is the possibility of malicious (or more likely just untested) code somehow getting into our app, even though we're using shrink-wrapped versions. It means we'll have to be much more careful with the way we proxy the npm registry.

As a secondary point, I would have thought immutable packages would allow for much better caching behaviour, so reduce load on the registry itself and speed up npm for everybody.

 

--
Richard Marr

 

--

[Alex Kocharin]

20.12.2013, 00:13, "Sam Roberts" <s...@strongloop.com>:

> On Wed, Dec 18, 2013 at 5:42 PM, Alex Kocharin <al...@kocharin.ru> wrote:
>

>> О©╫So if something needs to be republished, maintainer will be
>> О©╫forced to change version number (or add a build number although it's now
>> О©╫ignored by npm).

>
> I've found force republish invaluable in the last weeks when publishes
> to npmjs.org silently half-work, leaving the tarball missing from
> cloudfare. I'd prefer a build number, but like you say, its not
> respected by npm, and I'm not really happy with having to use patch
> number as a publish-retry number.

What about using fourth number for it? Like "1.2.3-4"? If you document out what this versioning means, it'll be fine since immutability is much more valuable here than strict adherence to semver (which is proven to be a bad default anyway).

[Richard Marr]

> I'm not really happy with having to use patch
> number as a publish-retry number.

This is conflating two problems though... using mutability to work around a stability problem for the sake of tidyness. While it'd be untidy for (during a period of instability) the patch number to be used as a proxy for the number of publish-retries, it's nowhere near as untidy as having your code change silently without any warning. When the stability issue is fixed the mutability would still be there to give me headaches.

I appreciate that there are some workarounds, but having to check in all dependencies is also untidy. As is add an extra maintenance step that checksums dependencies and aborts if they have.

I was hoping to either be educated to the reason for npm's package mutability, or to find consensus around changing it... I don't feel like I'v got either of those yet... Anybody?

On Saturday, December 21, 2013, Alex Kocharin wrote:

20.12.2013, 00:13, "Sam Roberts" <s...@strongloop.com>:
> On Wed, Dec 18, 2013 at 5:42 PM, Alex Kocharin <al...@kocharin.ru> wrote:
>
>>  So if something needs to be republished, maintainer will be

>>  forced to change version number (or add a build number although it's now

>>  ignored by npm).
>
> I've found force republish invaluable in the last weeks when publishes
> to npmjs.org silently half-work, leaving the tarball missing from
> cloudfare. I'd prefer a build number, but like you say, its not
> respected by npm, and I'm not really happy with having to use patch
> number as a publish-retry number.

What about using fourth number for it? Like "1.2.3-4"? If you document out what this versioning means, it'll be fine since immutability is much more valuable here than strict adherence to semver (which is proven to be a bad default anyway).

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Richard Marr

[Mikeal Rogers]

On Dec 20, 2013, at 2:45AM, Richard Marr <richar...@gmail.com> wrote:

Mikeal,

To sum up the way I currently see it, switching to immutable packages would:

 * solve a whole class of subtle problems caused by unrequested code changes

sure.

 * allow aggressive caching, reducing the cost of the npm registry and making npm use faster for most use cases

this isn't an issue. the cache control can, and must, be proactively invalidated on _changes from the database for document urls anyway, it's trivial to do the same for tarball changes. it can literally cache forever so long as it responds to pro-active invlaidation.

 * change behaviour only for version-locked dependencies when that (and only that) specific patch is unpublished

i'm not disagreeing with you but when you say things like "change in behavior" you're sort of sugar coating the fact that packages will fail to install at a greater rate than they do now. this "change in behavior" is not trivial, there is no notice sent to someone when their package can no longer resolve a dependency, it will usually require someone to see a failed install, report an issue, and the maintainer to intervene. the only way to avoid this is to never version lock your dependencies which we know people don't do and that there are tens of thousands of packages in npm today with some number of version locked deps.

That's two HUGE wins, and one downside. The downside can be reduced into obscurity being very rare by making it hard to unpublish, so that authors only bother to do it if they genuinely need to... i.e. serious legal or security reasons... cases where it's actually the right thing to do to break dependent apps.

From what I've heard so far, force republish is mainly used to keep patch numbers slow and sequential, which seems like a much lower priority requirement than introducing behaviour changes silently and unexpectedly into dependent apps.

Please do contribute more if you have more time... I do want to understand the root cause of your replies, and please set me straight if I've missed or misunderstood any of your comments

I don't misunderstand you, I don't even think that we disagree on what would or would not happen and the potential wins and loses. What I don't think we are in alignment about is the severity of the up and down sides of the available options.

We don't get a clean slate. At this point in time we can't afford to make changes that might break parts of the existing ecosystem even if it makes the future a little better. We can only really entertain strategies that make current and future packages less prone to these problems without the potential to break existing ones.

Rich

On 19 December 2013 23:49, Mikeal Rogers <mikeal...@gmail.com> wrote:

They won't :)

Oh well :)

It's better for things to work than for everyone to agree.

-Mikeal

On Dec 19, 2013, at 3:47PM, Alex Kocharin <al...@kocharin.ru> wrote:

 

Well okay, just one silly argument. How will these people know that they're doing the wrong thing if nothing will ever break? :)

 

 

20.12.2013, 03:42, "Mikeal Rogers" <mikeal...@gmail.com>:

First off, if someone version locks they are already doing the wrong thing. Saying "make my package ignore bugfix releases" is almost always the wrong thing. In this case it's better to keep their package working for them since they clearly don't know what they're doing.

 

If they **really** didn't want any changes coming in that they didn't know about then they had two other options that would still work: 1) check the module in to git if they are deploying at app or shrinkwrap publish if it's not something being deployed 2) stick the md5 in package json to ensure nobody can give them another tarball for the same version.

 

We have to deal with what people are doing in practice when they don't necessarily understand the best practice and it's especially important when you maintain a common dependency to do what you can to keep everyone who relies on you working even when they don't do things correctly.

 

-Mikeal

[Mikeal Rogers]

That's a good question. To my knowledge everyone agrees that shrinkwrap should include the hashes.

[Alex Kocharin]

 

23.12.2013, 04:36, "Mikeal Rogers" <mikeal...@gmail.com>:

 * allow aggressive caching, reducing the cost of the npm registry and making npm use faster for most use cases

 

this isn't an issue. the cache control can, and must, be proactively invalidated on _changes from the database for document urls anyway, it's trivial to do the same for tarball changes. it can literally cache forever so long as it responds to pro-active invlaidation.

 

HTTP caches know nothing about _changes feed.

 

 

 * change behaviour only for version-locked dependencies when that (and only that) specific patch is unpublished

 

i'm not disagreeing with you but when you say things like "change in behavior" you're sort of sugar coating the fact that packages will fail to install at a greater rate than they do now. this "change in behavior" is not trivial, there is no notice sent to someone when their package can no longer resolve a dependency, it will usually require someone to see a failed install, report an issue, and the maintainer to intervene. the only way to avoid this is to never version lock your dependencies which we know people don't do and that there are tens of thousands of packages in npm today with some number of version locked deps.

 

Patching npm to treat "1.2.3" like "~1.2.3" as a fallback would solve this task. I know it's ugly, but it's no less ugly than unpublishing-republishing practice, because you essentially doing the same thing.

 

 

// alex

[Mikeal Rogers]

On Dec 22, 2013, at 4:45PM, Alex Kocharin <al...@kocharin.ru> wrote:

 

23.12.2013, 04:36, "Mikeal Rogers" <mikeal...@gmail.com>:

 * allow aggressive caching, reducing the cost of the npm registry and making npm use faster for most use cases

 

this isn't an issue. the cache control can, and must, be proactively invalidated on _changes from the database for document urls anyway, it's trivial to do the same for tarball changes. it can literally cache forever so long as it responds to pro-active invlaidation.

 

HTTP caches know nothing about _changes feed.

NPM can't sit behind a standard HTTP cache on a TTL because a document change must take effect immediately. The work going in to putting it behind a cache is using a _changes listener to invalidate the cache.

 

 

 * change behaviour only for version-locked dependencies when that (and only that) specific patch is unpublished

 

i'm not disagreeing with you but when you say things like "change in behavior" you're sort of sugar coating the fact that packages will fail to install at a greater rate than they do now. this "change in behavior" is not trivial, there is no notice sent to someone when their package can no longer resolve a dependency, it will usually require someone to see a failed install, report an issue, and the maintainer to intervene. the only way to avoid this is to never version lock your dependencies which we know people don't do and that there are tens of thousands of packages in npm today with some number of version locked deps.

 

Patching npm to treat "1.2.3" like "~1.2.3" as a fallback would solve this task. I know it's ugly, but it's no less ugly than unpublishing-republishing practice, because you essentially doing the same thing.

Any changes to npm client take ~2 years to distribute enough that you can deprecate the behavior in an old one.

 

 

// alex

[Alex Kocharin]

 

23.12.2013, 05:46, "Mikeal Rogers" <mikeal...@gmail.com>:

On Dec 22, 2013, at 4:45PM, Alex Kocharin <al...@kocharin.ru> wrote:

 

23.12.2013, 04:36, "Mikeal Rogers" <mikeal...@gmail.com>:

 * allow aggressive caching, reducing the cost of the npm registry and making npm use faster for most use cases

 

this isn't an issue. the cache control can, and must, be proactively invalidated on _changes from the database for document urls anyway, it's trivial to do the same for tarball changes. it can literally cache forever so long as it responds to pro-active invlaidation.

 

HTTP caches know nothing about _changes feed.

 

NPM can't sit behind a standard HTTP cache on a TTL because a document change must take effect immediately. The work going in to putting it behind a cache is using a _changes listener to invalidate the cache.

 

It's not possible to cache json documents (I'd argue with it though). But immutable tarballs will allow caching them indefinitely without revalidating, and it effectively cuts an amount of if-modified requests in a half.

 

 

 

[Mikeal Rogers]

For reference:

9768 packages lock 24538 dependencies in 57515 different versions of those packages.

https://gist.github.com/mikeal/8090801 (will work as soon as I can publish npmetrics, currently getting allocation errors).

-Mikeal

[Sam Roberts]

On Sat, Dec 21, 2013 at 4:01 AM, Alex Kocharin <al...@kocharin.ru> wrote:
> 20.12.2013, 00:13, "Sam Roberts" <s...@strongloop.com>:

>> I've found force republish invaluable in the last weeks when publishes
>> to npmjs.org silently half-work, leaving the tarball missing from
>> cloudfare. I'd prefer a build number, but like you say, its not
>> respected by npm, and I'm not really happy with having to use patch
>> number as a publish-retry number.
>
> What about using fourth number for it? Like "1.2.3-4"? If you document out what this versioning means, it'll be fine since immutability is much more valuable here than strict adherence to semver (which is proven to be a bad default anyway).

Hey Alex, not sure I follow.

If 1.2.3 is broken, publishing 1.2.3-4 doesn't help, semver says the
"-" is a release candidate for 1.2.3.

Or are you suggesting that I always publish with a "-", so all
versions published look like release candidates? That I always do
first publish as 1.2.3-0? I guess I could... but about half my
publishes work first time, and I hope this to get better as npmjs
infrastructure gets worked on.

Mostly, I just want to point out that while you can do many bad things
with --force, that if the version you are force installing over was
NEVER INSTALLABLE FROM npmjs.org... I don't see it as so harmful.

[Sean McArthur]

There are good reasons to lock a dependency at a version. I'd love to accept all patch versions, but unfortunately too often a patch version introduces a new bug, or changes the API, and breaks my apps/libs. If I know my app works with a lib at 1.2.3, then my workmates better be able to npm install and get the same working version.

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/fDs5-Wl3I8c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.

[Alex Kocharin]

23.12.2013, 08:09, "Sam Roberts" <s...@strongloop.com>:

> On Sat, Dec 21, 2013 at 4:01 AM, Alex Kocharin <al...@kocharin.ru> wrote:
>

>> О©╫20.12.2013, 00:13, "Sam Roberts" <s...@strongloop.com>:
>>> О©╫I've found force republish invaluable in the last weeks when publishes
>>> О©╫to npmjs.org silently half-work, leaving the tarball missing from
>>> О©╫cloudfare. I'd prefer a build number, but like you say, its not
>>> О©╫respected by npm, and I'm not really happy with having to use patch
>>> О©╫number as a publish-retry number.
>> О©╫What about using fourth number for it? Like "1.2.3-4"? If you document out what this versioning means, it'll be fine since immutability is much more valuable here than strict adherence to semver (which is proven to be a bad default anyway).

>
> Hey Alex, not sure I follow.
>
> If 1.2.3 is broken, publishing 1.2.3-4 doesn't help, semver says the
> "-" is a release candidate for 1.2.3.
>
> Or are you suggesting that I always publish with a "-", so all
> versions published look like release candidates?

No. I'm suggesting that you always publish with a "-", so it will look like a classic 4-digit versioning system.

I'm thinking about it a lot recently, since even though semver is nice for most of the uses, it isn't perfect, and imposing it into all npm packages was a mistake.

> That I always do first publish as 1.2.3-0? I guess I could... but about half my
> publishes work first time, and I hope this to get better as npmjs
> infrastructure gets worked on.
>
> Mostly, I just want to point out that while you can do many bad things
> with --force, that if the version you are force installing over was

> NEVER INSTALLABLE FROM npmjs.org... О©╫I don't see it as so harmful.

Yes, in that particular case it isn't.

[Alex Kocharin]

 

You made it sound like locking will guarantee that no bugs will be introduced. You don't hardcode sha1sum, do you? :-)

 

 

23.12.2013, 08:42, "Sean McArthur" <sean.m...@gmail.com>:

[Sean McArthur]

We do check the shasum, using https://npmjs.org/package/lockdown

[Alex Kocharin]

 

Oh, you're right, that would work. I wonder, how many packages you encounter that do not match shasum (== republished)?

 

About locking in general: unfortunately, too often a patch version is better than an original dependency, but someone writes a module, locks down a dependency and very rarely updates it. This is the issue I encountered very often, and that's exactly why I hate locking packages at all, preferring open ranges like ">= 1.2.3". It's often better to have a new version with possible bugs than to be stuck with a legacy one for a year.

 

 

23.12.2013, 10:04, "Sean McArthur" <sean.m...@gmail.com>:

[Mark Hahn]

I lock versions down because each upgrade would create a need for an exhaustive test of my app which takes a day or more.  I'm a one man shop and I have about 30 npm modules in my app.

[Richard Marr]

> i'm not disagreeing with you but when you say things like "change in behavior" you're sort
> of sugar coating the fact that packages will fail to install at a greater rate than they do now.

I think the core problem there isn't that removing force-republish would encourage greater use of unpublish, it's that npm allows anyone to unpublish so easily that it can be done for any reason they like... whether for a serious reason (legal issues, malware, etc) or on a whim, or drunk, or in a fit of anger over some community drama.

I don't want to go too far down the road of conflating these two problems, but I do think it's currently far too easy to unpublish modules. If a module has been provided to the community, and people start to depend on it, it should be hard to remove it.

Perhaps it'd be more natural to talk about making unpublishing hard before we talk about removing mutability.

BTW. Thanks Mikael, and everyone else who's contributed to this thread. I'm certainly learning new things.

Rich

--
Richard Marr

[Mark Hahn]

> and people start to depend on it, it should be hard to remove it. 

Are you referring  to some central authority you would have to petition to be allowed to remove?  I can't imagine any other process that could be implemented to stop me from removing my module.

In other words a computer can't make things harder except maybe to make you type "I want to unpublish this module" a thousand times.  And then some clever person might figure out how to script that.  :-)

[Alain Mouette]

Em 30-12-2013 14:10, Mark Hahn escreveu:

> and people start to depend on it, it should be hard to remove it. 

Are you referring  to some central authority you would have to petition to be allowed to remove?  I can't imagine any other process that could be implemented to stop me from removing my module.

In fact that is the essense of Public Licences like MIT

A working solution would be to work like a Version control system:  move that module's version to some frozen/not recomended/whatever branch but keep it there because some people may be using it. Either in production or for study only

That is very close to Github's working, which is already the repositoy of choice...

Alain

[Tim Caswell]

For what it does and what it was designed for, the current NPM central repository system works great.  It wasn't meant to be a secure system with evil actors that can't trust eachother, but must work together.  It does at least have basic authentication so that an author has to change evil after publishing a successful package. (Or get their password hacked).  NPM is a tool for sharing code and information, not for build dependencies for your production servers.  Yes we use is that way because it's not been a problem and it's very convenient, but it could be.

I'm designing a more secure system based on the immutable concepts of git trees, but there are other hard parts to solve related to discovery and the lack of central control. (plus I've been busy with other stuff)

Like I said before, you can get the security benefits of this new system by using git submodules or adding your dependencies to your parent git tree, but it's a lot of work to keep things up to date manually.  You have to remember to check for updates on some schedule and review all the changes to make sure no backdoors or bugs get introduced in the changes.

[Mark Hahn]

>  You have to remember to check for updates on some schedule 

It would be nice to have a "watch" feature that would email you about updates and even tell you when the contents of an existing version change.  This would go a long way to all the concerns on this thread.  In other words allow mutability but force transparency and notifications.

[Sean McArthur]

Gemnasium can alert you when your npm dependencies are not up-to-date: https://gemnasium.com

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/fDs5-Wl3I8c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.

[Mark Hahn]

Now Gemnasium needs to notify us when checksum of any old module changes.

[Mark Hahn]

By old I mean old version, not just updates.

[Alex Kocharin]

 

20.12.2013, 03:48, "Mikeal Rogers" <mikeal...@gmail.com>:

On Dec 19, 2013, at 3:21PM, Alex Kocharin <al...@kocharin.ru> wrote:

It resulted in huge node_modules folder checked in (with binary deps because nobody cared), they weren't updated at all (and were outdated for like a year).

 

this just requires some diligence and using dedupe often.

And the worst thing is that when people find bugs in modules, they actually commit their change directly to node_modules!

 

this is actually why we checkin our deps. we'll fix a module and deploy that fix before the maintainer has accepted it. we don't really see fixes that don't go back in to projects, working with open source maintainers is part of the culture.

 

I just got a bit of an issue with that.

 

Suppose you do have a certain fix that isn't going to make it to upstream any time soon. For example, it's too specific for your case, or for whatever else reason.

 

How do you update this module in the future? You can't just do npm update, because it'll remove your changes, right?

 

[Thom Blake]

You can maintain your updated version of the module in its own repository and change your package.json to point to that instead of the npm registry version.