What do people do for versionizing node modules within git?

[deitch]

You are building an app, it has dependencies on lots of modules that
you install via npm. But versionizing those in git seems redundant.
What do people here do? Do they keep node_modules as tracked part of
git? Or do they put it in .gitignore, and if so, then how do you know
which versions and dependencies you have?

[Marak Squires]

I keep node_modules/ in my .gitignore

npm will let you bundleDeps in node_modules, but I've found it's usually more straight forward to create an actual "vendor" folder so it's more explicit. 

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--

-- 

Marak Squires

Co-founder and Chief Evangelist

Nodejitsu, Inc.

marak....@gmail.com

[deitch]

So you manually create a vendor/ folder under the root of your tree,
parallel to node_modules, which has a list of all of the dependencies?
How do you structure it? Is it something automated?

Would it be as effective as every time you run npm install (or
uninstall), to run:

npm ls > vendor.txt

Of course, you need to do it manually. Why doesn't npm have an
automatic "here is a list of all the things currently installed"? It
can be updated by npm every time it runs, say every run of npm install
or uninstall automatically dumps the output of "npm ls" to .npmls or
similar in the same parent dir of node_modules parallel to it. isaacs?

> marak.squi...@gmail.com

[Marak Squires]

No, I just drop in the modules I want to vendor and require them directly from the vendor folder. 

-- 

Marak Squires

marak....@gmail.com

[Avi Deitcher]

So, you basically ignore the whole npm and node_modules. What does it buy you? From a git perspective, you are versionizing identically to node_modules. You just have to explicitly path as opposed to require('foo').

Avi Deitcher
a...@deitcher.net

[Marak Squires]

I feel this this conversation is going in four different directions and no one understands what the other person is talking about.

I give up.

[deitch]

LOL!

Back to earth: when you install modules 3rd-party, npm or vendor or
anyhow, do you put them under version control? You said yes.

On Dec 4, 11:15 pm, Marak Squires <marak.squi...@gmail.com> wrote:
> I feel this this conversation is going in four different directions and no
> one understands what the other person is talking about.
>
> I give up.
>
>
>
>
>
>
>
>
>
> On Sun, Dec 4, 2011 at 1:13 PM, Avi Deitcher <a...@deitcher.net> wrote:
> > So, you basically ignore the whole npm and node_modules. What does it buy
> > you? From a git perspective, you are versionizing identically to
> > node_modules. You just have to explicitly path as opposed to require('foo').
>

> > Avi Deitcher
> > a...@deitcher.net
>
> > --
> > Job Board:http://jobs.nodejs.org/
> > Posting guidelines:
> >https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> > You received this message because you are subscribed to the Google
> > Groups "nodejs" group.
> > To post to this group, send email to nod...@googlegroups.com
> > To unsubscribe from this group, send email to
> > nodejs+un...@googlegroups.com
> > For more options, visit this group at
> >http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --
> --
> Marak Squires
> Co-founder and Chief Evangelist
> Nodejitsu, Inc.

> marak.squi...@gmail.com

[Marak Squires]

No I didn't.

-- 

Marak Squires

marak....@gmail.com

[Avi Deitcher]

Now I *really* am confused. You said you add node_modules to .gitignore, but you put all of your modules under vendor anyways, and that is under git control?

[Andrew Chilton]

I think there are two conversations, but if I clarify what Marak is
saying, that might help.

He uses npm, it puts them in node_modules, this is the best way to do
this with npm installable modules. I don't think this is the issue
here.

@avi ... but if you want to bundle a module which _isn't_ npm
installable, you could put it in a 'vendor' folder, which you _do_
check in to Git

So, for example you might have the following structure:

* myapp.js
* vendor/a-module-that-isnt-npm-installable/*
* vendor/another-module-that-isnt-npm-installable/*
* node_modules/express/*
* node_modules/data2xml/*
* node_modules/everyauth/*

The two vendor folders _are_ checked in to Git. The three node_modules
folder are _not_ checked in. They are installed with npm.

Finally, you might have the following in your .gitignores :

node_modules
*~

Hope that helps,
Andy

--
Andrew Chilton
e: chi...@appsattic.com
w: http://www.appsattic.com/
p: +64 21 891 681

[Andrew Chilton]

On 5 December 2011 10:29, Avi Deitcher <a...@deitcher.net> wrote:
> Now I *really* am confused. You said you add node_modules to .gitignore, but
> you put all of your modules under vendor anyways, and that is under git
> control?

He puts the npm installable modules into node_modules. He puts the
non-npm installable modules into vendor.

node_modules is _not_ checked into Git. Vendor is. :)

Does that make sense?

Cheers,
Andy

[Avi Deitcher]

@Andy,

Clarifies yes. Which opens the original question, though: if you don't track node_modules in git (which I don't), then what do you use to keep track of what dependencies you have?

If it is just another npm module, package.json lists all dependencies. But what about inside an app?

Right now, I just do "npm ls > npmdeps" or some similar file, run it manually when I remember. And then I can easily reconstruct when I need.

But is there a better way to keep track of which node_modules an app is dependent upon?

[Marak Squires]

Yes, exactly. 

Sometimes you need a package which isn't available on NPM. This is when I suggest using /vendor/ folder, although npm does have the ability to bundle deps for you using the "bundleDependencies" property. 

I just haven't had much luck using "bundleDependencies", since it won't resolve your deps sub-deps.

[Avi Deitcher]

Got it. And how do you track which 25 npm packages you needed for your app, and their versions? Anything better than my kludge of 
npm ls > somefile

with somefile under git control?

[Marak Squires]

Yeah, use package.json "dependencies" property and npm install?

[Andrew Chilton]

On 5 December 2011 10:33, Avi Deitcher <a...@deitcher.net> wrote:
> @Andy,
>
> Clarifies yes. Which opens the original question, though: if you don't track
> node_modules in git (which I don't), then what do you use to keep track of
> what dependencies you have?
>
> If it is just another npm module, package.json lists all dependencies. But
> what about inside an app?

You do exactly the same, just have a package.json file which lists
your dependecies. You check package.json into Git and voila ... you're
tracking your dependencies. :)

For example, a snippet from one of my private projects is as follows:

"dependencies": {
"express" : ">= 2.5.1",
"stylus" : ">= 0.19.5",
"jade" : ">= 0.18.0",
"passgen" : ">= 1.0.0"
},

When I do a fresh checkout of my repo, I just do the following and
everything is exactly as I like it:

$ git checkout myproject
$ npm -d install

Try that and let me know how it goes.

Cheers,
Andy

[Avi Deitcher]

As the Genie said in Aladdin (thanks to Robin Williams), "well, I feel foolish". Or, better yet, Homer Simpson, "Doh!"

I never thought of that, always used package.json when building an npm installable module, never inside a standalone non-npm-targeted app. Never even checked it was doable.

You know what would be nice? If every time I ran "npm install" or "npm uninstall" it automatically updated package.json.

Thanks, Andy and Marak.

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--
Avi Deitcher
a...@deitcher.net

[Andrew Chilton]

On 5 December 2011 10:39, Avi Deitcher <a...@deitcher.net> wrote:
> You know what would be nice? If every time I ran "npm install" or "npm
> uninstall" it automatically updated package.json.

I don't think you'd want it to do that - well, let me rephrase - *I*
wouldn't want it to do that. I want to either be explicit with the
version "= 0.2.0" or less so with a minimum version such as ">=
0.2.0".

In another note, I found something useful in npm which is good for
targeting non-npm modules/applications. Put a "private" : true into
your package.json and it'll help you to stop accidentally uploading it
to npm. :) See http://npmjs.org/doc/registry.html for more info.

[Martin Cooper]

On Sun, Dec 4, 2011 at 1:39 PM, Avi Deitcher <a...@deitcher.net> wrote:
> As the Genie said in Aladdin (thanks to Robin Williams), "well, I feel
> foolish". Or, better yet, Homer Simpson, "Doh!"
>
> I never thought of that, always used package.json when building an npm
> installable module, never inside a standalone non-npm-targeted app. Never
> even checked it was doable.
>
> You know what would be nice? If every time I ran "npm install" or "npm
> uninstall" it automatically updated package.json.

For install, use 'npm install --save' and it will do that for you. I
don't believe it's supported for uninstall, though.

--
Martin Cooper

[Avi Deitcher]

Yeah, the more I think about it, th clearer it is that I wan *not* to norm install, bu rather to modify a package.json an run install against it.

Is it idempotent? Can I add one line, rerun, and have it know to install just that one added?

[Andrew Chilton]

Hi Avi,

On 5 December 2011 12:11, Avi Deitcher <a...@deitcher.net> wrote:
> Is it idempotent? Can I add one line, rerun, and have it know to install
> just that one added?

Try it and see. :)

The answer is yes, but seriously, just try it and see. You're using
Git, you're a careful guy, nothing will disappear and even if it did,
your VCS is your backup. :) Experimentation is the key, as is reading
the manual.

Cheers,
Andy

--
Andrew Chilton
e: chi...@appsattic.com
w: http://www.appsattic.com/

[Avi Deitcher]

My, if that ain't beautiful. I love it... almost as much as that transition to defense move someone showed me at the hockey rink 2 weeks. Almost.

Thanks, @marak @andy.

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--
Avi Deitcher
a...@deitcher.net

[deitch]

Well, almost. Because I used ">=" instead of "=", it upgraded my
connect and express when I reran, so I got the newer version of
bodyParser, which killed my extension to "text/plain". I have a simple
one so that the app will *always* find what it wants in req.body -
plaintext, JS objects, whatever - fully parsed. Could have used git to
go backwards, but better to just find it and fix it.

[Tim Caswell]

My simple workflow which works for me is to put all modules in
node_modules. (Yes, this includes npm installed ones and ones I
install manually) Then in my .gitignore I list explicitly the folders
for my npm modules (including the trailing /) I put my dependencies
in a package.json at the root of my app with explicit package
versions. This way I'm in 100% control of what modules I'm using.
They won't change and break stuff (because node module authors don't
always respect semver semantics).

<https://github.com/c9/nog/blob/master/.gitignore>
<https://github.com/c9/nog/blob/master/package.json#L14-19>
<https://github.com/c9/nog/tree/master/node_modules>

-Tim Caswell

[Jeff Barczewski]

Also if you want to install a module that is not in npm but it is in github, then you can use the tar ball url in your package.json.

For instance to specify a dependency using github tarball either to master or a particular tag:

"dependencies": {
   "YOURPROJ" : "https://github.com/jeffbski/YOURPROJECT/tarball/master" // latest
 }

OR

"dependencies": {
   "YOURPROJ" : "https://github.com/jeffbski/YOURPROJECT/tarball/v0.0.1" // tag v0.0.1
 }

Then just npm install will fetch it

[Avi Deitcher]

It is funny, I really never thought about using package.json for standalone projects until the group here pointed it out; it was always for npm packages as far as I knew. 

--

[Will]

For the Googlers of the future that reach this thread: the answer depends on the nature of your project.

• Do track `node_modules` in your repo if you are deploying an app.
  
• Do not track `node_modules` in your repo if your project is a reusable package meant to be consumed by other projects.

See this post for an explanation as to why this is the best practice.

Also, it's a good idea to register all dependencies in your `package.json`: a dependency can be any git repo, not only ones published in the npm registry.

[Brian Lalor]

On Nov 5, 2013, at 11:25 PM, Will <wma...@gmail.com> wrote:

For the Googlers of the future that reach this thread: the answer depends on the nature of your project.

• Do track `node_modules` in your repo if you are deploying an app.

This is not as solved a problem as Mikeal and others would like to think.  From an operations perspective, doing *any* compilation on deploy is a terrible idea.  It’s a HORRIBLE idea.  It’s a product of developers who are ignorant of deploying applications repeatably and at scale.  The only thing you should be deploying is binary packages, compiled for the target you’re deploying to, preferably in a system package (deb, rpm).  Your production systems shouldn’t even *have* gcc on them, and that’s required if you have to run “npm rebuild” for a deployment step.

npm’s dependency resolution and loading model is excellent, but if “npm shrinkwrap” — the only tool in the suite that’s supposed to help solve the floating transitive dependency problem — doesn’t do the job, then it needs to be fixed.  Checking dependencies into source control is a hack.

npm’s model has greatly simplified the way I do deployments.  Ruby’s gems suck, Python’s less onerous but still kind of a pain.  It’s amusing and painful to watch the Fedora folks try to bend node dependency model to fit their archaic packaging standards (do yourself a favor and just don’t use anything node-related from EPEL).  npm needs to adopt the Maven central repository’s roach motel contract: packages check in, but they never check out.  This combined with a reliable way to lock down dependency versions and and compile-once binary deployment packages means never having to run “npm install” or “npm rebuild” on a production box, no matter how old your application is.

--

Brian Lalor

bla...@bravo5.org

[greelgorke]

@Brian: i think what you describe isn't npm's issue. if you have a deployment beyond simple apps, then you need another tool and another process that fits it. shrinkwrap helps you with managing dependency versions, not dependencies. if you have need that level of deploymant safety, take maven or ant or whatever you like, let your CI server do pull the repo to your target, do the npm install/rebuild and pack it as tar or debian package or whatever, put it into your private repo. even java projects use rpm or dpkg for production installations. not maven

[Alex Kocharin]

 

Why do you think compilation on deploy is a horrible idea? Is it because of a big compilation time, or are you unsure that compilation will produce the same results on all hosts?

 

 

 

06.11.2013, 15:34, "Brian Lalor" <bla...@bravo5.org>:

--

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 

---

You received this message because you are subscribed to the Google Groups "nodejs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Avi Deitcher]

When a package goes to a production system, it should be ready to drop in. You should have versionized binary artifacts stored that can be dropped anywhere. The binaries should be literally identical across all similar systems. Otherwise reliable deploy and support are nearly impossible, and small variants can mean your tests ran against a non-identical binary, making deploy unreliable.

I consult on business ops in tech, deploying anything that isn't 100% binary identical across all instances is big trouble (which, of course, led to some good consulting revenue for me to fix :-) ).

However, if you have CI with a compilation stage, your best practice might be:

a) Do not include node_modules, just package.json in version control.

b) Do not deploy to production, but rather to your CI system, which does the build and creates a packaged up artifact, versionizes and stores it, and runs all your integration tests. 

c) Deploy to prod (or dev or UAT or whatever) from the artifact, so you are guaranteed identical.

The problem is many devs on a small scale (which is what most startups are) build locally and then push directly to your PaaS via git, jitsu, etc. I don't have an easy solution for the above, although some of those CI offerings (circleci, etc.) are reasonable solutions.

Personally, I think how those PaaS use your package.json to rebuild node_modules is bad practice.

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/mqVj8f47P_U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.

[Matt]

From an ops perspective, having gcc/clang on a live server is considered a security weakness. It's a bit of a dated philosophy, but it does prevent hackers with shell access from compiling random binaries on the box.

Also yes, compile on deploy can be a bit slow. I don't have a personal problem with it up to a few servers, but if I was deploying to 10s+ of servers I would try to use debs/rpms for the extra features they offer.

[José F. Romaniello]

We do not version node_modules for our deploying apps but we use shrinkwrap and packaging all deps in CI after running all tests using http://github.com/carlos8f/bundle-deps. This little cli utility move all dependencies to bundleDependencies.

The CI script is something along these lines:

npm install

npm test

npm prune --production

bundle-deps

npm version patch $BUILD_NUMBER

npm pack

the result artifact is a npm pack with all dependencies inside. During deployment we use "npm install module.tgz" and yes this compile native dependencies. 

We never have problems with things compiled differently but it is something I would like to review in the future. I blame linux distributions for breaking binary compatibility, this make building everything ahead of time complicated.

I don't like to version node_modules because:

1- messy commits with tons of file changes

2- if you clone the repo in other platform it will probabily not work because..... native dependencies and binary compatibility 

Right now we are in the process of replicating npm in a cloud vm (its almost finished).

2013/11/6 Will <wma...@gmail.com>

--