really confused about which node_modules folder is being used by my application

[vesper8]

Hello fellow programmers,

I'm pretty new to node.js and I've just inherited a massive project that's built in node and I need to prepare it for small team development.

I apologize in advance for my newbieness.. I hope to become much better at node.js in the coming months (I have no choice)

I plan to use SVN and wanted to segment the existing project into many smaller repos to lessen the chance of conflicts when being working on by multiple people.

One thing I want to do is move the node_modules out of the main application repo because it causes commits and deployments to take waaay too long because of how massive the node_modules folder is.

I understand that the node_modules can be anywhere above the application root so right now I've set it up so it looks like this:

Before:

/var/myProjects/ProjectX/node_modules/

/var/myProjects/ProjectX/app.js

/var/myProjects/ProjectX/everything else

After:

/var/myProjects/node_modules/

/var/myProjects/ProjectX/app.js

/var/myProjects/ProjectX/everything else

I run my app and everything seems to be working fine. 

Now here's my confusion. I wanted to make sure that having moved the node_modules folder one level up from my application root really worked. So I went ahead and renamed the node_modules folder to "test".

And low and behold.. my application doesn't break! It doesn't stop working.. which has me very puzzled. Where is it loading the dependencies from??

I noticed that /home/myUser/.npm/ appears to have all the same node_modules as the ones that were in my application. Is it loading it from there? How can I verify this?

Is what I did ok? Moving node_modules up one folder I mean.

Really appreciate your time. Thank you

[mscdex]

On Monday, November 18, 2013 4:57:23 AM UTC-5, vesper8 wrote:

And low and behold.. my application doesn't break! It doesn't stop working.. which has me very puzzled. Where is it loading the dependencies from??

You can do `npm ls` in your project's root directory to see where modules are installed.

[vesper8]

I just tried that.. npm ls. And it is giving me a bunch of errors about unmet dependencies.

It's weird that my app seems to be working fine despite all the missing dependencies. But I did just notice that a few things appear to be broken.

Now the question is, why isn't it finding the node_modules that is just above the application root? I thought the default behavior is to look for a node_modules in the CWD, and if none is found it crawls up the root and looks in those folders.

As it stands the node_modules folder is two levels above the project root so it should find it easily?

If I'm wrong in my assumptions, then how can I point it to the folder. That is, without having to modify hundreds of paths accross my project.

On Monday, November 18, 2013 4:57:23 AM UTC-5, vesper8 wrote:

[Alex Kocharin]

If you launch your application and rename (or even remove) node_modules later, your application will still be working because of various caches.

Answering an original question, yes, node_modules above the application root should work fine, although default behaviour is to look for a node_modules in cwd AND all folders up to the root, so if you split your modules between several upper folders, it'll still work.

[vesper8]

Yea.. I am noticing that and it is driving me crazy actually.

I was actually able to confirm (because of debug errors) that the application has succesfully found the node_modules folder by crawling up two levels. So I guess npm ls is not particularly sincere about the missing dependencies. Could it be that it only looks for them in the CWD the "npm ls" is run from? It, unlike node, doesn't crawl up to locate the node_modules.

Anyway... it found the node_modules up two folders.. which is good. I moved it again just to test if it would locate it again. And it's driving me crazy because the error text keeps refering to node_modules in the folder where I moved it out of. So I guess it's cached somehow.

Does anyone know how I can fully clear the cache? I am using PM2 to start my processes and restarting them doesn't clear that stuborn cache. Even stopping and deleting the processes completely and then starting them up again doesn't clear it.

[vesper8]

Ok... so I figured out all I needed to do to clear that stuborn cache was make any little change to the application. Simply adding a whitespace and re-saving the app.js in the project root caused the app to recrawl and locate the node_modules folder.

I can also now confirm that it does indeed crawl up and up and up until it locates a node_modules folder. Which is great.. that's exactly what I needed to confirm. Now I can move the node_modules out of the main project repo and svn commits are now 10x faster.

Thanks for the help guys :)

[Ryan Schmidt]

On Nov 18, 2013, at 03:57, vesper8 wrote:

> I plan to use SVN and wanted to segment the existing project into many smaller repos to lessen the chance of conflicts when being working on by multiple people.
>
> One thing I want to do is move the node_modules out of the main application repo because it causes commits and deployments to take waaay too long because of how massive the node_modules folder is.

The node_modules directory should be in the same directory that contains your package.json file.

> I understand that the node_modules can be anywhere above the application root so right now I've set it up so it looks like this:
>
> Before:
> /var/myProjects/ProjectX/node_modules/
> /var/myProjects/ProjectX/app.js
> /var/myProjects/ProjectX/everything else
>
> After:
> /var/myProjects/node_modules/
> /var/myProjects/ProjectX/app.js
> /var/myProjects/ProjectX/everything else

The purpose of having a node_modules folder in each project is so that each project can independently manage its own dependencies.

Imagine ProjectX uses express 2.x, and so does ProjectY. So ProjectX/node_modules contains express 2.x and so does ProjectY/node_modules. Now you plan to move to a centralized node_modules folder. What happens when you want to upgrade to express 3.x, which necessitates changes to your project’s code? Are you prepared to make those changes to all of your projects simultaneously?

[vesper8]

Oh I understand the purpose. And I'm sorry if I didn't explain myself properly but I do not plan to have a centralized node_modules folder. I am still going to have one node_modules folder per project. I simply did not want to have it inside the same SVN repo because it makes the repo weigh 100mb instead of 2mb, and it makes commits and deployments much longer than they need to be. Since the node_modules folder doesn't change or hardly ever changes, I prefer to isolate it and have it be in its own SVN repo.

But that's just me and how I segment my projects into multiple repos. Maybe it's not the best practice but it's how I learned to work when working as a small to medium team of developers and it's worked well so far.

Def. thanks for the feedback though I appreciate the intention

[mscdex]

On Monday, November 18, 2013 2:37:50 PM UTC-5, vesper8 wrote:

I simply did not want to have it inside the same SVN repo because it makes the repo weigh 100mb instead of 2mb, and it makes commits and deployments much longer than they need to be.

You should be able to add node_modules to your .svnignore file and put that file in your svn repo and svn will not commit that directory (this is how .gitignore for git works also).

[Martin Alix]

I agree, ignoring is the way to go...
Can you try "svn propset svn:ignore node_modules ." from your project directory?