Troubles with running npm install in the default heroku-buildpack-nodejs with private Github repos

[roma...@googlemail.com]

Hi guys,

We are using private Github repositories for our Node.js apps and now facing an issue switching over from CloudFoundry version 1 to version 2 in our environment.

Before we simply ran 'npm install' before pushing the app into CF and everything worked as it should.

With introduction of the buildpacks in CF 2, the default heroku-buildpack-nodejs which is used to deploy a Node.js app will always run 'npm install' as part of the deployment process and fail because it does not have credentials to access private repositories. 

The document located at

http://docs.cloudfoundry.com/docs/using/deploying-apps/javascript/

mentions that "You do not need to run npm install before deploying your application. Cloud Foundry will run it for you when your application is pushed. If you would prefer to run npm install and create a node_modules folder inside of your application, this is also supported." But it does not say how to make the buildpack not to run 'npm install' - and after spending some time looking at the scripts contained in the https://github.com/cloudfoundry/heroku-buildpack-nodejs I could not spot any conditionals for passing by the 'npm install' execution.

If there is no easier way, I see the following options with getting this issue resolved for us:

1. incorporate the ability to bypass 'npm install' into a forked version of the Heroku buildpack

2. figure a way to supply auth credentials to that buildpack (through CF) so that the buildpack can actually clone the private repos 

1. - could be based either on a simple test of a presence of the 'node_modules' directory (not the most intelligent way I admit) or some sort of parsing files uploaded with the application in order to figure whether some sort of a deployment option 'no npm install' has been provided.

2. - either passing over a .netrc file (ugly) or a Github auth token in some way. The latter seems to have been tackled with the following buildpack in the Heroku world: https://github.com/timshadel/heroku-buildpack-github-netrc 

Alas I could not find any way a buildpack can access user-defined environment variables reading this document:

http://docs.cloudfoundry.com/docs/using/deploying-apps/buildpacks.html

Any help or hints on this topic will be greatly appreciated!

With kind regards,

Roman

[Andy Piper]

I think in v1 if you did NOT include an npm-shrinkwrap.json it would not execute npm install on the staging side.

However, since I think either package.json or npm-shrinkwrap.json are part of the autoselection of the Node buildpack at staging time in v2, that may no longer be the case.

You could try excluding those files, but passing a --buildpack=https://github.com/cloudfoundry/heroku-buildpack-nodejs to the cf push command? *disclaimer, I have not tried this.

[Roman Revyakin]

Hi Andy,

Thank you for your hint, unfortunately it does not work: for some reason the CF2 still tries to execute the 'compile' stage of the build pack, even though in my understanding it should have stopped executing the build pack once the 'detect' stage has failed:

```

#!/usr/bin/env bash

# bin/detect <build-dir>

if [ -f $1/package.json ]; then

  echo "Node.js" && exit 0

else

  echo "no" && exit 1

fi

```

Here's the error and the stack trace:

```

-----> Downloaded app package (12M)

Initialized empty Git repository in /tmp/buildpacks/heroku-buildpack-nodejs/.git/

cat: /tmp/staged/app/package.json: No such file or directory

cat: /tmp/staged/app/package.json: No such file or directory

-----> Resolving engine versions

       WARNING: No version of Node.js specified in package.json, see:

       https://devcenter.heroku.com/articles/nodejs-versions

       Using Node.js version: 0.10.15

       Using npm version: 1.2.30

-----> Fetching Node.js binaries

-----> Vendoring node into slug

-----> Installing dependencies with npm

       npm ERR! install Couldn't read dependencies  

       npm ERR! Error: ENOENT, open '/tmp/staged/app/package.json'

       npm ERR! If you need help, you may report this log at:

       npm ERR!     <http://github.com/isaacs/npm/issues>

       npm ERR! or email it to:

       npm ERR!     <np...@googlegroups.com>

       npm ERR! System Linux 3.0.0-32-virtual

       npm ERR! command "/tmp/node-node-PyHJ/bin/node" "/tmp/node-npm-FAYD/cli.js" "install" "--production"

       npm ERR! cwd /tmp/staged/app

       npm ERR! node -v v0.10.15

       npm ERR! npm -v 1.2.30

       npm ERR! path /tmp/staged/app/package.json   

       npm ERR! code ENOENT

       npm ERR! errno 34

       npm ERR!

       npm ERR! Additional logging details can be found in:

       npm ERR!     /tmp/staged/app/npm-debug.log   

       npm ERR! not ok code 0

 !     Failed to install --production dependencies with npm

/var/vcap/packages/dea_next/buildpacks/lib/installer.rb:16:in `compile': Buildpack compilation step failed: (RuntimeError)

        from /var/vcap/packages/dea_next/buildpacks/lib/buildpack.rb:141:in `block in compile_with_timeout'

        from /usr/lib/ruby/1.9.1/timeout.rb:68:in `timeout'

        from /var/vcap/packages/dea_next/buildpacks/lib/buildpack.rb:140:in `compile_with_timeout'

        from /var/vcap/packages/dea_next/buildpacks/lib/buildpack.rb:131:in `block in stage_application'

        from /var/vcap/packages/dea_next/buildpacks/lib/buildpack.rb:127:in `chdir'

        from /var/vcap/packages/dea_next/buildpacks/lib/buildpack.rb:127:in `stage_application'

        from /var/vcap/packages/dea_next/buildpacks/bin/run:10:in `<main>'

```

Best regards,

Roman

[Balachander Keelapudi]

I believe option 1 is recommended ! 

[Andy Piper]

Probably best to log an issue against our copy of the buildpack https://github.com/cloudfoundry/heroku-buildpack-nodejs so that it is on the development radar - feel free to send a pull request too.

--
Andy Piper | Farnborough, Hampshire (UK)
blog: http://andypiper.co.uk   |   skype: andypiperuk
twitter: @andypiper  |  images: http://www.flickr.com/photos/andypiper

[James Bayer]

So your app already includes the dependencies in the `node_modules` directory on the client side before the push correct? Maybe those files are being excluded during the upload?

If you `npm install` with an already populated `node_modules` directory, doesn't that become a no-op for the modules that already exist in your app?

--

Thank you,

James Bayer

[Shawn Nielsen]

Our organization experienced a similar issue to the one you're facing.   We ended up creating a private fork of the current heroku-buildpack-nodejs and added a 'cfNpmInstall' property flag which can be placed in the package.json file.    When the buildpack runs it checks that flag to determine whether or not we want to use the npm install during staging.   If the option is not set, it simply uses the node_modules that are uploaded.  

This approach gives us the flexibility of deploying with or without the node_modules and also allows us to run builds that are ready to deploy at any time without an npm registry dependency.   If there's interest in this approach, we could submit a pull request.

[Dr Nic Williams]

Shawn, perhaps create the PR if in part to explain what you did. As a bonus, it might get merged in :)

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--

Dr Nic Williams

Stark & Wayne LLC - consultancy for Cloud Foundry users

http://drnicwilliams.com

http://starkandwayne.com

cell +1 (415) 860-2185

twitter @drnic

[Roman Revyakin]

Hi Shawn,

Your solution seems like a more intelligent alternative to what we did. We also forked the heroku-buildpack-nodejs but modified the 'compile' script so that it does not perform 'npm install' if it detects a 'node_modules' directory:

```
diff --git a/bin/compile b/bin/compile
index 3df26b0..4fd6513 100755
--- a/bin/compile
+++ b/bin/compile
@@ -201,11 +201,15 @@ INCLUDE_PATH="$VENDORED_NODE/include"
 export CPATH="$INCLUDE_PATH"
 export CPPPATH="$INCLUDE_PATH"
 
-# install dependencies with npm
-echo "-----> Installing dependencies with npm"
-run_npm "install --production"
-run_npm "rebuild"
-echo "Dependencies installed" | indent
+if [ -d "$BUILD_DIR/node_modules" ]; then
+    echo "-----> Skip npm install: '$BUILD_DIR/node_modules' dir detected"
+else
+    # install dependencies with npm
+    echo "-----> Installing dependencies with npm"
+    run_npm "install --production"
+    run_npm "rebuild"
+    echo "Dependencies installed" | indent
+fi
 
 echo "-----> Building runtime environment"
 mkdir -p $BUILD_DIR/.profile.d
```

I do hesitate to create a PR for such a workaround - and probably yours will be a better solution indeed - so I agree with Dr. Nic. - submit yours!

Best regards,
Roman

[Shawn Nielsen]

I went ahead and submitted the pull request.   Let me know if you have any questions on it.    The default in the pull request is to use npm install, with the extension to disable with the "cfNpmInstall":false engine flag:

https://github.com/cloudfoundry/heroku-buildpack-nodejs

[Shawn Nielsen]

I went ahead and submitted the pull request.   Let me know if you have any questions on it.    The default in the pull request is to use npm install, with the extension to disable with the "cfNpmInstall":false engine flag:

https://github.com/cloudfoundry/heroku-buildpack-nodejs