Hi Guys, I'm working for a startup using Node to build mobile BaaS integrations for mobile. When building our Node apps, we've been trying to follow the pattern of small apps, where functionality is broken out into many node modules. This works out pretty well, until it comes to managing these dependencies across projects. A lot of our customers are large enterprises, who aren't comfortable open sourcing every module they create. Some of these modules would be little benefit to the open source community, some contain proprietary business logic..
Before I go on, it's worth referencing Mikeal's blog post: http://www.mikealrogers.com/posts/nodemodules-in-git.html where it's suggested we check in node_modules into git when deploying. I think this works fine when deploying to a VPS or similar, but we deploy to PaaS. We're both a PaaS vendor, and we also support alternative PaaS deploy targets. We have two 'deploy targets' - development, where people deploy frequently to test and stage, and live - which is where production apps get launched from. Even if we did check node_modules into git, we've still got the problem of looking up and resolving these dependencies in a neat manner during development, prior to check in.
So, here's what I've tried so far:
The first attempt involved private git, which worked up to a point. The difficulty arises in the 'npm install -d' stage. The PaaS attempts to resolve dependencies in a private git repository, but unless the user running the NPM command's public key is stored as a deploy key in our repository, the dependency retrieval fails. In a load balanced PaaS, maintaining this myriad of public keys is something we'd like to avoid. There's also a few other disadvantages to using git to store these dependencies - versioning isn't as neat as in NPM itself when specifying depdendencies.
Enter our private install of the NPM registry. This works fine up to a point - we're able to push our node modules to the relevant registry by using package.json's publishConfig - great! The problem arises when it comes to specifying these modules as dependencies in package.json. This would be fine when deploying to a VPS where we have shell, and can script our deploy. We can specify the registry to use, e.g. npm install -d --registry http://someRegistry In a PaaS environment, however, this isn't going to work so well. The installation of dependencies is automated, and relies entirely on package.json - which can specify the dependency we require..: "dependencies" : { "request" : "2.11.2", "somethingPriavete" : "1.0" } ..however not where to find it! So we try to find 'somethingPrivate' in registry.npmjs.org, and fail. So how do we architect our private dependencies? Is there something I'm missing?
If this isn't something currently solved by NPM, how should managing private registries in package.json look in the future? Could there be some kind of array of (preferred?)registries property for the current project, the default being registry.npmjs.org? "registries" : ["http://myprivateregistry", "http://registry.npmjs.org"] Maybe registry.npmjs.org should be an implied registry source, and never needs to be explicit? Am I jumping the gun a bit here, maybe this just polluting the package.json namespace?
I'd welcome any thoughts on how to best manage these private dependencies, as I'm not 100% satisfied with any of the approaches I've tried to date.
The approach we are using is a variation on what you described above.
I'm not 100% happy with it either but is working for us on Heroku. We
specify the dependencies using the npm
"https://user:p...@github.com/user/package/tarball/v0.5.6" format.
With tagging of releases you get a specific version and you don't need
to worry about maintaining SSH keys but you do exposed a username and
password in your package.json. Might work as a stop-gap measure.
On Thu, Oct 25, 2012 at 6:20 AM, Cian <c...@cianclarke.com> wrote:
> Hi Guys,
> I'm working for a startup using Node to build mobile BaaS integrations for
> mobile. When building our Node apps, we've been trying to follow the pattern
> of small apps, where functionality is broken out into many node modules.
> This works out pretty well, until it comes to managing these dependencies
> across projects.
> A lot of our customers are large enterprises, who aren't comfortable open
> sourcing every module they create. Some of these modules would be little
> benefit to the open source community, some contain proprietary business
> logic..
> Before I go on, it's worth referencing Mikeal's blog post:
> http://www.mikealrogers.com/posts/nodemodules-in-git.html > where it's suggested we check in node_modules into git when deploying. I
> think this works fine when deploying to a VPS or similar, but we deploy to
> PaaS. We're both a PaaS vendor, and we also support alternative PaaS deploy
> targets.
> We have two 'deploy targets' - development, where people deploy frequently
> to test and stage, and live - which is where production apps get launched
> from.
> Even if we did check node_modules into git, we've still got the problem of
> looking up and resolving these dependencies in a neat manner during
> development, prior to check in.
> So, here's what I've tried so far:
> The first attempt involved private git, which worked up to a point. The
> difficulty arises in the 'npm install -d' stage. The PaaS attempts to
> resolve dependencies in a private git repository, but unless the user
> running the NPM command's public key is stored as a deploy key in our
> repository, the dependency retrieval fails.
> In a load balanced PaaS, maintaining this myriad of public keys is something
> we'd like to avoid. There's also a few other disadvantages to using git to
> store these dependencies - versioning isn't as neat as in NPM itself when
> specifying depdendencies.
> Enter our private install of the NPM registry. This works fine up to a point
> - we're able to push our node modules to the relevant registry by using
> package.json's publishConfig - great! The problem arises when it comes to
> specifying these modules as dependencies in package.json.
> This would be fine when deploying to a VPS where we have shell, and can
> script our deploy. We can specify the registry to use, e.g. npm install -d
> --registry http://someRegistry > In a PaaS environment, however, this isn't going to work so well. The
> installation of dependencies is automated, and relies entirely on
> package.json - which can specify the dependency we require..:
> "dependencies" : {
> "request" : "2.11.2",
> "somethingPriavete" : "1.0"
> }
> ..however not where to find it! So we try to find 'somethingPrivate' in
> registry.npmjs.org, and fail.
> So how do we architect our private dependencies? Is there something I'm
> missing?
> If this isn't something currently solved by NPM, how should managing private
> registries in package.json look in the future? Could there be some kind of
> array of (preferred?)registries property for the current project, the
> default being registry.npmjs.org?
> "registries" : ["http://myprivateregistry", "http://registry.npmjs.org"]
> Maybe registry.npmjs.org should be an implied registry source, and never
> needs to be explicit?
> Am I jumping the gun a bit here, maybe this just polluting the package.json
> namespace?
> I'd welcome any thoughts on how to best manage these private dependencies,
> as I'm not 100% satisfied with any of the approaches I've tried to date.
Hi Daniel - Good idea! I hadn't considered using basic auth as the way in/out. As you say, it's not perfect - but it's certainly a start, and avoids the headache of managing keys! This would mean we'd need to migrate to Github Private for our SCM in this case, but that doesn't strike me as a bad thing :-) Maybe this will make for a good stopgap, but I'd still like to get this working with NPM as the persistance rather than git at some point! Thanks! Cian
On Thursday, October 25, 2012 3:58:29 PM UTC+1, Daniel Rinehart wrote:
> The approach we are using is a variation on what you described above. > I'm not 100% happy with it either but is working for us on Heroku. We > specify the dependencies using the npm > "https://user:p...@github.com/user/package/tarball/v0.5.6" format. > With tagging of releases you get a specific version and you don't need > to worry about maintaining SSH keys but you do exposed a username and > password in your package.json. Might work as a stop-gap measure.
> On Thu, Oct 25, 2012 at 6:20 AM, Cian <ci...@cianclarke.com <javascript:>> > wrote: > > Hi Guys, > > I'm working for a startup using Node to build mobile BaaS integrations > for > > mobile. When building our Node apps, we've been trying to follow the > pattern > > of small apps, where functionality is broken out into many node modules. > > This works out pretty well, until it comes to managing these > dependencies > > across projects. > > A lot of our customers are large enterprises, who aren't comfortable > open > > sourcing every module they create. Some of these modules would be little > > benefit to the open source community, some contain proprietary business > > logic..
> > Before I go on, it's worth referencing Mikeal's blog post: > > http://www.mikealrogers.com/posts/nodemodules-in-git.html > > where it's suggested we check in node_modules into git when deploying. I > > think this works fine when deploying to a VPS or similar, but we deploy > to > > PaaS. We're both a PaaS vendor, and we also support alternative PaaS > deploy > > targets. > > We have two 'deploy targets' - development, where people deploy > frequently > > to test and stage, and live - which is where production apps get > launched > > from. > > Even if we did check node_modules into git, we've still got the problem > of > > looking up and resolving these dependencies in a neat manner during > > development, prior to check in.
> > So, here's what I've tried so far:
> > The first attempt involved private git, which worked up to a point. The > > difficulty arises in the 'npm install -d' stage. The PaaS attempts to > > resolve dependencies in a private git repository, but unless the user > > running the NPM command's public key is stored as a deploy key in our > > repository, the dependency retrieval fails. > > In a load balanced PaaS, maintaining this myriad of public keys is > something > > we'd like to avoid. There's also a few other disadvantages to using git > to > > store these dependencies - versioning isn't as neat as in NPM itself > when > > specifying depdendencies.
> > Enter our private install of the NPM registry. This works fine up to a > point > > - we're able to push our node modules to the relevant registry by using > > package.json's publishConfig - great! The problem arises when it comes > to > > specifying these modules as dependencies in package.json. > > This would be fine when deploying to a VPS where we have shell, and can > > script our deploy. We can specify the registry to use, e.g. npm install > -d > > --registry http://someRegistry > > In a PaaS environment, however, this isn't going to work so well. The > > installation of dependencies is automated, and relies entirely on > > package.json - which can specify the dependency we require..: > > "dependencies" : { > > "request" : "2.11.2", > > "somethingPriavete" : "1.0" > > } > > ..however not where to find it! So we try to find 'somethingPrivate' in > > registry.npmjs.org, and fail. > > So how do we architect our private dependencies? Is there something I'm > > missing?
> > If this isn't something currently solved by NPM, how should managing > private > > registries in package.json look in the future? Could there be some kind > of > > array of (preferred?)registries property for the current project, the > > default being registry.npmjs.org? > > "registries" : ["http://myprivateregistry", " > http://registry.npmjs.org"] > > Maybe registry.npmjs.org should be an implied registry source, and > never > > needs to be explicit? > > Am I jumping the gun a bit here, maybe this just polluting the > package.json > > namespace?
> > I'd welcome any thoughts on how to best manage these private > dependencies, > > as I'm not 100% satisfied with any of the approaches I've tried to date.
On Thu, Oct 25, 2012 at 12:19 PM, Cian <c...@cianclarke.com> wrote:
> Hi Daniel - Good idea! I hadn't considered using basic auth as the way
> in/out. As you say, it's not perfect - but it's certainly a start, and
> avoids the headache of managing keys!
> This would mean we'd need to migrate to Github Private for our SCM in this
> case, but that doesn't strike me as a bad thing :-)
> Maybe this will make for a good stopgap, but I'd still like to get this
> working with NPM as the persistance rather than git at some point!
> Thanks!
> Cian
> On Thursday, October 25, 2012 3:58:29 PM UTC+1, Daniel Rinehart wrote:
>> The approach we are using is a variation on what you described above.
>> I'm not 100% happy with it either but is working for us on Heroku. We
>> specify the dependencies using the npm
>> "https://user:p...@github.com/user/package/tarball/v0.5.6" format.
>> With tagging of releases you get a specific version and you don't need
>> to worry about maintaining SSH keys but you do exposed a username and
>> password in your package.json. Might work as a stop-gap measure.
>> On Thu, Oct 25, 2012 at 6:20 AM, Cian <ci...@cianclarke.com> wrote:
>> > Hi Guys,
>> > I'm working for a startup using Node to build mobile BaaS integrations
>> > for
>> > mobile. When building our Node apps, we've been trying to follow the
>> > pattern
>> > of small apps, where functionality is broken out into many node modules.
>> > This works out pretty well, until it comes to managing these
>> > dependencies
>> > across projects.
>> > A lot of our customers are large enterprises, who aren't comfortable
>> > open
>> > sourcing every module they create. Some of these modules would be little
>> > benefit to the open source community, some contain proprietary business
>> > logic..
>> > Before I go on, it's worth referencing Mikeal's blog post:
>> > http://www.mikealrogers.com/posts/nodemodules-in-git.html >> > where it's suggested we check in node_modules into git when deploying. I
>> > think this works fine when deploying to a VPS or similar, but we deploy
>> > to
>> > PaaS. We're both a PaaS vendor, and we also support alternative PaaS
>> > deploy
>> > targets.
>> > We have two 'deploy targets' - development, where people deploy
>> > frequently
>> > to test and stage, and live - which is where production apps get
>> > launched
>> > from.
>> > Even if we did check node_modules into git, we've still got the problem
>> > of
>> > looking up and resolving these dependencies in a neat manner during
>> > development, prior to check in.
>> > So, here's what I've tried so far:
>> > The first attempt involved private git, which worked up to a point. The
>> > difficulty arises in the 'npm install -d' stage. The PaaS attempts to
>> > resolve dependencies in a private git repository, but unless the user
>> > running the NPM command's public key is stored as a deploy key in our
>> > repository, the dependency retrieval fails.
>> > In a load balanced PaaS, maintaining this myriad of public keys is
>> > something
>> > we'd like to avoid. There's also a few other disadvantages to using git
>> > to
>> > store these dependencies - versioning isn't as neat as in NPM itself
>> > when
>> > specifying depdendencies.
>> > Enter our private install of the NPM registry. This works fine up to a
>> > point
>> > - we're able to push our node modules to the relevant registry by using
>> > package.json's publishConfig - great! The problem arises when it comes
>> > to
>> > specifying these modules as dependencies in package.json.
>> > This would be fine when deploying to a VPS where we have shell, and can
>> > script our deploy. We can specify the registry to use, e.g. npm install
>> > -d
>> > --registry http://someRegistry >> > In a PaaS environment, however, this isn't going to work so well. The
>> > installation of dependencies is automated, and relies entirely on
>> > package.json - which can specify the dependency we require..:
>> > "dependencies" : {
>> > "request" : "2.11.2",
>> > "somethingPriavete" : "1.0"
>> > }
>> > ..however not where to find it! So we try to find 'somethingPrivate' in
>> > registry.npmjs.org, and fail.
>> > So how do we architect our private dependencies? Is there something I'm
>> > missing?
>> > If this isn't something currently solved by NPM, how should managing
>> > private
>> > registries in package.json look in the future? Could there be some kind
>> > of
>> > array of (preferred?)registries property for the current project, the
>> > default being registry.npmjs.org?
>> > "registries" : ["http://myprivateregistry",
>> > "http://registry.npmjs.org"]
>> > Maybe registry.npmjs.org should be an implied registry source, and never
>> > needs to be explicit?
>> > Am I jumping the gun a bit here, maybe this just polluting the
>> > package.json
>> > namespace?
>> > I'd welcome any thoughts on how to best manage these private
>> > dependencies,
>> > as I'm not 100% satisfied with any of the approaches I've tried to date.
On Thursday, October 25, 2012 11:20:45 AM UTC+1, Cian wrote:
> Hi Guys, > I'm working for a startup using Node to build mobile BaaS integrations for > mobile. When building our Node apps, we've been trying to follow the > pattern of small apps, where functionality is broken out into many node > modules. This works out pretty well, until it comes to managing these > dependencies across projects. > A lot of our customers are large enterprises, who aren't comfortable open > sourcing every module they create. Some of these modules would be little > benefit to the open source community, some contain proprietary business > logic..
> Before I go on, it's worth referencing Mikeal's blog post: > http://www.mikealrogers.com/posts/nodemodules-in-git.html > where it's suggested we check in node_modules into git when deploying. I > think this works fine when deploying to a VPS or similar, but we deploy to > PaaS. We're both a PaaS vendor, and we also support alternative PaaS deploy > targets. > We have two 'deploy targets' - development, where people deploy frequently > to test and stage, and live - which is where production apps get launched > from. > Even if we did check node_modules into git, we've still got the problem of > looking up and resolving these dependencies in a neat manner during > development, prior to check in.
> So, here's what I've tried so far:
> The first attempt involved private git, which worked up to a point. The > difficulty arises in the 'npm install -d' stage. The PaaS attempts to > resolve dependencies in a private git repository, but unless the user > running the NPM command's public key is stored as a deploy key in our > repository, the dependency retrieval fails. > In a load balanced PaaS, maintaining this myriad of public keys is > something we'd like to avoid. There's also a few other disadvantages to > using git to store these dependencies - versioning isn't as neat as in NPM > itself when specifying depdendencies.
> Enter our private install of the NPM registry. This works fine up to a > point - we're able to push our node modules to the relevant registry by > using package.json's publishConfig - great! The problem arises when it > comes to specifying these modules as dependencies in package.json. > This would be fine when deploying to a VPS where we have shell, and can > script our deploy. We can specify the registry to use, e.g. npm install -d > --registry http://someRegistry > In a PaaS environment, however, this isn't going to work so well. The > installation of dependencies is automated, and relies entirely on > package.json - which can specify the dependency we require..: > "dependencies" : { > "request" : "2.11.2", > "somethingPriavete" : "1.0" > } > ..however not where to find it! So we try to find 'somethingPrivate' in > registry.npmjs.org, and fail. > So how do we architect our private dependencies? Is there something I'm > missing?
> If this isn't something currently solved by NPM, how should managing > private registries in package.json look in the future? Could there be some > kind of array of (preferred?)registries property for the current project, > the default being registry.npmjs.org? > "registries" : ["http://myprivateregistry", "http://registry.npmjs.org > "] > Maybe registry.npmjs.org should be an implied registry source, and never > needs to be explicit? > Am I jumping the gun a bit here, maybe this just polluting the > package.json namespace?
> I'd welcome any thoughts on how to best manage these private dependencies, > as I'm not 100% satisfied with any of the approaches I've tried to date.
