NPM Repository on an air-gapped network? N2, N3, ???

[Phil Beiler]

We do all our development on an air-gapped network, mainly using simple Maven repositories, This works great, as it is extremely easy to sneaker-net individual files or even collections of files to the air-gapped network.

I started using N3 to support the new NPM requirement, but that was a complete failure for both maven and NPM artifacts. It appears that N3 did not actually seem to use any of the cached artifacts.  I went back to N2, but have not had much more success. 

This is what I tired to do, but have numerous issues:

1. Setup an NPM proxy on an internet enabled machine.
2. Install a bunch of npm modules – which appear to be cached in NEXUS and NPM works as expected.
3. Shut down nexus.
4. Create a tar ball of NEXUS (sonatype work) and sneaker net it to our air-gapped network
5. Install the tarball in the exact same location as the internet facing machine.
6. Start NEXUS So far, so good!


Some weird things:

1. We have an “hacked” attempt at mirroring registry.npmjs.org – so we have an actual DNS entry for that site, but NEXUS cannot talk to it because it is not HTTPS – so we get a handshake error in the UI

· This seems fine, but because NEXUS will serve out some limited number of the NPM modules

· However, If I change the proxy URL in nexus – NEXUS will NOT vend any more artifacts.

2. When I try to install most of the previously installed modules (from the cache in N2), NPM fails -- For example: Npm install gulp

· It seems to pull down a bunch of modules, but fails on chalk-1.1.1.tgz

· It is NOT in the repository, verified by browsing.

· It is NOT even in the repository on the internet enabled machine, even after doing a npm install chalk – which said success – but still NOT in the repository…

3. When I say npm install instanbul – I get the “Repository proxy mode BLOCKED_AUTO

- Which make sense, but not really..

- Yes, there is no connectivity, but all of the modules I’m asking for were asked for on the internet enabled machine, and they should be cached.

I'm getting very frustrated with this problem!  I have used N2 for Maven artifacts for years, but have had no success with NPM. 

Can I ask if or how anyone else deals with this problem? Maybe I am using the product completely wrong, and it not intended to be used as a disconnected cache?

Thanks for any assistance. Phil

[Peter Lynch]

Hi Phil,

On Mon, Jan 4, 2016 at 11:42 AM, Phil Beiler <phil....@gmail.com> wrote:

We do all our development on an air-gapped network, mainly using simple Maven repositories, This works great, as it is extremely easy to sneaker-net individual files or even collections of files to the air-gapped network.

I started using N3 to support the new NPM requirement, but that was a complete failure for both maven and NPM artifacts. It appears that N3 did not actually seem to use any of the cached artifacts.  I went back to N2, but have not had much more success. 

Nexus 3 uses a different storage technology than Nexus 2. You cannot move storage files around in the same manner. Our story for migrating content from one nexus 3 instance to another is not yet complete.

This is what I tired to do, but have numerous issues:

1. Setup an NPM proxy on an internet enabled machine.
2. Install a bunch of npm modules – which appear to be cached in NEXUS and NPM works as expected.
3. Shut down nexus.
4. Create a tar ball of NEXUS (sonatype work) and sneaker net it to our air-gapped network
5. Install the tarball in the exact same location as the internet facing machine.
6. Start NEXUS So far, so good!

When Nexus 2x downloads remote NPM packages, it reads in metadata contained in those packages and specific metadata URLs in the remote server. Then it stores this metadata in a database.

While you may copy the NPM package contents from one Nexus 2 NPM repository to another, you are not copying the metadata and so Nexus determines that it needs to go remote to get the files.

Instead copy the NPM proxy repository storage source contents into an NPM hosted repository. Then after starting up Nexus, manually schedule a task called Rebuild hosted npm metadata to have Nexus read in the storage contents and rebuild the metadata. After the task completes, try your build again.

Anytime you add npm packages manually in this manner, rerun the scheduled task.

Some weird things:

1. We have an “hacked” attempt at mirroring registry.npmjs.org – so we have an actual DNS entry for that site, but NEXUS cannot talk to it because it is not HTTPS – so we get a handshake error in the UI

· This seems fine, but because NEXUS will serve out some limited number of the NPM modules

· However, If I change the proxy URL in nexus – NEXUS will NOT vend any more artifacts.

In the air-gapped network, do not use a proxy repository, use a hosted repository as described above.

 

2. When I try to install most of the previously installed modules (from the cache in N2), NPM fails -- For example: Npm install gulp

· It seems to pull down a bunch of modules, but fails on chalk-1.1.1.tgz

· It is NOT in the repository, verified by browsing.

· It is NOT even in the repository on the internet enabled machine, even after doing a npm install chalk – which said success – but still NOT in the repository…

Some packages contain hard coded URLs to external servers that are different from the official registry. This could be what is happening.

The debug log from npm should confirm what URLs are being used. Use the -ddd  option to npm client to get the most verbose output.

https://support.sonatype.com/hc/en-us/articles/213465048-Why-does-npm-client-need-access-to-URLs-other-than-my-private-registry-

 

3. When I say npm install instanbul – I get the “Repository proxy mode BLOCKED_AUTO

- Which make sense, but not really..

- Yes, there is no connectivity, but all of the modules I’m asking for were asked for on the internet enabled machine, and they should be cached.

As before, please try a hosted repository processed with a scheduled task.

I'm getting very frustrated with this problem!  I have used N2 for Maven artifacts for years, but have had no success with NPM. 

Can I ask if or how anyone else deals with this problem? Maybe I am using the product completely wrong, and it not intended to be used as a disconnected cache?

Thanks for any assistance. Phil

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/653e735b-a31a-4681-ab09-a84265263cd9%40glists.sonatype.com.
For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

[Brian Blain]

So if there is no longer a sonatype-work directory, how do we export and import files to a offline network?