How to populate Nexus when Central blocked by firewall?

[jeffjensen]

Hello,

My new customer's security group does not allow servers to reach the Internet.  As an example for OS updates, they copy yum repos to an internal server and then all servers pull from there.

What have others found as good solutions (workarounds?) to this firewall blockage for populating Nexus?

[Nikola Milutinovic]

Hi Jeff.

 

If your company has a security policy requiring all inbound content, intended for distribution (YUM repos, Windows Update,…), to land on a secure host, inside some sort of DMZ, that is acceptable. They can do the same with Maven Central.

 

You should ask them to mirror Maven Central on a host in that “DMZ” and then you can proxy that on your Nexus. Nexus is much more than just a proxy, so there is a case for its use, even if the Maven Central is mirrored.

 

Nix.

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/f5908a95-85c6-4b46-8c77-217a3453a37b%40glists.sonatype.com.
For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

[Henrik Ridder]

Hi Jeff,

I’m in a similar situation and i solved it by have a identical setup on each side and then made a script that take ICS offline, tar together a “diff" of the ICS files that i want to copy to NICS.
Basically i have a timestamp file from when i did the last diff, and then using find -cnewer 

Also using exclude file options in tar, with all the files/dir i don't want to be part of the copy.

So something like

find sonatype-work/nexus/storage -type f -cnewer .last_check | tar -vzcf nexus-diff.tar.gz -X new_excludes —files-from -

I guess rsync would do the same thing instead of tar.

Then put it on a thumb drive and copy it to NICS and untar it.

But you have to re-index everything after you copied the files and you have to shutdown the ICS before you make the rsync or tar.

I missed that once and it was no fun.

For the re-index i just run a schedule task in Nexus.

It works, but it's a lot of overhead work.
But I can’t connect the two networks so that’s the way I did it.
I hope i explain it in an understandably way?!

/Henrik

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/6c81b8b0fb734125a68fb78b31913080%40nldcpvmlbx0002.levi9.com.

[MATHUS Baptiste]

By experience, you don't want to rsync the whole central repo if you don't want to get automatically banned.
At least, contact/wait for sonatype answer here before hammering Central.

My 2 cents
Cheers

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/E950ADC6-1691-4062-BC00-7844D1273191%40team11.org.

[Henrik Ridder]

Sorry if I made it look like im talking about rsync the whole central.

Never do that!

I was talking about rsync your ICS vs the NICS instead of using tar to do the same. 

I myself do not use rsync.

Hope that cleared up any confusion.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/ADBE2176A3D94349BB72EEAFDE21D651018E54E262%40sv-t-vnw-ex102.mipih.net.

[Peter Lynch]

Hi everyone,

This issue comes up every now and then. Great to hear these real-world solutions listed from our community. 

Sonatype has listed similar ideas in an article about Nexus in the DMZ.

There is no single solution to work with this restriction, but Nexus design should be flexible enough for you to come up with a way to address the concerns of a security team.

-Peter

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/D132C547-C8B7-488B-AA74-B6EFC82CABAF%40team11.org.

[Rich Seddon]

Just wanted to chime in here...

Henrik's solution is the best one, provided that you can run your build outside of your DMZ.    If you can, just set up a Nexus instance outside the DMZ, run your build(s) through it, and then you'll have what you need.  Because Nexus store's artifacts on disk in Maven 2 repository format you can just copy the storage directory of the central proxy repository outside the DMZ to an instance inside.

After doing this run "update indexes" on the destination repository, this will allow UI search in Nexus to find the new artifacts.

Rich

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/D132C547-C8B7-488B-AA74-B6EFC82CABAF%40team11.org.

[Manso, Francisco]

If your company's IT blocks sonatype central is for a reason. 

Reinventing the wheel is kind of not a good idea. 

I rather present a case to my IT department about the need to access sonatype central.

Once they allow an iP to access, that is it you have your approved internal sonatype repository, that your internal Devekopmenr Nexus can point to. 

If your company have policies about what you can or not download, use Nexus procurement that allows signing off every download so you are legally covered. 

My 2 cents 

- Francisco

Sent from my iPhone.

Please excuse my typos and clever auto-corrections.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/CALX_hVx%3D8gxDG3__yJqbPYCPKE92A1d%2Bw50JsZZdOO3kXSNwew%40mail.gmail.com.

[Manfred Moser]

The other problem of course is that this idea of only having the "right" things available is totally transient and time dependent. It relies on the old idea of a "Golden Repository" and as anybody that has tried that will tell you... it doesnt work.

http://blog.sonatype.com/2013/10/golden-repository/

Unfortunately in practice a lot of people still attempt to follow this approach anyway. I have seen a too many companies having their own custom scripts for provisioning repositories. I even ended up starting to write my own open source tool. It is available here https://github.com/simpligility/maven-repository-tools It works but has a few problematic limitations (like no javadoc and sources being provisioned yet) that I will have to implement but lack the time to do so. I am however more than happy to take pull requests if somebody wants to take it to the next level and collaborate on this. Once its a bit better and some of the planned features are implemented I am happy to cut a release and push to central.

Manfred

Manso, Francisco wrote on 2015-01-12 08:38

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/1E474EFF-DAF0-439E-A7D9-9353A110EB28%40emc.com.

[Jamie Jones]

All,

  I have tackled similar issues for previous clients who either have development lans or disconnected vm farms that lacked outside connectivity.

I've typically done combination of diffs, tars and finds as mentioned above, but did use an rsync solution for a while.  However, I eventually ran into problems with different servers running incompatible versions of rsync, and general complexity.

From a "purity" standpoint, rsync is the way to go, and just maintain a one way sync.  But that can be a bit more trouble than it's worth, in my real world experience.

Best of luck!

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/20150112170145.93FE347C0B20%40dd7514.kasserver.com.

[Jeff]

For what it's worth, we have a secure development environment (SDE) at out work and they have locked down external access, but have allowed direct access to lab networks generally on a per-request basis.  All our builds happen inside the SDE (requiring Nexus access).  Since QA/production delivery needs access outside the SDE for testing and such, the build artifacts must be accessible in both places.

We ended up locating our Nexus server outside the SDE (where Maven Central is accessible).  IT opened ports for any server/workstation needing access to Nexus.  This allowed pushing builds for consumption by anyone and still kept outside access blocked.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/CAGMEYLRwneNhgvWwTApm04xysR06wtO77uPMm_r5-kgs1z7jyg%40mail.gmail.com.

[Jeff Jensen]

Thank you all for your ideas and responses.

With some extra explanations and requests, they agreed to open a port to the Central.  w00t!

So the next question is which ip to use (in US)?  Is 23.235.40.209:80 the best one, or is their a different balancer or other to use?

(I didn't see anything in the FAQ or other site pages)

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/CAPJK9%3DJB%2B5kfZkXXAU9XxYRG%3DL3u0NxpvfQ3CJt3YTwH4r4gmg%40mail.gmail.com.

[Jamie Jones]

I'd also be interested in current information about how central is load balanced and located.  The latest information I could find was from 2011 I believe.

http://blog.sonatype.com/2011/07/the-central-repository-is-getting-faster-are-you-ready-for-the-new-ips/#.VLU1iCvF-4I

If I remember from my previous research, central will balance to whichever is faster/less loaded/whatever.  uk.maven.org will always go to the uk instance (or instance's' now, not sure what the current topology looks like).  However, I could not find a way to force to only the US servers.

An update on current setup (is this in the "cloud" these days? Are they multiple europe servers? Is there a us/north america-only load balancer address?) would be helpful.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/CAKumxBzcr%3DEn12M5jM4cNe0enk10eYXKX9%2BCfmoMweydiZkEhA%40mail.gmail.com.

[Brian Fox]

Hi Jamie,

Central is all distributed via a CDN now so all the previous urls would be cnames to the CDN.

Additional reading:

http://blog.sonatype.com/2012/07/we-just-kicked-central-performance-and-availability-up-a-notch-with-edgecast/#.VLU7G6YgGK4

https://support.sonatype.com/entries/20871391-Where-is-Central-Hosted-

http://www.marketwired.com/press-release/Sonatype-Partners-With-EdgeCast-Networks-Serve-Billions-Requests-per-Year-From-1681244.htm

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/CAGMEYLTm6QiEVRda0O02o2FXTzcm57JY-CAtJ%2BPteJp6BuSLug%40mail.gmail.com.

[Jamie Jones]

Brian, 

Does that mean that vanity urls, like the former uk.maven.org are deprecated too?  All of the balancing and control has now been handed over to edgecast, so there is no way to force a specific cdn, correct? (I know these questions go against the very definition and usefulness of CDN's, but from a build-repeatability sense, I have customers who don't trust that "this server" will be the same as "that server", and have interest in "pinning" their builds.

[Jason Swank]

Hi Jamie,

On Tue, Jan 13, 2015 at 03:43:55PM +0000, Jamie Jones wrote:
> Does that mean that vanity urls, like the former uk.maven.org are
> deprecated too?

Deprecated, but with no intent of removing them.

The canonical URLs are:

https://repo1.maven.org
https://repo1.maven.apache.org

> All of the balancing and control has now been handed over to edgecast,

Just a nitpick- we've been using Fastly for the last 20 months.

> so there is no way to force a specific cdn, correct? (I know these
> questions go against the very definition and usefulness of CDN's, but
> from a build-repeatability sense, I have customers who don't trust that
> "this server" will be the same as "that server", and have interest in
> "pinning" their builds.

That is mostly correct. We don't have control over the IP addresses
used by the CDN POPs, and they are dynamic.

One alternative that we've directed folks to for this purpose is the
(also "deprecated, but no intent of removal") is "Secure Central".
This is available at a fixed IP address, and requires use of an
authentication token.

https://support.sonatype.com/entries/20871391-Where-is-Central-Hosted-
https://support.sonatype.com/entries/22527071-Configuring-Nexus-for-Secure-Access-to-Maven-Central

Hope this helps,
Jason

[Jamie Jones]

Jason, what's the details with secure central? hosted location/etc? is it manged via the same as the other CDN data (aka, artifacts pushed to them all more or less simultaneously)?  Sorry for all the questions, but as this thread shows, the various points of documentation out there all hold portions of the current configuration, and this thread has been a good Rosetta stone to access the details.

[Jason Swank]

Hi Jamie,

On Tue, Jan 13, 2015 at 04:30:48PM +0000, Jamie Jones wrote:
> Jason, what's the details with secure central? hosted location/etc? is it
> manged via the same as the other CDN data (aka, artifacts pushed to them
> all more or less simultaneously)?

Secure central is running on several nodes our colocated environment
behind a FIPS-compliant load balancer. It is updated along with the CDN.

> Sorry for all the questions, but as this thread shows, the various
> points of documentation out there all hold portions of the current
> configuration, and this thread has been a good Rosetta stone to access
> the details.

No apology required :)

Jason

[Rich Seddon]

In case it helps, I've issued a token for you for secure central.

Rich

Jason

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/20150113172700.GE27097%40sonatype.com.

[Jeff Jensen]

I'm hoping "you" means "me", but my guess it means "Jason".  :-)  

May we receive one too to prevent the CDN IP change issue?

[Manfred Moser]

Also just to clarify .. the documentation for the Central Repository is located at 

http://central.sonatype.org/

If anybody feels like more of this sort of information needs to go onto that site please let us know.

There is a LOT of information there already though.

manfred

Hi Jamie,

On Tue, Jan 13, 2015 at 04:30:48PM +0000, Jamie Jones wrote:
> Jason, what's the details with secure central? hosted location/etc? is it
> manged via the same as the other CDN data (aka, artifacts pushed to them
> all more or less simultaneously)?

Secure central is running on several nodes our colocated environment
behind a FIPS-compliant load balancer.  It is updated along with the CDN.

> Sorry for all the questions, but as this thread shows, the various
> points of documentation out there all hold portions of the current
> configuration, and this thread has been a good Rosetta stone to access
> the details.

No apology required :)

Jason

-- 
You received this message because you are subscribed to the Google Groups "Nexus
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit
https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/20150113172700.GE27097%40sonatype.com.