Single URL/Tag for pulling and pushing docker images from and to Sonatype Nexus 3

[Mario Wirth]

Hi community,

in this helpful video here it is demonstrated how Nexus can be used as a private docker registry: https://www.youtube.com/watch?v=Z2jH9LgeeI8
I am wondering why you recommend to use different ports for pulling and pushing. I understand that it is necessary to forward the pull and push request to different Nexus Docker repositories. But instead of different ports I use this apache vhost configuration to split the traffic by the used http protocol method:

<VirtualHost *:443>
 SSLEngine on

 SSLCertificateFile "/etc/ssl/certs/nexus.sprd.net.crt"
 SSLCertificateKeyFile "/etc/ssl/private/nexus.sprd.net.key"

 RewriteEngine on
 #### pull ####
 RewriteCond %{REQUEST_METHOD} GET
 RewriteRule ^/(.*) /docker-pull/$1 [PT]

 #### push ####
 #HEAD checks if the layer is already in the private registry. I forward this to the push repo to make sure all layers are uploaded
 #to this particular repo. Otherwise docker does reject pushing if some layers are missing (which are in fact in another nexus repos only and just merged by the repository group)
 #forward snapshots
 RewriteCond %{REQUEST_METHOD} (HEAD|POST|PATCH|PUT|DELETE)
 RewriteCond %{REQUEST_URI} .*/snapshot/.*
 RewriteRule ^/(.*) /docker-push-snapshot/$1 [PT]
 #forward non-snapshots
 RewriteCond %{REQUEST_METHOD} (HEAD|POST|PATCH|PUT|DELETE)
 RewriteRule ^/(.*) /docker-push/$1 [PT]

 ## Proxy rules
 ProxyRequests Off
 ProxyPass /docker-pull http://localhost:8083
 <Location /docker-pull>
   ProxyPassReverse http://localhost:8083
 </Location>
 ProxyPass /docker-push http://localhost:8082
 <Location /docker-push>
   ProxyPassReverse http://localhost:8082
 </Location>

 ProxyPass /docker-push-snapshot http://localhost:8085
 <Location /docker-push-snapshot>
   ProxyPassReverse http://localhost:8085
 </Location>

 ServerName myregistry.sprd.net
 ServerAdmin ad...@example.com
 RequestHeader set X-Forwarded-Proto "https"

 ## Logging
 ErrorLog "| /usr/bin/rotatelogs -l /var/log/apache2/registry-error.%Y.%m.%d 86400"
 CustomLog "| /usr/bin/rotatelogs -l /var/log/apache2/registry-access.%Y.%m.%d 86400" combined
</VirtualHost>

Explanation of the ports:
- 8082 -> hosted repository with deployment policy: "Disable redeploy"
- 8085 -> hosted repository with deployment policy: "Allow redeploy"
- 8084 -> proxy repository for docker hub (only used via repository group)
- 8083 -> repository group which contains 1. docker-releases, 2. docker-snapshot and 3. docker-hub repos in that order


The ssl offloading is done by Apache2 not by Nexus
As you can see, only requests with GET method are forwarded to the repository group for read access. Requests which use the other listed methods are forwarded to the hosted release repository or snapshot repository, it depends if the word "snapshot" is part of the sent url. (The snapshot is only an internal convention, it reminds me to Maven snapshots...)

With this solution I can i.e. pull an image from docker-hub through Nexus
docker pull myregistry.sprd.net/ubuntu:16.04

I can also pull this image:
docker push myregistry.sprd.net/ubuntu:16.04
This one will end up in the release repository
If I use the word "snapshot" in my tag path i.e.
docker push myregistry.sprd.net/snapshot/ubuntu:16.04
this image and all related layers will be stored in my snapshot repository.

In my opinion this approach is much easier to handle then your suggestion with different ports.
I cannot believe that I am the only one who has the idea for this solution. Is there something I missed and could break later?
My tests are all successful so far. But also the guys from Artifactory recommend different ports for pulling and pushing: https://www.youtube.com/watch?v=014ZXoJnDys&feature=youtu.be&list=PLY0Zjn5rFo4PR0MqN1MsqXG-t4izCcuUx

So, do you see any problems with my simplified approach? Please let me know!
Thank you in advance!

Regards,

Mario

[Jeffry Hesse]

This approach is fine, and I'm glad it works for you. We don't have it currently documented, mainly because we try and put the simplest setup possible (least amount of moving parts) in our docs. 

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/c9e20167-7b9a-4924-9452-bb546224b4fa%40glists.sonatype.com.
For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

[Mika Mattila]

Hi Mario,

I like your idea of using the same port for push and pull. A setup similar to your's seems to work for me as well. 

However, there are a couple things which did not work for me:

- docker search from nexus (either proxied or hosted)

- push to a proxied private docker registry (it gave this: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "\n<!DOCTYPE html>\n<html>\n<head>\n  <title>404 - Nexus Repository Manager</title>\n )

Did you manage to overcome these issues?

Thanks,

Mika

[Mario Wirth]

Hi Mika,

first of all we use the same-port-for-pull-and-push-approach for more than one year now and it works really great. But we don’t use the snapshot repository I mentioned and allow a redeployment on the release repository instead. This simplifies the usage further.

To your questions:

1. I guess you mean the cli tool docker search. This works for me and it searches the proxied repos (docker hub) as well as our hosted repository. But the output order is arbitrary and we prefer using the nexus web ui. Do you have an error message or so?

2. A push to a proxied docker registry is not possible by design. You can only write to hosted repositories.

Best,

Mario

-- 
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/76e81a53-fa75-42e6-88cf-b09406f8334a%40glists.sonatype.com.

[Mika Mattila]

Thanks Mario for your quick response!

Actually, I managed to solve the problem #1. Docker search requires this to be enabled in Nexus config: 

Enable Docker V1 API: Allow clients to use the V1 API to interact with this Repository

About problem #2: ok, I'll use a hosted repo instead.

BR,

Mika

[Mika Mattila]

Hi again,

I'm wondering how I can achieve having both docker and the nexus itself working via httpd reverse proxy. Currently I have this kind of config:

<VirtualHost *:443>

   SSLEngine on

   SSLCertificateFile    /usr/local/apache2/conf/server.crt

   SSLCertificateKeyFile /usr/local/apache2/conf/server.key

   AllowEncodedSlashes NoDecode

   RewriteEngine on

   ProxyPass /nexus http://nexus_nexus_1:8081/nexus

   ProxyPassReverse /nexus http://nexus_nexus_1:8081/nexus

   # pull

   RewriteCond %{REQUEST_METHOD} GET

   RewriteRule ^/(.*) /docker-pull/$1 [PT]

   # push

   RewriteCond %{REQUEST_METHOD} (HEAD|POST|PATCH|PUT|DELETE)

   RewriteRule ^/(.*) /docker-push/$1 [PT]

   ProxyPass /docker-pull http://nexus_nexus_1:8082 nocanon

   ProxyPassReverse /docker-pull http://nexus_nexus_1:8082

   ProxyPass /docker-push http://nexus_nexus_1:8083 nocanon

   ProxyPassReverse /docker-push http://nexus_nexus_1:8083

   ServerName repo.testdomain.com

   RequestHeader set X-Forwarded-Proto "https"

   # Logging

   ErrorLog logs/nexus/error.log

   CustomLog logs/nexus/access.log common

</VirtualHost>

So I have Nexus GUI running in port 8081, docker group in port 8082 and nexus hosted docker repo in port 8083. I want the same URL to be used for both push and pull and context path /nexus to point to the Nexus GUI. (And maybe later on add some YUM repo proxy as well in Nexus.)

But with this setup, the nexus GUI does not work. Docker pull and push seems to be fine.

I think the problem is that RewriteRule takes precedence over ProxyPass, but I haven't been able to figure out how to fix this.

Any suggestions?

Thanks,

Mika

[Rich Seddon]

Docker repositories must run on context path "/", they cannot run on any other context path. This constraint is imposed by docker, not Nexus.  This is the reason that docker repositories have connector ports.  We wish this wasn't necessary, but it is.

So you're going to have to either use a different port in apache for the docker repository, or use a virtual host so that the docker repository can run on context path "/".

https://support.sonatype.com/hc/en-us/articles/115013153887-Docker-Repository-Configuration-and-Client-Connection

[Mika Mattila]

Hi Rich,

Yes, I know about this constraint in Docker. But I think it's not convenient to use different ports for push and pull, so that's why I applied the RewriteRule idea posted by Mario.

My only problem was about reserving "/" for docker and at the same time have "/nexus" accessible for the nexus gui.

I ended up adding one more RewriteRule in Apache config, which seems to do what I wanted.

----- 8< -----

   RewriteEngine on

   RewriteRule ^/nexus(.*) /nexus$1 [PT]

   ProxyPass /nexus http://nexus_nexus_1:8081/nexus

   <Location /nexus>

      ProxyPassReverse http://nexus_nexus_1:8081/nexus

   </Location>

   # pull

  

----- 8< -----  

So this setup seems to work now okay.

BR,

Mika

[Peter Lynch]

On Wed, Sep 19, 2018 at 6:24 AM Mika Mattila <mika.p....@gmail.com> wrote:

Hi Rich,

Yes, I know about this constraint in Docker. But I think it's not convenient to use different ports for push and pull, so that's why I applied the RewriteRule idea posted by Mario.

My only problem was about reserving "/" for docker and at the same time have "/nexus" accessible for the nexus gui.

I ended up adding one more RewriteRule in Apache config, which seems to do what I wanted.

Another bit of info is that all Docker requests will be sent to paths starting with either "/v1" or "/v2" - which gives you another bit to filter on should you need it.

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/CAPk7iCQZVDx27oGeDdVsiqCZc5_vG1Kt2tAMQXXyUWoZdUwEGg%40mail.gmail.com.

[Holger Falk]

Hi Mario,

I am trying to implement your approach myself, but i am running into issues when it comes to authentication with the nexus hostes docker repository.

These are my rewrite rules and it seems like "docker login" is caught by the pull rule and authentication is then proxied to the docker-hub proxy on port 8083, making it impossible for me to push to the hosted repository:

# Rewrite rules

# Docker registry rewrites

#### pull ####
RewriteCond %{REQUEST_METHOD} GET

RewriteCond %{HTTP_USER_AGENT} docker

RewriteRule ^/(.*) /docker-pull/$1 [PT]

#### push ####

RewriteCond %{REQUEST_METHOD} (HEAD|POST|PATCH|PUT|DELETE)

RewriteCond %{HTTP_USER_AGENT} docker
RewriteRule ^/(.*) /docker-push/$1 [PT]

################################################################################
# Proxy rules
ProxyRequests Off
SSLProxyEngine On

SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off
AllowEncodedSlashes NoDecode

ProxyPass           /docker-pull http://localhost:8083

ProxyPassReverse    /docker-pull http://localhost:8083

ProxyPass           /docker-push http://localhost:8086
ProxyPassReverse    /docker-push http://localhost:8086

ProxyPass           / https://localhost:8443/ nocanon
ProxyPassReverse    / https://localhost:8443/

Do you have any ideas why this is not working for me?

Regards,

Holger

[Mario Wirth]

Hi Holger,

please remove these rules from the virtualhost configuration:

ProxyPass           / https://localhost:8443/ nocanon
ProxyPassReverse    / https://localhost:8443/

and create a 2nd virtual host configuration for it. In the end you will have 2 hosts:

i.e. 

myregistry.example.com for your docker requests

mynexus.example.com for the other artifact types like maven, npm…

We use this setup for more than 2 years now.

Hope this will help. 

Best,

Mario

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/8c9ef193-2dfa-4ea1-9505-256256c942be%40glists.sonatype.com.