So close - docker push through Apache SSL proxy

997 views
Skip to first unread message

Michael Hoffmann

unread,
Nov 13, 2017, 12:59:22 AM11/13/17
to Nexus Users
Hi all, once more.

I think am *so* close to having everything Nexus working just the way I want : YUM plays nice, NuGet was no issues whatsoever (in fact Nexus helped me resolve a documented NuGet bug through offering me a nice work-around), PyPi works acceptably both ways.

The last missing piece is Docker, and there I have pull working nicely. Only push is still elusive.

The setup:

- OS is Oracle Linux 7.4
- Front-end Apache reverse proxy holding the SSL certificate, company issues. CA chains are now all installed both at the OS as well as the Java keystore level
- Nexus is the back-end, of course, using merely HTTP via localhost

I have 3 repositories in Nexus

1) ncdocker as group (HTTP port 55555)
2) dockerhub as proxy
3) dockernabcert as hosted (HTTP port 55556)

Due to unresolved issues (one thing after another) with basic auth passthrough, I've opened up anonymous by unchecking Force Basic Authentication. V1 compatibility is checked. Allow Deploy turned on in the hosted repo.

I have an /etc/httpd/conf.d/r_proxy.conf which has this

<IfModule mod_proxy.c>
        SSLProxyEngine on

        ProxyRequests Off
        ProxyPreserveHost On

        <Proxy *>
                Require all granted
        </Proxy>

    Header always set "Docker-Distribution-Api-Version" "registry/2.0"
    Header onsuccess set "Docker-Distribution-Api-Version" "registry/2.0"
    RequestHeader set X-Forwarded-Proto "https"

    ProxyPass               /repos  http://localhost:8080
    ProxyPassReverse        /repos  http://localhost:8080
    ProxyPass        /v2 http://localhost:55555/v2
    ProxyPassReverse    /v2 http://localhost:55555/v2
    ProxyPass        /v1 http://localhost:55555/v1
    ProxyPassReverse    /v1 http://localhost:55555/v1
    ProxyPass        /nexus http://localhost:8081/
    ProxyPassReverse    /nexus http://localhost:8081/
    ProxyPass        / http://localhost:8081/
    ProxyPassReverse    / http://localhost:8081/

    Timeout 2400
    ProxyTimeout 2400
    ProxyBadHeader Ignore

</IfModule>

This is the result of many, many Googles. The /repos entry is for the legacy reposync site, which has now been bundled away into a simple and minimal Apache container.

The /v1 allows "docker search" to work as well. /v2 allows pull and such to work.

All Docker running systems have /etc/docker/daemon.json with

{
  "registry-mirrors": ["https://repository.ourdomain"],
  "insecure-registries": ["repository.ourdomain"]
}

With this I can do:

$ docker login -u myaccount repository.ourdomain
$ docker search repository.ourdomain/centos
$ docker pull centos:latest

The final piece of the puzzle is:

docker tag myimage repository.ourdomain/myimage:mytag
docker push repository.ourdomain/myimage:mytag
docker rmi <myimage ID>
docker pull myimage:mytag

That one still eludes me with some strange error. When I try the "push" I get textual output of a 404, which when properly formatted as HTML amounts to:

<!DOCTYPE html>
<html>
<head>
  <title>404 - Nexus Repository Manager</title>
  <meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"/>


  <!--[if lt IE 9]>
  <script>(new Image).src=\"https://repository.ourdomain/favicon.ico?3.6.0-02\"</script>
  <![endif]-->
  <link rel=\"icon\" type=\"image/png\" href=\"https://repository.ourdomain/favicon-32x32.png?3.6.0-02\" sizes=\"32x32\">
  <link rel=\"mask-icon\" href=\"https://repository.ourdomain/safari-pinned-tab.svg?3.6.0-02\" color=\"#5bbad5\">
  <link rel=\"icon\" type=\"image/png\" href=\"https://repository.ourdomain/favicon-16x16.png?3.6.0-02\" sizes=\"16x16\">
  <link rel=\"shortcut icon\" href=\"https://repository.ourdomain/favicon.ico?3.6.0-02\">
  <meta name=\"msapplication-TileImage\" content=\"https://repository.ourdomain/mstile-144x144.png?3.6.0-02\">
  <meta name=\"msapplication-TileColor\" content=\"#00a300\">

  <link rel=\"stylesheet\" type=\"text/css\" href=\"https://repository.ourdomain/static/css/nexus-content.css?3.6.0-02\"/>
</head>
<body>
<div class=\"nexus-header\">
  <a href=\"https://repository.ourdomain\">
    <div class=\"product-logo\">
      <img src=\"https://repository.ourdomain/static/images/nexus.png?3.6.0-02\"/>
    </div>
    <div class=\"product-id\">
      <div class=\"product-id__line-1\">
        <span class=\"product-name\">Nexus Repository Manager</span>
      </div>
      <div class=\"product-id__line-2\">
        <span class=\"product-spec\">OSS 3.6.0-02</span>
      </div>
    </div>
  </a>
</div>

<div class=\"nexus-body\">
  <div class=\"content-header\">
    <img src=\"https://repository.ourdomain/static/rapture/resources/icons/x32/exclamation.png?3.6.0-02\"/>
    <span class=\"title\">Error 404</span>
    <span class=\"description\">Not Found</span>
  </div>
  <div class=\"content-body\">
    <div class=\"content-section\">
      Not Found
    </div>
      </div>
</div>
</body>
</html>

While at the same time I see in the Apache ssl_access_log

10.52.94.150 - - [13/Nov/2017:16:18:02 +1100] "POST /v2/myimage/blobs/uploads/ HTTP/1.1" 404 1953
10.52.94.150 - - [13/Nov/2017:16:18:02 +1100] "POST /v2/myimage/blobs/uploads/ HTTP/1.1" 404 1953

And in ssl_request_log
[13/Nov/2017:16:18:02 +1100] 10.52.94.150 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "POST /v2/myimage/blobs/uploads/ HTTP/1.1" 1953
[13/Nov/2017:16:18:02 +1100] 10.52.94.150 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "POST /v2/myimage/blobs/uploads/ HTTP/1.1" 1953

At the same time this shows up in the Nexus request.log
10.52.94.150 - - [13/Nov/2017:16:18:02 +1100] "GET /repository/ncdocker/v2/ HTTP/1.1" 401 113 2 "docker/17.09.0-ce go/go1.8.3 git-commit/afdb6d4 kernel/3.10.0-693.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.09.0-ce \(linux\))"
10.52.94.150 - - [13/Nov/2017:16:18:02 +1100] "GET /repository/ncdocker/v2/token?scope=repository%3Amyimage%3Apush%2Cpull&service=https%3A%2F%2Frepository.ourdomain%2Ftoken HTTP/1.1" 200 60 2 "docker/17.09.0-ce go/go1.8.3 git-commit/afdb6d4 kernel/3.10.0-693.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.09.0-ce \(linux\))"
10.52.94.150 - anonymous [13/Nov/2017:16:18:02 +1100] "POST /repository/ncdocker/v2/myimage/blobs/uploads/ HTTP/1.1" 404 1953 3 "docker/17.09.0-ce go/go1.8.3 git-commit/afdb6d4 kernel/3.10.0-693.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.09.0-ce \(linux\))"
10.52.94.150 - anonymous [13/Nov/2017:16:18:02 +1100] "POST /repository/ncdocker/v2/myimage/blobs/uploads/ HTTP/1.1" 404 1953 1 "docker/17.09.0-ce go/go1.8.3 git-commit/afdb6d4 kernel/3.10.0-693.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.09.0-ce \(linux\))"
10.52.94.150 - anonymous [13/Nov/2017:16:18:02 +1100] "HEAD /repository/ncdocker/v2/myimage/blobs/sha256:c65c4cb3c7fa55459f81a0a113d1e2867a10251ca868fc3dc63c0491023317fb HTTP/1.1" 200 0 4 "docker/17.09.0-ce go/go1.8.3 git-commit/afdb6d4 kernel/3.10.0-693.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.09.0-ce \(linux\))"

What befuddles me is those paths. Everything seems to transit both the web proxy and into Nexus, but then Nexus complains about not finding the paths to upload - well, of course not, should it be the one to create them? It's a POST after all. Previously I had gotten 401s with basic authentication not going through, hence allowing anonymous for debugging.

What's the last missing link that I just don't seem to get?

Thanks
Mike

Sam Gleske

unread,
Nov 13, 2017, 1:29:52 PM11/13/17
to Michael Hoffmann, Nexus Users
Check out this issue https://github.com/SergK/nexus3-ssl/issues/3

It dual purposes nginx to act as a web proxy and as a docker frontend for pushing and pulling images.  Basically, you need to proxypass on the condition if the word "docker" is in the User Agent of the client.  This way if you're using the docker client the user is directly interacting with the docker backend.

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/4dbcaeed-a555-4665-bbed-f90ca7c81b00%40glists.sonatype.com.
For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

Michael Hoffmann

unread,
Nov 14, 2017, 12:12:09 AM11/14/17
to Nexus Users, mike_hof...@yahoo.com
From reading that threat it's sounding like a request for the ability to be added in Nexus? At any rate, now, being the web server non-admin I am, I just need to figure out how this is done in Apache. Switching to nginx is unfortunately not an option at this time.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.

Nikola Milutinovic

unread,
Nov 14, 2017, 4:04:23 AM11/14/17
to nexus...@glists.sonatype.com
I'd say that you are close :-)

What you need is a way to target your hosted repo in case of a PUSH and your group repo in case of GET. Docker cannot have different paths for thos two operations (unlike Maven). So, some trickery is required.

The suggested trick with detecting the user agent should work, but I see no reason to actually differentiate Docker client from cURL or any such agent. Why not just route differently, based on the operation?

You can use mod_rewrite for that:

RewriteEngine On

RewriteCond REQUEST_METHOD GET
RewriteRule /v(1|2)/(.*) http://localhost:55555/v$1/$2 [PL]

RewriteCond REQUEST_METHOD POST
RewriteRule /v(1|2)/(.*) http://localhost:55556/v$1/$2 [PL]

Haven't tested this, so use with caution and read the manual on mod_rewrite.

Nix.
--

Met vriendelijke groet / Kind regards,

Nikola Milutinović

Java Software Architect

cid:image001.png@01D1DB56.9D5D68B0

Address: Trifkovićev trg 6, 21000 Novi Sad, Serbia

Tel.: +31 20 6701 947 | +381 21 2155 500

Mobile: +381 64 2202824

Skype: nikola.milutinovic

Internet:www.levi9.com

 

Chamber of commerce Levi9 Holding: 34221951

Chamber of commerce Levi9 IT Services BV: 34224746

Michael Hoffmann

unread,
Nov 16, 2017, 4:13:27 PM11/16/17
to Nexus Users, n.milu...@levi9.com
Hi,

For the last 2 days I have been tearing my increasingly sparse hair out. Trying to get rewriting working in Apache. Should be straight-forward, but it simply is not doing it. As if it's not "hitting" the config. I see the module is loaded with httpd -M, but even with LogLevel debug and tests to force everything to go through the rewrite module, I see nothing in the log files. AllowOverride is set to All.

I'm about ready to just give in and go with nginx, finally, like all the cool kids are doing it.

Brian Fox

unread,
Nov 16, 2017, 4:28:59 PM11/16/17
to Michael Hoffmann, Nexus Users, n.milu...@levi9.com
For some things, nginx is just easier. Oldie but relevant: http://blog.sonatype.com/2008/10/nginx-is-centrals-new-friend/

To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.

To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/930c4574-b4dd-4215-8c48-f6659b5fc409%40glists.sonatype.com.

Nikola Milutinovic

unread,
Nov 17, 2017, 4:33:03 AM11/17/17
to nexus...@glists.sonatype.com
The reason why I pointed you to Apache mod_rewrite is because you are using Apache. You could switch to nginx, but Apache is also doable.

I have not tested the config fragment I sent you. And, yes, there was a typo in it: flags need to be comma separated. So, the first step is to somehow debug what it is doing. It's been a while since I played with Apache, so you'll have to dig a bit, start with some simpler rewrite, like:

RewriteRule /v1/(.*) http://localhost:55555/v1/$1 [P,L]

Or even

RewriteRule /v1/(.*) http://localhost:55555/v1/$1 [R=302,L]

The last one will give an external redirect, in the form of "HTTP 302" response. Then add more "tricks" into it. You should eventually get to a working thing. Also, do consult http://httpd.apache.org/docs/current/mod/mod_rewrite.html

To correct my previous fragment:

RewriteEngine On

RewriteCond REQUEST_METHOD GET
RewriteRule ^/v(1|2)/(.*) http://localhost:55555/v$1/$2 [P,L]

RewriteCond REQUEST_METHOD POST
RewriteRule ^/v(1|2)/(.*) http://localhost:55556/v$1/$2 [P,L]


Nix.

Michael Hoffmann

unread,
Nov 18, 2017, 2:46:39 AM11/18/17
to Nexus Users
I should have used nginx from the start :-P

Finally, at long last, everything (well just about) is working the way I want it to. What's still not playing nice is basic authentication, but I understand that is an ongoing issue with Nexus, which is why the option to turn it off and allow anonymous was introduced?

My nginx.conf contains this (some stuff removed)

...


http {
    ...

    upstream nexus {
        server localhost:8081;
    }

    upstream registry {
        server localhost:55555;
    }

    upstream privateregistry {
        server localhost:55556;
    }

    server {
        listen       80 default_server;
        server_name  _;
        return 301 https://malbec.centaur.id.au$request_uri;
    }

    map $request_method $registrytouse {
        default "privateregistry";
        (GET) "registry";
    }

    server {
        ...

        # disable any limits to avoid HTTP 413 for large file uploads
        client_max_body_size 0;

        # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
        chunked_transfer_encoding on;

        location / {
            if ($http_user_agent ~ docker ) {
                proxy_pass http://$registrytouse;
            }
            proxy_pass http://nexus;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto "https";
        }

       ...
    }
}





Michael Hoffmann

unread,
Nov 18, 2017, 2:53:38 AM11/18/17
to Nexus Users
Argh, I butterfingered the Post button too soon and there's no Edit Post function? Or I'm too tired to see it.

At any rate, with that config file, I can do things like

$ docker build -t mytestimage .
$ docker tag mytestimage malbec.centaur.id.au/library/mytestimage:latest
$ docker push malbec.centaur.id.au/library/mytestimage:latest

And on another machine do

$ docker pull mytestimage:latest

without having to specify the registry server or port, therefore making it transparent to the user whether are using our Nexus repo or Docker Hub, as they continue to be able to do

$ docker pull ubuntu:latest

or what have you. (As I think I'd said, we distribute an appropriate daemon.json through Puppet).

Thanks to all for the pointers and support!

Carlo Reggiani

unread,
May 24, 2018, 4:28:22 AM5/24/18
to Nexus Users
Great! It's working also for me for push/pull in my repo!

Only a problem: no way to search images in Docker Hub using the group repo configured (I'm using the same 55555 http port for group repo):

PS C:\Users> docker search nexq.xxxx.it/jboss
NAME                DESCRIPTION         STARS               OFFICIAL            AUTOMATED

Nexus access.log show a correct v1 protocol (GET /v1/search?q=jboss&n=25):

xxx.xxx.xxx.xxx - - [24/May/2018:10:24:40 +0200] "GET /v1/_ping HTTP/1.1" 200 0 "-" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.9.87-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \x5C(windows\x5C))" "-"
xxx.xxx.xxx.xxx - - [24/May/2018:10:24:40 +0200] "GET /v1/_ping HTTP/1.1" 200 0 "-" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.9.87-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \x5C(windows\x5C))" "-"
xxx.xxx.xxx.xxx - admin [24/May/2018:10:24:40 +0200] "GET /v1/search?q=jboss&n=25 HTTP/1.1" 200 221 "-" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.9.87-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \x5C(windows\x5C))" "-"

Any idea?

PS: using Nexus OSS 3.10

Carlo

Carlo Reggiani

unread,
May 24, 2018, 4:40:47 AM5/24/18
to Nexus Users
... also not able to pull from the nexus group repo:

PS C:\Users> docker pull nexq.xxxx.it/hello-world
Using default tag: latest
Error response from daemon: manifest for nexq.cadit.it/hello-world:latest not found


nginx log:

xxx.xxx.xxx.xxx - - [24/May/2018:10:38:23 +0200] "GET /v2/ HTTP/1.1" 401 113 "-" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.9.87-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \x5C(windows\x5C))" "-"
xxx.xxx.xxx.xxx - admin [24/May/2018:10:38:23 +0200] "GET /v2/hello-world/manifests/latest HTTP/1.1" 404 120 "-" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.9.87-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \x5C(windows\x5C))" "-"

In the Nexus's access.log:

xxx.xxx.xxx.xxx - - [24/May/2018:10:39:44 +0200] "GET /repository/cadit-docker/v2/ HTTP/1.0" 401 113 2 "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.9.87-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \(windows\))"
xxx.xxx.xxx.xxx - admin [24/May/2018:10:39:44 +0200] "GET /repository/cadit-docker/v2/hello-world/manifests/latest HTTP/1.0" 404 120 4 "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.9.87-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \(windows\))"

Using the Docker Hub registry directly it's work :(

PS C:\Users> docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
9bb5a5d4561a: Pull complete
Digest: sha256:f5233545e43561214ca4891fd1157e1c3c563316ed8e237750d59bde73361e77
Status: Downloaded newer image for hello-world:latest

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/137404b5-c269-4422-bde4-d886b1d8daf5%40glists.sonatype.com.

For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.


--

Rich Seddon

unread,
May 24, 2018, 3:18:32 PM5/24/18
to Nexus Users
You need to have v1 API support enabled on the repository in order for docker search to work (there isn't a v2 search API in docker yet).  

Also be sure that your docker hub proxy repository is using the docker hub index in it's settings.

Other than that, I would suggest checking the nexus.log, you may have connectivity issues from nexus to docker hub.
Reply all
Reply to author
Forward
0 new messages