Jib & Quay: 500s, 400s, and 401s

374 views
Skip to first unread message

Kurtis Mullins

unread,
Oct 1, 2019, 9:33:04 AM10/1/19
to quay-sig
Good morning,

@hbjastad opened up an issue on Github regarding some obvious issues between Jib and Quay. Since we do not currently have a public Github or Jira, I am opening up conversation here to include others in the conversation and to avoid overloading Jib developers with more noise than necessary until we can determine there is an issue on Jib's end.

The essential issues:
- On Quay v3.0.5 with the Jib "helloworld" example, Quay returns HTTP 500 responses. One theory is that this is related to the parallelization used by Jib. It appears that issuing the flag `-Djib.serialize=true` is a work-around for the problem but runs the operations in serial.
- Similarly, sporadic HTTP 400 errors are encountered as documented in the related Github Issue https://github.com/GoogleContainerTools/jib/issues/2013.
- After an upgrade to Quay v3.1.0, it appears that each push is encountering a single HTTP 401 response. (Not verified on my end)

References / Links:
- Github Issue concerning Quay 400s: https://github.com/GoogleContainerTools/jib/issues/2013

I am going to reach out to @hbjastad to request bringing the conversation over here until we can verify the issue is specifically related to Jib. If anyone has suggestions or more information in the mean-time, feel free to chime in!

Havard Bjastad

unread,
Oct 2, 2019, 2:29:25 AM10/2/19
to quay-sig
Hi Kurtis, thanks for picking up the threads here. I think you summarized everything well, just a small correction - the 401 that we're getting with Quay v3.1.0 doesn't happen for every run. Yesterday, I experienced it in 1 or 2 out of 10 runs, when I tried again now I got it in 4 out of 10.

The 401 happens when downloading the base image, here is the log:

```
$ mvn -Djava.util.logging.config.file=logging.properties -Djib.console=plain  compile jib:build
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building helloworld 1a
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ helloworld ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ helloworld ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- jib-maven-plugin:1.6.1:build (default-cli) @ helloworld ---
[INFO]
[INFO] Containerizing application to QUAY-REGISTRY/fellestjenester/helloworld...
[WARNING] Base image 'QUAY-REGISTRY/r2d2/distroless_java' does not use a specific image digest - build may not be reproducible
[INFO] Retrieving registry credentials for QUAY-REGISTRY...
[INFO] Getting base image QUAY-REGISTRY/r2d2/distroless_java...
[INFO] Building dependencies layer...
[INFO] Building resources layer...
[INFO] Building classes layer...
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST  --------------
GET https://QUAY-REGISTRY/v2/r2d2/distroless_java/manifests/latest
Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json
Accept-Encoding: gzip
User-Agent: jib 1.6.1 jib-maven-plugin Google-HTTP-Java-Client/1.27.0 (gzip)
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -H 'Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 1.6.1 jib-maven-plugin Google-HTTP-Java-Client/1.27.0 (gzip)' -- 'https://QUAY-REGISTRY/v2/r2d2/distroless_java/manifests/latest'
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST  --------------
GET https://QUAY-REGISTRY/v2/
Accept:
Accept-Encoding: gzip
User-Agent: jib 1.6.1 jib-maven-plugin Google-HTTP-Java-Client/1.27.0 (gzip)
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 1.6.1 jib-maven-plugin Google-HTTP-Java-Client/1.27.0 (gzip)' -- 'https://QUAY-REGISTRY/v2/'
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
HTTP/1.1 401 UNAUTHORIZED
Server: nginx/1.12.1
Date: Wed, 02 Oct 2019 06:16:54 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 4
Connection: close
Docker-Distribution-API-Version: registry/2.0
WWW-Authenticate: Bearer realm="https://QUAY-REGISTRY/v2/auth",service="QUAY-REGISTRY"
Oct 02, 2019 8:16:54 AM com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: Total: 4 bytes
Oct 02, 2019 8:16:54 AM com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: true
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST  --------------
GET https://QUAY-REGISTRY/v2/auth?service=QUAY-REGISTRY&scope=repository:fellestjenester/helloworld:pull,push&scope=repository:r2d2/distroless_java:pull
Accept: */*
Accept-Encoding: gzip
Authorization: <Not Logged>
User-Agent: jib 1.6.1 jib-maven-plugin Google-HTTP-Java-Client/1.27.0 (gzip)
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -H 'Accept: */*' -H 'Accept-Encoding: gzip' -H 'Authorization: <Not Logged>' -H 'User-Agent: jib 1.6.1 jib-maven-plugin Google-HTTP-Java-Client/1.27.0 (gzip)' -- 'https://QUAY-REGISTRY/v2/auth?service=QUAY-REGISTRY&scope=repository:fellestjenester/helloworld:pull,push&scope=repository:r2d2/distroless_java:pull'
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
HTTP/1.1 200 OK
Server: nginx/1.12.1
Date: Wed, 02 Oct 2019 06:16:54 GMT
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Content-Length: 1160
Connection: close
Docker-Content-Digest: sha256:4d70e98aa99a53afe4acc661593fee3b0f8c92d5cfd4129db4cc77b85a3012d5
X-Frame-Options: DENY
Strict-Transport-Security: max-age=63072000; preload
Oct 02, 2019 8:16:54 AM com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: Total: 1,160 bytes
Oct 02, 2019 8:16:54 AM com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: {
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1013,
      "digest": "sha256:2ee039e7a421877e71af02cbe1ac11228f07df61fbba8d5d2b8876e9cd023036"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 654467,
         "digest": "sha256:e8d8785a314f385d3675a017f4e2df1707c528c06e7a7989663fdab4900bd8ff"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 7330260,
         "digest": "sha256:e005d777a298a3529b1c8cf890883359e050cc966089ce84fea4d17b111907db"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 643663,
         "digest": "sha256:3e010093287c245d72a774033b4cddd6451a820bfbb1948c97798e1838858dd2"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 42538785,
         "digest": "sha256:408ad02cbf46b38f36ac42c09d911cf359a5be4dc3fee6e5956b1e4f96e98bdf"
      }
   ]
}
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST  --------------
GET https://QUAY-REGISTRY/v2/r2d2/distroless_java/blobs/sha256:2ee039e7a421877e71af02cbe1ac11228f07df61fbba8d5d2b8876e9cd023036
Accept:
Accept-Encoding: gzip
User-Agent: jib 1.6.1 jib-maven-plugin Google-HTTP-Java-Client/1.27.0 (gzip)
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 1.6.1 jib-maven-plugin Google-HTTP-Java-Client/1.27.0 (gzip)' -- 'https://QUAY-REGISTRY/v2/r2d2/distroless_java/blobs/sha256:2ee039e7a421877e71af02cbe1ac11228f07df61fbba8d5d2b8876e9cd023036'
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
HTTP/1.1 302 FOUND
Server: nginx/1.12.1
Date: Wed, 02 Oct 2019 06:16:54 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 2813
Connection: close
Location: https://QUAY-REGISTRY/_storage_proxy/eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImFhZjg4ZWYzMzVjYjhmYWE0MDY1ZmQ5ODNkMmY4MDA5YjI0YzAyMmQxZmZiMTlkOWJjZDVlNWM5ODcwYTIwNWMifQ.eyJhY2Nlc3MiOlt7Imhvc3QiOiJzMy5wb2xpdGlldC5ubyIsInNjaGVtZSI6Imh0dHBzIiwidHlwZSI6InN0b3JhZ2Vwcm94eSIsInVyaSI6InF1YXktcHJvZC9kYXRhc3RvcmFnZS9yZWdpc3RyeS9zaGEyNTYvMmUvMmVlMDM5ZTdhNDIxODc3ZTcxYWYwMmNiZTFhYzExMjI4ZjA3ZGY2MWZiYmE4ZDVkMmI4ODc2ZTljZDAyMzAzNj9TaWduYXR1cmU9MndZT0QwYSUyQkRBa0slMkY5bFFKVU91SXBZdGk4byUzRCZFeHBpcmVzPTE1Njk5OTc2MTQmQVdTQWNjZXNzS2V5SWQ9ODk4M1U4Rk42VVc3Ukg5NVNHM00ifV0sImNvbnRleHQiOnt9LCJhdWQiOiJodWIucG9saXRpZXQubm8iLCJleHAiOjE1Njk5OTcwNDQsImlzcyI6InF1YXkiLCJpYXQiOjE1Njk5OTcwMTQsIm5iZiI6MTU2OTk5NzAxNCwic3ViIjoic3RvcmFnZXByb3h5In0.lOiZVHicYf6id2FEi2MJxAYmuIR69gkImwccTFP03hyYC_jVXRUh3Jc40uhJhyNa4YMLGmy9rlg2MuvUlN_mRqdoBEn4wf8xEZcj931RZnimiw048tTrSPGupBYxTn4N3WCzKnczHMizj-JlqJtcP7SwaHfu3wdUmwFdHeYgDI6JjV5qPzmm9TQRLL4KbT2D64prMsTMpX5tWcWeivYa_Pxzmv8fSLlvJAmHi7cBLMy5OJAYYKJiSAAZCCnxwWM4E3jNMahJOjbAG6txvcU5xreKXjRzoiCKcUSbuCo361yIBwM764jXKq75HHpeC16E0wzGe6jJMRPBXySmaZYAtQ/https/S3-STORAGE/quay-prod/datastorage/registry/sha256/2e/2ee039e7a421877e71af02cbe1ac11228f07df61fbba8d5d2b8876e9cd023036?Signature=2wYOD0a%2BDAkK%2F9lQJUOuIpYti8o%3D&Expires=1569997614&AWSAccessKeyId=8983U8FN6UW7RH95SG3M
Accept-Ranges: bytes
Docker-Content-Digest: sha256:2ee039e7a421877e71af02cbe1ac11228f07df61fbba8d5d2b8876e9cd023036
Cache-Control: max-age=31536000
X-Frame-Options: DENY
Strict-Transport-Security: max-age=63072000; preload
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST  --------------
GET https://QUAY-REGISTRY/_storage_proxy/eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImFhZjg4ZWYzMzVjYjhmYWE0MDY1ZmQ5ODNkMmY4MDA5YjI0YzAyMmQxZmZiMTlkOWJjZDVlNWM5ODcwYTIwNWMifQ.eyJhY2Nlc3MiOlt7Imhvc3QiOiJzMy5wb2xpdGlldC5ubyIsInNjaGVtZSI6Imh0dHBzIiwidHlwZSI6InN0b3JhZ2Vwcm94eSIsInVyaSI6InF1YXktcHJvZC9kYXRhc3RvcmFnZS9yZWdpc3RyeS9zaGEyNTYvMmUvMmVlMDM5ZTdhNDIxODc3ZTcxYWYwMmNiZTFhYzExMjI4ZjA3ZGY2MWZiYmE4ZDVkMmI4ODc2ZTljZDAyMzAzNj9TaWduYXR1cmU9MndZT0QwYSUyQkRBa0slMkY5bFFKVU91SXBZdGk4byUzRCZFeHBpcmVzPTE1Njk5OTc2MTQmQVdTQWNjZXNzS2V5SWQ9ODk4M1U4Rk42VVc3Ukg5NVNHM00ifV0sImNvbnRleHQiOnt9LCJhdWQiOiJodWIucG9saXRpZXQubm8iLCJleHAiOjE1Njk5OTcwNDQsImlzcyI6InF1YXkiLCJpYXQiOjE1Njk5OTcwMTQsIm5iZiI6MTU2OTk5NzAxNCwic3ViIjoic3RvcmFnZXByb3h5In0.lOiZVHicYf6id2FEi2MJxAYmuIR69gkImwccTFP03hyYC_jVXRUh3Jc40uhJhyNa4YMLGmy9rlg2MuvUlN_mRqdoBEn4wf8xEZcj931RZnimiw048tTrSPGupBYxTn4N3WCzKnczHMizj-JlqJtcP7SwaHfu3wdUmwFdHeYgDI6JjV5qPzmm9TQRLL4KbT2D64prMsTMpX5tWcWeivYa_Pxzmv8fSLlvJAmHi7cBLMy5OJAYYKJiSAAZCCnxwWM4E3jNMahJOjbAG6txvcU5xreKXjRzoiCKcUSbuCo361yIBwM764jXKq75HHpeC16E0wzGe6jJMRPBXySmaZYAtQ/https/S3-STORAGE/quay-prod/datastorage/registry/sha256/2e/2ee039e7a421877e71af02cbe1ac11228f07df61fbba8d5d2b8876e9cd023036?Signature=2wYOD0a%2BDAkK/9lQJUOuIpYti8o%3D&Expires=1569997614&AWSAccessKeyId=8983U8FN6UW7RH95SG3M
Accept:
Accept-Encoding: gzip
User-Agent: jib 1.6.1 jib-maven-plugin Google-HTTP-Java-Client/1.27.0 (gzip)
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 1.6.1 jib-maven-plugin Google-HTTP-Java-Client/1.27.0 (gzip)' -- 'https://QUAY-REGISTRY/_storage_proxy/eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImFhZjg4ZWYzMzVjYjhmYWE0MDY1ZmQ5ODNkMmY4MDA5YjI0YzAyMmQxZmZiMTlkOWJjZDVlNWM5ODcwYTIwNWMifQ.eyJhY2Nlc3MiOlt7Imhvc3QiOiJzMy5wb2xpdGlldC5ubyIsInNjaGVtZSI6Imh0dHBzIiwidHlwZSI6InN0b3JhZ2Vwcm94eSIsInVyaSI6InF1YXktcHJvZC9kYXRhc3RvcmFnZS9yZWdpc3RyeS9zaGEyNTYvMmUvMmVlMDM5ZTdhNDIxODc3ZTcxYWYwMmNiZTFhYzExMjI4ZjA3ZGY2MWZiYmE4ZDVkMmI4ODc2ZTljZDAyMzAzNj9TaWduYXR1cmU9MndZT0QwYSUyQkRBa0slMkY5bFFKVU91SXBZdGk4byUzRCZFeHBpcmVzPTE1Njk5OTc2MTQmQVdTQWNjZXNzS2V5SWQ9ODk4M1U4Rk42VVc3Ukg5NVNHM00ifV0sImNvbnRleHQiOnt9LCJhdWQiOiJodWIucG9saXRpZXQubm8iLCJleHAiOjE1Njk5OTcwNDQsImlzcyI6InF1YXkiLCJpYXQiOjE1Njk5OTcwMTQsIm5iZiI6MTU2OTk5NzAxNCwic3ViIjoic3RvcmFnZXByb3h5In0.lOiZVHicYf6id2FEi2MJxAYmuIR69gkImwccTFP03hyYC_jVXRUh3Jc40uhJhyNa4YMLGmy9rlg2MuvUlN_mRqdoBEn4wf8xEZcj931RZnimiw048tTrSPGupBYxTn4N3WCzKnczHMizj-JlqJtcP7SwaHfu3wdUmwFdHeYgDI6JjV5qPzmm9TQRLL4KbT2D64prMsTMpX5tWcWeivYa_Pxzmv8fSLlvJAmHi7cBLMy5OJAYYKJiSAAZCCnxwWM4E3jNMahJOjbAG6txvcU5xreKXjRzoiCKcUSbuCo361yIBwM764jXKq75HHpeC16E0wzGe6jJMRPBXySmaZYAtQ/https/S3-STORAGE/quay-prod/datastorage/registry/sha256/2e/2ee039e7a421877e71af02cbe1ac11228f07df61fbba8d5d2b8876e9cd023036?Signature=2wYOD0a%2BDAkK/9lQJUOuIpYti8o%3D&Expires=1569997614&AWSAccessKeyId=8983U8FN6UW7RH95SG3M'
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
HTTP/1.1 401 Unauthorized
Server: nginx/1.12.1
Date: Wed, 02 Oct 2019 06:16:54 GMT
Content-Type: text/html
Content-Length: 195
Connection: keep-alive
Oct 02, 2019 8:16:54 AM com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: Total: 195 bytes
Oct 02, 2019 8:16:54 AM com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: <html>
<head><title>401 Authorization Required</title></head>
<body bgcolor="white">
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.12.1</center>
</body>
</html>
Oct 02, 2019 8:16:54 AM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
HTTP/1.1 200 OK
Server: nginx/1.12.1
Date: Wed, 02 Oct 2019 06:16:54 GMT
Content-Type: application/json
Content-Length: 1206
Connection: close
Cache-Control: no-cache, no-store, must-revalidate
X-Frame-Options: DENY
Strict-Transport-Security: max-age=63072000; preload
Oct 02, 2019 8:16:54 AM com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: Total: 1,206 bytes
Oct 02, 2019 8:16:54 AM com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: {"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImFhZjg4ZWYzMzVjYjhmYWE0MDY1ZmQ5ODNkMmY4MDA5YjI0YzAyMmQxZmZiMTlkOWJjZDVlNWM5ODcwYTIwNWMifQ.eyJhY2Nlc3MiOlt7InR5cGUiOiJyZXBvc2l0b3J5IiwibmFtZSI6ImZlbGxlc3RqZW5lc3Rlci9oZWxsb3dvcmxkIiwiYWN0aW9ucyI6WyJwdXNoIiwicHVsbCJdfSx7InR5cGUiOiJyZXBvc2l0b3J5IiwibmFtZSI6InIyZDIvZGlzdHJvbGVzc19qYXZhIiwiYWN0aW9ucyI6WyJwdWxsIl19XSwiY29udGV4dCI6eyJlbnRpdHlfa2luZCI6InVzZXIiLCJraW5kIjoidXNlciIsInZlcnNpb24iOjIsImNvbS5hcG9zdGlsbGUucm9vdCI6IiRkaXNhYmxlZCIsInVzZXIiOiJoYmowMTMiLCJlbnRpdHlfcmVmZXJlbmNlIjoiYTZjYWY4NjktNWI2Yi00YjY3LWE0YjktMTA1NTUwNTMyZDMxIiwiY29tLmFwb3N0aWxsZS5yb290cyI6eyJyMmQyL2Rpc3Ryb2xlc3NfamF2YSI6IiRkaXNhYmxlZCIsImZlbGxlc3RqZW5lc3Rlci9oZWxsb3dvcmxkIjoiJGRpc2FibGVkIn19LCJhdWQiOiJodWIucG9saXRpZXQubm8iLCJleHAiOjE1NzAwMDA2MTQsImlzcyI6InF1YXkiLCJpYXQiOjE1Njk5OTcwMTQsIm5iZiI6MTU2OTk5NzAxNCwic3ViIjoiaGJqMDEzIn0.sLLyay994DFwGpXrqxNDwDl_A_4dWjr7KrZbsaK_6pIJOVCu6O7RPI9MeJ3Ss8SqtZOolgmA8SBAo-yJu6nzB6WWj7JM0MkftRtVDw02zQWkjNiFnzjry06sQOT5KXyEQyowm0NiSSt5pKPZPtx4iA77QJEieerVDYN2Shu0hqByKEFSR4OT3cFVLMVFcCu9BKzyqcynIU3Fn44a9GDCtNKTqDg7Do488dXVG6kP083v8UBjBuKXMYqspNL_IBkOxrN2EKvKXpjn5jcpthB04rFNGlJW6LbpASogF947vnxqn4jXGL6TdS6NsXIgEcFoYSoMJKRpiCq34irWNJYQWQ"}
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.190 s
[INFO] Finished at: 2019-10-02T08:16:54+02:00
[INFO] Final Memory: 21M/261M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.6.1:build (default-cli) on project helloworld: com.google.cloud.tools.jib.api.RegistryUnauthorizedException: Unauthorized for QUAY-REGISTRY/r2d2/distroless_java: 401 Unauthorized
[ERROR] <html>
[ERROR] <head><title>401 Authorization Required</title></head>
[ERROR] <body bgcolor="white">
[ERROR] <center><h1>401 Authorization Required</h1></center>
[ERROR] <hr><center>nginx/1.12.1</center>
[ERROR] </body>
[ERROR] </html>
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
```

Chanseok Oh

unread,
Oct 2, 2019, 12:22:39 PM10/2/19
to quay-sig
Jib dev here. Things are happening in parallel, so I'll parse the log and explain what has transpired. This will show that Quay unexpectedly and randomly returns 401.

First, there are (basically) two threads running in parallel: A) pulling the base image; and B) authenticating with Quay for later push.

Below, I have paired all the requests and responses correctly. No dangling/missing/mismatches request-response pairs. 

# Thread A: pulling the base image

  1. Requests the manifest.
      GET .../v2/r2d2/distroless_java/manifests/latest

  2. Gets a response. Manifest returned. Obviously, the image is public and no auth is required. The manifest says the container config JSON is sha256:2ee039e7....

   "config": {
      ...
      "digest": "sha256:2ee039e7a421877e71af02cbe1ac11228f07df61fbba8d5d2b8876e9cd023036"
   },

  3. As the next step, requests to get the container config JSON (sha256:2ee039e7...).
      GET .../v2/r2d2/distroless_java/blobs/sha256:2ee039e7a421877e71af02cbe1ac11228f07df61fbba8d5d2b8876e9cd023036

  4. Gets the response 302 to redirect (returning Location: header). Obviously, the base image requires no auth.
      Location: .../_storage_proxy/eyJhbGciOiJSUzI1NiIsInR5cCI6I...

  5. Requests to get the container config JSON at the location given by the server.
      GET .../_storage_proxy/eyJhbGciOiJSUzI1NiIsInR5cCI6I...

  6. The server unexpectedly returns 401 for the base image that requires no auth.


# Thread B: preparing for push application layers

  1. Starts the auth flow for pushing to the target repository.

  2. Gets the auth challenge response, as expected. This is the only response that returns WWW-Authenticate: header.
      WWW-Authenticate: Bearer realm="https://QUAY-REGISTRY/v2/auth",service="QUAY-REGISTRY"

  3. As the server directed, goes to the auth URL and authenticates using username/password (provided with the Authorization: header). This is the first and last time in the log Jib ever sends the username/password info.
      Authorization: <Not Logged>
      GET .../v2/auth?...
       &scope=repository:fellestjenester/helloworld:pull,push
       &scope=repository:r2d2/distroless_java:pull

  4. Auth succeeded and gets the token (JSON) response.
      {"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVC..."}

Hope this helps.

-Chanseok Oh

Joey Schorr

unread,
Oct 2, 2019, 1:32:36 PM10/2/19
to Chanseok Oh, quay-sig
So all the 401s are occurring on the storage proxy endpoint?

If so, its likely something in the URL isn't being encoded properly.

--
You received this message because you are subscribed to the Google Groups "quay-sig" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quay-sig+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/c7269e3b-57d7-4baf-b061-c05d104a4e26%40googlegroups.com.

Chanseok Oh

unread,
Oct 2, 2019, 2:03:11 PM10/2/19
to quay-sig
I think that is a likely cause. I see that there is a difference in the query string values (blob fetch URL) returned from "Location:" and the normalized URL that Jib tries.

?Signature=2wYOD0a%2BDAkK%2F9lQJUOuIpYti8o%3D&Expires=

?Signature=2wYOD0a%2BDAkK/9lQJUOuIpYti8o%3D&Expires=

"%2F" in this case is normalized to "/", as "/" is a valid character according to the standard. (For example, http://example.com/document/?uri=http://user:pass...@example.com/?foo=bar is perfectly valid.) Conforming servers should have no problem handling valid characters including "/", "?", ":", "@", etc. And perhaps Quay treats the "%2F" literally as "%" + "2" + F" instead of decoding it to "/"?

If the "Signature" parameter value is dynamically generated, this also explains the sporadic 401 failures. I suspect it happens only when the value includes characters like "/", "?", etc.

To unsubscribe from this group and stop receiving emails from it, send an email to quay...@googlegroups.com.

Joey Schorr

unread,
Oct 2, 2019, 2:05:10 PM10/2/19
to Chanseok Oh, quay-sig
On Wed, Oct 2, 2019 at 2:03 PM 'Chanseok Oh' via quay-sig <quay...@googlegroups.com> wrote:
I think that is a likely cause. I see that there is a difference in the query string values (blob fetch URL) returned from "Location:" and the normalized URL that Jib tries.

?Signature=2wYOD0a%2BDAkK%2F9lQJUOuIpYti8o%3D&Expires=

?Signature=2wYOD0a%2BDAkK/9lQJUOuIpYti8o%3D&Expires=

"%2F" in this case is normalized to "/", as "/" is a valid character according to the standard. (For example, http://example.com/document/?uri=http://user:pass...@example.com/?foo=bar is perfectly valid.) Conforming servers should have no problem handling valid characters including "/", "?", ":", "@", etc. And perhaps Quay treats the "%2F" literally as "%" + "2" + F" instead of decoding it to "/"?

If the "Signature" parameter value is dynamically generated, this also explains the sporadic 401 failures. I suspect it happens only when the value includes characters like "/", "?", etc.

Exactly. We could look into changing the signature encoding so that it doesn't generate characters like that, but Jib should also respect them as well.
 
To unsubscribe from this group and stop receiving emails from it, send an email to quay-sig+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/66891419-05e7-45c7-ac04-37a323ba6bcf%40googlegroups.com.

Chanseok Oh

unread,
Oct 2, 2019, 2:16:07 PM10/2/19
to quay-sig
Exactly. We could look into changing the signature encoding so that it doesn't generate characters like that, but Jib should also respect them as well.

I don't really disagree, but the Apache HttpClient library folks don't really agree on this. We had a somewhat similar situation with Microsoft before and had stumbled upon this thread: https://github.com/apache/httpcomponents-client/commit/8c04c6ae5e5ba1432e40684428338ce68431766b#r33969408

After reading the thread, I came to agree with the Apache folks. Jib is using the mainstream Apache library, so I'm afraid it is unlikely that we will be able to purse the Apache folks to change their behavior, which is perfectly sound and doing its own best effort to handle URL parsing business. We will try to see what we can do in Jib, but I urge you to make Quay conform to work with the standard.

Joey Schorr

unread,
Oct 2, 2019, 2:17:00 PM10/2/19
to Chanseok Oh, quay-sig
I'll file a ticket and look into it.

To unsubscribe from this group and stop receiving emails from it, send an email to quay-sig+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/ec9ca51e-d2d7-4a07-8834-871cfe37719e%40googlegroups.com.

Charles Moulliard

unread,
Oct 28, 2019, 3:49:45 PM10/28/19
to quay-sig
Is the ticket opened ? When will it be fixed as we are stuck also to use redhat registry server - https://github.com/GoogleContainerTools/jib/issues/2106#issuecomment-547089392 ?

Joey Schorr

unread,
Oct 29, 2019, 11:22:27 AM10/29/19
to Charles Moulliard, quay-sig
The ticket is opened, but no solution has been found and, to be honest, won't likely be solved for a very long time. The reality is: the only good solution either requires Jib to fix their bad implementation (which they say they will not do) OR a new custom compilation of nginx on our end to add additional modules (which is a hard sell at this time), so I don't see any resolution coming for a while.

To unsubscribe from this group and stop receiving emails from it, send an email to quay-sig+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/7abc06a9-b9ba-468b-942c-d3a7328a972f%40googlegroups.com.

Chanseok Oh

unread,
Oct 29, 2019, 11:34:01 AM10/29/19
to quay-sig
Joey,

Have you identified what exactly should be fixed in nginx? If you could explain to me in a bit more details what's wrong with nginx and how this should be fixed, I can file a bug against nginx. And maybe I could even contribute a fix for them.

From my understanding, the bug is basically that the server works differently for "https://registry.redhat.io?foo=bar" and "https://registry.redhat.io?foo=%62%61%72" even though they are equivalent, which means the server doesn't conform to the HTTP standard. But I don't have any details than that.

And it is not like we (Jib devs) are averse to implement a temporary workaround in Jib. For that to happen, we would probably need to ditch the Java Apache HttpClient altogether, which would be a massive undertaking that is not likely to happen. So we are in a deadlock. But I would like to make the world better by implementing a proper fix for the greater good. If nginx is non-conformant, I would like to fix it in the right way.

-Chanseok

Joey Schorr

unread,
Oct 29, 2019, 11:38:03 AM10/29/19
to Chanseok Oh, quay-sig
On Tue, Oct 29, 2019 at 11:34 AM 'Chanseok Oh' via quay-sig <quay...@googlegroups.com> wrote:
Joey,

Have you identified what exactly should be fixed in nginx? If you could explain to me in a bit more details what's wrong with nginx and how this should be fixed, I can file a bug against nginx. And maybe I could even contribute a fix for them.

It is not a bug in nginx, unfortunately. The bug lies in Jib (or, more accurately, Apache lib, as you know). In order for us the workaround the bug, we need additional nginx modules to do more hacky things, which means getting a custom-compiled nginx version for our use, which had a bunch of internal process around it, unfortunately.
 

From my understanding, the bug is basically that the server works differently for "https://registry.redhat.io?foo=bar" and "https://registry.redhat.io?foo=%62%61%72" even though they are equivalent, which means the server doesn't conform to the HTTP standard. But I don't have any details than that.

Correct, but that is Jib sending us the bad data; nginx is just using it accordingly, and thus when the URL gets to the downstream storage engine, its encoding doesn't match what was originally sent and signed.
 

And it is not like we (Jib devs) are averse to implement a temporary workaround in Jib. For that to happen, we would probably need to ditch the Java Apache HttpClient altogether, which would be a massive undertaking that is not likely to happen.

Yeah, exactly, which is a highly unfortunate solution.
 
So we are in a deadlock. But I would like to make the world better by implementing a proper fix for the greater good. If nginx is non-conformant, I would like to fix it in the right way.

If only... I'm still looking into alternative solutions internally right now, but as there are a million other things going on, I can't promise any movement in the near term future :/
 
To unsubscribe from this group and stop receiving emails from it, send an email to quay-sig+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/0c3b8f0f-7813-4be8-b5b5-37c573b1a4f1%40googlegroups.com.

Chanseok Oh

unread,
Oct 29, 2019, 11:50:05 AM10/29/19
to quay-sig
Correct, but that is Jib sending us the bad data; nginx is just using it accordingly, and thus when the URL gets to the downstream storage engine, its encoding doesn't match what was originally sent and signed.
Please understand that there is not much Jib can do about this. What Jib does is to get the value of the `Location:` header (the URL the Quay wants Jib to redirect to), and Jib just tells the Apache HttpClient "okay, I got the URL from Quay. I need to call the endpoint. Apache, go to http://registry.redhat.io/foo=%62%61%72a and fetch data to me" and then the Apache client does the normal URL parsing on the URL I gave in the sound and correct way. The Apache client does the usual complex URL parsing and normalization and in the end sends "http://registry.redhat.io/foo=bar" instead of "http://registry.redhat.io/foo=%62%61%72". In this sense, none of Jib or the Apache client is doing anything wrong, and I honestly think it is unfair to blame Jib for "bad implementation" or "sending bad data" (which I retrained from bringing up). I hope you understand this situation and we find some workaround or fix that can resolve this deadlock.

Joey Schorr

unread,
Oct 29, 2019, 11:50:50 AM10/29/19
to Chanseok Oh, quay-sig
I should add: This is only related to the use of Jib and storage proxying.

The other issues (500s and 400s) are being looked into, although the 400s is likely due to Jib sending us an invalid request; for those, an example of the failing request is needed for us to debug.

Joey Schorr

unread,
Oct 29, 2019, 11:55:00 AM10/29/19
to Chanseok Oh, quay-sig
On Tue, Oct 29, 2019 at 11:50 AM 'Chanseok Oh' via quay-sig <quay...@googlegroups.com> wrote:
Correct, but that is Jib sending us the bad data; nginx is just using it accordingly, and thus when the URL gets to the downstream storage engine, its encoding doesn't match what was originally sent and signed.
Please understand that there is not much Jib can do about this. What Jib does is to get the value of the `Location:` header (the URL the Quay wants Jib to redirect to), and Jib just tells the Apache HttpClient "okay, I got the URL from Quay. I need to call the endpoint. Apache, go to http://registry.redhat.io/foo=%62%61%72a and fetch data to me" and then the Apache client does the normal URL parsing on the URL I gave in the sound and correct way. The Apache client does the usual complex URL parsing and normalization and in the end sends "http://registry.redhat.io/foo=bar" instead of "http://registry.redhat.io/foo=%62%61%72". In this sense, none of Jib or the Apache client is doing anything wrong, and I honestly think it is unfair to blame Jib for "bad implementation" or "sending bad data" (which I retrained from bringing up). I hope you understand this situation and we find some workaround or fix that can resolve this deadlock.

I understand that, but at the same time: we've explicitly chosen to encode the URL in a specific way to ensure we can pass it along, verbatim, to the downstream storage engine. Since the Apache HttpClient is changing the URL's encoding in certain areas, that operation is therefore failing. I'm not blaming Apache HttpClient, mind you: I'm simply saying that it is the root cause of the problem, and while I don't expect it to be changed, this bug only does affect Jib (and, presumably, any other client that may use the Apache lib).

I fully admit how we're performing the proxying is a hack, but then, there wasn't really another performant option. That's the crux of issue: I'm looking for a better solution that maintains performance (via use of nginx) while also respecting the fact that URLs cannot be trusted to maintain their encoded values without modification. To do so means re-encoding the downstream URL in a format that Apache will ignore (such as URL-safe base64), but to do so also requires additional nginx modules to do the decoding of said values in nginx, which, as stated above, means getting a custom-compiled nginx for our use, which will take some time.
 
To unsubscribe from this group and stop receiving emails from it, send an email to quay-sig+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/8aa4b69b-2792-4149-9b25-97924660abca%40googlegroups.com.

Chanseok Oh

unread,
Oct 29, 2019, 5:54:42 PM10/29/19
to quay-sig
Just in case if this matters:

In https://github.com/GoogleContainerTools/jib/issues/2106#issuecomment-547593721, the query string value doesn't seem like a signature thing but some other pairs of values for the key `_auth_` for passing an auth token. Thought this may go into a different execution path and I just wanted to make sure both issues are fixed when you design a permanent solution.

Reply all
Reply to author
Forward
0 new messages