Push with Google jib

[Christopher Cooper]

My team wants to push to quay using Google jib from Gradle. It seems that oci/2.2 docker is not supported still. Where is that on the road map? We would like to only need 1 tool to push our images.

[2020-07-20T20:38:08.433Z] > com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push image manifest for quay.io/xxx/xxx:xxx but failed because: Registry may not support pushing OCI Manifest or Docker Image Manifest Version 2, Schema 2 | If this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new

Please advise,
Thanks
Chris

[Joey Schorr]

Quay.io supports Docker 2 Schema 2 since January, so we'll need logs to determine why Jib is unable to use it.

OCI support is currently being tested behind a feature flag, but is not generally available.

--
You received this message because you are subscribed to the Google Groups "quay-sig" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quay-sig+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/5653bba8-0a8c-44ad-895c-6cf2e1e28ef9o%40googlegroups.com.

[Christopher Cooper]

I'll work on getting some more log information as this is all the output we got from the run in jenkins. Is there something we need to enable in our repository to get this to work? I've read conflicting reports about whether or not jib can push to quay, but it seems mostly like it can't. FWIW, the exact same jib build can push to our artifactory instance, so I'm inclined to think the issue is on the side of quay.

[Bill Dettelback]

Hi Chris- we've had one other person using jib last year who had some issues- so any details would be helpful so we can try to identify where things are failing.

Bill

To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/CAJc%2BtgCAkH3xRa4FNfKUED92cEGxT_7b_O%3D-oJK3opzRPtnQPA%40mail.gmail.com.

-- 
Bill Dettelback
Engineering Manager - Red Hat Quay, OpenShift Image Registry
M: 732-991-0052
https://www.redhat.com/en/technologies/cloud-computing/quay
Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.

[Christopher Cooper]

Strangely enough, one of our devs re-enabled it today and it started working. Is it possible there are some nodes behind a load balancer that don't support the schema and we were hitting them before?

You received this message because you are subscribed to a topic in the Google Groups "quay-sig" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/quay-sig/7Pp80zhdgzM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to quay-sig+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/33f95dd6-b0d6-ba03-425b-55dddfb35880%40redhat.com.

[Joey Schorr]

No. If the problem reappears, please make sure to get logs.

To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/CAJc%2BtgAup6YbbtLyOqveRN3-TMnegiXmRJtfJEdK15jrV0dTDg%40mail.gmail.com.

[Chanseok Oh]

Chris,

For getting more log, try adding the -X flag to mvn. At least it should output the entire stack trace as in here. For example, it should show an error code and a detailed error message from Quay.io.

Failed to execute goal com.google.cloud.tools:jib-maven-plugin:2.2.0:build (default-cli) on project workflow-java-sdk-process-cloud: Tried to push image manifest for quay.io/your_image but failed because: Registry may not support pushing OCI Manifest or Docker Image Manifest Version 2, Schema 2 | If this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new: 400 BAD REQUEST

[ERROR] {"errors":[{"code":"MANIFEST_INVALID","detail":{"message":"Attempt to create an invalid manifest. Please report this issue."},"message":"manifest invalid"}]}

Anyways, if the error is intermittent, I think it's possible that you may be hitting this issue. (This is just a speculation. Your issue may be a different one.) Just in case, make sure you use the latest Jib (2.4.0).

According to the Jib code, what's certain is that Quay either returned 400 Bad Request or 415 UNSUPPORTED MEDIA TYPE. At the same time, the registry error was either MANIFEST_INVALID or TAG_INVALID.

[Christopher Cooper]

I am getting the above error message as well.

[2020-07-24T09:10:54.846Z] 05:10:54.659 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Caused by: com.google.api.client.http.HttpResponseException: 400 BAD REQUEST
[2020-07-24T09:10:54.846Z] 05:10:54.659 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] {"errors":[{"code":"MANIFEST_INVALID","detail":{"message":"Attempt to create an invalid manifest. Please report this issue."},"message":"manifest invalid"}]}

We never hit this with our artifactory pushes, so my assumption is there must be some issue on the quay side. Is the only workaround to serialize the pushes? Can't you search your logs for this error to determine cause?

To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/b8d2fb76-cb82-443e-9509-df987ff4d5f3n%40googlegroups.com.

[Chanseok Oh]

The concurrency issue on the Quay side may happen only when you are pushing multiple tags (i.e., you have at least one tag specified in Jib's `<to><tags>`), if what the Quay dev speculated is correct. If so, I think pushing one tag at a time is a much better workaround than using -Djib.serialize=true outright because it can be terribly slow (unless you only serialize the second re-run when you hit the error). (BTW, I'm the Jib dev who had done the analysis on the GitHub issue.)

If you are intermittently hitting even if you are pushing only one tag (i.e., you don't have `<to><tags>` at all), then I guess it would be another interesting clue for the Quay devs. Even if so, I am leaning heavily toward an issue on the Quay side too, from past experience and history with Jib on Quay, especially when an error is intermittent. I also do hope Quay fixes the issue soon.

[Christopher Cooper]

So are you suggesting multiple "to" blocks with a single tag for each? We are using multiple tags.

To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/85831177-cdfb-41e7-ba32-593278ae6ecfn%40googlegroups.com.

[Chanseok Oh]

Jib doesn't support multiple <to> blocks. The best you can do is to somehow parameterize <to><image> (without <to><tags>, of course) and run "jib:build" multiple times. I expect running Jib multiple times is not that bad, since subsequent "jib:build" runs will be really fast; all the blobs (layers and container config) should have been cached locally and uploaded remotely, and Jib just needs to push a new tag (pushing a manifest, strictly speaking).

For parameterizing <to><image>, you can use any standard and feasible ways you can use in Maven. Some examples:

- Use system properties or Maven session properties. That is, <to><image>quay.io/.../repo:${my.parameterized.tag}</image></to>, and run mvn -Dmy.parameterized.tag=awesome-tag jib:build followed by another mvn -Dmy.parameterized.tag=another-tag jib:build

- Use Maven profiles.

- Use -Djib.to.image. For example, mvn -Djib.to.image=quay.io/.../repo:awesome-tag jib:build followed by another jib:build run. 

[Joey Schorr]

Without logs including the repository namespace, name and tag, we can't know what the error being raised is. My guess is that it is hitting the same concurrency issue reported before, but I cannot be sure.

To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/cf5ea874-6f93-4711-8e4f-7bcaa5e4145en%40googlegroups.com.

[Christopher Cooper]

I'm using gradle not maven. It definitely supports multiple to blocks since we're already using those to publish to both quay and artifactory.

I don't really feel comfortable posting the information you are asking for in this thread, Joey.

I don't mind sending them to you directly from my personal email or my IBM account, but it's starting to blur the line between free and paid support of this community.

To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/CAP2w_6eNbcoNcYjGFKs%2BnwSqOan-FDz0uAk1MvXdQmkJugiwLg%40mail.gmail.com.

[Joey Schorr]

Hi Christopher,

Feel free to email our support ticket system then, if you like: sup...@quay.io

[Christopher Cooper]

Thanks I'll send support an email.

I will say that using multiple "to" blocks in gradle produces the same unreliable behavior. It still fails occasionally.

[Chanseok Oh]

I'd appreciate getting updates here from anyone once the issue is resolved (or any periodic updates really). Other Jib users are affected as well, and the Jib team wants to provide them with up-to-date information.

[Joey Schorr]

It is due to the concurrent push of the tags pointing to the same manifest; I don't know why Jib insists on re-pushing the same manifest multiple times, but by doing so at the exact same time, it is hitting a corner case in our duplication check code. I'm working on a fix which I hope we can have deployed by next week (and available for RHQ 3.4), but in the interim, it can be worked around by simply turning off concurrent push.

On Wed, Jul 29, 2020 at 11:39 AM 'Chanseok Oh' via quay-sig <quay...@googlegroups.com> wrote:

I'd appreciate getting updates here from anyone once the issue is resolved (or any periodic updates really). Other Jib users are affected as well, and the Jib team wants to provide them with up-to-date information.

--
You received this message because you are subscribed to the Google Groups "quay-sig" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quay-sig+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/a72247fb-2920-4c4d-8158-9636b611e3b8o%40googlegroups.com.

[Chanseok Oh]

> I don't know why Jib insists on re-pushing the same manifest multiple times

AFAIK, pushing a manifest is the only way to create a tag. So, pushing the manifest multiple times to create multiple tags. If you know any other way, let us know. We can update our implementation right away, as I think it would be a better optimization than pushing a manifest multiple times.

[Joey Schorr]

Right... my bad.... I forgot that in V2 protocol you have to repush the manifest :/

To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/91d16807-b3d8-4299-9c43-21b6a95e64e9n%40googlegroups.com.

[Joey Schorr]

Hi everyone,

The fix has been deployed to Quay.io. Please let us know if you encounter any further issues.

Thanks,

Joey Schorr

[Christopher Cooper]

Thanks for that!

Funny I was actually just about to email to ask about it.

We'll test out ASAP.

You received this message because you are subscribed to a topic in the Google Groups "quay-sig" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/quay-sig/7Pp80zhdgzM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to quay-sig+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/CAP2w_6cj967p2YypsB6tEj54BwtfXh7t%2BoPm-BhnK%2BSokioSVQ%40mail.gmail.com.

[Christopher Cooper]

I did give few testes and the performance is getting better, we can push 4 images  sometime now, but still have very often fail build with following error

 Execution failed for task ':controller:operator:jib'.

 > com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: 500 Internal Server Error

   <html>

     <head>

       <title>Internal Server Error</title>

     </head>

     <body>

       <h1><p>Internal Server Error</p></h1>

 

     </body>

   </html>

[Joey Schorr]

I know... I responded on the git issue. We're looking into it now.

[Joey Schorr]

Okay, I think I see the primary issue. We'll issue a fix for testing today and likely deploy early next week.

[Christopher Cooper]

Hi,

Is there any update on the 500 error?

Thanks,

Chris

[Bill Dettelback]

Hi Chris-

I believe this latest fix was deployed- can you please check again if you see the issue?

Bill

To view this discussion on the web visit https://groups.google.com/d/msgid/quay-sig/CAJc%2BtgAW%3DFwUQ-a%2BFCiDPLz7oqaMQ8C_JCOM5y4FPGw1wibw8w%40mail.gmail.com.

[Christopher Cooper]

Looks great, thanks quay team!