Very slow images publishing to GitLab

[Evgeny]

Hello, 

I have not found any discussions related to this topic, so maybe it is some local problem on my laptop. 

The problem is that container images publishing to GitLab container registry using jib gradle plugin is extremely slow and cannot be completed at all. 

To minimize and try to troubleshot the problem I created very simple few lines hello world java project which has 0 dependencies. Jar file size is just few Kb. 

I created Docker file for this project and tried to build and push it to gitlab registry using "docker build / docker push". Push took 1-2 minutes and images are successfully published. 

A

fter that I deleted pushed container images on gitlab and tried to push images using jib Gradle plugin. Push got stuck on 87.7% and is not moving forward. Eventually build is failed with timeout error. 

I tried to enable debug logging for gradle and jib plugin and have not noticed anything suspicious: pull base image, preparing application layer, start pushing... And that is it.  

Does anybody who use gitlab and publishes images there have similar problems? 

Thank you 

[Evgeny]

I forgot to mentioned that I am using 1.5.1 version of the plugin. 

[Chanseok Oh]

Hi Evgeny,

I suspect this is a network/connectivity issue.

Are you pushing to a prviate non-HTTPS registry (and therefore setting allowInsecureRegistries)? Is it same when using 1.4.0 (old version)?

-Chanseok

--
You received this message because you are subscribed to the Google Groups "Jib users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jib-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jib-users/4c19ce2a-2f82-453a-9a86-71deb366a0cb%40googlegroups.com.

[Evgeny]

Hi, 

Thank you for all the responses and suggestions. I tried to downgrade to 1.4.0, result was the same: long wait and timeout error. 

When I returned back 1.5.1, one of my images (my app has 3 of them), was published, other 2 failed with the same timeout error. 

I tried to enable allowInsecureRegistries and publishing became more stable - the rest 2 artifacts were published correctly and it took reasonable time. I am not sure how exactly it helped because before enabling this option I have not seen any communication errors related to HTTPS in debug log - just silent wait and timeout error in the end.

Anyway, I will continue playing with it. 

To unsubscribe from this group and stop receiving emails from it, send an email to jib-...@googlegroups.com.

[Evgeny]

Hi, 

Thank you for all the responses and suggestions. I tried to downgrade to 1.4.0, result was the same: long wait and timeout error. 

When I returned back 1.5.1, one of my images (my app has 3 of them), was published, other 2 failed with the same timeout error. 

I tried to enable allowInsecureRegistries and publishing became more stable - the rest 2 artifacts were published correctly and it took reasonable time. I am not sure how exactly it helped because before enabling this option I have not seen any communication errors related to HTTPS in debug log - just silent wait and timeout error in the end.

Anyway, I will continue playing with it. 

On Saturday, August 31, 2019 at 1:00:06 PM UTC-7, Chanseok Oh wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to jib-...@googlegroups.com.

[Chanseok Oh]

Too early to speculate about allowInsecureRegistries, but in the end, you shouldn't be enabling it. I was trying to see if I can get a quick signal from it.

The surefire way is to look at a detailed network trace. Carefully follow these instructions. Note, one user reported that to reliably make the underlying HTTP library to generate network logs, `--no-daemon` is required for Gradle and also needed to downgrade Gradle to 4.10.3. Also add "--debug" to make Jib log more information. If you succeed, you'll see (in a serialized manner) logs like this.

I still suspect this is a network timeout issue, but we'll see. A network trace will capture enough information. Strip out any sensitive information (if any) from the log before attaching it as a file. It'd be helpful if you do this for both 1.5.1 and 1.4.0.

-Chanseok

[Michel Raizor]

Evgeny,

we have an issue about that on github (not exactly, i think, but similar).

See that: https://github.com/GoogleContainerTools/jib/issues/1917

Like Chanseok, I think it's a network problem, but it's not clear. Please, attach your logs with debug on gradle.

I'm use gitlab, but Harbor to registry. I don't have any problem (maven).Nowadays, I'm use 1.5.0 in my workspace, but HTTPS. In past, on 1.4 version, I'm used to use HTTP with JFROG Artifactory and didn't have any problem.

[Chanseok Oh]

Evgeny,

Have you captured and looked into a network trace? We (Jib dev) would like to know what really happened to understand why Docker took only a minute or two. Thanks!

-Chanseok

[Evgeny]

Hi everybody, 

Really sorry for a silence. I was out and did not have much chances to play with it. I finally got a chance to try it again today. I disabled allowInsecureRegistries back to false and try to publish containers - everything worked fine without delays. I am still using 1.5.1 version of Jib gradle plugin. Not sure what caused this slowness before. I tend to think that something was not right on gitlab side. 

Anyway. I will keep an eye on it and publish all the requested debug info if problem starts reproducing again. 

[Chanseok Oh]

Hi Evgeny,

Thanks for the update. Glad to hear it's fine now. You may want to do a final check though, because subsequent Jib runs can be fast for many reasons.

- Use a totally different base image (just for the sake of testing) not on GitLab. Jib completely skips downloading base images if they already exist on the target repository. Also, cross-repo BLOb mount may skip pushing base image layers within a registry.

- Delete Jib's base image cache to force Jib to download base images. On Linux, it is $HOME/.cache/google-cloud-tools-java/jib

- Add a new dependency to your project, so that Jib builds a different application layer. Jib skips pushing layers that already exist in the target repository. Completely emptying the GitLab target repository may help, but some registries still keep deleted layers for some time.

-Chanseok

[Chanseok Oh]

Hi Evgeny,

This may actually be a serious issue in Jib >= 1.5.0.

https://github.com/GoogleContainerTools/jib/issues/1970#issuecomment-530647211

However, 1.4.0 isn't affected. You said 1.4.0 also failed, but since you mentioned some builds succeeded after trying 1.4.0, so I suspect that 1.4.0 may have successfully uploaded at least some layers. And once most of the layers are uploaded, subsequent Jib builds should be fast (whether 1.4.0 or 1.5.0+) because it will skip uploading layers that already exist in the registry.

In any case, it is certain that 1.5.0+ becomes really slow or even hang on slow networks.

[Evgeny]

Hey, 

You were right. Problem is kind of reproduced if publishing of brand new images is performed. Previously published images are published fast even if they are removed from gitlab and published again. 

When I created a brand new GitLab project and tried to publish images again with default Jib plugin settings, publishing speed became slow again. But what I see now is not as bad as it used to be before. There is no timeout error and images are published eventually. It just takes long time. 

Not sure if logs are still useful. I attached them anyway just in case. 

As I can see, 1.6.0 is going to be releases soon. I can try this version once it is released. 

[Chanseok Oh]

Jib 1.6.0 is released, which fixes the slow network throughput: https://groups.google.com/forum/#!topic/jib-users/dAeMKsNEvrQ