Pushing large docker image to private repo

[Patrick]

Hello,

I'm having trouble pushing a rather large image to the private docker.illumina.com repository. Its size is about 40GB (containing a database-file). I always get the Gateway-Timeout-Error:
"Error pushing to registry: Server error: 504 trying to push <myrepo/img-name> blob - sha256:69a50398cdbae2bbb9fb0229bfc237b5b4c7657[...]"
(I removed the repository and shortened the hash here.)

Does anybody have any suggestions how to tackle this?
I've been trying docker client- and deamon/server- versions: 1.0.1, 1.6.2, 1.7.0
Client and Server API-Version: 1.12, 1.18, 1.19
the result was always the same. It seems to be related to the proxy of the repository (see https://github.com/docker/docker/issues/1266). Since buffering and uploading the image takes some time it seems to me that the server just rejects it due to timeout.

Do you have any suggestion on how to solve this? I've been thinking to gzip the db-file (about 18Gb) and push it. But then it will have to be unzipped every time the image is spinned up on Basespace, which is not really desirable.

Best wishes,
Patrick

[Patrick]

Sorry wrong Link. This one is the one i actuallay wanted to refer to: https://github.com/docker/docker-registry/issues/900

[Mayank Tyagi]

Hey Patrick,

Docker in general runs into issues with large images and/or layers, this is an especially large image as well.  We recommend keeping images within 10 GB if possible to avoid issues with the docker client.

The database that you are including with your docker image should be made available in another location if possible.  One option is to host it in an Amazon S3 bucket and link to it from within your docker image as needed.  This functionality is currently supported but will be restricted in the future when we introduce a mechanism to host dependency data within BaseSpace.  For now, we would recommend hosting this data outside of the docker image in order to reduce size so that the docker client can handle it.

Please let us know if you run into any further issues or if we can help you at all with this solution.

Sincerely,

The BaseSpace Platform Team

[Patrick]

Dear Mayank,

thank you very much for your quick and helpful reply.
I will discuss those options with my superior. I will let you know if I need any help or information.

Best wishes,
Patrick