Is Dockerfile and other source for container images available?

[Jeremy Lewi]

Hi Google Team,

I'd like to run TensorFlow Federated on GKE. Unfortunately it looks like this requires Python 3.9 (federated/issues/3109). It looks like Python 3.7 is shipping in the Deep Learning Container Images.

Is the Dockerfile and other source files published anywhere? I'd just as soon modify what the DL containers are doing rather than rebuild it or reverse engineer it.

Thanks

J

[James Webber]

I have the same question. It would be great to know the process behind building these images so I can build new VMs that need other dependencies.

I can install my own requirements manually and save an image, but it would be really nice to have an automated process. I would think that these images are built with some kind of automated tooling, but it doesn't seem to be public.

[Jonas]

I actually have the exact same question.

[Yuxuan Chen]

Hello,

The source code for Deep Learning Containers is not available. Later versions of Python are coming in 2023. In the meantime, feel free to build a container using DL containers as a derivative!

[James Webber]

Can you provide any hint about how the images are built? I can learn the tooling necessary to build my own, if there was any guidance. In particular I want to make sure my images are compatible with GCE and GCP services, that they start JupyterLab on startup, and that they configure a GPU when present. But it'd be great to customize what gets installed in an automated way.

[Yuxuan Chen]

We use a combination of Pip and Conda to install JupyterLab, Python, etc. on top of an Ubuntu base Docker image. Configuring the GPU is a little bit outside of the scope of the container image itself, as it also requires some Docker run flags, but CUDA toolkit is installed via the Nvidia base CUDA containers for our GPU images. To get JupyterLab to run on startup, we define a Docker entrypoint with a command to run the jupyter service.

Hope this helps!

[James Webber]

But this produces a Docker image, not a VM image? I was able to produce a comparable customized Docker image (using `docker inspect` to get the commands for the DL container) but running a docker container on GCE is a little cumbersome compared to a VM. For one thing I can't configure attached storage to be automatically available, the way that the DL images do.

Are the VM images created via the Docker images using some other tool? I assume it is not done manually!

[Yuxuan Chen]

Yes, this produces a Docker image. I thought you were asking about Docker images, since Deep Learning Container images (produced with Dockerfiles) are all Docker container images.

VM images are https://cloud.google.com/compute/docs/images#custom_images and we create these with Packer, not Docker. The rest of the stuff about using Conda/Pip to install dependencies and setting up startup services for Jupyter and such still holds true.

Hope this helps!