Running rez inside a docker container

[Mina Ragaie]

Hello world!

I've been a high level rez user for a couple of years but I've never involved in the low level details or initial rez setup at a studio. I also very new to Docker. This is mainly a learning project for me.

My first goal is to run a DCC (no GUI) on a server (my desktop) within a containerized, rez-configured environment and serve that to a client (My laptop). Eventually I'll look into how to scale that setup (and possibly run a DCC in GUI mode) but for now just starting with 2 test machines (Ehm 1 actually for the first pass).

For my initial tests I've successfully ran python shell (as a stand-in for a DCC) within a containerized rez configured environment. (Just `docker run -it`  with `rez env` as it's CMD for now)

First issue I stumbled upon is versions:
- Rez design relies on immutable released versions that are available for `rez env` to resolve against.
- With Docker the tendency seems to be make your images as small as possible and create a container from the image for just the version you need. 

I'm not sure what would be a good approach here?

- Build larger images that have all previously release versions for all packages? (sounds slow and monolithic)
- Install rez packages to a docker volume? ( Don't know enough about volumes to know if this sound design)
- The premise is faulty? Use either rez or docker not both?
- Some different setup altogether?

Thanks!

Mina

[Allan Johns]

So there are a number of different things you can do.

1: Just mount your rez package repo dir(s) into your container. The upside to this is that images stay small, and you don't need different images to use different packages. The downside is that you're pulling package source from some filesystem outside of your container, which doesn't really make it very contained! But, whatever works for you. At Method we do this a lot of the time. https://docs.docker.com/storage/bind-mounts/

2: You can use rez-cp to copy all the packages that a context uses, into another package repo. Then, when constructing the image (via dockerfile), you would COPY this dedicated package repo into your container. Now, you can perform a rez-env in the container, and point it at this repo. The image now contains only the package you're using.

As of version 2.60.0, you can do this:

    ]$ rez-env pkgA pkgB -o - | rez-context --print-resolve --su - | xargs -i rez-cp --variant-uri {} --dest-path ./mytestrepo

Ie:

* resolve an env, and write it to stdout

* read a context from stdin, and print the URIs of each variant

* use rez-cp to copy each variant into package repo ./mytestrepo

In your dockerfile then, you may want to run the following commands:

    mkdir /packages  # make local rez pkg repo

    rez-env pkgA pkgB -o - | rez-context --print-resolve --su - | xargs -i rez-cp --variant-uri {} --dest-path /packages

    rez-env --nl --paths /packages -o runtime.rxt

Now, to run your command in docker, you'd run "rez-env -i runtime.rxt -- somecommand".

All that of course depends on your packages being relocatable. If you have compiled packages that do any rpathed linking for eg, that may not be true.

3: rez-build stuff directly into your container. Note that you'd have to rez-build all the dependencies in bottom-to-top order. Rez-build doesn't fetch dependencies or anything like that. Ultimately I'd like to have a rez-install tool that does that - it would perform much the same function as pip for eg - but that's a larger topic and there's a lot of work to be done there.

4: Another approach could be to do something similar to (2), but copy the packages to the docker host instead of the image, and to bind mount that local host path. That would mean you don't need different images for each resolve, but you would also avoid mounting to shared storage somewhere. However you'd have to ensure that this package repo is available on every host your container might run on.

In short, rez and docker aren't mutually exclusive at all. What you want to do specifically depends on a number of factors and hopefully the suggestions above get you on your way.

Hth

A

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/83334df3-88c3-41e6-abfa-95404db80b0a%40googlegroups.com.

[Christian]

hi Mina,

I think it depends on what exactly you want to achieve with your rez + containers setup

• if you have a central rez package repo and want to put a rez package's payload (i.e. the binaries of Maya) into a container then you can simply use something like docker run --rm -ti maya:2020.0.1 /opt/maya/bin/maya as command in the package.py. Using Docker as a "sandbox" to run cmdline tools for interactive use is a pretty common thing (without rez you would usually create a shell alias or use a Makefile to invoke the "docker run -ti ...". Here's a great example).
  
  
• if you want to simulate a larger rez installation in a container then everything that gets changed in the container's filesystem while it's running (i.e. new rez packages that you add) should go to a volume (as it's not persistent and will also use up RAM). -> i.e. create a volume for the package repo.
  
  
• if you want to rez-build from a container to a central package repo (i.e. on a fileserver) you could either mount the path to it as a volume or connect from inside the container via smb/nfs. Tricky part here can be (posix) permissions - usually making sure that the container user is set to a uid/gid that match a real directory user helps (i.e. in a Dockerfile: USER 65534:65534 ).
  
  
• if you want to build a rez package in a container that does not have access to your package repo it's possible to copy data off a container with docker cp <container>:<sourcepath> <destpath>. Could also be used in a Makefile and/or build command for a rez package. Recent versions of Docker also have a nice new feature to copy build results to the host system: docker build -o out <destpath> So you can do something like this:
  
  
  Dockerfile:
  FROM debian:10 as builder
  ...
  gcc -o /app.exe app.c

  FROM scratch
  COPY --from-builder /app.exe /
  
  shell:
  $ docker build -o output ./Dockerfile
  $ ls output/
  app.exe
  
• more a theoretical thought: if you would want to run everything in a container (rez runtime for resolves, payload of every rez package version as a container - not sure if that's a real usecase) then you would probably still want put your package repo on a shared volume (nfs or so, not a container volume). And you would need to find a way to start other containers from within that rez container. Which might be a usecase for dind (Docker in Docker) or a Kubernetes operator. Again, just a thought, not advice.
  
• ... (combinations of the above options and many other options depending on the exact usecase) ...

> Eventually I'll look into how to scale that setup (and possibly run a DCC in GUI mode)

Running basic GUI apps is relatively simple (by doing X11 Forwarding to the host -> env var DISPLAY=<ip of the host>:0.0 ), it's more tricky for hardware support (GPU, etc). There are however a lot of tricks to make things work by running privileged containers and mapping certain system files (like device files) as volumes into containers. But it feels hacky and sometimes only works properly if the container is based on the same Linux distro as the host, which breaks some of Docker's advantages. (Just as an example, last week I've spent some time trying to containerize bcc-tools, which need the Kernel headers of the host. I ended up installing kernel-headers on all hosts so that I can map them as read-only volume to the containers).

Unfortunately there doesn't seem to be much traction behind supporting complex GUI apps in Docker/oci containers. It seems to be more regarded as a server technology.

My current opinion is: Looking at Fedora Silverblue, Red Hat's test field for the Linux Desktop of the future, the vision clearly seems to be that the Linux root filesystem is read-only with few exceptions like the home dirs or /var (similar to MacOS Catalina) and everything that the user usually installs on top of the base system runs in a container. But there are two container technologies in use: Docker/oci containers for combinations of command line tools ("developer toolboxes") and Flatpak apps for GUI apps. In contrast to Docker/oci containers, there seems to be a lot of traction behind making Flatpak "containers" work for complex GUI apps (like the freedesktop sdk runtime). So I think a realistic (future) scenario for DCCs could be: Use Docker/oci containers to build software. For Desktop use, package the build results into Flatpak apps. For server use (renderfarm, cloud rendering) run either Flatpak or Docker/oci containers (Docker/oci containers are probably better suited as long as you don't need GPU rendering as they work with less dependencies and are more flexible to orchestrate). And maybe orchestrate all that (Docker builds, Docker runs, Flatpak runs) using rez. :-)

Best,

Chris

[Mina Ragaie]

Thanks for the detailed reply.

I am leaning towards option (2). Let rez figure out the needed environment and use docker to build an image only as big as it needs to be. Relocatable packages make sense with containers and I intend to use them exclusively. (As for rpath linking I am new to C/C++ and I'll just have to cross that bridge when I get to it).

Option (3) is what I was exploring before I posted and it became clear that without a `pip` like solution it's not very manageable.

Mina

To unsubscribe from this group and stop receiving emails from it, send an email to rez-c...@googlegroups.com.

[Mina Ragaie]

Thanks Christian!

I haven't ruled out building to a volume as I would like to be able to replicate both the build and runtime resolve. Being new to docker I'm still trying to figure it out though.

But here's my theory:
Unless I missed something, I see no reason not to attempt the 2nd option Allan suggested with a volume dedicated for built packages. The volume would only be needed in the build stage of multi stage Dockerfile.

1- Build my rez packages from within a container to a volume. All packages build to the same volume.
2- Pass the packages requested by the user to a `docker build` command.
3- In a multi stage Dockerfile:

        - stage 1: mount the packages volume and use `rez-cp` to copy needed packages the container.

        - stage 2: run `rez-env` and launch the DCC.

Thanks for the info regarding Flatpak that sounds very interesting :) I'll have to put a pin in it till I get to GUI portion.
For orchestration I imagined I'll be using Kubernetes and/or Istio (Disclaimer: I know nothing about both) not rez but I would be overstretching if I attempt any of that before I figure out the containerization first. Taking it one step at a time.

Mina

[Christian]

But here's my theory:

should work perfectly (via a multi stage Dockerfile)

For orchestration I imagined I'll be using Kubernetes and/or Istio (Disclaimer: I know nothing about both) 

I think Kubernetes is a bit off-topic from your main question. Kubernetes is more for running complex web applications that consist of many server components (microservices) that each run in their own containers. You can also run jobs, which almost feels like a render manager.

But I would highly encourage anyone looking into containers to play with it. It extends Docker a lot and can come handy even for small stuff even when only running on a local workstation (and not a multi node cluster).

On Linux I would recommend k3s for a 3 minutes installation. Or with Docker Desktop on Mac/Win it can get enabled with one checkbox in the Docker Desktop settings.

Istio is something different - it runs on top of Kubernetes and solves some problems that you may have when you run a large microservices environment. It basically creates a central management interface to an environment that is otherwise very decentralised (by injecting a proxy server into every container). I wouldn't get started with it before having an overall understanding of Kubernetes.

My recommendation for getting started with Kubernetes would be:

- install k3s (or enable Kubernetes in Docker Desktop)

- learn the Kubernetes Basics (the official tutorial is a good starting point)

- learn Helm (the Kubernetes package manager, which is both handy for installing things into Kubernetes and also to define your own container deployments - similar to docker-compose)

- (other things if needed/interesting, like monitoring with prometheus or service mesh with istio)

But as said, this is pretty off-topic from your main question :-)

[Mina Ragaie]

But as said, this is pretty off-topic from your main question :-)

Yeah, I purposely kept the original question to a limited scope :)

A microservice architecture is the ultimate goal here just not on the horizon for the first few phases. I'd imagine each rez package would become a microservice in the end. I wonder if the setup suggested above would be suited for that end goal?

In all cases, I plan to give it a try before I throwing Kubernetes and a complex microservice setup into the mix. This is a learning project after all and it helps to limit the scope :)

Mina

[Mina Zakie]

I have a working solution. C&C much appreciated!

I tried putting it into words but code proved to be a far more concise and less ambiguous way of describing the solution. 

Questions/notes on the solution below:
1 - Correct me if I'm wrong but, with the use of a `packages` volume shared across all containers, I see no need for generating a `runtime.rxt` as I originally intended.
2 - The shared `packages` volume will grow in size as more versions are released. How much of a concern is that? is there a way in rez to safely deprecate older versions?
3 - The shell commands needed to run the solution are rather finicky. I'm hoping that is something Kubernetes can help simplify.
4 - One obvious way to improve this solution is having 2 different "base" images for release/runtime. The runtime image would exclude cmake, curl, g++, git and tar. I'll keep it simple for now.
5 -  I have so far avoided bringing users into the equation for simplicity. Saving that for a later phase.

Here are the files I am using:
/some/local/path/

.

├── base

│   ├── base.bashrc

│   └── Dockerfile

├── rez

│   ├── Dockerfile

│   └── rezconfig.py

└── studio

    ├── release.Dockerfile

    └── runtime.Dockerfile

/some/local/path/base/Dockerfile

FROM alpine

RUN apk update && apk add bash cmake curl g++ git python2 tar

COPY base.bashrc /root/.bashrc

CMD ["bash"]

/some/local/path/base/base.bashrc (Just niceties for interactive use)

alias ll="ls -l"

export PS1="[\[\033[32m\]\u@\h\[\033[00m\]] \[\033[90m\]\t\[\033[00m\] \[\033[34m\]\w\[\033[00m\] \[\033[33m\]$(git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ (\1)/')\[\033[00m\] \[\033[36m\]$ \[\033[00m\]"

/some/local/path/rez/Dockerfile

FROM base AS appbuilder

ARG version=2.60.0

COPY rezconfig.py /studio/etc/rezconfig.py

ENV REZ_CONFIG_FILE /studio/etc/rezconfig.py

RUN mkdir -p /usr/src/rez \

    && curl -SL https://github.com/nerdvegas/rez/archive/${version}.tar.gz \

    | tar -xzC /usr/src/rez --strip 1 \

    && python /usr/src/rez/install.py -v /opt/rez \

    && mkdir -p /studio/packages/int \

    && mkdir -p /studio/packages/ext \

    && mkdir -p /studio/packages/pypi \

    && /opt/rez/bin/rez/rez-bind -r platform \

    && /opt/rez/bin/rez/rez-bind -r arch \

    && /opt/rez/bin/rez/rez-bind -r os \

    && /opt/rez/bin/rez/rez-bind -r python \

    && /opt/rez/bin/rez/rez-bind -r gcc \

    && /opt/rez/bin/rez/rez-bind -r cmake \

    && rm -rf /usr/src/rez

ENV REZ_CONFIG_FILE /studio/etc/rezconfig.py

ENV PATH /opt/rez/bin/rez:$PATH

RUN echo "source /opt/rez/completion/complete.sh" >> .bashrc

CMD ["bash"]

/some/local/path/rez/rezconfig.py (More or less. Modified for brevity)

release_packages_path = "/studio/packages"

packages_path = [

    "${HOME}/packages",

    "${HOME}/packages/int",

    "${HOME}/packages/ext",

    "${HOME}/packages/pypi",

    "{}/packages".format(release_packages_path),

    "{}/packages/int".format(release_packages_path),

    "{}/packages/ext".format(release_packages_path),

    "{}/packages/pypi".format(release_packages_path),

]

package_preprocess_mode = "after"

def package_preprocess_function(this, data):

    """Build to int/ext/pypi based on data.get("group")."""

    # Code omitted for brevity.

Now, Here's is the tricky bit:
Volumes can't be mounted during a `docker build`.
There is a `RUN --mount` syntax but it can only mount read only path from within the build context.
`CMD rez-release` (& not `RUN rez-release`) must be used, if you want to be able to build to a volume.

As a result, I am using 2 docker files one for `release` and one for `runtime`.

/some/local/path/studio/release.Dockerfile (This is used to release to a volume **see shell commands below**)

FROM rez AS appbuilder

WORKDIR /tmp

COPY . .

CMD ["/bin/bash", "-c", "rez-release --skip-repo-errors"]

/some/local/path/studio/runtime.Dockerfile (This builds an image based on packages requested by the user at **runtime**)

FROM rez as context_resolver

CMD rez-env ${PKGS} -- ${COMMAND}

Here's how to use the files above:
1 - Builds the base images.

cd /some/local/path/
sudo docker build -t base ./base
sudo docker build -t rez ./rez
sudo docker build -f ./studio/runtime.Dockerfile -t studio ./studio

2 - Create the `packages` volume that will be mounted to *all* docker containers moving forward.

sudo docker volume create packages

3 - Release to the volume by building an image and running a "release" container.

sudo docker build -f ./studio/release.Dockerfile -t pkg_name /path/to/dev/packages/int/pkg_name

sudo docker run --rm -v packages:/studio/packages -it pkg_name

4 - On receiving a request from the user (REST API?) with a specific packages and a command to run we spin a container (Kubernetes?).

sudo docker run --rm -e PKGS="python pkg_name" -e COMMAND="python" -v packages:/studio/packages -it studio

I think I'll give Kubernetes a try next and see if it can streamline the solution.

If you think the concept is flawed, raise your hand! This is a learning project!

Thanks!

Mina