Persisting Docker images in NFS

[Jay Lau]

Hi,

One question related to docker image: As I do not want to create a local repo, so I'm pulling all images in NFS, but seems it is not working.....

[root@db03b04 sysconfig]# pwd
/etc/sysconfig
[root@db03b04 sysconfig]# cat docker
# /etc/sysconfig/docker
#
# Other arguments to pass to the docker daemon process
# These will be parsed by the sysv initscript and appended
# to the arguments list passed to docker -d
 
other_args="--selinux-enabled  -b=br100 -g=/nfs"  <<<<<<<<<<<<<< Persist all images in NFS
#other_args="--selinux-enabled -g=/root/gyliu -b=br100"

Based on my test, it seems that we cannot use this way, the test result is as following:
1) Pull image from one node and image was persisted in NFS, but the images cannot be listed if do not restart docker on another docker server which can also access the NFS
2) Create a docker container on one node and the docker container status is active, but it was OFF on other docker servers.

Any comments? Is this a docker limitation?

Thanks,

Jay

[Frederick F. Kautz IV]

Out of curiosity, does the vfs driver work on NFS?

docker --storage-driver="vfs"

--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jérôme Petazzoni]

Yes, it should work just fine on NFS.

--

@jpetazzo

Latest blog post: http://jpetazzo.github.io/2014/06/23/docker-ssh-considered-evil/

[Jay Lau]

Thanks jerome and Frederick for the quick response!

Can you please show some steps for how to configure NFS as you are experts for docker? I'm now searching something related to vfs to see if it can help.

Thanks,

Jay

--

Thanks,

Jay

[Jay Lau]

I did have a try with following configuration but seems still does not work.

root@mesos008:~# cat /etc/default/docker
DOCKER_OPTS="-g=/nfs --storage-driver=vfs"   # /nfs is an NFS

The problem still exists: the docker images pulled from one host is not visible to another host sharing the same NFS.

Jerome and Frederick,

Can you please share more of your experience on this? It would be great if you can include some steps for how to set up and how to test. ;-)

Thanks,

Jay

--

Thanks,

Jay

[Jérôme Petazzoni]

Hi Jay,

Sorry, I had misunderstood the question.

Here is a better answer: "Yes, you can put the images and containers on NFS; but if you want to share that NFS storage with other Docker Engines, there are some restrictions."

Here are the caveats:

- only one engine may pull images (otherwise, you can have conflicts);

- other engines have to be restarted to "see" new images;

- when you restart an engine, it will detect the containers executed by others.

Have you considered using docker load/save, and put the image tarballs on the NFS storage?

(This is inspired by the design of https://github.com/blake-education/dogestry)

[Jay Lau]

Thanks Jerom for the details explanation! Yes, the  caveats is same as my test. For this one:

- when you restart an engine, it will detect the containers executed by others.

[Jay] Others will not able the detect the status of the container, so one running docker container on a docker server might be marked as unavailable in other docker servers.

For docker load/save, are you meaning the following:

1) Docker save the tar ball in an NFS

2) Docker load the tar ball to local docker server

3) Create docker containers based on the new local image

Is it the process? If it is, then it might be not as what I want ;-(, what I want is using NFS as a "local repo" for docker images and make the docker images in NFS available to all docker servers. But because of the limitations that you listed, seems we do not have a better way to handle this, right?

I see that you are from docker team, so does docker team has any plan to make my case work via NFS? ;-)

Thanks,

Jay

Thanks,

Jay

--

Thanks,

Jay

[Jérôme Petazzoni]

Hi,

On Thu, Jul 17, 2014 at 1:27 AM, Jay Lau <jay.l...@gmail.com> wrote:

- when you restart an engine, it will detect the containers executed by others.

[Jay] Others will not able the detect the status of the container, so one running docker container on a docker server might be marked as unavailable in other docker servers.

That's correct.

 

For docker load/save, are you meaning the following:

1) Docker save the tar ball in an NFS

2) Docker load the tar ball to local docker server

3) Create docker containers based on the new local image

Is it the process?

Yes.

 

If it is, then it might be not as what I want ;-(, what I want is using NFS as a "local repo" for docker images and make the docker images in NFS available to all docker servers. But because of the limitations that you listed, seems we do not have a better way to handle this, right?

You are right: it will incur a copy (from the NFS tree to the local Docker storage) when you "docker load".

 

I see that you are from docker team, so does docker team has any plan to make my case work via NFS? ;-)

I'm not aware of specific plans in that area (cloud-like object stores are more fashionable those days than shared NFS mounts :-))

I don't know the storage driver internals well enough to know how hard it would be to implement concurrent access, sorry!

[Jay Lau]

Thanks Jerome, you are always very helpful! ;-)

Clear now, yes as you said, cloud-like object stores are more fashionable those days than shared NFS mounts, we are planning to switch to this solution.

Thanks,

Jay

--

Thanks,

Jay