SeaweedFS on Docker

[Jesper Zedlitz]

Hi. 

Is anyone successfully running SeaweedFS using multipe Docker containers? Building the Docker image was quite easy. After building the image I followed the instructions to set-up a Failover Master Server like this:

docker run --name master1 -it --rm weed master

docker run --name master2 -it --rm --link master1:master1 weed master -peers=master1:9333

docker run --name master3 -it --rm --link master2:master2 weed master -peers=master2:9333

docker run --name volume1 -it --rm --link master1:master1 weed volume -mserver=master1:9333

docker run --name volume2 -it --rm --link master2:master2 weed volume -mserver=master2:9333

docker run --name volume3 -it --rm --link master3:master2 weed volume -mserver=master3:9333

Now I have six weed instances running:

$ docker ps

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES

861ef0a488f6        weed                "weed volume -mserver"   6 minutes ago       Up 6 minutes        8080/tcp, 9333/tcp   volume3

e9839818c850        weed                "weed volume -mserver"   6 minutes ago       Up 6 minutes        8080/tcp, 9333/tcp   volume2

80d136b52275        weed                "weed volume -mserver"   6 minutes ago       Up 6 minutes        8080/tcp, 9333/tcp   volume1

f807339d7edc        weed                "weed master -peers=m"   6 minutes ago       Up 6 minutes        8080/tcp, 9333/tcp   master3

32dead850592        weed                "weed master -peers=m"   7 minutes ago       Up 6 minutes        8080/tcp, 9333/tcp   master2

0fee57ee2eeb        weed                "weed master"            7 minutes ago       Up 7 minutes        8080/tcp, 9333/tcp   master1

To add data I need the IP address of the first master:

docker inspect `docker ps |grep master1 | cut -c 1-12` |grep IPAddress

In my case it is 172.17.0.2

However, when I try to add data I get an error message:

$ curl http://172.17.0.2:9333/dir/assign

{"error":"Cannot grow volume group! Failed to assign 1: Post http://127.0.0.1:8080/admin/assign_volume: dial tcp 127.0.0.1:8080: getsockopt: connection refused"}

It seems that weed mixes up addresses of the connected master and volume nodes.

[Chris Lu]

I do not use docker myself. But it seems the weed volume and master are missing the IP address setting, which will default to 127.0.0.1. Since there are running on different docker instances, they can not see each other.

Chris

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jesper Zedlitz]

> I do not use docker myself. >

Docker is a really convenient way to test complicated scenarios. For
example, I plan to
a) set up a clustered SeaweedFS installation,
b) start a benchmark in an additional Docker container, and
c) while the benchmark is running stop some of the containers in the
cluster
to check SeaweedFS' correct reaction to hardware/network problems.

> But it seems the weed volume and master are missing the IP address > setting, which will default to 127.0.0.1. Since there are running on
> different docker instances, they can not see each other.
>

Would it be possible to specify the interface (e.g., eth0) to bind? That
way the correct IP address could be determined.

Another possiblity seems to be the "-ip" option:

docker run --name volume1 -it --rm --link master1:master1 weed volume

-mserver=master1:9333 -ip volume1

However, weed nodes exchange their host name not the IP address. In a
Docker scenario the weed master usually cannot resolv the names of other
nodes. If the volume node would transmit its IP address it would work.
Or are there other usage scenarios where that behaviour leads to problems?

Jesper

[Chris Lu]

There is an "-ip.bind" option. Please use "weed master -h" or "weed volume -h" to check it out.

Chris

[Jesper Zedlitz]

Hi Chris.

Thank you for the quick response.

> There is an "-ip.bind" option. Please use "weed master -h" or "weed
> volume -h" to check it out.
>

This is a helpful advice. I think I figured out how to run a clustered
SeaweedFS using Docker. The trick is that a different Dockerfile is
needed. In its entryfile the parameter -ip `hostname -i` is added. I am
going to optimize the Dockerfile in the next days and share it.

Jesper

[Jesper Zedlitz]

Attached are the promised files for running SeaweedFS using Docker. All parameters are passed through to weed. Depending on the command the entrypoint.sh script adds parameters to link containers. 

To start two master processes and two volume process the following commands can be used:

docker run -d --name master1 -v /tmp/weedtest/m1:/data weed master

docker run -d --name master2 -v /tmp/weedtest/m2:/data --link master1:master weed master

docker run -d --name volume1 -v /tmp/weedtest/v1:/data --link master1:master weed volume

docker run -d --name volume2 -v /tmp/weedtest/v1:/data --link master1:master weed volume

With only one more command you can run a benchmark with this cluster:

docker run -it --rm --link master1:master  weed benchmark -server master:9333

[Chris Lu]

Thanks!

Is it possible to send me a pull request to merge the code into the git repo? 

https://github.com/chrislusf/seaweedfs.git

Or the seaweedfs wiki repo?

https://github.com/chrislusf/seaweedfs.wiki.git

Chris

[David C.]

Hi,

Do you have any idea how to run a cluster of servers with this configuration in Docker? Would be like this:
-  Per instance 1 SeaweedFS master and X volumes (OR 1 master server and then X volume server with only volumes)

-  Connect each master between the instances. (OR connect the volume servers to the unique master server)

Tried it with your configuration, and hard-writting the host ips, but when the master or the volumes of other instances try to connect to another master instance, it connects but the master instance replaces the "public ip" with the "docker ip".

Without docker this is really straightforward. Maybe i'm overcomplicating but want to know if that way is possible. Maybe with Docker Swarm could work, but seems harder to setup.

Thanks!