Scylla Docker on AWS

[Madhukar Thota]

I am using ansible to deploy scylladb docker container on multiple AWS Nodes. I am not seeing any issue when i ran on single node but when i ran the container on multiple hosts with Ec2Snitch enabled i am seeing following errors.

Error:

Try --help.

2018-01-24 03:31:57,456 INFO exited: scylla (exit status 2; not expected)

2018-01-24 03:31:59,459 INFO spawned: 'scylla' with pid 1891

tuning /sys/devices/virtual/block/md0

already tuned: /sys/devices/virtual/block/md0/queue/scheduler

tuning: /sys/devices/virtual/block/md0/queue/nomerges 2

warning: unable to tune /sys/devices/virtual/block/md0/queue/nomerges to 2

tuning /sys/devices/pci0000:00/0000:00:1e.0/nvme/nvme0/nvme0n1

already tuned: /sys/devices/pci0000:00/0000:00:1e.0/nvme/nvme0/nvme0n1/queue/scheduler

tuning: /sys/devices/pci0000:00/0000:00:1e.0/nvme/nvme0/nvme0n1/queue/nomerges 2

warning: unable to tune /sys/devices/pci0000:00/0000:00:1e.0/nvme/nvme0/nvme0n1/queue/nomerges to 2

tuning /sys/devices/virtual/block/md0

error: too many positional options have been specified on the command line

/etc/scylla.d/docker.conf file in the container shows the multiple nodes without comma for seeds and adding two ip address for listen-address, rpc-address, 

SCYLLA_DOCKER_ARGS="--listen-address 10.10.12.89 10.10.8.149 --rpc-address 10.10.12.89 10.10.8.149 --seed-provider-parameters seeds=10.10.12.89 10.10.8.149 --blocked-reactor-notify-ms 999999999"

ansible docker_container section:

- name: scylladb container

docker_container:

name: scylladb

image: scylladb/scylla:2.1-rc2

pull: true

ulimits:

- memlock:-1:-1

- nofile:262144:262144

- core:-1:-1

- as:-1:-1

- nproc:65536:65536

network_mode: host

command: --developer-mode 0

env:

CPUSET: "--cpuset 1-7"

ports:

- "9042:9042"

- "7000:7000"

- "7001:7001"

- "7199:7199"

- "9180:9180"

- "9100:9100"

- "9160:9160"

volumes:

- "/etc/scylla/scylla.yaml:/etc/scylla/scylla.yaml"

- "/var/lib/scylla:/var/lib/scylla"

- "/etc/scylla/cadb.pem:/etc/scylla/cadb.pem"

- "/etc/scylla/db.crt:/etc/scylla/db.crt"

- "/etc/scylla/db.key:/etc/scylla/db.key"

restart_policy: always

recreate: true

state: started

How the container able to discover the nodes without specifying the seed nodes and why it adding mutliple address for listen-address, rpc-address? Any help  on this issue?

[Pekka Enberg]

Hello,

If you did not specify "listen address" yourself, the Docker image will use "hostname -i" to get one for you:

https://github.com/scylladb/scylla/blob/master/dist/docker/redhat/scyllasetup.py#L37

Same for RPC address and seed nodes. I assume that "hostname -i" returns multiple IP addresses on that node  -- multiple NIC interfaces perhaps? What's the reason for not configuring both listen address and seed nodes via Ansible?

Regards,

- Pekka 

[Madhukar Thota]

I was able to resolve the issue by command line options

command: --developer-mode 0 --listen-address {{ansible_all_ipv4_addresses[0]}} --seeds {% for h in groups['tag_application_scylladb'] %}{{ hostvars[h].ec2_private_ip_address}}{% if not loop.last %},{% endif %}{% endfor %} --broadcast-address {{ansible_all_ipv4_addresses[0]}} --broadcast-rpc-address {{ansible_all_ipv4_addresses[0]}}

[Pekka Enberg]

Hi,

On Wed, Jan 24, 2018 at 7:13 PM, Madhukar Thota <madhuka...@gmail.com> wrote:

I was able to resolve the issue by command line options

command: --developer-mode 0 --listen-address {{ansible_all_ipv4_addresses[0]}} --seeds {% for h in groups['tag_application_scylladb'] %}{{ hostvars[h].ec2_private_ip_address}}{% if not loop.last %},{% endif %}{% endfor %} --broadcast-address {{ansible_all_ipv4_addresses[0]}} --broadcast-rpc-address {{ansible_all_ipv4_addresses[0]}}

Great, thanks for sharing the solution!

I opened a ticket on the incomprehensible error message:

https://github.com/scylladb/scylla/issues/3149

- Pekka