Cannot create cluster on ScyllaDB

[guillaume mendlevitch]

Hello everyone,

I just got started with ScyllaDB (on Windows 10, using Docker). Everything is fine when I create a node (docker run --name Node_X -d scylladb/scylla:4.1.0)

But when I create another one that I link to the 1st one (docker run --name Node_Y -d scylladb/scylla:4.1.0 --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X)").

It returns an error message: "nodetool: Unable to connect to Scylla API server: java.net.ConnectException: Connection refused (Connection refused) See 'nodetool help' or 'nodetool help '".

Could someone please advise me on that matter?

Many thanks for your help.

Guillaume

One can find the logs for the 2nd node below:

running: (['/opt/scylladb/scripts/scylla_dev_mode_setup', '--developer-mode', '1'],) running: (['/opt/scylladb/scripts/scylla_io_setup'],) 2020-12-28 14:07:02,470 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. 2020-12-28 14:07:02,470 INFO Included extra file "/etc/supervisord.conf.d/node-exporter.conf" during parsing 2020-12-28 14:07:02,471 INFO Included extra file "/etc/supervisord.conf.d/rsyslog.conf" during parsing 2020-12-28 14:07:02,471 INFO Included extra file "/etc/supervisord.conf.d/scylla-housekeeping.conf" during parsing 2020-12-28 14:07:02,471 INFO Included extra file "/etc/supervisord.conf.d/scylla-jmx.conf" during parsing 2020-12-28 14:07:02,471 INFO Included extra file "/etc/supervisord.conf.d/scylla-server.conf" during parsing 2020-12-28 14:07:02,471 INFO Included extra file "/etc/supervisord.conf.d/sshd-server.conf" during parsing 2020-12-28 14:07:02,504 INFO RPC interface 'supervisor' initialized 2020-12-28 14:07:02,504 CRIT Server 'inet_http_server' running without any HTTP authentication checking 2020-12-28 14:07:02,504 INFO supervisord started with pid 26 2020-12-28 14:07:03,507 INFO spawned: 'scylla' with pid 29 2020-12-28 14:07:03,509 INFO spawned: 'scylla-housekeeping' with pid 30 2020-12-28 14:07:03,511 INFO spawned: 'sshd' with pid 31 2020-12-28 14:07:03,512 INFO spawned: 'scylla-jmx' with pid 32 2020-12-28 14:07:03,514 INFO spawned: 'rsyslog' with pid 33 2020-12-28 14:07:03,515 INFO spawned: 'node-exporter' with pid 34 Generating public/private ed25519 key pair. Your identification has been saved in /etc/ssh/ssh_host_ed25519_key. Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub. The key fingerprint is: SHA256:h6ScBFQ3ZuJpK3F0GZUTfOmLinXX1+LxW7B8rxOx5FM root@f881e3d82b89 The key's randomart image is: +--[ED25519 256]--+ | .o.+ B=oo . | | + B..+ o | | . * . + | | * = . . o E| | . = S .. =.+.| | . ..o o.Ooo| | o o . .oBo| | . . o.+| | .+o| +----[SHA256]-----+ time="2020-12-28T14:07:03Z" level=info msg="Starting node_exporter (version=0.17.0, branch=HEAD, revision=f6f6194a436b9a63d0439abc585c76b19a206b21)" source="node_exporter.go:82" time="2020-12-28T14:07:03Z" level=info msg="Build context (go=go1.11.2, user=root@322511e06ced, date=20181130-15:51:33)" source="node_exporter.go:83" time="2020-12-28T14:07:03Z" level=info msg="Enabled collectors:" source="node_exporter.go:90" time="2020-12-28T14:07:03Z" level=info msg=" - arp" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - bcache" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - bonding" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - conntrack" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - cpu" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - diskstats" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - edac" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - entropy" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - filefd" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - filesystem" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - hwmon" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - infiniband" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - interrupts" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - ipvs" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - loadavg" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - mdadm" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - meminfo" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - netclass" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - netdev" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - netstat" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - nfs" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - nfsd" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - sockstat" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - stat" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - textfile" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - time" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - timex" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - uname" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - vmstat" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - xfs" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - zfs" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg="Listening on :9100" source="node_exporter.go:111" /etc/scylla.d/docker.conf: line 1: docker: command not found FATAL: Exception during startup, aborting: boost::wrapexceptboost::bad_lexical_cast (bad lexical cast: source type value could not be interpreted as target) 2020-12-28 14:07:03,984 INFO exited: scylla (exit status 7; not expected) Connecting to http://localhost:10000 Starting the JMX server Generating public/private rsa key pair. Your identification has been saved in /etc/ssh/ssh_host_rsa_key. Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub. The key fingerprint is: SHA256:q+U05wLwl2LeoE98MrGp7MHuEpk4021nzP+TKeIVZbM root@f881e3d82b89 The key's randomart image is: +---[RSA 4096]----+ | | | | | + | | . o o | | o + =. S.E | |+ =.o.@+oo | | o oo*BBB .o | | .o.+oX=++ | | +*o+..+o. | +----[SHA256]-----+ Could not load host key: /etc/ssh/ssh_host_ecdsa_key JMX is enabled to receive remote connections on port: 7199 2020-12-28 14:07:05,423 INFO spawned: 'scylla' with pid 73 2020-12-28 14:07:05,424 INFO success: scylla-housekeeping entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-12-28 14:07:05,424 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-12-28 14:07:05,424 INFO success: scylla-jmx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-12-28 14:07:05,424 INFO success: rsyslog entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-12-28 14:07:05,424 INFO success: node-exporter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) /etc/scylla.d/docker.conf: line 1: docker: command not found FATAL: Exception during startup, aborting: boost::wrapexceptboost::bad_lexical_cast (bad lexical cast: source type value could not be interpreted as target) 2020-12-28 14:07:05,979 INFO exited: scylla (exit status 7; not expected) 2020-12-28 14:07:07,986 INFO spawned: 'scylla' with pid 181 /etc/scylla.d/docker.conf: line 1: docker: command not found FATAL: Exception during startup, aborting: boost::wrapexceptboost::bad_lexical_cast (bad lexical cast: source type value could not be interpreted as target) 2020-12-28 14:07:08,525 INFO exited: scylla (exit status 7; not expected) multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 1319, in do_open encode_chunked=req.has_header('Transfer-encoding')) File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1252, in request self._send_request(method, url, body, headers, encode_chunked) File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1298, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1247, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1026, in _send_output self.send(msg) File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 966, in send self.connect() File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 938, in connect (self.host,self.port), self.timeout, self.source_address) File "/opt/scylladb/python3/lib64/python3.7/socket.py", line 728, in create_connection raise err File "/opt/scylladb/python3/lib64/python3.7/socket.py", line 716, in create_connection sock.connect(sa) OSError: [Errno 99] Cannot assign requested address
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/opt/scylladb/python3/lib64/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 64, in get_url return urllib.request.urlopen(path).read().decode('utf-8') File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 525, in open response = self._open(req, data) File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 543, in _open '_open', req) File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 503, in _call_chain result = func(*args) File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 1347, in http_open return self.do_open(http.client.HTTPConnection, req) File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 1321, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [Errno 99] Cannot assign requested address> """
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 197, in args.func(args) File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 123, in check_version current_version = sanitize_version(get_api('/storage_service/scylla_release_version'))
File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 82, in get_api return get_json_from_url("http://" + api_address + path) File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 74, in get_json_from_url retval = result.get(timeout=5) File "/opt/scylladb/python3/lib64/python3.7/multiprocessing/pool.py", line 657, in get raise self._value urllib.error.URLError: <urlopen error [Errno 99] Cannot assign requested address> 2020-12-28 14:07:12,202 INFO spawned: 'scylla' with pid 216 /etc/scylla.d/docker.conf: line 1: docker: command not found FATAL: Exception during startup, aborting: boost::wrapexceptboost::bad_lexical_cast (bad lexical cast: source type value could not be interpreted as target) 2020-12-28 14:07:12,727 INFO exited: scylla (exit status 7; not expected) 2020-12-28 14:07:13,729 INFO gave up: scylla entered FATAL state, too many start retries too quickly

[Tzach Livyatan]

Hi 

The error code below is similar to the one reported at

https://github.com/scylladb/scylla/issues/5638

Not sure if you can control fs.aio-max-nr, but you can try using less cores per node using --smp 2 option

Tzach

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/ad2134c3-0df3-4625-b30b-82de4263dba0n%40googlegroups.com.

[Avi Kivity]

Is docker installed in a non-regular path?

From "docker: command not found" I infer that the call to "docker inspect" did not succeed.

--

[Pekka Enberg]

Hi,

I don't know why the "docker" command appears in the logs. It's certainly not supposed to be called _within_ the container.

But I think the real issue is that we have Scylla server failing to start up because of:

  FATAL: Exception during startup, aborting: boost::wrapexceptboost::bad_lexical_cast (bad lexical cast: source type value could not be interpreted as target)

Similarly, housekeeping fails to work because of:

  [Errno 99] Cannot assign requested address

Perhaps there's something wrong with the auto-detected listen address... What does the following command return while the container is running?

  docker exec -it some-scylla hostname -i

If the container has already been stopped, you can also just run:

  docker run -it centos:7 /usr/bin/hostname -i

- Pekka

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/8e823d50-347f-9fe8-d207-f33e174cd884%40scylladb.com.

[Avi Kivity]

On 1/5/21 10:15 AM, Pekka Enberg wrote:

Hi,

I don't know why the "docker" command appears in the logs. It's certainly not supposed to be called _within_ the container.

The docker command appears within the invocation, quoted

docker run --name Node_Y -d scylladb/scylla:4.1.0 --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X)").

What happens is that the "docker inspect" command fails, and we get a --seeds="whatever error message was returned".

It's also possible that it succeeded, but returned a value that was not expected (i.e. there was no IPAddress associated with the container).

Either way, the docker setup is broken.

But I think the real issue is that we have Scylla server failing to start up because of:

  FATAL: Exception during startup, aborting: boost::wrapexceptboost::bad_lexical_cast (bad lexical cast: source type value could not be interpreted as target)

That's expected, if you pass garbage to --seeds, you get back amplified garbage from boost.

[guillaume mendlevitch]

Hello everyone,

Thank you for your help!

Here is what I've got:

Perhaps there's something wrong with the auto-detected listen address... What does the following command return while the container is running?

  docker exec -it some-scylla hostname -i

--> 172.17.0.2

If the container has already been stopped, you can also just run:

  docker run -it centos:7 /usr/bin/hostname -i

--> 172.17.0.2  

So if the docker is broken, is there any way to fix it then? I really don't know much about Scylla, sorry...

Many thanks,

Guillaume

[Pekka Enberg]

Hi,

The IP address looks fine and Avi is correct that the issue is with "docker inspect" that fails, which is we pass an invalid address to the Docker image.

I don't have any experience with Docker on Windows so I am not sure what the problem is. However, as "docker run" clearly works, perhaps this is a shell issue? What are you using on Windows? As a workaround, you could try running the "docker inspect" command by itself and paste that IP address to the "docker run" command. IOW,

  docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X

and then

  docker run --name Node_Y -d scylladb/scylla:4.1.0 --seeds=<paste IP addres here>

Btw,looking at the original command,isn't there a stray ")" at the end of the command:

  docker run --name Node_Y -d scylladb/scylla:4.1.0 --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X)").

Shouldn't that be:

  docker run --name Node_Y -d scylladb/scylla:4.1.0 --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X)"

Regards,

- Pekka

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/ee38558e-d876-46b3-8023-104e43f94d05n%40googlegroups.com.

[Ivan Prisyazhnyy]

I have tested your commands. They work well on Windows.

$ docker run --name node1 -it scylladb/scylla:4.1.0
$ docker run --name node2 -it scylladb/scylla:4.1.0 --seeds="$(docker
inspect --format='{{ .NetworkSettings.IPAddress }}' node1)"

What is not working for you is that part:

echo "$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' node1)"

Just replace with seeds part with IP address of the node1

$ docker run --name node2 -it scylladb/scylla:4.1.0
--seeds="ip_address_of_the_node1"

> To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAGUyND-mF%3D7jwSpEfkXjU3A1W3CtxWXgN_4qkS%2B2BzNu-pOnsQ%40mail.gmail.com.

[guillaume mendlevitch]

Hi everyone,

It just tried this solution and it seems to be working! 

Thank you all for your help! It's such a relief.

Have a good day,

[Saif Jakati]

what changes you made I'm also facing this same issue but I'm using Ubuntu 20.04