Scylla University - S101 - Cluster doesn't start from Docker

[Roman Kunin]

The lesson S101: Scylla Essentials – Overview of Scylla and NoSQL Basics  High Availability  Lab 1 – Overview and Setup

The following commands fail to start a cluster.

```

docker run --name Node_X -d scylladb/scylla:4.1.0

docker run --name Node_Y -d scylladb/scylla:4.1.0 --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X)"

docker run --name Node_Z -d scylladb/scylla:4.1.0 --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X)"

```

```

> docker exec -it Node_Z nodetool status

nodetool: Unable to connect to Scylla API server: java.net.ConnectException: Connection refused (Connection refused)

See 'nodetool help' or 'nodetool help <command>'.

```

[Tzach Livyatan]

Hi 

First, please try again after a minute. It may take some time for the process to start.

Seconds, run 

> docker ps

To see if the instance are running

and

> docker logs Node_Z

And look for error messages that can help understand the problem.

Regards

Tzach

 

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/4d3685b2-35b1-4ef4-aa85-8d061375d933n%40googlegroups.com.

[Roman Kunin]

There is a problem.

docker logs Node_X returns this:

Scylla version 4.1.0-0.20200618.b443b2574a8 with build-id 650a4b0c03ae680a9fbf7166c992e99e29b37123 starting ...
command used: "/usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --developer-mode=1 --overprovisioned --listen-address 192.168.16.3 --rpc-address 192.168.16.3 --seed-provider-parameters seeds=192.168.16.3 --blocked-reactor-notify-ms 999999999"
parsed command line options: [log-to-syslog: 0, log-to-stdout: 1, default-log-level: info, network-stack: posix, developer-mode: 1, overprovisioned, listen-address: 192.168.16.3, rpc-address: 192.168.16.3, seed-provider-parameters: seeds=192.168.16.3, blocked-reactor-notify-ms: 999999999]
ERROR 2021-08-04 23:37:25,044 [shard 2] seastar - Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application
ERROR 2021-08-04 23:37:25,044 [shard 19] seastar - Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application
2021-08-04 23:37:25,179 INFO exited: scylla (exit status 1; not expected)
2021-08-04 23:37:27,183 INFO spawned: 'scylla' with pid 171
Scylla version 4.1.0-0.20200618.b443b2574a8 with build-id 650a4b0c03ae680a9fbf7166c992e99e29b37123 starting ...
command used: "/usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --developer-mode=1 --overprovisioned --listen-address 192.168.16.3 --rpc-address 192.168.16.3 --seed-provider-parameters seeds=192.168.16.3 --blocked-reactor-notify-ms 999999999"
parsed command line options: [log-to-syslog: 0, log-to-stdout: 1, default-log-level: info, network-stack: posix, developer-mode: 1, overprovisioned, listen-address: 192.168.16.3, rpc-address: 192.168.16.3, seed-provider-parameters: seeds=192.168.16.3, blocked-reactor-notify-ms: 999999999]
ERROR 2021-08-04 23:37:27,592 [shard 15] seastar - Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application
ERROR 2021-08-04 23:37:27,592 [shard 17] seastar - Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application
ERROR 2021-08-04 23:37:27,592 [shard 13] seastar - Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application
2021-08-04 23:37:27,729 INFO exited: scylla (exit status 1; not expected)
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 1319, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 938, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/opt/scylladb/python3/lib64/python3.7/socket.py", line 728, in create_connection
    raise err
  File "/opt/scylladb/python3/lib64/python3.7/socket.py", line 716, in create_connection
    sock.connect(sa)
OSError: [Errno 99] Cannot assign requested address

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/scylladb/python3/lib64/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 64, in get_url
    return urllib.request.urlopen(path).read().decode('utf-8')
  File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 1347, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 1321, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 99] Cannot assign requested address>
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 197, in <module>
    args.func(args)
  File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 123, in check_version
    current_version = sanitize_version(get_api('/storage_service/scylla_release_version'))
  File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 82, in get_api
    return get_json_from_url("http://" + api_address + path)
  File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 74, in get_json_from_url
    retval = result.get(timeout=5)
  File "/opt/scylladb/python3/lib64/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
urllib.error.URLError: <urlopen error [Errno 99] Cannot assign requested address>
2021-08-04 23:37:31,174 INFO spawned: 'scylla' with pid 236
Scylla version 4.1.0-0.20200618.b443b2574a8 with build-id 650a4b0c03ae680a9fbf7166c992e99e29b37123 starting ...
command used: "/usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --developer-mode=1 --overprovisioned --listen-address 192.168.16.3 --rpc-address 192.168.16.3 --seed-provider-parameters seeds=192.168.16.3 --blocked-reactor-notify-ms 999999999"
parsed command line options: [log-to-syslog: 0, log-to-stdout: 1, default-log-level: info, network-stack: posix, developer-mode: 1, overprovisioned, listen-address: 192.168.16.3, rpc-address: 192.168.16.3, seed-provider-parameters: seeds=192.168.16.3, blocked-reactor-notify-ms: 999999999]
ERROR 2021-08-04 23:37:31,511 [shard 8] seastar - Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application
2021-08-04 23:37:31,651 INFO exited: scylla (exit status 1; not expected)
2021-08-04 23:37:32,653 INFO gave up: scylla entered FATAL state, too many start retries too quickly

ср, 4 авг. 2021 г. в 16:13, Tzach Livyatan <tz...@scylladb.com>:

You received this message because you are subscribed to a topic in the Google Groups "ScyllaDB users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scylladb-users/891ppg9o7ZA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAEB7%3D3YiXFy-0zsyT4k8ahrGeu5WammEBdNoWPWwq%2Bt64zLtMg%40mail.gmail.com.

[Roman Kunin]

Everything worked after I added fs.aio-max-nr = 1048576 in my /etc/sysctl.conf file.

It should be in the lesson.

чт, 5 авг. 2021 г. в 09:53, Roman Kunin <mos...@gmail.com>:

[Tzach Livyatan]

On Thu, Aug 5, 2021 at 3:39 AM Roman Kunin <mos...@gmail.com> wrote:

Everything worked after I added fs.aio-max-nr = 1048576 in my /etc/sysctl.conf file.

Thanks for looking into this.

We will add this to the lesson!

Regards

Tzach

 

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CA%2BA5EZxtrtgeQ%2B1mRsXTRQttS%2B8aMoX7ufpN4_k6-eZ_PoenFA%40mail.gmail.com.

[Moreno Garcia]

I know this is docker so it applies to the host, not to the containers.

https://github.com/scylladb/scylla/issues/5981

https://github.com/scylladb/scylla/issues/9096

But we see this in the field a lot of times.

https://scylladb-corporate.slack.com/archives/C3AE85S07/p1613766859041200

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAEB7%3D3YaU%3DOOCmo-%2B%2BqOg4d8UqxDBier%2BYc8Bne7fGxxQRtdGQ%40mail.gmail.com.

--

--

Moreno Garcia

Head of Solutions Architects
https://calendly.com/moreno-scylla
Check our Scylla Summit 2019 Presentations on Youtube