ScyllaDB cannot be started after IP change

810 views
Skip to first unread message

Alexandr Porunov

<alexandr.porunov@gmail.com>
unread,
Nov 15, 2017, 2:14:51 PM11/15/17
to ScyllaDB users
Hello,

I had a scylladb cluster of 3 nodes. It was working good. Then I had to change IP addresses on all 3 nodes. I changed them. Also, I changed all related IP addresses in /etc/scylla/scylla.yaml

I tried to restart all nodes but they couldn't find each other. I tried to run scylla_setup on all nodes again but it didn't help.
Then I remove scylladb from all nodes and removed all data from directory with scylladb data and scylladb commitlog. Then I installed scylladb again on all nodes with proper configuration and run scylla_setup again. After that I tried to start a new cluster but it cannot be started.

Here is what I get when I try to start any node:

systemctl status scylla-server.service -l
● scylla-server.service - Scylla Server
   Loaded: loaded (/usr/lib/systemd/system/scylla-server.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2017-11-15 21:05:47 EET; 4s ago
  Process: 1395 ExecStopPost=/usr/lib/scylla/scylla_stop (code=exited, status=0/SUCCESS)
  Process: 1389 ExecStart=/usr/bin/scylla $SCYLLA_ARGS $SEASTAR_IO $DEV_MODE $CPUSET (code=dumped, signal=ABRT)
  Process: 1378 ExecStartPre=/usr/lib/scylla/scylla_prepare (code=exited, status=0/SUCCESS)
 Main PID: 1389 (code=dumped, signal=ABRT)
   Status: "creating snitch"

Nov 15 21:05:46 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 15 21:05:46 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 15 21:05:46 localhost.localdomain systemd[1]: scylla-server.service failed.
Nov 15 21:05:47 localhost.localdomain systemd[1]: scylla-server.service holdoff time over, scheduling restart.
Nov 15 21:05:47 localhost.localdomain systemd[1]: start request repeated too quickly for scylla-server.service
Nov 15 21:05:47 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 15 21:05:47 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 15 21:05:47 localhost.localdomain systemd[1]: scylla-server.service failed.

Can this error be fixed somehow?

Best regards,
Alexandr

Tzach Livyatan

<tzach@scylladb.com>
unread,
Nov 15, 2017, 2:20:55 PM11/15/17
to ScyllaDB users
The old IPs probably still live in the system tables as ghosts, making the nodes keep calling their old, dead friends in vain.
If you do not have data on the cluster, make sure to clean the data directory, including the system table and restart
If you do have data, follow the replace procedure

Regards
Tzach

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/b889c633-1475-4cfd-a6f0-17488374d35f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Alexandr Porunov

<alexandr.porunov@gmail.com>
unread,
Nov 15, 2017, 3:29:07 PM11/15/17
to ScyllaDB users
But I don't have any data. I already removed everything from each data directory. I reinstalled scylladb. Now I can't start any node because of that error.
Because of that I can't check the status of nodes with "node tool status".

# nodetool status
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.

I don't have an option to start the cluster and change nodes one by one. I need to run a new cluster and forgot about old cluster. I don't have an ability to run old cluster because I removed all data.

I am sure that the information about the old cluster is stored somewhere except the data directory. I just need to find that data and remove it to run a new cluster.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.

Tomer Sandler

<tomer@scylladb.com>
unread,
Nov 15, 2017, 3:37:34 PM11/15/17
to scylladb-users@googlegroups.com
Did you also change the cluster name?
Dis you try to keep the old name and see if it works? 
What about the RAID0 created on the old cluster?  if you are using the same node with the same disk and the RAID is still active you won't be able to complete the scylla_setup. 

--
Tomer Sandler
ScyllaDB

(Sent from my android)

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.

Alexandr Porunov

<alexandr.porunov@gmail.com>
unread,
Nov 15, 2017, 3:40:38 PM11/15/17
to ScyllaDB users
Here is additional logs:

# journalctl -xe
                                                              #3  0x00007ff721a2ce25 start_thread (libpthread.so.0)
                                                              #4  0x00007ff72175a34d __clone (libc.so.6)
                                                              
                                                              Stack trace of thread 1837:
                                                              #0  0x00007ff721a3370d read (libpthread.so.0)
                                                              #1  0x00000000004ed5f6 n/a (scylla)
                                                              #2  0x00000000004eebcf n/a (scylla)
                                                              #3  0x000000000059a54e n/a (scylla)
                                                              #4  0x00007ff721a2ce25 start_thread (libpthread.so.0)
                                                              #5  0x00007ff72175a34d __clone (libc.so.6)
-- Subject: Process 1833 (scylla) dumped core
-- Defined-By: systemd
-- Documentation: man:core(5)
-- 
-- Process 1833 (scylla) crashed and dumped core.
-- 
-- This usually indicates a programming error in the crashing program and
-- should be reported to its vendor as a bug.
Nov 15 22:39:17 localhost.localdomain systemd[1]: scylla-server.service holdoff time over, scheduling restart.
Nov 15 22:39:17 localhost.localdomain systemd[1]: start request repeated too quickly for scylla-server.service
Nov 15 22:39:17 localhost.localdomain systemd[1]: Failed to start Scylla Server.
-- Subject: Unit scylla-server.service has failed
-- Defined-By: systemd
-- 
-- Unit scylla-server.service has failed.
-- 
-- The result is failed.
Nov 15 22:39:17 localhost.localdomain systemd[1]: Dependency failed for Scylla JMX.
-- Subject: Unit scylla-jmx.service has failed
-- Defined-By: systemd
-- 
-- Unit scylla-jmx.service has failed.
-- 
-- The result is dependency.
Nov 15 22:39:17 localhost.localdomain systemd[1]: Job scylla-jmx.service/start failed with result 'dependency'.
Nov 15 22:39:17 localhost.localdomain systemd[1]: Dependency failed for Run Scylla Housekeeping daily mode.
-- Subject: Unit scylla-housekeeping-daily.timer has failed
-- Defined-By: systemd
-- 
-- Unit scylla-housekeeping-daily.timer has failed.
-- 
-- The result is dependency.
Nov 15 22:39:17 localhost.localdomain systemd[1]: Job scylla-housekeeping-daily.timer/start failed with result 'dependency'.
Nov 15 22:39:17 localhost.localdomain systemd[1]: Dependency failed for Run Scylla Housekeeping restart mode.
-- Subject: Unit scylla-housekeeping-restart.timer has failed
-- Defined-By: systemd
-- 
-- Unit scylla-housekeeping-restart.timer has failed.
-- 
-- The result is dependency.
Nov 15 22:39:17 localhost.localdomain systemd[1]: Job scylla-housekeeping-restart.timer/start failed with result 'dependency'.
Nov 15 22:39:17 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 15 22:39:17 localhost.localdomain systemd[1]: scylla-server.service failed.

Alexandr Porunov

<alexandr.porunov@gmail.com>
unread,
Nov 15, 2017, 3:43:09 PM11/15/17
to ScyllaDB users
Tomer, 

The cluster name is the same as the old one.
I didn't use RAID0 and don't use it now. I have only one disk with one directory for scylladb data.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.

Glauber Costa

<glauber@scylladb.com>
unread,
Nov 15, 2017, 4:16:30 PM11/15/17
to ScyllaDB users
Did you remove system tables and commitlog as well, or just user data?

Another possibility is that you forgot to update the seeds and is
still using some of the old IPs as seeds.


On Wed, Nov 15, 2017 at 3:29 PM, Alexandr Porunov
> https://groups.google.com/d/msgid/scylladb-users/ae9bc703-d8a1-4a87-8f3d-31d49d98c3db%40googlegroups.com.

Alexandr Porunov

<alexandr.porunov@gmail.com>
unread,
Nov 15, 2017, 4:44:51 PM11/15/17
to ScyllaDB users
Where does system tables are stored? 

I have next config options in my scylla.yaml:

data_file_directories:
    - /srv/node/d1/scylla/data

commitlog_directory: /srv/node/d1/scylla/commitlog

Both /srv/node/d1/scylla/data and /srv/node/d1/scylla/commitlog directories are empty.

Glauber Costa

<glauber@scylladb.com>
unread,
Nov 15, 2017, 4:55:02 PM11/15/17
to ScyllaDB users
Inside data. If you deleted everything under data, and not just data/yourstuff it is safe to say they are gone.

What about your seeds ?

To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.

Alexandr Porunov

<alexandr.porunov@gmail.com>
unread,
Nov 15, 2017, 5:01:32 PM11/15/17
to ScyllaDB users
Yes I deleted everything from those directories on all nodes. Right now these directories are empty on all nodes. 
Each node now have new IP addresses in /etc/scylla/scylla.yaml 

Now seeds have this string on all nodes:
- seeds: "10.10.10.22,10.10.10.23,10.10.10.24"

Glauber Costa

<glauber@scylladb.com>
unread,
Nov 15, 2017, 5:16:51 PM11/15/17
to ScyllaDB users
Best send us the full logs then.

The systemd excerpts are not saying much.

You can grab them with journalctl -u scylla-server and then pipe them somewhere

To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.

Alexandr Porunov

<alexandr.porunov@gmail.com>
unread,
Nov 15, 2017, 5:21:50 PM11/15/17
to ScyllaDB users
Here is my logs:


-- Logs begin at Thu 2017-11-16 00:15:36 EET, end at Thu 2017-11-16 00:17:01 EET. --
Nov 16 00:15:40 localhost.localdomain systemd[1]: Starting Scylla Server...
Nov 16 00:15:41 localhost.localdomain scylla_prepare[777]: irqbalance is not running
Nov 16 00:15:41 localhost.localdomain scylla_prepare[777]: Setting a physical interface eth0...
Nov 16 00:15:41 localhost.localdomain scylla_prepare[777]: Setting mask 00000001 in /proc/irq/19/smp_affinity
Nov 16 00:15:41 localhost.localdomain scylla_prepare[777]: Setting mask 00000001 in /sys/class/net/eth0/queues/rx-0/rps_cpus
Nov 16 00:15:41 localhost.localdomain scylla_prepare[777]: Setting net.core.rps_sock_flow_entries to 32768
Nov 16 00:15:41 localhost.localdomain scylla_prepare[777]: Setting limit 32768 in /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
Nov 16 00:15:41 localhost.localdomain scylla_prepare[777]: Trying to enable ntuple filtering HW offload for eth0...not supported
Nov 16 00:15:41 localhost.localdomain scylla_prepare[777]: Setting mask 00000001 in /sys/class/net/eth0/queues/tx-0/xps_cpus
Nov 16 00:15:41 localhost.localdomain scylla_prepare[777]: Writing '4096' to /proc/sys/net/core/somaxconn
Nov 16 00:15:41 localhost.localdomain scylla_prepare[777]: Writing '4096' to /proc/sys/net/ipv4/tcp_max_syn_backlog
Nov 16 00:15:42 localhost.localdomain scylla_prepare[777]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:15:42 localhost.localdomain scylla_prepare[777]: tuning: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/scheduler noop
Nov 16 00:15:42 localhost.localdomain scylla_prepare[777]: tuning: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/nomerges 2
Nov 16 00:15:42 localhost.localdomain scylla_prepare[777]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:15:42 localhost.localdomain scylla[1043]: Scylla version 2.0.0-0.20170929.e265c91 starting ...
Nov 16 00:15:42 localhost.localdomain scylla[1043]:  [shard 0] init - Only 476 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
Nov 16 00:15:42 localhost.localdomain scylla[1043]:  [shard 0] snitch_logger - Property file /etc/scylla/cassandra-rackdc.properties is incomplete. Some obligatory fields are missing.
Nov 16 00:15:42 localhost.localdomain scylla[1043]:  [shard 0] snitch_logger - Failed to parse a properties file (/etc/scylla/cassandra-rackdc.properties). Halting...
Nov 16 00:15:42 localhost.localdomain scylla[1043]:  [shard 0] seastar - Exiting on unhandled exception: locator::bad_property_file_error (std::exception)
Nov 16 00:15:42 localhost.localdomain scylla[1043]: scylla: ./seastar/core/sharded.hh:327: seastar::sharded<Service>::~sharded() [with Service = seastar::httpd::http_server]: Assertion `_instances.empty()' failed.
Nov 16 00:15:42 localhost.localdomain scylla[1043]: Aborting on shard 0.
Nov 16 00:15:42 localhost.localdomain scylla[1043]: Backtrace:
Nov 16 00:15:42 localhost.localdomain scylla[1043]: 0x00000000004eef58
Nov 16 00:15:42 localhost.localdomain scylla[1043]: 0x00000000004ef075
Nov 16 00:15:42 localhost.localdomain scylla[1043]: 0x00000000004ef123
Nov 16 00:15:42 localhost.localdomain scylla[1043]: 0x00007fe1206a65df
Nov 16 00:15:42 localhost.localdomain scylla[1043]: 0x00007fe1203091f6
Nov 16 00:15:42 localhost.localdomain scylla[1043]: 0x00007fe12030a8e7
Nov 16 00:15:42 localhost.localdomain scylla[1043]: 0x00007fe120302265
Nov 16 00:15:42 localhost.localdomain scylla[1043]: 0x00007fe120302311
Nov 16 00:15:42 localhost.localdomain scylla[1043]: 0x000000000090274d
Nov 16 00:15:42 localhost.localdomain scylla[1043]: 0x00000000004751e4
Nov 16 00:15:42 localhost.localdomain scylla[1043]: 0x00007fe1202f5c04
Nov 16 00:15:42 localhost.localdomain scylla[1043]: 0x00000000004e8d9e
Nov 16 00:15:43 localhost.localdomain systemd[1]: scylla-server.service: main process exited, code=dumped, status=6/ABRT
Nov 16 00:15:43 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 16 00:15:43 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 16 00:15:43 localhost.localdomain systemd[1]: scylla-server.service failed.
Nov 16 00:15:44 localhost.localdomain systemd[1]: scylla-server.service holdoff time over, scheduling restart.
Nov 16 00:15:44 localhost.localdomain systemd[1]: Starting Scylla Server...
Nov 16 00:15:44 localhost.localdomain scylla_prepare[1115]: irqbalance is not running
Nov 16 00:15:44 localhost.localdomain scylla_prepare[1115]: Setting a physical interface eth0...
Nov 16 00:15:44 localhost.localdomain scylla_prepare[1115]: Setting mask 00000001 in /proc/irq/19/smp_affinity
Nov 16 00:15:44 localhost.localdomain scylla_prepare[1115]: Setting mask 00000001 in /sys/class/net/eth0/queues/rx-0/rps_cpus
Nov 16 00:15:44 localhost.localdomain scylla_prepare[1115]: Setting net.core.rps_sock_flow_entries to 32768
Nov 16 00:15:44 localhost.localdomain scylla_prepare[1115]: Setting limit 32768 in /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
Nov 16 00:15:44 localhost.localdomain scylla_prepare[1115]: Trying to enable ntuple filtering HW offload for eth0...not supported
Nov 16 00:15:44 localhost.localdomain scylla_prepare[1115]: Setting mask 00000001 in /sys/class/net/eth0/queues/tx-0/xps_cpus
Nov 16 00:15:44 localhost.localdomain scylla_prepare[1115]: Writing '4096' to /proc/sys/net/core/somaxconn
Nov 16 00:15:44 localhost.localdomain scylla_prepare[1115]: Writing '4096' to /proc/sys/net/ipv4/tcp_max_syn_backlog
Nov 16 00:15:45 localhost.localdomain scylla_prepare[1115]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:15:45 localhost.localdomain scylla_prepare[1115]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/scheduler
Nov 16 00:15:45 localhost.localdomain scylla_prepare[1115]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/nomerges
Nov 16 00:15:45 localhost.localdomain scylla_prepare[1115]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:15:45 localhost.localdomain scylla[1144]: Scylla version 2.0.0-0.20170929.e265c91 starting ...
Nov 16 00:15:45 localhost.localdomain scylla[1144]:  [shard 0] init - Only 476 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
Nov 16 00:15:45 localhost.localdomain scylla[1144]:  [shard 0] snitch_logger - Property file /etc/scylla/cassandra-rackdc.properties is incomplete. Some obligatory fields are missing.
Nov 16 00:15:45 localhost.localdomain scylla[1144]:  [shard 0] snitch_logger - Failed to parse a properties file (/etc/scylla/cassandra-rackdc.properties). Halting...
Nov 16 00:15:45 localhost.localdomain scylla[1144]: scylla: ./seastar/core/sharded.hh:327: seastar::sharded<Service>::~sharded() [with Service = seastar::httpd::http_server]: Assertion `_instances.empty()' failed.
Nov 16 00:15:45 localhost.localdomain scylla[1144]: Aborting on shard 0.
Nov 16 00:15:45 localhost.localdomain scylla[1144]: Backtrace:
Nov 16 00:15:45 localhost.localdomain scylla[1144]: 0x00000000004eef58
Nov 16 00:15:45 localhost.localdomain scylla[1144]: 0x00000000004ef075
Nov 16 00:15:45 localhost.localdomain scylla[1144]: 0x00000000004ef123
Nov 16 00:15:45 localhost.localdomain scylla[1144]: 0x00007f10b959b5df
Nov 16 00:15:45 localhost.localdomain scylla[1144]: 0x00007f10b91fe1f6
Nov 16 00:15:45 localhost.localdomain scylla[1144]: 0x00007f10b91ff8e7
Nov 16 00:15:45 localhost.localdomain scylla[1144]: 0x00007f10b91f7265
Nov 16 00:15:45 localhost.localdomain scylla[1144]: 0x00007f10b91f7311
Nov 16 00:15:45 localhost.localdomain scylla[1144]: 0x000000000090274d
Nov 16 00:15:45 localhost.localdomain scylla[1144]: 0x00000000004751e4
Nov 16 00:15:45 localhost.localdomain scylla[1144]: 0x00007f10b91eac04
Nov 16 00:15:45 localhost.localdomain scylla[1144]: 0x00000000004e8d9e
Nov 16 00:15:45 localhost.localdomain scylla[1144]:  [shard 0] seastar - Exiting on unhandled exception: locator::bad_property_file_error (std::exception)
Nov 16 00:15:45 localhost.localdomain systemd[1]: scylla-server.service: main process exited, code=dumped, status=6/ABRT
Nov 16 00:15:45 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 16 00:15:45 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 16 00:15:45 localhost.localdomain systemd[1]: scylla-server.service failed.
Nov 16 00:15:46 localhost.localdomain systemd[1]: scylla-server.service holdoff time over, scheduling restart.
Nov 16 00:15:46 localhost.localdomain systemd[1]: Starting Scylla Server...
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: irqbalance is not running
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: Setting a physical interface eth0...
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: Setting mask 00000001 in /proc/irq/19/smp_affinity
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: Setting mask 00000001 in /sys/class/net/eth0/queues/rx-0/rps_cpus
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: Setting net.core.rps_sock_flow_entries to 32768
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: Setting limit 32768 in /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: Trying to enable ntuple filtering HW offload for eth0...not supported
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: Setting mask 00000001 in /sys/class/net/eth0/queues/tx-0/xps_cpus
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: Writing '4096' to /proc/sys/net/core/somaxconn
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: Writing '4096' to /proc/sys/net/ipv4/tcp_max_syn_backlog
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/scheduler
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/nomerges
Nov 16 00:15:46 localhost.localdomain scylla_prepare[1200]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:15:46 localhost.localdomain scylla[1211]: Scylla version 2.0.0-0.20170929.e265c91 starting ...
Nov 16 00:15:46 localhost.localdomain scylla[1211]:  [shard 0] init - Only 476 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
Nov 16 00:15:46 localhost.localdomain scylla[1211]:  [shard 0] snitch_logger - Property file /etc/scylla/cassandra-rackdc.properties is incomplete. Some obligatory fields are missing.
Nov 16 00:15:46 localhost.localdomain scylla[1211]:  [shard 0] snitch_logger - Failed to parse a properties file (/etc/scylla/cassandra-rackdc.properties). Halting...
Nov 16 00:15:46 localhost.localdomain scylla[1211]: scylla: ./seastar/core/sharded.hh:327: seastar::sharded<Service>::~sharded() [with Service = seastar::httpd::http_server]: Assertion `_instances.empty()' failed.
Nov 16 00:15:46 localhost.localdomain scylla[1211]: Aborting on shard 0.
Nov 16 00:15:46 localhost.localdomain scylla[1211]: Backtrace:
Nov 16 00:15:46 localhost.localdomain scylla[1211]: 0x00000000004eef58
Nov 16 00:15:46 localhost.localdomain scylla[1211]: 0x00000000004ef075
Nov 16 00:15:46 localhost.localdomain scylla[1211]: 0x00000000004ef123
Nov 16 00:15:46 localhost.localdomain scylla[1211]: 0x00007f1e3ac355df
Nov 16 00:15:46 localhost.localdomain scylla[1211]: 0x00007f1e3a8981f6
Nov 16 00:15:46 localhost.localdomain scylla[1211]: 0x00007f1e3a8998e7
Nov 16 00:15:46 localhost.localdomain scylla[1211]: 0x00007f1e3a891265
Nov 16 00:15:46 localhost.localdomain scylla[1211]: 0x00007f1e3a891311
Nov 16 00:15:46 localhost.localdomain scylla[1211]: 0x000000000090274d
Nov 16 00:15:46 localhost.localdomain scylla[1211]: 0x00000000004751e4
Nov 16 00:15:46 localhost.localdomain scylla[1211]: 0x00007f1e3a884c04
Nov 16 00:15:46 localhost.localdomain scylla[1211]: 0x00000000004e8d9e
Nov 16 00:15:46 localhost.localdomain scylla[1211]:  [shard 0] seastar - Exiting on unhandled exception: locator::bad_property_file_error (std::exception)
Nov 16 00:15:47 localhost.localdomain systemd[1]: scylla-server.service: main process exited, code=dumped, status=6/ABRT
Nov 16 00:15:47 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 16 00:15:47 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 16 00:15:47 localhost.localdomain systemd[1]: scylla-server.service failed.
Nov 16 00:15:47 localhost.localdomain systemd[1]: scylla-server.service holdoff time over, scheduling restart.
Nov 16 00:15:47 localhost.localdomain systemd[1]: Starting Scylla Server...
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: irqbalance is not running
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: Setting a physical interface eth0...
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: Setting mask 00000001 in /proc/irq/19/smp_affinity
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: Setting mask 00000001 in /sys/class/net/eth0/queues/rx-0/rps_cpus
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: Setting net.core.rps_sock_flow_entries to 32768
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: Setting limit 32768 in /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: Trying to enable ntuple filtering HW offload for eth0...not supported
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: Setting mask 00000001 in /sys/class/net/eth0/queues/tx-0/xps_cpus
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: Writing '4096' to /proc/sys/net/core/somaxconn
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: Writing '4096' to /proc/sys/net/ipv4/tcp_max_syn_backlog
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/scheduler
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/nomerges
Nov 16 00:15:47 localhost.localdomain scylla_prepare[1220]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:15:47 localhost.localdomain scylla[1231]: Scylla version 2.0.0-0.20170929.e265c91 starting ...
Nov 16 00:15:47 localhost.localdomain scylla[1231]:  [shard 0] init - Only 476 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
Nov 16 00:15:47 localhost.localdomain scylla[1231]:  [shard 0] snitch_logger - Property file /etc/scylla/cassandra-rackdc.properties is incomplete. Some obligatory fields are missing.
Nov 16 00:15:47 localhost.localdomain scylla[1231]:  [shard 0] snitch_logger - Failed to parse a properties file (/etc/scylla/cassandra-rackdc.properties). Halting...
Nov 16 00:15:47 localhost.localdomain scylla[1231]: scylla: ./seastar/core/sharded.hh:327: seastar::sharded<Service>::~sharded() [with Service = seastar::httpd::http_server]: Assertion `_instances.empty()' failed.
Nov 16 00:15:47 localhost.localdomain scylla[1231]: Aborting on shard 0.
Nov 16 00:15:47 localhost.localdomain scylla[1231]: Backtrace:
Nov 16 00:15:47 localhost.localdomain scylla[1231]: 0x00000000004eef58
Nov 16 00:15:47 localhost.localdomain scylla[1231]: 0x00000000004ef075
Nov 16 00:15:47 localhost.localdomain scylla[1231]: 0x00000000004ef123
Nov 16 00:15:47 localhost.localdomain scylla[1231]: 0x00007f4fc8b575df
Nov 16 00:15:47 localhost.localdomain scylla[1231]: 0x00007f4fc87ba1f6
Nov 16 00:15:47 localhost.localdomain scylla[1231]: 0x00007f4fc87bb8e7
Nov 16 00:15:47 localhost.localdomain scylla[1231]: 0x00007f4fc87b3265
Nov 16 00:15:47 localhost.localdomain scylla[1231]: 0x00007f4fc87b3311
Nov 16 00:15:47 localhost.localdomain scylla[1231]: 0x000000000090274d
Nov 16 00:15:47 localhost.localdomain scylla[1231]: 0x00000000004751e4
Nov 16 00:15:47 localhost.localdomain scylla[1231]: 0x00007f4fc87a6c04
Nov 16 00:15:47 localhost.localdomain scylla[1231]: 0x00000000004e8d9e
Nov 16 00:15:47 localhost.localdomain scylla[1231]:  [shard 0] seastar - Exiting on unhandled exception: locator::bad_property_file_error (std::exception)
Nov 16 00:15:48 localhost.localdomain systemd[1]: scylla-server.service: main process exited, code=dumped, status=6/ABRT
Nov 16 00:15:48 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 16 00:15:48 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 16 00:15:48 localhost.localdomain systemd[1]: scylla-server.service failed.
Nov 16 00:15:48 localhost.localdomain systemd[1]: scylla-server.service holdoff time over, scheduling restart.
Nov 16 00:15:48 localhost.localdomain systemd[1]: Starting Scylla Server...
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: irqbalance is not running
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: Setting a physical interface eth0...
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: Setting mask 00000001 in /proc/irq/19/smp_affinity
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: Setting mask 00000001 in /sys/class/net/eth0/queues/rx-0/rps_cpus
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: Setting net.core.rps_sock_flow_entries to 32768
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: Setting limit 32768 in /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: Trying to enable ntuple filtering HW offload for eth0...not supported
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: Setting mask 00000001 in /sys/class/net/eth0/queues/tx-0/xps_cpus
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: Writing '4096' to /proc/sys/net/core/somaxconn
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: Writing '4096' to /proc/sys/net/ipv4/tcp_max_syn_backlog
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/scheduler
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/nomerges
Nov 16 00:15:49 localhost.localdomain scylla_prepare[1240]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:15:49 localhost.localdomain scylla[1251]: Scylla version 2.0.0-0.20170929.e265c91 starting ...
Nov 16 00:15:49 localhost.localdomain scylla[1251]:  [shard 0] init - Only 476 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
Nov 16 00:15:49 localhost.localdomain scylla[1251]:  [shard 0] snitch_logger - Property file /etc/scylla/cassandra-rackdc.properties is incomplete. Some obligatory fields are missing.
Nov 16 00:15:49 localhost.localdomain scylla[1251]:  [shard 0] snitch_logger - Failed to parse a properties file (/etc/scylla/cassandra-rackdc.properties). Halting...
Nov 16 00:15:49 localhost.localdomain scylla[1251]:  [shard 0] seastar - Exiting on unhandled exception: locator::bad_property_file_error (std::exception)
Nov 16 00:15:49 localhost.localdomain scylla[1251]: scylla: ./seastar/core/sharded.hh:327: seastar::sharded<Service>::~sharded() [with Service = seastar::httpd::http_server]: Assertion `_instances.empty()' failed.
Nov 16 00:15:49 localhost.localdomain scylla[1251]: Aborting on shard 0.
Nov 16 00:15:49 localhost.localdomain scylla[1251]: Backtrace:
Nov 16 00:15:49 localhost.localdomain scylla[1251]: 0x00000000004eef58
Nov 16 00:15:49 localhost.localdomain scylla[1251]: 0x00000000004ef075
Nov 16 00:15:49 localhost.localdomain scylla[1251]: 0x00000000004ef123
Nov 16 00:15:49 localhost.localdomain scylla[1251]: 0x00007fae5a07f5df
Nov 16 00:15:49 localhost.localdomain scylla[1251]: 0x00007fae59ce21f6
Nov 16 00:15:49 localhost.localdomain scylla[1251]: 0x00007fae59ce38e7
Nov 16 00:15:49 localhost.localdomain scylla[1251]: 0x00007fae59cdb265
Nov 16 00:15:49 localhost.localdomain scylla[1251]: 0x00007fae59cdb311
Nov 16 00:15:49 localhost.localdomain scylla[1251]: 0x000000000090274d
Nov 16 00:15:49 localhost.localdomain scylla[1251]: 0x00000000004751e4
Nov 16 00:15:49 localhost.localdomain scylla[1251]: 0x00007fae59ccec04
Nov 16 00:15:49 localhost.localdomain scylla[1251]: 0x00000000004e8d9e
Nov 16 00:15:49 localhost.localdomain systemd[1]: scylla-server.service: main process exited, code=dumped, status=6/ABRT
Nov 16 00:15:49 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 16 00:15:49 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 16 00:15:49 localhost.localdomain systemd[1]: scylla-server.service failed.
Nov 16 00:15:51 localhost.localdomain systemd[1]: scylla-server.service holdoff time over, scheduling restart.
Nov 16 00:15:51 localhost.localdomain systemd[1]: start request repeated too quickly for scylla-server.service
Nov 16 00:15:51 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 16 00:15:51 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 16 00:15:51 localhost.localdomain systemd[1]: scylla-server.service failed.
Nov 16 00:16:56 localhost.localdomain systemd[1]: Starting Scylla Server...
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: irqbalance is not running
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: Setting a physical interface eth0...
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: Setting mask 00000001 in /proc/irq/19/smp_affinity
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: Setting mask 00000001 in /sys/class/net/eth0/queues/rx-0/rps_cpus
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: Setting net.core.rps_sock_flow_entries to 32768
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: Setting limit 32768 in /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: Trying to enable ntuple filtering HW offload for eth0...not supported
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: Setting mask 00000001 in /sys/class/net/eth0/queues/tx-0/xps_cpus
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: Writing '4096' to /proc/sys/net/core/somaxconn
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: Writing '4096' to /proc/sys/net/ipv4/tcp_max_syn_backlog
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/scheduler
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/nomerges
Nov 16 00:16:56 localhost.localdomain scylla_prepare[1290]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:16:56 localhost.localdomain scylla[1301]: Scylla version 2.0.0-0.20170929.e265c91 starting ...
Nov 16 00:16:57 localhost.localdomain scylla[1301]:  [shard 0] init - Only 476 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
Nov 16 00:16:57 localhost.localdomain scylla[1301]:  [shard 0] snitch_logger - Property file /etc/scylla/cassandra-rackdc.properties is incomplete. Some obligatory fields are missing.
Nov 16 00:16:57 localhost.localdomain scylla[1301]:  [shard 0] snitch_logger - Failed to parse a properties file (/etc/scylla/cassandra-rackdc.properties). Halting...
Nov 16 00:16:57 localhost.localdomain scylla[1301]:  [shard 0] seastar - Exiting on unhandled exception: locator::bad_property_file_error (std::exception)
Nov 16 00:16:57 localhost.localdomain scylla[1301]: scylla: ./seastar/core/sharded.hh:327: seastar::sharded<Service>::~sharded() [with Service = seastar::httpd::http_server]: Assertion `_instances.empty()' failed.
Nov 16 00:16:57 localhost.localdomain scylla[1301]: Aborting on shard 0.
Nov 16 00:16:57 localhost.localdomain scylla[1301]: Backtrace:
Nov 16 00:16:57 localhost.localdomain scylla[1301]: 0x00000000004eef58
Nov 16 00:16:57 localhost.localdomain scylla[1301]: 0x00000000004ef075
Nov 16 00:16:57 localhost.localdomain scylla[1301]: 0x00000000004ef123
Nov 16 00:16:57 localhost.localdomain scylla[1301]: 0x00007f3f447975df
Nov 16 00:16:57 localhost.localdomain scylla[1301]: 0x00007f3f443fa1f6
Nov 16 00:16:57 localhost.localdomain scylla[1301]: 0x00007f3f443fb8e7
Nov 16 00:16:57 localhost.localdomain scylla[1301]: 0x00007f3f443f3265
Nov 16 00:16:57 localhost.localdomain scylla[1301]: 0x00007f3f443f3311
Nov 16 00:16:57 localhost.localdomain scylla[1301]: 0x000000000090274d
Nov 16 00:16:57 localhost.localdomain scylla[1301]: 0x00000000004751e4
Nov 16 00:16:57 localhost.localdomain scylla[1301]: 0x00007f3f443e6c04
Nov 16 00:16:57 localhost.localdomain scylla[1301]: 0x00000000004e8d9e
Nov 16 00:16:57 localhost.localdomain systemd[1]: scylla-server.service: main process exited, code=dumped, status=6/ABRT
Nov 16 00:16:57 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 16 00:16:57 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 16 00:16:57 localhost.localdomain systemd[1]: scylla-server.service failed.
Nov 16 00:16:57 localhost.localdomain systemd[1]: scylla-server.service holdoff time over, scheduling restart.
Nov 16 00:16:57 localhost.localdomain systemd[1]: Starting Scylla Server...
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: irqbalance is not running
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: Setting a physical interface eth0...
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: Setting mask 00000001 in /proc/irq/19/smp_affinity
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: Setting mask 00000001 in /sys/class/net/eth0/queues/rx-0/rps_cpus
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: Setting net.core.rps_sock_flow_entries to 32768
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: Setting limit 32768 in /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: Trying to enable ntuple filtering HW offload for eth0...not supported
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: Setting mask 00000001 in /sys/class/net/eth0/queues/tx-0/xps_cpus
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: Writing '4096' to /proc/sys/net/core/somaxconn
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: Writing '4096' to /proc/sys/net/ipv4/tcp_max_syn_backlog
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/scheduler
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/nomerges
Nov 16 00:16:57 localhost.localdomain scylla_prepare[1310]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:16:57 localhost.localdomain scylla[1321]: Scylla version 2.0.0-0.20170929.e265c91 starting ...
Nov 16 00:16:57 localhost.localdomain scylla[1321]:  [shard 0] init - Only 476 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
Nov 16 00:16:57 localhost.localdomain scylla[1321]:  [shard 0] snitch_logger - Property file /etc/scylla/cassandra-rackdc.properties is incomplete. Some obligatory fields are missing.
Nov 16 00:16:57 localhost.localdomain scylla[1321]:  [shard 0] snitch_logger - Failed to parse a properties file (/etc/scylla/cassandra-rackdc.properties). Halting...
Nov 16 00:16:57 localhost.localdomain scylla[1321]: scylla: ./seastar/core/sharded.hh:327: seastar::sharded<Service>::~sharded() [with Service = seastar::httpd::http_server]: Assertion `_instances.empty()' failed.
Nov 16 00:16:57 localhost.localdomain scylla[1321]: Aborting on shard 0.
Nov 16 00:16:57 localhost.localdomain scylla[1321]: Backtrace:
Nov 16 00:16:57 localhost.localdomain scylla[1321]: 0x00000000004eef58
Nov 16 00:16:57 localhost.localdomain scylla[1321]: 0x00000000004ef075
Nov 16 00:16:57 localhost.localdomain scylla[1321]: 0x00000000004ef123
Nov 16 00:16:57 localhost.localdomain scylla[1321]: 0x00007f13909195df
Nov 16 00:16:57 localhost.localdomain scylla[1321]: 0x00007f139057c1f6
Nov 16 00:16:57 localhost.localdomain scylla[1321]: 0x00007f139057d8e7
Nov 16 00:16:57 localhost.localdomain scylla[1321]: 0x00007f1390575265
Nov 16 00:16:57 localhost.localdomain scylla[1321]: 0x00007f1390575311
Nov 16 00:16:57 localhost.localdomain scylla[1321]: 0x000000000090274d
Nov 16 00:16:57 localhost.localdomain scylla[1321]: 0x00000000004751e4
Nov 16 00:16:57 localhost.localdomain scylla[1321]: 0x00007f1390568c04
Nov 16 00:16:57 localhost.localdomain scylla[1321]: 0x00000000004e8d9e
Nov 16 00:16:57 localhost.localdomain scylla[1321]:  [shard 0] seastar - Exiting on unhandled exception: locator::bad_property_file_error (std::exception)
Nov 16 00:16:58 localhost.localdomain systemd[1]: scylla-server.service: main process exited, code=dumped, status=6/ABRT
Nov 16 00:16:58 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 16 00:16:58 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 16 00:16:58 localhost.localdomain systemd[1]: scylla-server.service failed.
Nov 16 00:16:58 localhost.localdomain systemd[1]: scylla-server.service holdoff time over, scheduling restart.
Nov 16 00:16:58 localhost.localdomain systemd[1]: Starting Scylla Server...
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: irqbalance is not running
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: Setting a physical interface eth0...
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: Setting mask 00000001 in /proc/irq/19/smp_affinity
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: Setting mask 00000001 in /sys/class/net/eth0/queues/rx-0/rps_cpus
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: Setting net.core.rps_sock_flow_entries to 32768
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: Setting limit 32768 in /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: Trying to enable ntuple filtering HW offload for eth0...not supported
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: Setting mask 00000001 in /sys/class/net/eth0/queues/tx-0/xps_cpus
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: Writing '4096' to /proc/sys/net/core/somaxconn
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: Writing '4096' to /proc/sys/net/ipv4/tcp_max_syn_backlog
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/scheduler
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/nomerges
Nov 16 00:16:58 localhost.localdomain scylla_prepare[1330]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:16:58 localhost.localdomain scylla[1341]: Scylla version 2.0.0-0.20170929.e265c91 starting ...
Nov 16 00:16:58 localhost.localdomain scylla[1341]:  [shard 0] init - Only 476 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
Nov 16 00:16:58 localhost.localdomain scylla[1341]:  [shard 0] snitch_logger - Property file /etc/scylla/cassandra-rackdc.properties is incomplete. Some obligatory fields are missing.
Nov 16 00:16:58 localhost.localdomain scylla[1341]:  [shard 0] snitch_logger - Failed to parse a properties file (/etc/scylla/cassandra-rackdc.properties). Halting...
Nov 16 00:16:58 localhost.localdomain scylla[1341]: scylla: ./seastar/core/sharded.hh:327: seastar::sharded<Service>::~sharded() [with Service = seastar::httpd::http_server]: Assertion `_instances.empty()' failed.
Nov 16 00:16:58 localhost.localdomain scylla[1341]: Aborting on shard 0.
Nov 16 00:16:58 localhost.localdomain scylla[1341]: Backtrace:
Nov 16 00:16:58 localhost.localdomain scylla[1341]: 0x00000000004eef58
Nov 16 00:16:58 localhost.localdomain scylla[1341]: 0x00000000004ef075
Nov 16 00:16:58 localhost.localdomain scylla[1341]: 0x00000000004ef123
Nov 16 00:16:58 localhost.localdomain scylla[1341]: 0x00007fde446b15df
Nov 16 00:16:58 localhost.localdomain scylla[1341]: 0x00007fde443141f6
Nov 16 00:16:58 localhost.localdomain scylla[1341]: 0x00007fde443158e7
Nov 16 00:16:58 localhost.localdomain scylla[1341]: 0x00007fde4430d265
Nov 16 00:16:58 localhost.localdomain scylla[1341]: 0x00007fde4430d311
Nov 16 00:16:58 localhost.localdomain scylla[1341]: 0x000000000090274d
Nov 16 00:16:58 localhost.localdomain scylla[1341]: 0x00000000004751e4
Nov 16 00:16:58 localhost.localdomain scylla[1341]: 0x00007fde44300c04
Nov 16 00:16:58 localhost.localdomain scylla[1341]: 0x00000000004e8d9e
Nov 16 00:16:58 localhost.localdomain scylla[1341]:  [shard 0] seastar - Exiting on unhandled exception: locator::bad_property_file_error (std::exception)
Nov 16 00:16:59 localhost.localdomain systemd[1]: scylla-server.service: main process exited, code=dumped, status=6/ABRT
Nov 16 00:16:59 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 16 00:16:59 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 16 00:16:59 localhost.localdomain systemd[1]: scylla-server.service failed.
Nov 16 00:16:59 localhost.localdomain systemd[1]: scylla-server.service holdoff time over, scheduling restart.
Nov 16 00:16:59 localhost.localdomain systemd[1]: Starting Scylla Server...
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: irqbalance is not running
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: Setting a physical interface eth0...
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: Setting mask 00000001 in /proc/irq/19/smp_affinity
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: Setting mask 00000001 in /sys/class/net/eth0/queues/rx-0/rps_cpus
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: Setting net.core.rps_sock_flow_entries to 32768
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: Setting limit 32768 in /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: Trying to enable ntuple filtering HW offload for eth0...not supported
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: Setting mask 00000001 in /sys/class/net/eth0/queues/tx-0/xps_cpus
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: Writing '4096' to /proc/sys/net/core/somaxconn
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: Writing '4096' to /proc/sys/net/ipv4/tcp_max_syn_backlog
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/scheduler
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/nomerges
Nov 16 00:16:59 localhost.localdomain scylla_prepare[1350]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:17:00 localhost.localdomain scylla[1361]: Scylla version 2.0.0-0.20170929.e265c91 starting ...
Nov 16 00:17:00 localhost.localdomain scylla[1361]:  [shard 0] init - Only 476 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
Nov 16 00:17:00 localhost.localdomain scylla[1361]:  [shard 0] snitch_logger - Property file /etc/scylla/cassandra-rackdc.properties is incomplete. Some obligatory fields are missing.
Nov 16 00:17:00 localhost.localdomain scylla[1361]:  [shard 0] snitch_logger - Failed to parse a properties file (/etc/scylla/cassandra-rackdc.properties). Halting...
Nov 16 00:17:00 localhost.localdomain scylla[1361]: scylla: ./seastar/core/sharded.hh:327: seastar::sharded<Service>::~sharded() [with Service = seastar::httpd::http_server]: Assertion `_instances.empty()' failed.
Nov 16 00:17:00 localhost.localdomain scylla[1361]: Aborting on shard 0.
Nov 16 00:17:00 localhost.localdomain scylla[1361]: Backtrace:
Nov 16 00:17:00 localhost.localdomain scylla[1361]: 0x00000000004eef58
Nov 16 00:17:00 localhost.localdomain scylla[1361]: 0x00000000004ef075
Nov 16 00:17:00 localhost.localdomain scylla[1361]: 0x00000000004ef123
Nov 16 00:17:00 localhost.localdomain scylla[1361]: 0x00007f749ceb05df
Nov 16 00:17:00 localhost.localdomain scylla[1361]: 0x00007f749cb131f6
Nov 16 00:17:00 localhost.localdomain scylla[1361]: 0x00007f749cb148e7
Nov 16 00:17:00 localhost.localdomain scylla[1361]: 0x00007f749cb0c265
Nov 16 00:17:00 localhost.localdomain scylla[1361]: 0x00007f749cb0c311
Nov 16 00:17:00 localhost.localdomain scylla[1361]: 0x000000000090274d
Nov 16 00:17:00 localhost.localdomain scylla[1361]: 0x00000000004751e4
Nov 16 00:17:00 localhost.localdomain scylla[1361]: 0x00007f749caffc04
Nov 16 00:17:00 localhost.localdomain scylla[1361]: 0x00000000004e8d9e
Nov 16 00:17:00 localhost.localdomain scylla[1361]:  [shard 0] seastar - Exiting on unhandled exception: locator::bad_property_file_error (std::exception)
Nov 16 00:17:00 localhost.localdomain systemd[1]: scylla-server.service: main process exited, code=dumped, status=6/ABRT
Nov 16 00:17:00 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 16 00:17:00 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 16 00:17:00 localhost.localdomain systemd[1]: scylla-server.service failed.
Nov 16 00:17:00 localhost.localdomain systemd[1]: scylla-server.service holdoff time over, scheduling restart.
Nov 16 00:17:00 localhost.localdomain systemd[1]: Starting Scylla Server...
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: irqbalance is not running
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: Setting a physical interface eth0...
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: Setting mask 00000001 in /proc/irq/19/smp_affinity
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: Setting mask 00000001 in /sys/class/net/eth0/queues/rx-0/rps_cpus
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: Setting net.core.rps_sock_flow_entries to 32768
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: Setting limit 32768 in /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: Trying to enable ntuple filtering HW offload for eth0...not supported
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: Setting mask 00000001 in /sys/class/net/eth0/queues/tx-0/xps_cpus
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: Writing '4096' to /proc/sys/net/core/somaxconn
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: Writing '4096' to /proc/sys/net/ipv4/tcp_max_syn_backlog
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/scheduler
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: already tuned: /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb/queue/nomerges
Nov 16 00:17:00 localhost.localdomain scylla_prepare[1370]: tuning /sys/devices/pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
Nov 16 00:17:00 localhost.localdomain scylla[1381]: Scylla version 2.0.0-0.20170929.e265c91 starting ...
Nov 16 00:17:00 localhost.localdomain scylla[1381]:  [shard 0] init - Only 476 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
Nov 16 00:17:00 localhost.localdomain scylla[1381]:  [shard 0] snitch_logger - Property file /etc/scylla/cassandra-rackdc.properties is incomplete. Some obligatory fields are missing.
Nov 16 00:17:00 localhost.localdomain scylla[1381]:  [shard 0] snitch_logger - Failed to parse a properties file (/etc/scylla/cassandra-rackdc.properties). Halting...
Nov 16 00:17:00 localhost.localdomain scylla[1381]: scylla: ./seastar/core/sharded.hh:327: seastar::sharded<Service>::~sharded() [with Service = seastar::httpd::http_server]: Assertion `_instances.empty()' failed.
Nov 16 00:17:00 localhost.localdomain scylla[1381]: Aborting on shard 0.
Nov 16 00:17:00 localhost.localdomain scylla[1381]: Backtrace:
Nov 16 00:17:00 localhost.localdomain scylla[1381]: 0x00000000004eef58
Nov 16 00:17:00 localhost.localdomain scylla[1381]: 0x00000000004ef075
Nov 16 00:17:00 localhost.localdomain scylla[1381]: 0x00000000004ef123
Nov 16 00:17:00 localhost.localdomain scylla[1381]: 0x00007f81e11685df
Nov 16 00:17:00 localhost.localdomain scylla[1381]: 0x00007f81e0dcb1f6
Nov 16 00:17:00 localhost.localdomain scylla[1381]: 0x00007f81e0dcc8e7
Nov 16 00:17:00 localhost.localdomain scylla[1381]: 0x00007f81e0dc4265
Nov 16 00:17:00 localhost.localdomain scylla[1381]: 0x00007f81e0dc4311
Nov 16 00:17:00 localhost.localdomain scylla[1381]: 0x000000000090274d
Nov 16 00:17:00 localhost.localdomain scylla[1381]: 0x00000000004751e4
Nov 16 00:17:00 localhost.localdomain scylla[1381]: 0x00007f81e0db7c04
Nov 16 00:17:00 localhost.localdomain scylla[1381]: 0x00000000004e8d9e
Nov 16 00:17:00 localhost.localdomain scylla[1381]:  [shard 0] seastar - Exiting on unhandled exception: locator::bad_property_file_error (std::exception)
Nov 16 00:17:01 localhost.localdomain systemd[1]: scylla-server.service: main process exited, code=dumped, status=6/ABRT
Nov 16 00:17:01 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 16 00:17:01 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 16 00:17:01 localhost.localdomain systemd[1]: scylla-server.service failed.
Nov 16 00:17:01 localhost.localdomain systemd[1]: scylla-server.service holdoff time over, scheduling restart.
Nov 16 00:17:01 localhost.localdomain systemd[1]: start request repeated too quickly for scylla-server.service
Nov 16 00:17:01 localhost.localdomain systemd[1]: Failed to start Scylla Server.
Nov 16 00:17:01 localhost.localdomain systemd[1]: Unit scylla-server.service entered failed state.
Nov 16 00:17:01 localhost.localdomain systemd[1]: scylla-server.service failed.

Alexandr Porunov

<alexandr.porunov@gmail.com>
unread,
Nov 15, 2017, 5:30:11 PM11/15/17
to ScyllaDB users
Ow not. My mistake. I didn't read those logs. Now I see the problem is that I have deleted /etc/scylla/ when I was re-installing the whole cluster but I forgot to clone /etc/scylla/cassandra-rackdc.properties
Sorry for this stupid mistake. Thank you all for helping me! I appreciate it.

Best regards,
Alexandr


On Thursday, November 16, 2017 at 12:16:51 AM UTC+2, Glauber Costa wrote:

Tzach Livyatan

<tzach@scylladb.com>
unread,
Nov 16, 2017, 3:24:04 AM11/16/17
to ScyllaDB users
Hi Alexander
Thanks for letting us know. Glad the problem was solved.

Cheers
Tzach

To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages