Help me launch a test cluster

392 views
Skip to first unread message

Alex

<shestakov.am@gmail.com>
unread,
May 22, 2021, 2:59:43 PM5/22/21
to ScyllaDB users
Hello everyone,

Does anyone know what the problem is with the launch?

I have two servers:

192.168.232.100 (all ports is opened, firewall inactive):
cluster_name: 'CC Cluster'
num_tokens: 256
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "192.168.232.100"
listen_address: 192.168.232.100
broadcast_address: 192.168.232.100
native_transport_port: 9042
native_shard_aware_transport_port: 19042
#native_transport_port_ssl: 9142
#native_shard_aware_transport_port_ssl: 19142
read_request_timeout_in_ms: 5000
write_request_timeout_in_ms: 2000
cas_contention_timeout_in_ms: 1000
endpoint_snitch: GossipingPropertyFileSnitch
rpc_address: 192.168.232.100
rpc_port: 9160
api_port: 10000
api_address: 192.168.232.100
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
broadcast_rpc_address: 192.168.232.100
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
commitlog_total_space_in_mb: -1
murmur3_partitioner_ignore_msb_bits: 12
api_ui_dir: /opt/scylladb/swagger-ui/dist/
api_doc_dir: /opt/scylladb/api/api-doc/

dc=main_data_center
rack=rack11

Datacenter: main_data_center
============================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load       Tokens       Owns    Host ID                               Rack
UN  192.168.232.100  172.64 KB  256          ?       48b094fa-544b-46b1-9be2-2c2be8a3cec3  rack11

192.168.232.101 (all ports is opened, firewall inactive):

cluster_name: 'CC Cluster'
num_tokens: 256
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "192.168.232.100"
listen_address: 192.168.232.101
broadcast_address: 192.168.232.101
native_transport_port: 9042
native_shard_aware_transport_port: 19042
#native_transport_port_ssl: 9142
#native_shard_aware_transport_port_ssl: 19142
read_request_timeout_in_ms: 5000
write_request_timeout_in_ms: 2000
cas_contention_timeout_in_ms: 1000
endpoint_snitch: GossipingPropertyFileSnitch
rpc_address: 192.168.232.101
rpc_port: 9160
api_port: 10000
api_address: 192.168.232.101
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
broadcast_rpc_address: 192.168.232.101
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
commitlog_total_space_in_mb: -1
murmur3_partitioner_ignore_msb_bits: 12
api_ui_dir: /opt/scylladb/swagger-ui/dist/
api_doc_dir: /opt/scylladb/api/api-doc/

when I started server 192.168.232.101:

Cluster Information:
        Name: CC Cluster
        Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch
        DynamicEndPointSnitch: disabled
        Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
        Schema versions:
                3e82e237-010d-387d-bc99-6a9cae0cf76e: [192.168.232.100]
                59adb24e-f3cd-3e02-97f0-5b395827453f: [192.168.232.101]

Log file:
May 22 18:39:43 scylla2 scylla[73125]: Scylla version 4.4.1-0.20210406.00da6b5e9 with build-id 49017ad539e40dba20fa9b82837090165772bf66 starting ...
May 22 18:39:44 scylla2 scylla[73125]: command used: "/usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --developer-mode=1 --lock-memory=1"
May 22 18:39:44 scylla2 scylla[73125]: parsed command line options: [log-to-syslog: 1, log-to-stdout: 0, default-log-level: info, network-stack: posix, developer-mode: 1, lock-memory: 1]
May 22 18:39:44 scylla2 scylla[73125]:  [shard 0] init - installing SIGHUP handler
May 22 18:39:44 scylla2 scylla[73125]:  [shard 0] init - Scylla version 4.4.1-0.20210406.00da6b5e9 with build-id 49017ad539e40dba20fa9b82837090165772bf66 starting ...
May 22 18:39:44 scylla2 scylla[73125]:  [shard 0] init - Only 476 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
May 22 18:39:44 scylla2 scylla[73125]:  [shard 0] init - starting prometheus API server
May 22 18:39:44 scylla2 scylla[73125]:  [shard 0] init - starting tokens manager
May 22 18:39:44 scylla2 scylla[73125]:  [shard 0] init - starting migration manager notifier
May 22 18:39:44 scylla2 scylla[73125]:  [shard 0] init - creating tracing
May 22 18:39:44 scylla2 scylla[73125]:  [shard 0] init - creating snitch
May 22 18:39:44 scylla2 scylla[73125]:  [shard 0] init - determining DNS name
May 22 18:39:44 scylla2 scylla[73125]:  [shard 0] init - starting API server
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] init - Scylla API server listening on 192.168.232.101:10000 ...
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] init - initializing storage service
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] init - starting per-shard database core
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] init - I/O Scheduler is not properly configured! This is a non-supported setup, and performance is expected to be unpredictably bad.
                                        Reason found: none of --max-io-requests, --io-properties and --io-properties-file are set.
                                       To properly configure the I/O Scheduler, run the scylla_io_setup utility shipped with Scylla.
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] init - creating and verifying directories
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Populating Keyspace system_schema
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Keyspace system_schema: Reading CF view_virtual_columns id=08843b63-45dc-3be2-9798-a0418295cfaa version=c777531c-15f7-326f-8ebe-39fd0265c8c9
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Keyspace system_schema: Reading CF computed_columns id=cc7c7069-3740-33c1-92a4-c3de78dbd2c4 version=2b8c4439-de76-31e0-807f-3b7290a975d7
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Keyspace system_schema: Reading CF aggregates id=924c5587-2e3a-345b-b10c-12f37c1ba895 version=4b53e92c-0368-3d5c-b959-2ec1bfd1a59f
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Keyspace system_schema: Reading CF functions id=96489b79-80be-3e14-a701-66a0b9159450 version=329ed804-55b3-3eee-ad61-d85317b96097
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Keyspace system_schema: Reading CF views id=9786ac1c-dd58-3201-a7cd-ad556410c985 version=5b58bb47-96e7-3f57-accf-0bfca4dbbc6e
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Keyspace system_schema: Reading CF indexes id=0feb57ac-311f-382f-ba6d-9024d305702f version=99c40462-8687-304e-abe3-2bdbef1f25aa
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Keyspace system_schema: Reading CF types id=5a8b1ca8-6602-3f77-a045-9273d308917a version=de51b2ce-5e4d-3b7d-a75f-2204332ce8d1
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Keyspace system_schema: Reading CF triggers id=4df70b66-6b05-3251-95a1-32b54005fd48 version=582d7071-1ef0-37c8-adc6-471a13636139
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Keyspace system_schema: Reading CF dropped_columns id=5e7583b5-f3f4-3af1-9a39-b7e1d6f5f11f version=7426bc6c-4c2f-3200-8ad8-4329610ed59a
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Keyspace system_schema: Reading CF columns id=24101c25-a2ae-3af7-87c1-b40ee1aca33f version=d33236d4-9bdd-3c09-abf0-a0bc5edc2526
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Keyspace system_schema: Reading CF scylla_tables id=5d912ff1-f759-3665-b2c8-8042ab5103dd version=38e5e56b-155d-39b5-a0d2-8e5dcad42a26
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Keyspace system_schema: Reading CF tables id=afddfb9d-bc1e-3068-8056-eed6c302ba09 version=b6240810-eeb7-36d5-9411-43b2d68dddab
May 22 18:39:45 scylla2 scylla[73125]:  [shard 0] database - Keyspace system_schema: Reading CF keyspaces id=abac5682-dea6-31c5-b535-b3d6cffd0fb6 version=e79ca8ba-6556-3f7d-925a-7f20cf57938c
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Populating Keyspace system
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF compaction_history id=b4dbb7b4-dc49-3fb5-b3bf-ce6e434832ca version=25cae56a-8d75-39f3-a146-3756ab4981c7
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF batchlog id=0290003c-977e-397c-ac3e-fdfdc01d626b version=e2a2e804-49e4-3597-9f16-39fd9475835c
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF size_estimates id=618f817b-005f-3678-b8a4-53f3930b8e86 version=6d4dbcff-f05b-3dc3-95ad-f79f7b10504d
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF range_xfers id=55d76438-4e55-3f8b-9f6e-676d4af3976d version=dd15a078-409b-350d-9bef-c5f3520832d8
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF peer_events id=59dfeaea-8db2-3341-91ef-109974d81484 version=7f874a05-72b8-3c21-9acb-baa164fc351a
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF peers id=37f71aca-7dc2-383b-a706-72528af04d4f version=f6f6871f-8c86-3eca-ac0b-2a2e848e395d
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF scylla_local id=2972ec7f-fb20-38dd-aac1-d876f2e3fcbd version=5f0b407d-eedf-3845-a48e-dbb9673d10e1
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF schema_aggregates id=a5fc57fc-9d6c-3bfd-a3fc-01ad54686fea version=d06c934f-9a69-374c-86f1-523ab26fb05c
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF paxos id=b7b7f0c2-fd0a-3410-8c05-3ef614bb7c2d version=6dd372de-72fb-3e1b-9b8c-f97738a67fe9
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF compactions_in_progress id=55080ab0-5d9c-3886-90a4-acb25fe1f77b version=540263f1-40db-3869-8a38-3baadedc222d
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF local id=7ad54392-bcdd-35a6-8417-4e047860b377 version=7fa82c2e-5b67-37dd-8e5e-2079e18f1536
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF schema_columnfamilies id=45f5b360-24bc-3f83-a363-1034ea4fa697 version=7f7a5fea-07b7-304d-a6f8-f7aec72cabdb
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF hints id=2666e205-73ef-38b3-90fe-fecf96e8f0c7 version=0b74fdd1-e96d-309e-a14e-a5bcd7ac885d
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF sstable_activity id=5a1ff267-ace0-3f12-8563-cfae6103c65e version=bdec57a3-b234-334b-be9c-7b1f33113995
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF IndexInfo id=9f5c6374-d485-3229-9a0a-5094af9ad1e3 version=bbb3743b-351f-3023-b4fc-09a9be37d529
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF large_partitions id=8a7fe624-96b0-34b1-b90e-f71bddcdd2d3 version=04fa9920-9369-3a96-be39-6dd9fdc816b6
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF scylla_table_schema_history id=0191a53e-40f0-31d4-9171-b0d19ffb17b4 version=2cc890b8-9ade-3afc-ab63-043c8017608e
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF large_cells id=ead8bbc5-f146-3ae1-9f71-0b11f9a1d296 version=0d4a6937-a781-3f87-a192-b4e4a4a40acf
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF views_builds_in_progress id=b7f2c108-78cd-3c80-9cd5-d609b2bd149c version=b905cce5-1474-3085-819a-7592453e2fb9
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF built_views id=4b3c50a9-ea87-3d76-9101-6dbc9c38494a version=a28aa0b3-6def-30a7-9fbe-ce78b3f3c9b9
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF scylla_views_builds_in_progress id=a04c7bfd-1e13-36c9-a44d-f22da352281d version=e3fb736c-6956-3990-a31d-9a482279e3fc
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF truncated id=38c19fd0-fb86-3310-a4b7-0d0cc66628aa version=6da9a85c-7ae0-3917-aead-54a2f65a57a8
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF cdc_local id=0bcaffd4-0c83-3ead-ad13-dc1d5015b77c version=0aa4d3a2-ed95-3ecd-aba0-cd75622ad290
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF clients id=ca0f635d-8630-3609-8d93-a1fc06f2a5e5 version=51273ebf-e6f9-3cd5-b455-ea09ad29796f
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF schema_keyspaces id=b0f22357-4458-3cdb-9631-c43e59ce3676 version=84d28cef-59f9-34a9-9be5-31c684997f03
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF schema_columns id=296e9c04-9bec-3085-827d-c17d3df2122a version=53cfbb66-ee0d-37be-97bb-b2afb3746f85
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF schema_triggers id=0359bc71-7123-3ee1-9a4a-b9dfb11fc125 version=75e4ec4c-76da-3f59-984e-66810ba0c62b
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF large_rows id=40550f66-0858-39a0-9430-f27fc08034e9 version=aee3acb0-7926-317b-848e-db7bc3721695
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF schema_usertypes id=3aa75225-4f82-350b-8d5c-430fa221fa0a version=3c53e6bb-56d5-3618-a0b2-4ff5f18127d7
May 22 18:39:46 scylla2 scylla[73125]:  [shard 0] database - Keyspace system: Reading CF schema_functions id=d1b675fe-2b50-3ca4-8e49-c0f81989dcad version=a50959dd-2be0-3e4e-8bb5-cf40796e256a
May 22 18:39:47 scylla2 scylla[73125]:  [shard 0] init - starting gossip
May 22 18:39:47 scylla2 scylla[73125]:  [shard 0] init - seeds={192.168.232.100}, listen_address=192.168.232.101, broadcast_address=192.168.232.101
May 22 18:39:47 scylla2 scylla[73125]:  [shard 0] init - starting storage proxy
May 22 18:39:47 scylla2 scylla[73125]:  [shard 0] init - starting migration manager
May 22 18:39:47 scylla2 scylla[73125]:  [shard 0] init - starting query processor
May 22 18:39:47 scylla2 scylla[73125]:  [shard 0] init - initializing batchlog manager
May 22 18:39:48 scylla2 scylla[73125]:  [shard 0] legacy_schema_migrator - Moving 0 keyspaces from legacy schema tables to the new schema keyspace (system_schema)
May 22 18:39:48 scylla2 scylla[73125]:  [shard 0] legacy_schema_migrator - Dropping legacy schema tables
May 22 18:39:49 scylla2 scylla[73125]:  [shard 0] legacy_schema_migrator - Completed migration of legacy schema tables
May 22 18:39:49 scylla2 scylla[73125]:  [shard 0] init - loading system sstables
May 22 18:39:49 scylla2 scylla[73125]:  [shard 0] init - loading non-system sstables
May 22 18:39:49 scylla2 scylla[73125]:  [shard 0] init - starting view update generator
May 22 18:39:49 scylla2 scylla[73125]:  [shard 0] init - discovering staging sstables
May 22 18:39:49 scylla2 scylla[73125]:  [shard 0] init - setting up system keyspace
May 22 18:39:50 scylla2 scylla[73125]:  [shard 0] init - starting commit log
May 22 18:39:50 scylla2 scylla[73125]:  [shard 0] init - initializing migration manager RPC verbs
May 22 18:39:50 scylla2 scylla[73125]:  [shard 0] init - initializing storage proxy RPC verbs
May 22 18:39:50 scylla2 scylla[73125]:  [shard 0] init - starting streaming service
May 22 18:39:50 scylla2 scylla[73125]:  [shard 0] init - starting hinted handoff manager
May 22 18:39:50 scylla2 scylla[73125]:  [shard 0] init - starting messaging service
May 22 18:39:50 scylla2 scylla[73125]:  [shard 0] repair - Setting max_repair_memory=49912217, max_repair_memory_per_range=33554432, max_repair_ranges_in_parallel=1
May 22 18:39:50 scylla2 scylla[73125]:  [shard 0] init - starting storage service
May 22 18:39:50 scylla2 scylla[73125]:  [shard 0] storage_service - Loading persisted ring state
May 22 18:39:50 scylla2 scylla[73125]:  [shard 0] storage_service - initial_contact_nodes={192.168.232.100}, loaded_endpoints={}, loaded_peer_features=0
May 22 18:39:50 scylla2 scylla[73125]:  [shard 0] storage_service - Checking remote features with gossip
May 22 18:39:50 scylla2 scylla[73125]:  [shard 0] messaging_service - Starting Messaging Service on port 7000
May 22 18:39:50 scylla2 scylla[73125]:  [shard 0] gossip - Gossip shadow round started with nodes={192.168.232.100}
May 22 18:39:51 scylla2 scylla[73125]:  [shard 0] gossip - Gossip shadow round finisehd with nodes_talked={192.168.232.100}
May 22 18:39:51 scylla2 scylla[73125]:  [shard 0] gossip - Feature check passed. Local node 192.168.232.101 features = {CDC, COMPUTED_COLUMNS, CORRECT_COUNTER_ORDER, CORRECT_IDX_TOKEN_IN_SECONDARY_INDEX, CORRECT_NON>
May 22 18:39:51 scylla2 scylla[73125]:  [shard 0] storage_service - Checking bootstrapping/leaving/moving nodes: ok (check_for_endpoint_collision)
May 22 18:39:52 scylla2 scylla[73125]:  [shard 0] schema_tables - Schema version changed to 59adb24e-f3cd-3e02-97f0-5b395827453f
May 22 18:39:52 scylla2 scylla[73125]:  [shard 0] storage_service - Starting up server gossip
May 22 18:39:54 scylla2 scylla[73125]:  [shard 0] compaction - [Compact system.local 1982ff70-bb2d-11eb-a0e4-000000000000] Compacting [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/mc-2-big-Data>
May 22 18:39:55 scylla2 scylla[73125]:  [shard 0] compaction - [Compact system.local 1982ff70-bb2d-11eb-a0e4-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/>
May 22 18:39:56 scylla2 scylla[73125]:  [shard 0] storage_service - Not updating host ID 48b094fa-544b-46b1-9be2-2c2be8a3cec3 for 192.168.232.100 because it's mine
May 22 18:39:56 scylla2 scylla[73125]:  [shard 0] gossip - removed 192.168.232.100 from _seeds, updated _seeds list = {}
May 22 18:39:56 scylla2 scylla[73125]:  [shard 0] gossip - InetAddress 192.168.232.100 is now UP, status = NORMAL
May 22 18:40:04 scylla2 scylla[73125]:  [shard 0] gossip - Gossip not settled after 1 polls.
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] gossip - Gossip settled after 1 extra polls; proceeding
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature CDC is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature COMPUTED_COLUMNS is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature CORRECT_IDX_TOKEN_IN_SECONDARY_INDEX is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature CORRECT_STATIC_COMPACT_IN_MC is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature DIGEST_FOR_NULL_VALUES is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature DIGEST_INSENSITIVE_TO_EXPIRY is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature HINTED_HANDOFF_SEPARATE_CONNECTION is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature LWT is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature MC_SSTABLE_FORMAT is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature MD_SSTABLE_FORMAT is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature NONFROZEN_UDTS is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature PER_TABLE_CACHING is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature PER_TABLE_PARTITIONERS is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature ROW_LEVEL_REPAIR is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature TRUNCATION_TABLE is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature VIEW_VIRTUAL_COLUMNS is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] format_selector - Selected mc sstables format
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] features - Feature UNBOUNDED_RANGE_TOMBSTONES is enabled
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] format_selector - Selected md sstables format
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] schema_tables - Schema version changed to 59adb24e-f3cd-3e02-97f0-5b395827453f
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] schema_tables - Schema version changed to 59adb24e-f3cd-3e02-97f0-5b395827453f
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] schema_tables - Schema version changed to 59adb24e-f3cd-3e02-97f0-5b395827453f
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] schema_tables - Schema version changed to 59adb24e-f3cd-3e02-97f0-5b395827453f
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for ring information
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] compaction - [Compact system.local 217acaf0-bb2d-11eb-a0e4-000000000000] Compacting [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/md-4-big-Data>
May 22 18:40:07 scylla2 scylla[73125]:  [shard 0] compaction - [Compact system.local 217acaf0-bb2d-11eb-a0e4-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/>
May 22 18:40:08 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:09 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:10 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:11 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:12 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:13 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:14 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:15 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:16 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:17 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:18 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:19 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:20 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:21 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete
May 22 18:40:22 scylla2 scylla[73125]:  [shard 0] storage_service - JOINING: waiting for schema information to complete

If I started next server:

cluster_name: 'CC Cluster'
num_tokens: 256
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "192.168.232.100"
listen_address: localhost
# broadcast_address: 1.2.3.4
native_transport_port: 9042
native_shard_aware_transport_port: 19042
read_request_timeout_in_ms: 5000
write_request_timeout_in_ms: 2000
cas_contention_timeout_in_ms: 1000
endpoint_snitch: GossipingPropertyFileSnitch
rpc_address: localhost
rpc_port: 9160
api_port: 10000
api_address: 127.0.0.1
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
commitlog_total_space_in_mb: -1
murmur3_partitioner_ignore_msb_bits: 12
api_ui_dir: /opt/scylladb/swagger-ui/dist/
api_doc_dir: /opt/scylladb/api/api-doc/

Log file:

May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - Scylla version 4.4.1-0.20210406.00da6b5e9 with build-id 49017ad539e40dba20fa9b82837090165772bf66 starting ...
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - Only 476 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting prometheus API server
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting tokens manager
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting migration manager notifier
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - creating tracing
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - creating snitch
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - determining DNS name
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting API server
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - Scylla API server listening on 127.0.0.1:10000 ...
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - initializing storage service
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting per-shard database core
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - I/O Scheduler is not properly configured! This is a non-supported setup, and performance is expected to be unpredictably bad.
                                       Reason found: none of --max-io-requests, --io-properties and --io-properties-file are set.
                                      To properly configure the I/O Scheduler, run the scylla_io_setup utility shipped with Scylla.
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - creating and verifying directories
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Populating Keyspace system_schema
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_schema: Reading CF view_virtual_columns id=08843b63-45dc-3be2-9798-a0418295cfaa version=c777531c-15f7-326f-8ebe-39fd0265c8c9
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_schema: Reading CF computed_columns id=cc7c7069-3740-33c1-92a4-c3de78dbd2c4 version=2b8c4439-de76-31e0-807f-3b7290a975d7
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_schema: Reading CF aggregates id=924c5587-2e3a-345b-b10c-12f37c1ba895 version=4b53e92c-0368-3d5c-b959-2ec1bfd1a59f
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_schema: Reading CF functions id=96489b79-80be-3e14-a701-66a0b9159450 version=329ed804-55b3-3eee-ad61-d85317b96097
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_schema: Reading CF views id=9786ac1c-dd58-3201-a7cd-ad556410c985 version=5b58bb47-96e7-3f57-accf-0bfca4dbbc6e
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_schema: Reading CF indexes id=0feb57ac-311f-382f-ba6d-9024d305702f version=99c40462-8687-304e-abe3-2bdbef1f25aa
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_schema: Reading CF types id=5a8b1ca8-6602-3f77-a045-9273d308917a version=de51b2ce-5e4d-3b7d-a75f-2204332ce8d1
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_schema: Reading CF triggers id=4df70b66-6b05-3251-95a1-32b54005fd48 version=582d7071-1ef0-37c8-adc6-471a13636139
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_schema: Reading CF dropped_columns id=5e7583b5-f3f4-3af1-9a39-b7e1d6f5f11f version=7426bc6c-4c2f-3200-8ad8-4329610ed59a
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_schema: Reading CF columns id=24101c25-a2ae-3af7-87c1-b40ee1aca33f version=d33236d4-9bdd-3c09-abf0-a0bc5edc2526
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_schema: Reading CF scylla_tables id=5d912ff1-f759-3665-b2c8-8042ab5103dd version=38e5e56b-155d-39b5-a0d2-8e5dcad42a26
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_schema: Reading CF tables id=afddfb9d-bc1e-3068-8056-eed6c302ba09 version=b6240810-eeb7-36d5-9411-43b2d68dddab
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_schema: Reading CF keyspaces id=abac5682-dea6-31c5-b535-b3d6cffd0fb6 version=e79ca8ba-6556-3f7d-925a-7f20cf57938c
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Populating Keyspace system
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF compaction_history id=b4dbb7b4-dc49-3fb5-b3bf-ce6e434832ca version=25cae56a-8d75-39f3-a146-3756ab4981c7
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF batchlog id=0290003c-977e-397c-ac3e-fdfdc01d626b version=e2a2e804-49e4-3597-9f16-39fd9475835c
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF size_estimates id=618f817b-005f-3678-b8a4-53f3930b8e86 version=6d4dbcff-f05b-3dc3-95ad-f79f7b10504d
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF range_xfers id=55d76438-4e55-3f8b-9f6e-676d4af3976d version=dd15a078-409b-350d-9bef-c5f3520832d8
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF peer_events id=59dfeaea-8db2-3341-91ef-109974d81484 version=7f874a05-72b8-3c21-9acb-baa164fc351a
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF peers id=37f71aca-7dc2-383b-a706-72528af04d4f version=f6f6871f-8c86-3eca-ac0b-2a2e848e395d
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF scylla_local id=2972ec7f-fb20-38dd-aac1-d876f2e3fcbd version=5f0b407d-eedf-3845-a48e-dbb9673d10e1
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF schema_aggregates id=a5fc57fc-9d6c-3bfd-a3fc-01ad54686fea version=d06c934f-9a69-374c-86f1-523ab26fb05c
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF paxos id=b7b7f0c2-fd0a-3410-8c05-3ef614bb7c2d version=6dd372de-72fb-3e1b-9b8c-f97738a67fe9
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF compactions_in_progress id=55080ab0-5d9c-3886-90a4-acb25fe1f77b version=540263f1-40db-3869-8a38-3baadedc222d
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF local id=7ad54392-bcdd-35a6-8417-4e047860b377 version=7fa82c2e-5b67-37dd-8e5e-2079e18f1536
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF schema_columnfamilies id=45f5b360-24bc-3f83-a363-1034ea4fa697 version=7f7a5fea-07b7-304d-a6f8-f7aec72cabdb
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF hints id=2666e205-73ef-38b3-90fe-fecf96e8f0c7 version=0b74fdd1-e96d-309e-a14e-a5bcd7ac885d
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF sstable_activity id=5a1ff267-ace0-3f12-8563-cfae6103c65e version=bdec57a3-b234-334b-be9c-7b1f33113995
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF IndexInfo id=9f5c6374-d485-3229-9a0a-5094af9ad1e3 version=bbb3743b-351f-3023-b4fc-09a9be37d529
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF large_partitions id=8a7fe624-96b0-34b1-b90e-f71bddcdd2d3 version=04fa9920-9369-3a96-be39-6dd9fdc816b6
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF scylla_table_schema_history id=0191a53e-40f0-31d4-9171-b0d19ffb17b4 version=2cc890b8-9ade-3afc-ab63-043c8017608e
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF large_cells id=ead8bbc5-f146-3ae1-9f71-0b11f9a1d296 version=0d4a6937-a781-3f87-a192-b4e4a4a40acf
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF views_builds_in_progress id=b7f2c108-78cd-3c80-9cd5-d609b2bd149c version=b905cce5-1474-3085-819a-7592453e2fb9
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF built_views id=4b3c50a9-ea87-3d76-9101-6dbc9c38494a version=a28aa0b3-6def-30a7-9fbe-ce78b3f3c9b9
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF scylla_views_builds_in_progress id=a04c7bfd-1e13-36c9-a44d-f22da352281d version=e3fb736c-6956-3990-a31d-9a482279e3fc
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF truncated id=38c19fd0-fb86-3310-a4b7-0d0cc66628aa version=6da9a85c-7ae0-3917-aead-54a2f65a57a8
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF cdc_local id=0bcaffd4-0c83-3ead-ad13-dc1d5015b77c version=0aa4d3a2-ed95-3ecd-aba0-cd75622ad290
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF clients id=ca0f635d-8630-3609-8d93-a1fc06f2a5e5 version=51273ebf-e6f9-3cd5-b455-ea09ad29796f
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF schema_keyspaces id=b0f22357-4458-3cdb-9631-c43e59ce3676 version=84d28cef-59f9-34a9-9be5-31c684997f03
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF schema_columns id=296e9c04-9bec-3085-827d-c17d3df2122a version=53cfbb66-ee0d-37be-97bb-b2afb3746f85
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF schema_triggers id=0359bc71-7123-3ee1-9a4a-b9dfb11fc125 version=75e4ec4c-76da-3f59-984e-66810ba0c62b
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF large_rows id=40550f66-0858-39a0-9430-f27fc08034e9 version=aee3acb0-7926-317b-848e-db7bc3721695
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF schema_usertypes id=3aa75225-4f82-350b-8d5c-430fa221fa0a version=3c53e6bb-56d5-3618-a0b2-4ff5f18127d7
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system: Reading CF schema_functions id=d1b675fe-2b50-3ca4-8e49-c0f81989dcad version=a50959dd-2be0-3e4e-8bb5-cf40796e256a
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting gossip
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - seeds={192.168.232.100}, listen_address=localhost, broadcast_address=127.0.0.1
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting storage proxy
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting migration manager
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting query processor
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - initializing batchlog manager
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] format_selector - Selected md sstables format
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] legacy_schema_migrator - Moving 0 keyspaces from legacy schema tables to the new schema keyspace (system_schema)
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] legacy_schema_migrator - Dropping legacy schema tables
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] legacy_schema_migrator - Completed migration of legacy schema tables
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - loading system sstables
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - loading non-system sstables
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Populating Keyspace system_distributed
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_distributed: Reading CF view_build_status id=5582b59f-8e4e-35e1-b913-3acada51eb04 version=e61ea49b-8dbc-3ff6-a69f-896f4d38b7bb
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_distributed: Reading CF cdc_streams_descriptions_v2 id=0bf73fd7-65b2-36b0-85e5-658131d5df36 version=58a18c32-0c5a-34ba-992f-7723dd0265c0
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_distributed: Reading CF cdc_generation_timestamps id=fdf455c4-cfec-3e00-9719-d7a45436c89d version=5777e043-a81a-3324-8803-b497c056849f
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] database - Keyspace system_distributed: Reading CF cdc_generation_descriptions id=ae653406-7b3e-34aa-af70-d5b90f837c67 version=95e9e159-839f-39b3-990f-8e8a04817ff8
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting view update generator
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - discovering staging sstables
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - setting up system keyspace
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting commit log
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - initializing migration manager RPC verbs
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - initializing storage proxy RPC verbs
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting streaming service
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting hinted handoff manager
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system.truncated 4684ec70-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/md-4-b>
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.computed_columns 46851380-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system_schema/computed_columns-cc7c7069374>
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.columns 46851381-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca>
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.scylla_tables 46851382-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system_schema/scylla_tables-5d912ff1f7593665b>
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system.compaction_history 46851383-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfc>
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting messaging service
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] repair - Setting max_repair_memory=49912217, max_repair_memory_per_range=33554432, max_repair_ranges_in_parallel=1
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] init - starting storage service
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] storage_service - Loading persisted ring state
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] storage_service - initial_contact_nodes={192.168.232.100}, loaded_endpoints={}, loaded_peer_features=0
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] storage_service - Checking remote features with gossip
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] messaging_service - Starting Messaging Service on port 7000
May 22 04:36:28 scylla3 scylla[7482]:  [shard 0] gossip - Gossip shadow round started with nodes={192.168.232.100}
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] gossip - Gossip shadow round finisehd with nodes_talked={192.168.232.100}
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] gossip - Feature check passed. Local node 127.0.0.1 features = {CDC, COMPUTED_COLUMNS, CORRECT_COUNTER_ORDER, CORRECT_IDX_TOKEN_IN_SECONDARY_INDEX, CORRECT_NON_COMPOU>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] storage_service - Checking bootstrapping/leaving/moving nodes: ok (check_for_endpoint_collision)
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] schema_tables - Schema version changed to f71ba4e8-545a-3c13-a15c-d889f1f181be
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] storage_service - Starting up server gossip
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.indexes 468e8960-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system_schema/indexes-0feb57ac311f382fba6d9024d3057>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.dropped_columns 468efe90-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system_schema/dropped_columns-5e7583b5f3f43>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.tables 468f25a0-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba0>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system.compaction_history 46851383-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system/compaction_history-b4dbb7b4>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.computed_columns 46851380-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/computed_column>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system.local 469232e0-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/md-12-big-Data>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.scylla_tables 46851382-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/scylla_tables-5d91>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system.truncated 4684ec70-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc6>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.columns 46851381-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/columns-24101c25a2ae3af7>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.functions 469678a0-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system_schema/functions-96489b7980be3e14a70166a0b>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.keyspaces 469678a1-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system_schema/keyspaces-abac5682dea631c5b535b3d6c>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.indexes 468e8960-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/indexes-0feb57ac311f382f>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.dropped_columns 468efe90-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/dropped_columns->
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.tables 468f25a0-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/tables-afddfb9dbc1e306880>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system.local 469232e0-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/m>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.types 469b5aa0-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.keyspaces 469678a1-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/keyspaces-abac5682dea6>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.functions 469678a0-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/functions-96489b7980be>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.triggers 469df2b0-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system_schema/triggers-4df70b666b05325195a132b5400>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.types 469b5aa0-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/types-5a8b1ca866023f77a045>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.view_virtual_columns 46a01590-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system_schema/view_virtual_columns-088>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.triggers 469df2b0-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/triggers-4df70b666b0532>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.aggregates 46a25f80-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system_schema/aggregates-924c55872e3a345bb10c12f>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.view_virtual_columns 46a01590-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/view_virtua>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.views 46a45b50-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.aggregates 46a25f80-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/aggregates-924c55872e>
May 22 04:36:29 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system_schema.views 46a45b50-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/views-9786ac1cdd583201a7cd>
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] gossip - No gossip backlog; proceeding
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature CDC is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature COMPUTED_COLUMNS is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature CORRECT_IDX_TOKEN_IN_SECONDARY_INDEX is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature CORRECT_STATIC_COMPACT_IN_MC is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature DIGEST_FOR_NULL_VALUES is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature DIGEST_INSENSITIVE_TO_EXPIRY is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature HINTED_HANDOFF_SEPARATE_CONNECTION is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature LWT is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature MC_SSTABLE_FORMAT is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature MD_SSTABLE_FORMAT is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature NONFROZEN_UDTS is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature PER_TABLE_CACHING is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature PER_TABLE_PARTITIONERS is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature ROW_LEVEL_REPAIR is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature TRUNCATION_TABLE is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature UNBOUNDED_RANGE_TOMBSTONES is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] features - Feature VIEW_VIRTUAL_COLUMNS is enabled
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] storage_service - Detected previous bootstrap failure; retrying
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] storage_service - JOINING: waiting for ring information
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] schema_tables - Schema version changed to f71ba4e8-545a-3c13-a15c-d889f1f181be
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] schema_tables - Schema version changed to 43a8aaae-72c9-3939-896b-f4a4755295dd
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] schema_tables - Schema version changed to 43a8aaae-72c9-3939-896b-f4a4755295dd
May 22 04:36:41 scylla3 scylla[7482]:  [shard 0] schema_tables - Schema version changed to 43a8aaae-72c9-3939-896b-f4a4755295dd
May 22 04:37:11 scylla3 scylla[7482]:  [shard 0] storage_service - JOINING: schema complete, ready to bootstrap
May 22 04:37:11 scylla3 scylla[7482]:  [shard 0] storage_service - JOINING: waiting for pending range calculation
May 22 04:37:11 scylla3 scylla[7482]:  [shard 0] storage_service - JOINING: calculation complete, ready to bootstrap
May 22 04:37:11 scylla3 scylla[7482]:  [shard 0] storage_service - Checking bootstrapping/leaving nodes: ok
May 22 04:37:11 scylla3 scylla[7482]:  [shard 0] storage_service - JOINING: getting bootstrap token
May 22 04:37:11 scylla3 scylla[7482]:  [shard 0] storage_service - Using previously saved tokens = {8691677480650276474, 8620457412695038357, 8666104302219232778, 8315206492745832013, 8163627526922029783, 8011305143>
May 22 04:37:11 scylla3 scylla[7482]:  [shard 0] init - starting system distributed keyspace
May 22 04:37:11 scylla3 scylla[7482]:  [shard 0] gossip - Waiting for pending range setup...
May 22 04:37:11 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system.local 5fa62e80-bab7-11eb-84a6-000000000000] Compacting [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/md-14-big-Data>
May 22 04:37:11 scylla3 scylla[7482]:  [shard 0] compaction - [Compact system.local 5fa62e80-bab7-11eb-84a6-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/m>
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] gossip - No gossip backlog; proceeding
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] token_metadata - sorted_tokens is empty in first_token_index!
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down gossiping
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] gossip - My status = UNKNOWN
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] gossip - No local state or state is in silent shutdown, not announcing shutdown
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] gossip - Disable and wait for gossip loop started
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] gossip - Disable and wait for gossip loop finished
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] gossip - Gossip is now stopped
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down gossiping was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down storage service notifications
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down storage service notifications was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down repair API
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down repair API was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down messaging service API
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down messaging service API was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down storage service messaging
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down storage service messaging was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down repair message handlers
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down repair message handlers was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down repair service
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down repair service was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down streaming service
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down streaming service was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down storage proxy RPC verbs
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down storage proxy RPC verbs was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down cdc
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down cdc was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down sstables format selector
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down sstables format selector was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down migration manager
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] migration_manager - stopping migration service
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down migration manager was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down database
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] compaction_manager - Asked to stop
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] compaction_manager - Stopping 0 ongoing compactions
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] compaction_manager - Stopped
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down database was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down sst_dir_semaphore
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down sst_dir_semaphore was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down messaging service
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down messaging service was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down API server
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down API server was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down migration manager notifier
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down migration manager notifier was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down prometheus API server
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down prometheus API server was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down sighup
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Shutting down sighup was successful
May 22 04:37:48 scylla3 scylla[7482]:  [shard 0] init - Startup failed: std::runtime_error (sorted_tokens is empty in first_token_index!)



Ivan Prisyazhnyy

<ivan@scylladb.com>
unread,
May 23, 2021, 3:25:00 AM5/23/21
to scylladb-users@googlegroups.com
In one of the cases I saw this issue the reason was is that you have your broadcast address in your seed list

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/6624073f-0a21-47a2-90d5-361058dd40d9n%40googlegroups.com.

Alex Shestakov

<shestakov.am@gmail.com>
unread,
May 23, 2021, 9:53:06 PM5/23/21
to scylladb-users@googlegroups.com
I removed all broadcast addresses like in https://docs.scylladb.com/operating-scylla/procedures/cluster-management/create_cluster/ , but nothing changed. First seed server indicate "[shard 0] gossip - InetAddress 192.168.232.101 is now UP, status = UNKNOWN", second server indicate "[shard 0] storage_service - JOINING: waiting for schema information to complete".

I really don't know how to start this cluster

You received this message because you are subscribed to a topic in the Google Groups "ScyllaDB users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scylladb-users/EIqSvue2NF0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAOZ5tkEPu6ZndjTRJ1vq3UBs4Gu7Oy2DP9gAKNb7B5ZzNhme5Q%40mail.gmail.com.


--
============================


Asias He

<asias@scylladb.com>
unread,
May 24, 2021, 6:21:06 AM5/24/21
to ScyllaDB users, Tomasz Grabiec
This is a schema sync issue. @Tomasz Grabiec do we have all the fixes in 4.4.1-0.20210406.00da6b5e9 ?

 

If I started next server:

cluster_name: 'CC Cluster'
num_tokens: 256
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "192.168.232.100"
listen_address: localhost
# broadcast_address: 1.2.3.4

If we set listen_address to localhost, we have to set broadcast_address. Your config iscorrect in the first example.
 

Alex Shestakov

<shestakov.am@gmail.com>
unread,
May 24, 2021, 10:46:07 AM5/24/21
to scylladb-users@googlegroups.com
Hello,

I installed 4.4.1-0.20210406.00da6b5e9. In this case I don't use a broadcast addresses, but I have "[shard 0] storage_service - JOINING: waiting for schema information to complete" error!

Seed server:

cluster_name: 'CC Cluster'
num_tokens: 256
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "192.168.232.100"
listen_address: 192.168.232.100
#broadcast_address: 192.168.232.100

native_transport_port: 9042
native_shard_aware_transport_port: 19042
#native_transport_port_ssl: 9142
#native_shard_aware_transport_port_ssl: 19142
read_request_timeout_in_ms: 5000
write_request_timeout_in_ms: 2000
cas_contention_timeout_in_ms: 1000
endpoint_snitch: GossipingPropertyFileSnitch
rpc_address: 192.168.232.100
rpc_port: 9160
api_port: 10000
api_address: 192.168.232.100
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
#broadcast_rpc_address: 192.168.232.100

partitioner: org.apache.cassandra.dht.Murmur3Partitioner
commitlog_total_space_in_mb: -1
murmur3_partitioner_ignore_msb_bits: 12
api_ui_dir: /opt/scylladb/swagger-ui/dist/
api_doc_dir: /opt/scylladb/api/api-doc/

Next node:
cluster_name: 'CC Cluster'
num_tokens: 256
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "192.168.232.100"
listen_address: 192.168.232.101
#broadcast_address: 192.168.232.101

native_transport_port: 9042
native_shard_aware_transport_port: 19042
#native_transport_port_ssl: 9142
#native_shard_aware_transport_port_ssl: 19142
read_request_timeout_in_ms: 5000
write_request_timeout_in_ms: 2000
cas_contention_timeout_in_ms: 1000

endpoint_snitch: GossipingPropertyFileSnitch

rpc_address: 192.168.232.101
rpc_port: 9160
api_port: 10000
api_address: 192.168.232.101
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
#broadcast_rpc_address: 192.168.232.101

partitioner: org.apache.cassandra.dht.Murmur3Partitioner
commitlog_total_space_in_mb: -1
murmur3_partitioner_ignore_msb_bits: 12

api_ui_dir: /opt/scylladb/swagger-ui/dist/
api_doc_dir: /opt/scylladb/api/api-doc/
You received this message because you are subscribed to a topic in the Google Groups "ScyllaDB users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scylladb-users/EIqSvue2NF0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAO1GqFbpehbM%2BB8ZTyZJii5-5H8VfXjxO3KO3vctMQz8StJnaA%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages