NoHostAvailable in cqlsh

[jason.aylward@gmail.com]

This might not be a ScyllaDB issue at all but after configuring three nodes in AWS to use the Ec2Snitch, I'm getting a NoHostAvailable error when trying to query a table through cqlsh.

I started with three Scylla v1.7.2 AMIs on i2.2xlarge instances.  The only editing I did to the /etc/scylla/scylla.yaml was to change the snitch to Ec2Snitch.

Restarting the servers using `sudo systemctl stop scylla-server` and `sudo systemctl start scylla-server` did not seem to work because cqlsh would not start on any of them.

`sudo reboot` did the trick though.  The data centers in `nodetool status` was updated to `us-east` and everything looked great.

Creating a keyspace and creating a table worked just fine and I could verify that each node could `describe keyspace`.

I then did a large export of data from my redis server.  Roughly 35million rows.  It took about 35 minutes and using the AWS monitoring, I could see that data was making it to all three nodes.

Now that the export has finished though, I can't seem to query it via cqlsh.

`nodetools tablestats` estimates 0 keys as well.

This is my first time setting up a multi-node cluster so I'm guessing I just missed a step somewhere.

If anything sounds out of the ordinary, I'm all ears.  Thanks!

[Vladislav Zolotarov]

Hi!

Could you describe the exact characteristics of the i2.2xlarge instances you created: I think the storage size is the only configurable part there.

Could you, please, send us your keyspace and column family schemas (DESC command output)?

Could you estimate the amount of data (in bytes) that you streamed to the scylla cluster?

Could you, please, describe in detail your steps starting from exporting your data from redis - how did you do that?

Could you also send us the output of 'journalctl -t scylla' command on all Nodes?

This should be enough to start with... :)

Thanks,

Vlad

 

[jason.aylward@gmail.com]

Hello and thanks for the response.  

Could you describe the exact characteristics of the i2.2xlarge instances you created: I think the storage size is the only configurable part there.

--  I used the default storage volumes for the i2.2xlarge:  1 Magnetic 10GB volume and 2 "Instance Store" volumes.  The cluster was 3 nodes.

Could you, please, send us your keyspace and column family schemas (DESC command output)?

-- (pasted at the bottom)

Could you estimate the amount of data (in bytes) that you streamed to the scylla cluster?

-- I'm going to guess 10-15 GBs.  I know it was about 35 million records, the records are relatively small.

Could you, please, describe in detail your steps starting from exporting your data from redis - how did you do that?

-- The script is all done in python using the cassandra-driver package.The export is handled by one instance which scans the Redis for the keys it wants and puts them on a multiprocessing queue.  Then each child process (32 total) pulls a key of the queue, gets the values for that key from Redis, creates a Cluster session and executes the insert.

Potential problem is creating a unique session for each insert but when I tried creating one session per child process, I ran into problems running the script for an hour without a timeout occurring and breaking the entire process.

Could you also send us the output of 'journalctl -t scylla' command on all Nodes?

-- I terminated the cluster yesterday but I'll be running some more tests today and I'll paste any updates here.

SCHEMA:

CREATE KEYSPACE record_store WITH replication = {'class': 'NetworkTopologyStrategy', 'US-EAST': '2'} AND durable_writes = true;

CREATE TABLE records (user uuid, 

                      attributes set<int>, 

                      items set<int>, 

                      primary key (user)) 

WITH bloom_filter_fp_chance = 0.01 

AND caching = '{"keys": "ALL", "rows_per_partition":"NONE"}' 

AND comment = '' 

AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'} 

AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'} 

AND dclocal_read_repair_chance = 0.0 

AND default_time_to_live = 0.0 

AND gc_grace_seconds = 604800 

AND max_index_interval = 2048 

AND memtable_flush_period_in_ms = 0 

AND min_index_interval = 128 

AND read_repair_chance = 0.1 

AND speculative_retry = '99.0PERCENTILE';

[jason.aylward@gmail.com]

I think I've resolved the issue.  Looks like it was a matter of case-sensitivity when defining the data center in the CREATE KEYSPACE.

replication = {'class': 'NetworkTopologyStrategy', 'US-EAST': '2'}

will successfully create a keyspace and any requested tables within the cluster but INSERT or SELECT on a table will yield NoHostAvailable

replication = {'class': 'NetworkTopologyStrategy', 'us-east': '2'}

will successfully create the keyspace and tables and actually allow you to use them.

Good times.

Thanks for helping out!

[Vladislav Zolotarov]

Hi!

I'm glad the issue is resolved. ;)

 

Good times.

Thanks for helping out!

You are welcome ;) (Not that I did much).

Enjoy scylla and let us know if you need anything.