Kubernetes - problems to deploy cassandra_ycsb workload

[Paweł Zięcik]

I want to benchmark a Kubernetes cluster using CBTOOL. I followed the user guide, manage to deploy nullworkload and capture metrics from the task. When I execute aiattach cassandra_ycsb, I am getting error:

Parallel run os command operation failure: Giving up on executing command "~/cb_restart_seed.sh" on hostname 172.17.0.3. Too many attempts (3).

/var/log/XXXX_remotescripts.log

Nov 19 09:20:48 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): The JAVA_VERSION was set to "8" Nov 19 09:20:48 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): The JAVA_HOME was set to "auto". Attempting to find the most recent in /opt/ibm Nov 19 09:20:49 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): The JAVA_HOME was set to "auto". Attempting to find the most recent in /usr/lib/jvm Nov 19 09:20:49 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): JAVA_HOME determined to be "/usr/lib/jvm//jre" Nov 19 09:20:49 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Cassandra token is " 85070591730234615865843651857942052864" Nov 19 09:20:49 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): No VMs with the "cassandra" role have been found on this AI Nov 19 09:20:49 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): The VMs with the "seed" role on this AI has the following IPs: 10.244.1.3,10.244.1.4 Nov 19 09:20:49 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Performing a quick check from cb-root-mykub-vm1-seed-ai-1 in order to decide on Cassan dra restart Nov 19 09:20:49 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Waiting for all nodes to become available... Nov 19 09:20:50 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Successfully contacted Cassandra with command "cassandra-cli -h 10.244.1.4 -f cassandr a-cli_list_keyspace.cassandra" through node 10.244.1.4 Nov 19 09:20:51 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Obtaining the node list for this Cassandra cluster by running "nodetool -u cassandra - pw cassandra -h 10.244.1.4 status"... Nov 19 09:20:53 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Nodes registered on the cluster: 0 out of 2 Nov 19 09:20:53 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Make sure that Keyspace "system" is present Nov 19 09:20:55 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Obtaining the node list for this Cassandra cluster by running "nodetool -u cassandra - pw cassandra -h 10.244.1.4 status"... Nov 19 09:20:56 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Nodes registered on the cluster: 0 out of 2 Nov 19 09:20:56 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Make sure that Keyspace "system" is present Nov 19 09:20:58 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): The exit code of "check_cassandra_cluster_state 10.244.1.4 1 1" was 1. Starting Cassan dra service on this seed... Nov 19 09:20:58 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Restarting service "cassandra", with command "sudo service cassandra restart", attempt 1 of 7... Nov 19 09:20:58 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Service "cassandra" was successfully restarted Nov 19 09:20:58 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Enabling service "cassandra", with command "sudo update-rc.d -f cassandra defaults"... Nov 19 09:22:38 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Port 9160 on host 10.244.1.4 was NOT found open after 21 attempts! Nov 19 09:22:39 cb-root-mykub-vm1-seed-ai-1_10-244-1-4 cloudbench EXP-11-19-2019-08-05-27-AM-UTC cb_common.sh (6475): Cassandra service failed to start on this seed

The image of ycsb I am using came from ibmcb docker hub. The cluster I am using is set up using https://kind.sigs.k8s.io/ so each Kubernetes node is a Docker container.