No host available use C++ driver with CASS_ERROR_LIB_NO_HOSTS_AVAILABLE error code

[Jet Li]

Hi, everyone
I have write a post yesterday to describe my situation, thanks yours reply. I will describe my error and code more detail here with a new post, please help me, thanks!
Driver: C/C++ driver
cassandra environment: I have three linux machine and install cassandra on each mancine, the ip is 100.142.31.176, 100.142.31.177 and 100.142.31.178; 100.142.31.176 is the seed node, in my cassandra.yaml, I configure all parameter I know need to config, include rpc address: 0.0.0.0, broad cast rpc address: localhost. all cassandra is up when use ./nodetool status, and can ssh each other when use ./cqpsh <ip>;
demo task: I run at least 10 task which will run in three node(average again on three node)
demo code describe: when a task on a node comming, then the demo code will do below thing:
1. create cluster:
m_cluster = cass_cluster_new();
cass_cluster_set_request_timeout(m_cluster, 50000);
cass_cluster_set_latency_aware_routing(m_cluster, cass_treue);
cass_cluster_set_core_connections_per_host(m_cluster, 50);
cass_cluster_set_max_connections_per_host(m_cluster, 100);
....
cass_cluster_set_contact_point(m_cluster, "127.0.0.1");
....
2. create session and connection...
CassSession* session = cass_session_new();
CassFuture* connect_future = cass_session_connect(session, m_cluster);
....
3. create keyspace belong the task..

phenomenon: there are some task failed, actually, each node include failed task and successfull task, the failed task failed with No hosts available when create keyspace(CassFuture* future = cass_session_execute(session, statement): it seems this return No hosts available error), the error code is: CASS_ERROR_LIB_NO_HOSTS_AVAILABLE

I am a new user of cassandra, can you help me? I indeed no idea about this, why some task on a node sucessful, but some failed... thanks very much.

[Neil Ablang]

On Fri, Nov 27, 2015 at 10:10 AM, Jet Li <jetli...@gmail.com> wrote:

Hi, everyone
I have write a post yesterday to describe my situation, thanks yours reply. I will describe my error and code more detail here with a new post, please help me, thanks!
Driver: C/C++ driver
cassandra environment: I have three linux machine and install cassandra on each mancine, the ip is 100.142.31.176, 100.142.31.177 and 100.142.31.178; 100.142.31.176 is the seed node, in my cassandra.yaml, I configure all parameter I know need to config, include rpc address: 0.0.0.0, broad cast rpc address: localhost. all cassandra is up when use ./nodetool status, and can ssh each other when use ./cqpsh <ip>;
demo task: I run at least 10 task which will run in three node(average again on three node)
demo code describe: when a task on a node comming, then the demo code will do below thing:
1. create cluster:
m_cluster = cass_cluster_new();
cass_cluster_set_request_timeout(m_cluster, 50000);
cass_cluster_set_latency_aware_routing(m_cluster, cass_treue);
cass_cluster_set_core_connections_per_host(m_cluster, 50);
cass_cluster_set_max_connections_per_host(m_cluster, 100);
....
cass_cluster_set_contact_point(m_cluster, "127.0.0.1");

>>> Neil ------> This may be your problem : cass_cluster_set_contact_point(m_cluster, "127.0.0.1") -- set listen_address to you server ip address for each server and set rpc_address to you server address for each server and try if works.  These parameters are found in your cassandra.yaml file. Then on you program set the address to one of you server IP address instead of the 127.0.0.1 in the code you presented 

....
2. create session and connection...
CassSession* session = cass_session_new();
CassFuture* connect_future = cass_session_connect(session, m_cluster);
....
3. create keyspace belong the task..

phenomenon: there are some task failed, actually, each node include failed task and successfull task, the failed task failed with No hosts available when create keyspace(CassFuture* future = cass_session_execute(session, statement): it seems this return No hosts available error), the error code is: CASS_ERROR_LIB_NO_HOSTS_AVAILABLE

I am a new user of cassandra, can you help me? I indeed no idea about this, why some task on a node sucessful, but some failed... thanks very much.

--
You received this message because you are subscribed to the Google Groups "DataStax C++ Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cpp-driver-us...@lists.datastax.com.

[Jet Li]

在 2015年11月27日星期五 UTC+8上午10:33:19，Neil Ablang写道：

> On Fri, Nov 27, 2015 at 10:10 AM, Jet Li <jetli...@gmail.com> wrote:
> Hi, everyone
>
> I have write a post yesterday to describe my situation, thanks yours reply. I will describe my error and code more detail here with a new post, please help me, thanks!
>
> Driver: C/C++ driver
>
> cassandra environment: I have three linux machine and install cassandra on each mancine, the ip is 100.142.31.176, 100.142.31.177 and 100.142.31.178; 100.142.31.176 is the seed node, in my cassandra.yaml, I configure all parameter I know need to config, include rpc address: 0.0.0.0, broad cast rpc address: localhost. all cassandra is up when use ./nodetool status, and can ssh each other when use ./cqpsh <ip>;
>
> demo task: I run at least 10 task which will run in three node(average again on three node)
>
> demo code describe: when a task on a node comming, then the demo code will do below thing:
>
> 1. create cluster:
>
> m_cluster = cass_cluster_new();
>
> cass_cluster_set_request_timeout(m_cluster, 50000);
>
> cass_cluster_set_latency_aware_routing(m_cluster, cass_treue);
>
> cass_cluster_set_core_connections_per_host(m_cluster, 50);
>
> cass_cluster_set_max_connections_per_host(m_cluster, 100);
>
> ....
>
> cass_cluster_set_contact_point(m_cluster, "127.0.0.1");
>
>
>
> >>> Neil ------> This may be your problem : cass_cluster_set_contact_point(m_cluster, "127.0.0.1") -- set listen_address to you server ip address for each server and set rpc_address to you server address for each server and try if works.  These parameters are found in your cassandra.yaml file. Then on you program set the address to one of you server IP address instead of the 127.0.0.1 in the code you presented 

>>>>Liwei: Hi, Neil, thanks for your fast reply, I have some doubt about your answer, if the reason is the contact point "127.0.0.1", I think maybe all task should failed, but some of task failed and some of task successful, and If I run less task(such 3 task and each node will include 1 task), all task can success. I tried change contact point from 127.0.0.1 to local server ip, but no use actually, thanks!

[Neil Ablang]

On Fri, Nov 27, 2015 at 10:59 AM, Jet Li <jetli...@gmail.com> wrote:

在 2015年11月27日星期五 UTC+8上午10:33:19，Neil Ablang写道：
> On Fri, Nov 27, 2015 at 10:10 AM, Jet Li <jetli...@gmail.com> wrote:
> Hi, everyone
>
> I have write a post yesterday to describe my situation, thanks yours reply. I will describe my error and code more detail here with a new post, please help me, thanks!
>
> Driver: C/C++ driver
>
> cassandra environment: I have three linux machine and install cassandra on each mancine, the ip is 100.142.31.176, 100.142.31.177 and 100.142.31.178; 100.142.31.176 is the seed node, in my cassandra.yaml, I configure all parameter I know need to config, include rpc address: 0.0.0.0, broad cast rpc address: localhost. all cassandra is up when use ./nodetool status, and can ssh each other when use ./cqpsh <ip>;
>
> demo task: I run at least 10 task which will run in three node(average again on three node)
>
> demo code describe: when a task on a node comming, then the demo code will do below thing:
>
> 1. create cluster:
>
> m_cluster = cass_cluster_new();
>
> cass_cluster_set_request_timeout(m_cluster, 50000);
>
> cass_cluster_set_latency_aware_routing(m_cluster, cass_treue);
>
> cass_cluster_set_core_connections_per_host(m_cluster, 50);
>
> cass_cluster_set_max_connections_per_host(m_cluster, 100);
>
> ....
>
> cass_cluster_set_contact_point(m_cluster, "127.0.0.1");
>
>
>
> >>> Neil ------> This may be your problem : cass_cluster_set_contact_point(m_cluster, "127.0.0.1") -- set listen_address to you server ip address for each server and set rpc_address to you server address for each server and try if works.  These parameters are found in your cassandra.yaml file. Then on you program set the address to one of you server IP address instead of the 127.0.0.1 in the code you presented 
>>>>Liwei: Hi, Neil, thanks for your fast reply, I have some doubt about your answer, if the reason is the contact point "127.0.0.1", I think maybe all task should failed, but some of task failed and some of task successful, and If I run less task(such 3 task and each node will include 1 task), all task can success. I tried change contact point from 127.0.0.1 to local server ip, but no use actually,  thanks!

>>>>>Neil - change the those params on the server too and restart your servers' casssandra service.. the listen_address i believe is what the servers use to exchange gossips, learning who is up or down etc,

[Jet Li]

在 2015年11月27日星期五 UTC+8上午11:13:46，Neil Ablang写道：

> On Fri, Nov 27, 2015 at 10:59 AM, Jet Li <jetli...@gmail.com> wrote:
> 在 2015年11月27日星期五 UTC+8上午10:33:19，Neil Ablang写道：
>
> > On Fri, Nov 27, 2015 at 10:10 AM, Jet Li <jetli...@gmail.com> wrote:
>
> > Hi, everyone
>
> >
>
> > I have write a post yesterday to describe my situation, thanks yours reply. I will describe my error and code more detail here with a new post, please help me, thanks!
>
> >
>
> > Driver: C/C++ driver
>
> >
>
> > cassandra environment: I have three linux machine and install cassandra on each mancine, the ip is 100.142.31.176, 100.142.31.177 and 100.142.31.178; 100.142.31.176 is the seed node, in my cassandra.yaml, I configure all parameter I know need to config, include rpc address: 0.0.0.0, broad cast rpc address: localhost. all cassandra is up when use ./nodetool status, and can ssh each other when use ./cqpsh <ip>;
>
> >
>
> > demo task: I run at least 10 task which will run in three node(average again on three node)
>
> >
>
> > demo code describe: when a task on a node comming, then the demo code will do below thing:
>
> >
>
> > 1. create cluster:
>
> >
>
> > m_cluster = cass_cluster_new();
>
> >
>
> > cass_cluster_set_request_timeout(m_cluster, 50000);
>
> >
>
> > cass_cluster_set_latency_aware_routing(m_cluster, cass_treue);
>
> >
>
> > cass_cluster_set_core_connections_per_host(m_cluster, 50);
>
> >
>
> > cass_cluster_set_max_connections_per_host(m_cluster, 100);
>
> >
>
> > ....
>
> >
>
> > cass_cluster_set_contact_point(m_cluster, "127.0.0.1");
>
> >
>
> >
>
> >
>
> > >>> Neil ------> This may be your problem : cass_cluster_set_contact_point(m_cluster, "127.0.0.1") -- set listen_address to you server ip address for each server and set rpc_address to you server address for each server and try if works.  These parameters are found in your cassandra.yaml file. Then on you program set the address to one of you server IP address instead of the 127.0.0.1 in the code you presented 
>
> >>>>Liwei: Hi, Neil, thanks for your fast reply, I have some doubt about your answer, if the reason is the contact point "127.0.0.1", I think maybe all task should failed, but some of task failed and some of task successful, and If I run less task(such 3 task and each node will include 1 task), all task can success. I tried change contact point from 127.0.0.1 to local server ip, but no use actually,  thanks!
>
>
>
> >>>>>Neil - change the those params on the server too and restart your servers' casssandra service.. the listen_address i believe is what the servers use to exchange gossips, learning who is up or down etc,
>

>>>>>Liwei: I set rpc address to localhost, and set rpc address to 0.0.0.0, broad cast rpc address to localhost, is it right?

[Neil Ablang]

Hi Liwei:

I think the cluster setup is not ok , i mean the configs

do the following:

on server 100.142.31.176 edit cassandra.yaml to have the ff:

listen_address: 100.142.31.176

rpc_address:100.142.31.176

on server 100.142.31.177 edit cassandra.yaml to have the ff:

listen_address: 100.142.31.177
rpc_address:100.142.31.177
- seeds: "100.142.31.176"

on server 100.142.31.178 edit cassandra.yaml to have the ff:
listen_address: 100.142.31.178
rpc_address:100.142.31.178
- seeds: "100.142.31.176"

leave the rpc_broadcast_address commented. use the default with # sign infront

allow the rpc port on the each server's firewall, tcp port 9160

restart  cassandra on each server

then use the ip address on of any of the servers for your program/code

hope this helps

[Jet Li]

Hi, Neil

Thank you very much, it's very nice of you!:) I will try it, thanks again
在 2015年11月27日星期五 UTC+8上午11:29:38，Neil Ablang写道：

[Neil Ablang]

Hi Liwei:

You are welcome.

Also you may need to allow more tcp ports in each server to pass the packets.

Please refer to this link for ports you have to allow on your servers to pass through packets:

http://docs.datastax.com/en/cassandra/2.0/cassandra/security/secureFireWall_r.html

You can be sure that the cluster is working when you run nodetool and it displays all your nodes (3 ip addresses in your case) showing the status of each node

Neil

[Jet Li]

Ok, I will, Thanks Neil :)
在 2015年11月27日星期五 UTC+8上午11:58:18，Neil Ablang写道：

[Jet Li]

Hi, Neil
I can work now after configure according you said, but I met an new problem describe below:
actually, I want the task still can work after one of cassandra node crush(it mean the cassandra process has be killed, but the node still will receive task), so I set contact point as seed node ip + local machine ip, but after I kill one cassandra process, all task of all the node failed.
PS: I set replication in the code, some data with 2 copy, some data for 3 copy:
createKeyspace = "CREATE KEYSPACE IF NOT EXISTS" + keyspaceName_rep2 + "WITH replication = {'class': 'SimpleStrategy', 'replication_fator': '2'}"
createKeyspace = "CREATE KEYSPACE IF NOT EXISTS" + keyspaceName_rep3 + "WITH replication = {'class': 'SimpleStrategy', 'replication_fator': '3'}"

在 2015年11月27日星期五 UTC+8上午11:58:18，Neil Ablang写道：

[Neil Ablang]

Hi Liwei,

You can put the ip addresses of all the nodes in an array to create a pool of connections as a practice in programming cassandra. Im not sure on c++ api, but i remember it is done that way too. Create a pool of connections. Its been a long time i didnt use the c++ api for its a lot easier and faster to develop in python especially when doing a lot of text manipulations.

Best Regards

Neil Ablang

[Jet Li]

Hi, Neil

actually, the api document said, we can use a string as contact point, such as "127.0.0.1"; "127.0.0.1,127.0.0.2"; "server1.com,server2.com" when do cass_cluster_set_contact_points, I write code like this, but it seems not work. I will consider your suggestion to create a connection pool, thanks!

Best wishes
Liwei
在 2015年11月28日星期六 UTC+8下午3:58:51，Neil Ablang写道：

[Neil Ablang]

Ill review the c++ api later and let you know

[Jet Li]

cool! thanks very much :)
在 2015年11月28日星期六 UTC+8下午4:33:06，Neil Ablang写道：

[Neil Ablang]

Hi Liwei,

Your code should look like this in creating the connection pool:

cass_cluster_set_contact_points(cluster, "100.142.31.176,100.142.31.177,100.142.31.178");

[Jet Li]

Hi, Neil

Yes, I do like this.
Only just that there will be a lot of node in my future work, so I just set seed node and local machine as the contact point, maybe I need set the routing for contact points.

Best wishes
Liwei

[Neil Ablang]

On my setup , I only use 3 or 5 at max for the connection pool. Thats is a cluster and one node knows everyone in the cluster and where to get the data you are looking for. No need to put all your node addresses I suppose

[Jet Li]

Hi, Neil

sorry for reply so late because some other issue.
yes, I set only two or three nodes as the connection pool, seed node and local machine ip. but my task still will not failed(and is all task failed) after I stop cassandra of one node.
in my environment, I have 3 nodes, 100.142.31.176(seed node),100.142.31.177,100.142.31.178, contact point in 100.142.31.176 is "100.142.31.176", contact point in 100.142.31.177 is "100.142.31.177,100.142.31.176", contact point in 100.142.31.178 is "100.142.31.178, 100.142.31.176". after I stop cassandra on 100.142.31.178, then all task failed after sometimes(already write some data). I do not know why, is this becuase of my replication configure? the configure is:

createKeyspace = "CREATE KEYSPACE IF NOT EXISTS" + keyspaceName_rep2 + "WITH replication = {'class': 'SimpleStrategy', 'replication_fator': '2'}"
createKeyspace = "CREATE KEYSPACE IF NOT EXISTS" + keyspaceName_rep3 + "WITH replication = {'class': 'SimpleStrategy', 'replication_fator': '3'}"

or there are some other parameter I need set?
在 2015年11月28日星期六 UTC+8下午9:17:52，Neil Ablang写道：

[Jet Li]

Hi, Neil

sorry, I forget the error code, the error code is: CASS_ERROR_SERVER_UNAVAILABLE(Unavailable)

sorry for reply so late because some other issue.
yes, I set only two or three nodes as the connection pool, seed node and local machine ip. but my task still will not failed(and is all task failed) after I stop cassandra of one node.
in my environment, I have 3 nodes, 100.142.31.176(seed node),100.142.31.177,100.142.31.178, contact point in 100.142.31.176 is "100.142.31.176", contact point in 100.142.31.177 is "100.142.31.177,100.142.31.176", contact point in 100.142.31.178 is "100.142.31.178, 100.142.31.176". after I stop cassandra on 100.142.31.178, then all task failed after sometimes(already write some data). I do not know why, is this becuase of my replication configure? the configure is:

createKeyspace = "CREATE KEYSPACE IF NOT EXISTS" + keyspaceName_rep2 + "WITH replication = {'class': 'SimpleStrategy', 'replication_fator': '2'}"
createKeyspace = "CREATE KEYSPACE IF NOT EXISTS" + keyspaceName_rep3 + "WITH replication = {'class': 'SimpleStrategy', 'replication_fator': '3'}"

or there are some other parameter I need set?
在 2015年11月28日星期六 UTC+8下午9:17:52，Neil Ablang写道：

[Neil Ablang]

Hi Liwei,

If you can send me a  snippet of  your source code (portion where you defined the contact point and creating the keyspaces) and cassandra.yaml of your servers, I can help you look further what are the issues

Best Regards,

Neil

[Jet Li]

Hi, Neil

below is some describe of my Demo code
void OpenSession()
{
judge is the session is null or not, not null then do some work and return.

new a cluster with cass_cluster_new api;
set request timeout use cass_cluster_set_request_timeout api;
set latency routing use cass_cluster_set_latency_aware_routing api;
set load balance use cass_cluster_set_load_balance_round_robin api;
new session use cass_cass_session_new api;

get point from configure file;//it like "100.142.31.177,100.142.31.176"
set contact point use cass_cluster_set_contact_points api;

connect session use cass_session_connect api;
...
some other error process operator
...
}
void OpenDB(const string& path)
{
OpenSession();
get the base keyspace name: keyspace_base
set keyspace replication 2: keyspace_rep2 = keyspace_base+"_rep2";
set keyspace replication 3: keyspace_rep3 = keyspace_base+"_rep3";
...
create key space...
create_key_space = "CREATE KEYSPACE IF NOT EXISTS " + keyspace_rep2
+ "WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': '2'} "";";
execute cass command use cass_session_execute api;
waiting for result sue cass_future_waitapi;
check result and do error process;
....
....
}

in cassandra.yaml, I set some paramerter and other parameter never set：
1. cluster name
2. seeds ip; 100.142.31.176
3. listen_adress: use the local machine ip of each node,like 100.142.31.177
4. rpc_adress: use the local machine ip of each node,like 100.142.31.177
5. commit log, cache and data path

result:
1. all task can success when all cassandra node is up
2. all task will failed after write some data if one cassandra node down

在 2015年12月2日星期三 UTC+8上午8:59:34，Neil Ablang写道：

[Jet Li]

Hi, Neil

below is some describe of my Demo code and cassandra.yaml file
void OpenSession()
{
if(m_session != NULL)
{
sessionNum ++;
return;
}

cluster = cass_cluster_new api();
cass_cluster_set_request_timeout(50000);
cass_cluster_set_latency_aware_routing api(cluster, cass_true);
cass_cluster_set_load_balance_round_robin(cluster);
m_seesion = cass_cass_session_new(cluster);

//contact point like "100.142.31.177,100.142.31.176"
const char* cintact_point = GetPointFromConfigFile();
cass_cluster_set_contact_points(cluster, cintact_point );

cass_session_connect(m_seesion, m_cluster);
...
//some other error process operator

...
}
void OpenDB(const string& path)
{
OpenSession();

//get the base keyspace name:
string keyspace_base = SetBaseName();
string keyspace_rep2 = keyspace_base+"_rep2";
string keyspace_rep3 = keyspace_base+"_rep3";
...
//create key space...
create_key_space = "CREATE KEYSPACE IF NOT EXISTS " + keyspace_rep2
+ "WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': '2'} "";";
Future result = cass_session_execute(m_seesion, create_key_space );
cass_future_wait(result);
//check result and do error process;
....
//create other key space and table

....
}

in cassandra.yaml, I set some paramerter and other parameter never set：
1. cluster name
2. seeds ip; 100.142.31.176
3. listen_adress: use the local machine ip of each node,like 100.142.31.177
4. rpc_adress: use the local machine ip of each node,like 100.142.31.177
5. commit log, cache and data path

result:
1. all task can success when all cassandra node is up
2. all task will failed after write some data if one cassandra node down

Best wishes
Liwei

[Michael Penick]

Is your application running queries at extremely high throughput?  

Also, I highly recommend you upgrade 2.3. Lots of fixes.

Mike

On Fri, May 13, 2016 at 4:43 AM, no3productionz <no3prod...@gmail.com> wrote:

Hi Liwei,

Were you able to resolve this?
I am encountering the same issue and would really appreciate some feedback.
I am using 2.1 version driver.

Thanks in advance!