Connecting to Cassandra from Kafka Subscriber Callback
20 views
Skip to first unread message
Shibha Malik
Apr 23, 2018, 11:34:23 AM4/23/18
to DataStax Java Driver for Apache Cassandra User Mailing List
Hi All,
I am trying to find out if there could be any concerns to connect to Cassandra using this driver from a Kafka Subscriber Callback method ?
Thanks,
Shibha
Alex Ott
Apr 23, 2018, 2:06:52 PM4/23/18
to java-dri...@lists.datastax.com
Session & Cluster objects could be shared are thread safe, so potentially they could be used from the callback method, but I think that you should use async methods so you won't block thread that is executing callback...
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
Shibha Malik
Apr 23, 2018, 2:42:13 PM4/23/18
to DataStax Java Driver for Apache Cassandra User Mailing List
Could we end up loosing data if we use async in any case?
On Monday, April 23, 2018 at 11:06:52 AM UTC-7, Alex Ott wrote:
Session & Cluster objects could be shared are thread safe, so potentially they could be used from the callback method, but I think that you should use async methods so you won't block thread that is executing callback...
On Mon, Apr 23, 2018 at 5:34 PM, Shibha Malik <shib...@gmail.com> wrote:
Hi All,I am trying to find out if there could be any concerns to connect to Cassandra using this driver from a Kafka Subscriber Callback method ?Thanks,Shibha
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
Shibha Malik
Apr 23, 2018, 2:57:01 PM4/23/18
to DataStax Java Driver for Apache Cassandra User Mailing List
Also do we have to take care of cassandra cluster management ? Do you provide a self cluster management(through spark configuration) if I use spark cassandra connector ?
Alex Ott
Apr 24, 2018, 2:13:53 AM4/24/18
to java-dri...@lists.datastax.com
It depends on how you implement the corresponding handler for futures returned by async methods...
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
Alex Ott
Apr 24, 2018, 2:15:15 AM4/24/18
to java-dri...@lists.datastax.com
what do you mean under cluster management? What operations?
P.S. There is #datastax-drivers channel on DataStax Academy Slack - it could be faster to get answers there...--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
Shibha Malik
Apr 24, 2018, 5:47:30 PM4/24/18
to DataStax Java Driver for Apache Cassandra User Mailing List
Hi,
I am getting the below error for my implementation. Please help.
As per the documention here : https://github.com/datastax/java-driver/tree/3.x/manual/pooling
"The driver communicates with Cassandra over TCP, using the Cassandra binary protocol. This protocol is asynchronous, which allows each TCP connection to handle multiple simultaneous requests:"
2018-04-24T14:32:10.28-0700 [APP/PROC/WEB/0] ERR sh: 1: /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh: Permission denied
2018-04-24T14:32:42.62-0700 [APP/PROC/WEB/0] ERR Exception in thread "cluster67-connection-reaper-0" Exception in thread "Timer-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
2018-04-24T14:32:58.31-0700 [APP/PROC/WEB/0] ERR java.lang.OutOfMemoryError: GC overhead limit exceeded
2018-04-24T14:33:15.31-0700 [APP/PROC/WEB/0] ERR Exception in thread "cluster95-worker-0" Exception in thread "cluster101-connection-reaper-0"
2018-04-24T14:34:18.17-0700 [APP/PROC/WEB/0] ERR Exception in thread "http-nio-8080-ClientPoller-1" Exception in thread "cluster6-scheduled-task-worker-0" Exception in thread "cluster92-scheduled-task-worker-0" Exception in thread "cluster107-connection-reaper-0" Exception in thread "cluster113-worker-0" Exception in thread "cluster23-scheduled-task-worker-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
On Monday, April 23, 2018 at 11:15:15 PM UTC-7, Alex Ott wrote:
what do you mean under cluster management? What operations?P.S. There is #datastax-drivers channel on DataStax Academy Slack - it could be faster to get answers there...
On Mon, Apr 23, 2018 at 8:57 PM, Shibha Malik <shib...@gmail.com> wrote:
Also do we have to take care of cassandra cluster management ? Do you provide a self cluster management(through spark configuration) if I use spark cassandra connector ?
On Monday, April 23, 2018 at 8:34:23 AM UTC-7, Shibha Malik wrote:Hi All,I am trying to find out if there could be any concerns to connect to Cassandra using this driver from a Kafka Subscriber Callback method ?Thanks,Shibha
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
Shibha Malik
Apr 24, 2018, 5:55:24 PM4/24/18
to DataStax Java Driver for Apache Cassandra User Mailing List
So could this be because I am creating a new cluster object when ever i consume mesage from Kafka ?
Can i use a single instance of this object through the application , will this solve the problem ?Let me know please
Alex Ott
Apr 25, 2018, 3:06:48 AM4/25/18
to java-dri...@lists.datastax.com
Please use only one Cluster & Session object - over your whole process. Their creation is expensive, and time consuming.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
Shibha Malik
Apr 30, 2018, 8:59:35 PM4/30/18
to java-dri...@lists.datastax.com
Hi Alex,
This problem is resolved after i used one instance of cluster and session. But i have a new query.
I am using prepared statements and creating Bound Statement for each data point ( data entry like key - value ) and calls session.execute over it.
So you suggest i try the Async to make the inserts faster or the above way is the suggested way to do it. Please help.
Thanks,
Shibha
Alex Ott
May 1, 2018, 3:05:23 AM5/1/18
to java-dri...@lists.datastax.com
Hi Shibha
The fastest way will be to use both bound statement & async operations, but it's really depends on your business rules - for example, do you commit Kafka offsets after you insert data into DB, or you can use listener on future on retry if operation wasn't successful, or something like.Shibha Malik
May 1, 2018, 10:07:47 PM5/1/18
to java-dri...@lists.datastax.com
Hi Alex,
Thanks for the reply . I have implemented a combination of async and bound and got better performance in terms of number of data points written per second .
I am not making more than 200 concurrent write operations to Cassandra but my Cassandra table is only able to handle 5 million records and simply stops responding ( select and insert queries both ) ....I am not sure why ..could this be because of too many async calls ? Or may be the partition is full ? Please suggest I
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
Sent from Gmail Mobile
Alex Ott
May 2, 2018, 3:08:48 AM5/2/18
to java-dri...@lists.datastax.com
Hi
What do you see in the system.log of this node? Can you check how many inflight request do you have at that point of time (see https://docs.datastax.com/en/developer/java-driver/3.5/manual/pooling/#monitoring-and-tuning-the-pool for example).To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--Sent from Gmail Mobile
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
Shibha Malik
May 2, 2018, 10:53:28 AM5/2/18
to java-dri...@lists.datastax.com
I am trying to get access to system log of Cassandra . How can I find out the number of inflight request ???
I have a total of 5 node cluster ...
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--Sent from Gmail Mobile
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
Alex Ott
May 2, 2018, 11:23:56 AM5/2/18
to java-dri...@lists.datastax.com
There is an example at the link that I've sent in my last email
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--Sent from Gmail Mobile
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--Sent from Gmail Mobile--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
Shibha Malik
May 2, 2018, 11:26:09 AM5/2/18
to java-dri...@lists.datastax.com
Ok I’ll check how many in flights requests I have but how to decide on what should be the number I should be not to go beyond or exceed ..?
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--Sent from Gmail Mobile
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--Sent from Gmail Mobile--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
Alex Ott
May 2, 2018, 12:57:16 PM5/2/18
to java-dri...@lists.datastax.com
If you look into begin of the page, then there is description of how connection pooling, etc. works, and depending on your Cassandra version you can define how many inflight requests you can issue (there is also default value, described in documentation).
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--Sent from Gmail Mobile
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--Sent from Gmail Mobile--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--Sent from Gmail Mobile--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
Shibha Malik
May 2, 2018, 2:03:36 PM5/2/18
to java-dri...@lists.datastax.com
Ok thank you . I will take a look...
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--Sent from Gmail Mobile
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--Sent from Gmail Mobile--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--Sent from Gmail Mobile--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
Shibha Malik
May 2, 2018, 2:47:35 PM5/2/18
to java-dri...@lists.datastax.com
I have my concurrent write set to max 200. The page you sent says it should not exceed more than 254 for remote hosts...So do you think number of concurrent writes to cassandra could still be a problem of table not responding ( I am executing count(*) from table and it just stops responding after like 5M records ) in my case
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--Sent from Gmail Mobile
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--Sent from Gmail Mobile--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--Sent from Gmail Mobile--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.
Alex Ott
May 2, 2018, 2:53:02 PM5/2/18
to java-dri...@lists.datastax.com
Ouch - don't do count(*) on big tables :-)
In context of documentation, REMOTE means the nodes in the another DC, relative to current DC where your application resides (if you use DCAware load balancing policy - it's used by default, but you may need to specify what DC your application in - if you have multiple DCs).Shibha Malik
May 2, 2018, 3:28:31 PM5/2/18
to java-dri...@lists.datastax.com
ok . How to count how data is inserted then...i understand and had an idea that since count is a very expensive operation especially when data size gets large.....but i need a count of total records in the table.....my table have just 4 columns ...its not a big table though but data that i will be dealing in future would be billions.....
Before trying on increasing my threshold.....i want to make sure that is not what is causing the delay....so i am trying my hands on SYNC executes...if that doesnt cause that problem, it would mean that number of in - flights requests is causing the problem....let me know if you think otherwise...
Thanks for all your help so far...Appreciate it a lot!
Alex Ott
May 2, 2018, 3:43:22 PM5/2/18
to java-dri...@lists.datastax.com
Cassandra isn't optimized for count(*) queries because it needs to scan whole cluster for this operation. Effective count could be done only if you limit your query only to one partition, but this is not a case...
Quite often, such counting is done via Spark, but not via CQL directly.Shibha Malik
May 2, 2018, 3:43:35 PM5/2/18
to java-dri...@lists.datastax.com
Btw i experienced the same issue with sync calls.....then it looks like number of in-flight writes is not the issue....but the volume of data is so large that my request from GUI Cassandra interface is not able to display the number because this count is a very very expensive operation......
Alex Ott
May 2, 2018, 3:57:23 PM5/2/18
to java-dri...@lists.datastax.com
If you need to display the count, then you need to do something with your data model, but this is not a topic for this mailing list...
Shibha Malik
May 2, 2018, 4:03:11 PM5/2/18
to java-dri...@lists.datastax.com
ok . Thanks for your help .
Reply all
Reply to author
Forward
0 new messages