Recommended query size limits on execute concurrent

[Bhuvan Rawal]

Hi,

We are using Datastax Python Driver 3.4.0. I did a POC on execute_concurrent and found it suitable as per the performance requirements I played around a bit and found good performance is achieved at 500 concurrency level using around 15 processes (so max 7500 concurrent queries on the cluster). There is a certain situation where I want to execute around 10,000 Rows with same concurrency. I would like to know what would be a better way to go about it :

1. Issue execute_concurrent for a subset of rows say (1,000 rows per execution 10 times) with 500 Concurrency .

2. Submit 10,000 rows in one shot with 500 Concurrency and wait for the result to collect.

I remember facing a situation that I faced where I got "maximum recursion depth exceeded" though I cannot recall what was the size of rows I submitted and concurrency Level. 

Best Regards,

Bhuvan

[Adam Holmberg]

I'm not sure what conditions would lead to deep recursion. I would be interested in hearing more about that.

Responding to your question: since you are using the same concurrency with each, I think the main consideration is peak memory in your client. If you go with option #1 you can write it so less queries and results are present at one time.

Option #2 is optimal if you're okay materializing all requests and results in memory at once -- it will keep the execution pipeline full until the end instead of breaking it into chunks.

If memory footprint is a concern in the second case, please note that execute_concurrent accepts generators for inputs, and can optionally return a generator for results (results_generator parameter here).

Regards,

Adam Holmberg

--
You received this message because you are subscribed to the Google Groups "DataStax Python Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-driver-u...@lists.datastax.com.

[Bhuvan Rawal]

Hi Adam,

Thanks for your reply! 

As I do not have much of an issue with keeping 10k results in memory I can possibly shoot all of the requests at once. Just a curious question, how far can one go with this approach? 1 Lac, 1 Million? 

I have a situation where I would be counting things and updating counters. 

Ill give an example :  for a count of 10 I can do  (1+1+1 .... till 10 ) in memory or issue 10 Counter updates for + 1 each these are the two extreme scenarios. Or say count till 5 in memory and issue 5+5 in two counter updates. The number of counter updates will be reduced as I increase the number of rows processed per unit execution and thats essentially the idea. Im still searching for the best tradeoff for this particular situation.

I could retrieve logs for deep recursion. Here is what I did then:

I used concurrency of 100 and param size 5000. It was a select query and each param returns around 15 Rows. So 75,000 rows would have been returned in this case overall.  I got below error for each query:

  File "cassandra/concurrent.py", line 191, in cassandra.concurrent.ConcurrentExecutorListResults._put_result (cassandra/concurrent.c:5381)

    with self._condition:

  File "/usr/lib/python2.7/threading.py", line 288, in __exit__

    return self.__lock.__exit__(*args)

  File "/usr/lib/python2.7/threading.py", line 215, in __exit__

    self.release()

  File "/usr/lib/python2.7/threading.py", line 212, in release

    self._note("%s.release(): non-final release", self)

RuntimeError: maximum recursion depth exceeded

On increasing concurrency to 500 (Which was pretty much safe as per docs) and row_size to 2500 this issue was resolved.

Regards,

Bhuvan

PS: I had used execute_concurrent_with_args

[Adam Holmberg]

As I do not have much of an issue with keeping 10k results in memory I can possibly shoot all of the requests at once. Just a curious question, how far can one go with this approach? 1 Lac, 1 Million? 

The key parameter is 'concurrency' -- that limits the number of requests in flight at any one time. Outside of that, you can submit as many requests as you are able to hold in memory along with respsones.

 Ill give an example :  for a count of 10 I can do  (1+1+1 .... till 10 ) in memory or issue 10 Counter updates for + 1 each these are the two extreme scenarios. .... The number of counter updates will be reduced as I increase the number of rows processed per unit execution and thats essentially the idea. Im still searching for the best tradeoff for this particular situation.

The answer here depends a lot on your application. It's a trade between real-time updates, how fast these events come in, and whether you are able to recover if your app stops in the middle of an accumulation interval. Only you know those constraints. All things equal, if you can aggregate some at this level, your server infrastructure will be less taxed by sending fewer requests.

Deep recursion:

I found a way this can happen if your pools are overloaded or diminished and there are 1K+ requests failed on execution. Can you confirm that you were running with 'raise_on_first_error=False'? Also, can you describe your server version and client configuration?

I've created this to improve this mode of failure: https://datastax-oss.atlassian.net/browse/PYTHON-585

Adam Holmberg

[Bhuvan Rawal]

Thanks Adam, I'd love to do some tests on concurrency and processes with hardware constraints and share them here.

Yes I switched off raise_on_first_error. I had used DSC 3.0.3 for the test. If you wish I'll try reproducing same on test env.

Best Regards,
Bhuvan

[Adam Holmberg]

Don't worry about reproducing that. I'm able to show that coming up under contrived circumstances. You may find the root cause is the pools are overloaded somehow. We'll use the ticket mentioned before to try to short circuit and improve the error.