No prepared statement with ID

[mediocregopher@gmail.com]

Hi all! We're having a small problem with Scylla that is happening intermittently, but first:

Docker Image: scylladb/scylla:3.1.1

Cluster size: 3 nodes

gocql version: ae2f7fc85f32

Issue

We're seeing the following error returned from gocql seemingly randomly:

No prepared statement with ID f1d8911ca7290a3db8e650fdb69e1839 found.

The first time it happened was under no load at  20:58:43, and the second time at 12:58:43 under somewhat high load. Both times with the same hash from the same client. The similarity of times makes me think there's some kind of scheduled job taking place on Scylla's side.

I've read in other places where people were seeing this happen that it might have to do with the prepared statement cache filling up, but we're only using two different queries currently (both INSERTs using ? placeholders for values, not batched or anything) so that seems unlikely. Is there any kind of visibility into the statement cache? Is that even a real thing? Any help would be appreciated here, it's not happening frequently but it's definitely something we want to know the cause of if possible. Thanks!

- Brian

[Shlomi Livne]

Brian Hi,

There is a prepared statements cache - so the "rumors" are correct that gets invalidated from time to time in some cases. There are actually hierarchically two - since with authorization there is a need to cache each prepared statement allowed per user/role login,

You can "see the caches" in action by enabling logging.

ndoetool setlogginglevel authorized_prepared_statements_cache trace

ndoetool setlogginglevel prepared_statements_cache trace

(by default both are set to info - like the rest of the system).

there are three variables in scylla.yaml 

# Validity period for permissions cache (fetching permissions can be an
# expensive operation depending on the authorizer, CassandraAuthorizer is
# one example). Defaults to 10000, set to 0 to disable.
# Will be disabled automatically for AllowAllAuthorizer.
# permissions_validity_in_ms: 10000

# Refresh interval for permissions cache (if enabled).
# After this interval, cache entries become eligible for refresh. Upon next
# access, an async reload is scheduled and the old value returned until it
# completes. If permissions_validity_in_ms is non-zero, then this also must have
# a non-zero value. Defaults to 2000. It's recommended to set this value to
# be at least 3 times smaller than the permissions_validity_in_ms.
# permissions_update_interval_in_ms: 2000

and also

permissions_cache_max_entries 1000 (that is the number of entries).

You can adjust the values of these configurations based on your setup and need.

I hope this helps, 

Shlomi

 

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/38c9cd1e-ea2a-462f-a04a-d0443a85bd96%40googlegroups.com.

[Brian Picciano]

Thanks for the detailed response Shlomi :) I'll enable tracing and see what happens. We do have authorization set up, but it seems like the statements should never really fall out of the cache at all, since there's only two of them. Do you think this is something wrong on our app-side or a bug in scylla?

You received this message because you are subscribed to a topic in the Google Groups "ScyllaDB users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scylladb-users/PGsaty8eB2s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAL5VWbf30Ua2F0bK%2BVeP5b5g3TUPidH%2BHd4fsY0Kqro0QT1x%2Bg%40mail.gmail.com.

[Shlomi Livne]

On Thu, Dec 5, 2019 at 6:41 PM Brian Picciano <mediocr...@gmail.com> wrote:

Thanks for the detailed response Shlomi :) I'll enable tracing and see what happens. We do have authorization set up, but it seems like the statements should never really fall out of the cache at all, since there's only two of them. Do you think this is something wrong on our app-side or a bug in scylla?

Your application is ok - the authorization cache will be invalidated (and with it the prepared cache). if you do not heavily depend on doing authorization changes (e.g. you have a 1 or 2 users that have a preset authorization and that is used all the time) than you can increase the intervals and it will make information cached used for longer period and as such invalidation will not happen. 

Please note that reppreparring prepared statements is automatically handled by the driver its behind the scene and most users/customers trust the driver todo its thing "blindly".

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAO2gzOvq2jEgkn5uN-k5h5ci56vtq1%2BKAz6reYe%2BTQYgFGq3jw%40mail.gmail.com.

[Brian Picciano]

Thanks for all your help Shlomi. Your last email made me think it's likely an error in gocql, and indeed I found this open issue: https://github.com/gocql/gocql/issues/1252. I'll try and keep up with that and see if anything comes of it.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAL5VWbccuCA6PBA7emVW-Uo9%2BSxViiMyrW3eeOKaridV4J1Jjw%40mail.gmail.com.