Question about index limitations

[Fritz Stauffacher]

I was hoping to use elassandra for a project that would create many new indexes. From my understanding it appears Cassandra best practices is to keep table count below 200 to 500 since each index creates a keyspace with a new table I suppose we are limited here to the 200 to 500 indexes based on our cluster?

Do you have any suggestions on how to scale out one cluster with 1,000's of unique indexes or is this a limitation that can not be worked around.

[Mitch Gitman]

I had a vague recollection about the table limit and came across a "Too many tables" antipattern:

https://docs.datastax.com/en/dse-planning/doc/planning/planningAntiPatterns.html#Toomanytables

However, this one statement doesn't quite make sense: "... since each index creates a keyspace with a new table." Keyspaces and tables are purely Cassandra concepts. A keyspace is simply a namespace that can have any number of tables.

From there, you can define one set of Elasticsearch mappings per table. And that single JSON mappings definition could contain any number of index definitions where for each index definition you're creating a Cassandra secondary index and (forgive me if I'm getting the terminology wrong myself) a reverse-lookup Elasticsearch index. So if your table has five non-primary-key columns that you want to be able to use as query parameters for Elasticsearch, then you can define five indices for that table.

As for what actually is the upper limit of indices per cluster, someone else would have to answer that. But even then, I've found it's a good idea not to have a single monolithic Cassandra cluster being shared by various workloads but rather to have one Cassandra cluster per domain, whether that domain involves one keyspace or a few closely related ones. 

On Fri, Aug 21, 2020 at 4:07 PM Fritz Stauffacher <fritz...@gmail.com> wrote:

I was hoping to use elassandra for a project that would create many new indexes. From my understanding it appears Cassandra best practices is to keep table count below 200 to 500 since each index creates a keyspace with a new table I suppose we are limited here to the 200 to 500 indexes based on our cluster?

Do you have any suggestions on how to scale out one cluster with 1,000's of unique indexes or is this a limitation that can not be worked around.

--
You received this message because you are subscribed to the Google Groups "Elassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elassandra+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elassandra/78e16372-48b9-4de7-a0a7-7b8ff051bb1an%40googlegroups.com.

[Vincent Royer]

For each elasticsearch index, you can specify the underlying keyspace (see settings) and the document type match the cassandra table. Thus, you can have many elasticsearch indices in one keyspace. 

Another option is to create virtual indices sharing the same table and same mapping, see doc.

Le 22 août 2020 à 05:31, Mitch Gitman <mgi...@gmail.com> a écrit :



To view this discussion on the web visit https://groups.google.com/d/msgid/elassandra/CAEJc6MX4_UE_xPQYz_2oUBDUvqTA7NiraSOSACmNNAws0PDo5Q%40mail.gmail.com.