Spark SQL doesn't allow empty IN in clauses

[Igor Markiewicz]

If we use an empty IN clause in sparkSession.sql("SELECT ... FROM ... WHERE ... IN ()") we're given org.apache.spark.sql.catalyst.ParseException (despite that the DataStax driver is fine with such query). Is it possible to use empty IN clauses with the spark-cassandra connector while selecting records and at the same time read them as a Dataframe?

[Alex Ott]

from the name of exception I would say that it's the Spark parser error, not related to connector... IIRC, SQL standard doesn't support such queries

On Sat, Nov 6, 2021 at 5:35 PM Igor Markiewicz <igom...@gmail.com> wrote:

If we use an empty IN clause in sparkSession.sql("SELECT ... FROM ... WHERE ... IN ()") we're given org.apache.spark.sql.catalyst.ParseException (despite that the DataStax driver is fine with such query). Is it possible to use empty IN clauses with the spark-cassandra connector while selecting records and at the same time read them as a Dataframe?

--
To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

--

With best wishes,                    Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)

[Igor Markiewicz]

I have the same impression, but

1. Isn't the spark-cassandra connector rewriting those Spark SQL rules?
2. Maybe there is another way to achive that, using the connector?

[Alex Ott]

Cassandra connector works on already parsed expression...

Create query programmatically, using Spark API

df = df.select(....)

if len(lst) > 0:

  df = df.filter(col("col1").isin(lst)

...