DELTA_SPARK_SESSION_NOT_SET exception on Delta 3.0.0/Spark 3.5.0

[Vadim Vararu]

Hi,

Trying to migrate a spark/delta job from AWS EMR with Spark 3.4.0/Delta 2.4.0 to a newer version that has got Spark 3.5.0/Delta 3.0.0.

Running a simplified piece of code like:

DeltaLog deltaLog = DeltaLog.forTable (
    jobContext.sparkSession (),

    "s3://bucket/path/to/delta/table"

);

long filesCount = 0L;

if (deltaLog.tableExists ()) {
  filesCount = deltaLog.currentSnapshot ().snapshot ().numOfFiles ();
}

Runs successfully on Spark 3.4.0/Delta 2.4.0 but fails with the exception below on Spark 3.5.0/Delta 3.0.0:

Caused by: java.lang.RuntimeException: org.apache.spark.sql.delta.DeltaIllegalStateException: [DELTA_SPARK_SESSION_NOT_SET] Active SparkSession not set.
at org.apache.spark.sql.delta.DeltaErrorsBase.sparkSessionNotSetException(DeltaErrors.scala:1501)
at org.apache.spark.sql.delta.DeltaErrorsBase.sparkSessionNotSetException$(DeltaErrors.scala:1500)
at org.apache.spark.sql.delta.DeltaErrors$.sparkSessionNotSetException(DeltaErrors.scala:3039)
at org.apache.spark.sql.delta.util.NonFateSharingFuture.get(DeltaThreadPool.scala:94)
at org.apache.spark.sql.delta.CheckpointProvider$$anon$2.createCheckpointProvider(CheckpointProvider.scala:128)
at org.apache.spark.sql.delta.LazyCompleteCheckpointProvider.underlyingCheckpointProvider$lzycompute(CheckpointProvider.scala:409)
at org.apache.spark.sql.delta.LazyCompleteCheckpointProvider.underlyingCheckpointProvider(CheckpointProvider.scala:409)
at org.apache.spark.sql.delta.LazyCompleteCheckpointProvider.allActionsFileIndexes(CheckpointProvider.scala:415)
at org.apache.spark.sql.delta.Snapshot.fileIndices$lzycompute(Snapshot.scala:125)

Tried to google DELTA_SPARK_SESSION_NOT_SET but found almost nothing on that.

Thanks,

Vadim.