Spark job not able to create a branch

[Ridampreet Jaggi]

I am trying to create a nessie branch from the spark job. The job fails with parse error.

Here is the config and spark.sql statement

spark = glueContext.spark_session.builder \
        .config("spark.jars.packages","org.apache.iceberg:iceberg-spark-runtime-3.5_2.12-1.5.0,org.projectnessie.nessie-integrations:nessie-spark-extensions-3.5_2.12-0.82.0") \
        .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,org.projectnessie.spark.extensions.NessieSparkSessionExtensions") \
        .config("spark.sql.catalog.nessie.uri", url) \
        .config("spark.sql.catalog.nessie.ref", ref) \
        .config("spark.sql.catalog.nessie.authentication.type", auth_type) \
        .config("spark.sql.catalog.nessie.catalog-impl", "org.apache.iceberg.nessie.NessieCatalog") \
        .config("spark.sql.catalog.nessie.warehouse", full_path_to_warehouse) \
        .config("spark.sql.catalog.nessie", "org.apache.iceberg.spark.SparkCatalog") \
        .config("spark.sql.legacy.parquet.int96RebaseModeInRead", "CORRECTED") \
        .config("spark.sql.legacy.parquet.int96RebaseModeInWrite", "CORRECTED") \
        .config("spark.sql.legacy.parquet.datetimeRebaseModeInRead", "CORRECTED") \
        .config("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "CORRECTED") \
        .getOrCreate()


spark.sql("CREATE BRANCH test IN nessie FROM main")

Thanks