add/remove columns in partition - delta table on aws

[anais rossetto]

Hi everyone,

I wish to change the partition columns of one of my table.

It is a table created on aws in delta format.I can't really find what is the process to do it, anyone having some helps for me please ?

I have tried to run a job and write the following:

my_table.write.format("delta")\

.mode("overwrite") \

.option("mergeSchema", "true")\

.option("compression", "gzip")\

.option("overwriteSchema", "true") \

.partitionBy(*PARTITIONS_COLS)\

.saveAsTable(table_name)

Then i see the transaction log file updated, and all the new path with the partitions created.

I have updated the shema on aws glue, i see the right column names as partition.

The problem is when I want to update the schema on aws glue it says: "INVALID_GLUE_SCHEMA: Delta Lake table schema in Glue does not match the most recent schema of the Delta Lake transaction log. Please ensure that you have the correct schema defined in Glue."

I have try to run a crawler but the error remains the same.

Any idea what I should do ?

The schema of the table and their partitions column are all defined in my repo and i need to keep them updated.