Merge operation using Delta Standalone

[Aayush Jain]

Hello everyone,

I have been exploring the Delta standalone library (https://docs.delta.io/latest/delta-standalone.html) for performing low level operations like Update, Delete, Merge with a Java standalone. I am looking to perform these operations with one Delta table as a source and the other being the target. 

I can see it is possible using DeltaTable APIs using scala (https://docs.delta.io/latest/api/scala/spark/io/delta/tables/DeltaTable.html). 

What I am trying to understand is if there is a possibility of performing the same using Delta standalone? Please let me know if anyone has tried a similar thing successfully.

Thanks & Regards,

Aayush

[Tathagata Das]

Hello Aayush, 

There are 2 viable approaches depending on the scale of your data

- If the scale of your data is small that it does not need distributed processing engine, then you can use deltalake PyPI package - https://pypi.org/project/deltalake/ (e.g. merge)

- If the scale is large, then it's best to use a distributed engine like Spark. In that case, you can use merge either using the DeltaTable API or directly SQL using `spark.sql("MERGE INTO ...")`

Delta Standalone was designed to provide basic read and write support for Delta for writing connectors for any engine, and advanced operations like deletes and merges were out of scope as implementing those efficiently were left to the engines to implement efficiently.

Side note: Our new project Delta Kernel is the successor to Delta Standalone. But the scope is still the same, no advanced operations.

Hope this helps.

--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/fb2d2f60-92f7-4763-872d-f5b3589b3e4bn%40googlegroups.com.