Is this possible?

[Drew Dwyer]

I'm experimenting with a Nessie + trino + iceberg implementation.

But i'd like to bulk ingest data to a table using flink, while performing deletes on the same table using trino, and i'd like to run nessie gc every 24h or so and remove the files from storage, all eventually without manual input and without corrupting the table.

Is this at all possible?

I was thinking something along the lines of 

- the ingestion job running on main

- the delete job creates a branch and merges on completion

- the gc job runs routinely, expires snapshots and removes orphan files

am i barking up the wrong tree?

[Dmitri Bourlatchkov]

Hi Drew,

Nessie ATM does not have content-aware merges [1], so the merge of deletes from a branch is very likely to conflict with ingestion on main (and require manual intervention to resolve).

Side note: please consider our Zulip Chat [2] for discussions :)

Cheers,

Dmitri.

[1] https://github.com/projectnessie/nessie/issues/2513

[2] https://project-nessie.zulipchat.com/

--
You received this message because you are subscribed to the Google Groups "projectnessie" group.
To unsubscribe from this group and stop receiving emails from it, send an email to projectnessi...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/projectnessie/8bc4bede-64d7-4d15-88f3-87ad73cf254cn%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.