I have a flink stream job writing new data to a partitioned iceberg table. In the background, I'm running a spark procedure to rewrite data files. However, I saw a lot of warnings below in the log of the compacting job. Once in production, I may need to do compaction to a table with several concurrent spark procedures, and I'm worried about the conflicts in the warning, and would like to know more under the hood.
22/10/12 11:30:26 WARN Tasks: Retrying task after failure: Cannot commit: Reference hash is out of date. Update the reference 'dev' and try again
org.apache.iceberg.exceptions.CommitFailedException: Cannot commit: Reference hash is out of date. Update the reference 'dev' and try again
at org.apache.iceberg.nessie.NessieTableOperations.doCommit(NessieTableOperations.java:196)
at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:133)
at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:317)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:295)
at org.apache.iceberg.actions.RewriteDataFilesCommitManager.commitFileGroups(RewriteDataFilesCommitManager.java:89)
at org.apache.iceberg.actions.RewriteDataFilesCommitManager.commitOrClean(RewriteDataFilesCommitManager.java:110)
at org.apache.iceberg.actions.RewriteDataFilesCommitManager$CommitService.lambda$start$0(RewriteDataFilesCommitManager.java:179)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.projectnessie.error.NessieReferenceConflictException: Key 'datafinger.flash_orderbooklv2' has conflicting put-operation from commit 'd414c5f96713526addb322065b4ca58c2507f0a9fa1ea3151f978bfbad707898'.
at org.projectnessie.error.ErrorCode.lambda$asException$1(ErrorCode.java:60)
at java.base/java.util.Optional.map(Unknown Source)
at org.projectnessie.error.ErrorCode.asException(ErrorCode.java:60)
at org.projectnessie.client.rest.ResponseCheckFilter.checkResponse(ResponseCheckFilter.java:56)
at org.projectnessie.client.rest.NessieHttpResponseFilter.filter(NessieHttpResponseFilter.java:34)
at org.projectnessie.client.http.HttpRequest.lambda$executeRequest$3(HttpRequest.java:157)
at java.base/java.util.ArrayList.forEach(Unknown Source)
at org.projectnessie.client.http.HttpRequest.executeRequest(HttpRequest.java:157)
at
org.projectnessie.client.http.HttpRequest.post(HttpRequest.java:196)
at org.projectnessie.client.http.HttpTreeClient.commitMultipleOperations(HttpTreeClient.java:188)
at jdk.internal.reflect.GeneratedMethodAccessor339.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.projectnessie.client.http.NessieHttpClient$ExceptionRewriter.invoke(NessieHttpClient.java:161)
at com.sun.proxy.$Proxy50.commitMultipleOperations(Unknown Source)
at org.projectnessie.client.http.v1api.HttpCommitMultipleOperations.commit(HttpCommitMultipleOperations.java:58)
at org.apache.iceberg.nessie.NessieTableOperations.doCommit(NessieTableOperations.java:188)
... 13 more