HTTP 413 Exception from Nessie

[Canope Nerda]

Hi folks,

I encountered the exception below when I used nessie as the catalog for my pipeline streaming data through flink to persist in the iceberg table. It seems to be an issue with the Nessie API. Any suggestions on how to configure the Nessie deployment? Thanks!

org.projectnessie.error.NessieBadRequestException: Bad Request (HTTP/400): HTTP 413 Request Entity Too Large (through reference chain: org.projectnessie.model.ImmutableOperations$Json["operations"]->java.util.ArrayList[0]->org.projectnessie.model.ImmutablePut$Json["content"]->org.projectnessie.model.ImmutableIcebergTable$Json["metadata"]->org.projectnessie.model.ImmutableGenericMetadata$Json["metadata"])

at org.projectnessie.error.ErrorCode.lambda$asException$1(ErrorCode.java:60)

at java.base/java.util.Optional.map(Unknown Source)

at org.projectnessie.error.ErrorCode.asException(ErrorCode.java:60)

at org.projectnessie.client.rest.ResponseCheckFilter.checkResponse(ResponseCheckFilter.java:56)

at org.projectnessie.client.rest.NessieHttpResponseFilter.filter(NessieHttpResponseFilter.java:34)

at org.projectnessie.client.http.HttpRequest.lambda$executeRequest$3(HttpRequest.java:157)

at java.base/java.util.ArrayList.forEach(Unknown Source)

at org.projectnessie.client.http.HttpRequest.executeRequest(HttpRequest.java:157)

at org.projectnessie.client.http.HttpRequest.post(HttpRequest.java:196)

at org.projectnessie.client.http.HttpTreeClient.commitMultipleOperations(HttpTreeClient.java:188)

at jdk.internal.reflect.GeneratedMethodAccessor184.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.base/java.lang.reflect.Method.invoke(Unknown Source)

at org.projectnessie.client.http.NessieHttpClient$ExceptionRewriter.invoke(NessieHttpClient.java:161)

at com.sun.proxy.$Proxy45.commitMultipleOperations(Unknown Source)

at org.projectnessie.client.http.v1api.HttpCommitMultipleOperations.commit(HttpCommitMultipleOperations.java:58)

at org.apache.iceberg.nessie.NessieTableOperations.doCommit(NessieTableOperations.java:188)

at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:133)

at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:317)

at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)

at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)

at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)

at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)

at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:295)

at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:325)

at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:287)

at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:229)

at org.apache.iceberg.flink.sink.IcebergFilesCommitter.initializeState(IcebergFilesCommitter.java:164)

at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)

at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:286)

at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)

at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:700)

at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)

at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:676)

at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:643)

at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)

at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:917)

at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)

at java.base/java.lang.Thread.run(Unknown Source)

Regards,
canopenerda

[Ajantha Bhat]

Hi Canopenerda,

a. What is the size of the latest table metadata JSON for this table? 
b. Looks to be the same issue as this (https://github.com/projectnessie/nessie/issues/5361)

In the upcoming Iceberg 1.1.0 version, we have fixed it (Iceberg will not send metadata JSON to Nessie) 

As a workaround,  maybe you can configure "nessie.transport.disable-compression"="true" from the Nessie client.
From the Flink, I guess it is just "transport.disable-compression" as I don't see a catalog name prefix via Flink Nessie configurations.

Thanks,
Ajantha

--
You received this message because you are subscribed to the Google Groups "projectnessie" group.
To unsubscribe from this group and stop receiving emails from it, send an email to projectnessi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/projectnessie/CAAQ4uE_T1vdB_CiFPkRuLxWwNrQtvgdTznv%2BQ3POMQxAO9sFJg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Canope Nerda]

Thanks Ajantha! Adding the config fixed the HTTP 413 issue. However, another issue surfaced. I only have one Flink job running and a transactional backend is used for Nessie. Would you mind helping me understand how the issue comes even when there is no concurrent commit?

org.apache.iceberg.exceptions.CommitStateUnknownException: Failed to execute POST request against 'http://nessie.nessie-ns.svc.cluster.local:19120/api/v1/trees/branch/dev/commit?expectedHash=b76384643cc39f9ffe5361622ee3d6621b6eacaaaeaac035813d23676a232206'.

Cannot determine whether the commit was successful or not, the underlying data files may or may not be needed. Manual intervention via the Remove Orphan Files Action can remove these files when a connection to the Catalog can be re-established if the commit was actually unsuccessful.

Please check to see whether or not your commit was successful before retrying this commit. Retrying an already successful operation will result in duplicate records or unintentional modifications.

At this time no files will be deleted including possibly unused manifest lists.

at org.apache.iceberg.nessie.NessieTableOperations.doCommit(NessieTableOperations.java:203)

at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:133)

at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:317)

at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)

at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)

at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)

at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)

at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:295)

at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:325)

at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:287)

at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:229)

at org.apache.iceberg.flink.sink.IcebergFilesCommitter.initializeState(IcebergFilesCommitter.java:164)

at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)

at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:286)

at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)

at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:700)

at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)

at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:676)

at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:643)

at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)

at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:917)

at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)

at java.base/java.lang.Thread.run(Unknown Source)

Caused by: org.projectnessie.client.http.HttpClientException: Failed to execute POST request against 'http://nessie.nessie-ns.svc.cluster.local:19120/api/v1/trees/branch/dev/commit?expectedHash=b76384643cc39f9ffe5361622ee3d6621b6eacaaaeaac035813d23676a232206'.

at org.projectnessie.client.http.HttpRequest.executeRequest(HttpRequest.java:183)

at org.projectnessie.client.http.HttpRequest.post(HttpRequest.java:196)

at org.projectnessie.client.http.HttpTreeClient.commitMultipleOperations(HttpTreeClient.java:188)

at jdk.internal.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.base/java.lang.reflect.Method.invoke(Unknown Source)

at org.projectnessie.client.http.NessieHttpClient$ExceptionRewriter.invoke(NessieHttpClient.java:161)

at com.sun.proxy.$Proxy47.commitMultipleOperations(Unknown Source)

at org.projectnessie.client.http.v1api.HttpCommitMultipleOperations.commit(HttpCommitMultipleOperations.java:58)

at org.apache.iceberg.nessie.NessieTableOperations.doCommit(NessieTableOperations.java:188)

... 23 more

Caused by: java.io.IOException: Error writing to server

at java.base/sun.net.www.protocol.http.HttpURLConnection.writeRequests(Unknown Source)

at java.base/sun.net.www.protocol.http.HttpURLConnection.writeRequests(Unknown Source)

at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)

at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)

at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)

at org.projectnessie.client.http.HttpRequest.executeRequest(HttpRequest.java:143)

... 32 more

Regards,
canopenerda

[Ajantha Bhat]

There is only one reason for a Nessie client to throw CommitStateUnknownException.
Which is whenever it gets a HttpClientException from the Nessie server as shown here. 

The below callstack looks to be a connection-related issue!

Caused by: java.io.IOException: Error writing to server
at java.base/sun.net.www.protocol.http.HttpURLConnection.writeRequests(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.writeRequests(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
at org.projectnessie.client.http.HttpRequest.executeRequest(HttpRequest.java:143)

[Canope Nerda]

Thanks Anantha. I checked the log from nessie pod and found the entry below. it seems the same 413 issue, but not surfaced to flink.

Regards,

canopenerda