org.projectnessie.client.rest.NessieBadRequestException: Bad Request (HTTP/400): Bad Request

[rei sivan]

Hi All,

I'm new to projectnessie and i'm trying to integrated it with spark 3.1.2

i have 2 pods on kubernetes one for spark and the other one for nessie INMEMORY deployment all the necessary jars is in spark jars folder.

the pods can communicate with each other and they under the same namespce.

here are the spark-shell input:

sh /opt/spark/bin/spark-shell    \

--master ${master}\

--name iceberg_test \

--deploy-mode client \

--conf spark.jars.ivy=/tmp/.ivy \

--conf spark.kubernetes.container.image=${image}\

--conf spark.kubernetes.namespace=${namespace} \

--conf spark.kubernetes.authenticate.subdmission.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt  \

--conf spark.driver.host=${host_ip} \

--conf spark.driver.bindAddress=${host_ip} \

--conf spark.driver.blockManager.port=7079 \

--conf spark.driver.port=7078 \

--conf spark.kubernetes.authenticate.submission.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token  \

--conf spark.kubernetes.authenticate.driver.serviceAccountName=${service_account}  \

--conf spark.dynamicAllocation.enabled=true \

--conf spark.dynamicAllocation.shuffleTracking.enabled=true \

--conf spark.dynamicAllocation.initialExecutors=1 \

--conf spark.dynamicAllocation.maxExecutors=50 \

--conf spark.dynamicAllocation.minExecutors=1 \

--conf spark.sql.execution.pyarrow.enabled=true \

--conf spark.sql.catalog.nessie=org.apache.iceberg.spark.SparkCatalog \

--conf spark.sql.catalog.nessie.catalog-impl=org.apache.iceberg.nessie.NessieCatalog \

--conf spark.sql.catalog.nessie.warehouse=/data/recordings/UC_tests/nessie_warehouse \

--conf spark.sql.catalog.nessie.uri=http://${HOST_IP}:${PORT}/api/v1 \

--conf spark.sql.catalog.nessie.ref=main \

--conf spark.sql.catalog.nessie.auth_type=NONE \

--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,org.projectnessie.spark.extensions.NessieSparkSessionExtensions

when I'm trying to  run a simple saveAsTable to nessie i get the following error:

org.apache.iceberg.shaded.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "errorCode" (class org.projectnessie.error.NessieError), not marked as ignorable (4 known properties: "serverStackTrace", "status", "reason", "message"])

 at [Source: (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 7, column: 2] (through reference chain: org.projectnessie.error.NessieError["errorCode"])

at org.apache.iceberg.shaded.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)

at org.apache.iceberg.shaded.com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:855)

at org.apache.iceberg.shaded.com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1212)

at org.apache.iceberg.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1604)

at org.apache.iceberg.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperties(BeanDeserializerBase.java:1554)

at org.apache.iceberg.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:440)

at org.apache.iceberg.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1322)

at org.apache.iceberg.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:331)

at org.apache.iceberg.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:164)

at org.apache.iceberg.shaded.com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2079)

at org.apache.iceberg.shaded.com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1453)

at org.projectnessie.client.rest.ResponseCheckFilter.decodeErrorObject(ResponseCheckFilter.java:85)

at org.projectnessie.client.rest.ResponseCheckFilter.checkResponse(ResponseCheckFilter.java:50)

at org.projectnessie.client.rest.NessieHttpResponseFilter.filter(NessieHttpResponseFilter.java:35)

at org.projectnessie.client.http.HttpRequest.lambda$executeRequest$6(HttpRequest.java:139)

at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

at org.projectnessie.client.http.HttpRequest.executeRequest(HttpRequest.java:139)

at org.projectnessie.client.http.HttpRequest.post(HttpRequest.java:178)

at org.projectnessie.client.ClientTreeApi.commitMultipleOperations(ClientTreeApi.java:186)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:564)

at org.projectnessie.client.NessieHttpClient$ExceptionRewriter.invoke(NessieHttpClient.java:210)

at com.sun.proxy.$Proxy36.commitMultipleOperations(Unknown Source)

at org.apache.iceberg.nessie.NessieTableOperations.doCommit(NessieTableOperations.java:114)

at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:126)

at org.apache.iceberg.BaseTransaction.commitCreateTransaction(BaseTransaction.java:249)

at org.apache.iceberg.BaseTransaction.commitTransaction(BaseTransaction.java:225)

at org.apache.iceberg.spark.source.StagedSparkTable.commitStagedChanges(StagedSparkTable.java:35)

at org.apache.spark.sql.execution.datasources.v2.TableWriteExecHelper.$anonfun$writeToTable$1(WriteToDataSourceV2Exec.scala:480)

at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1473)

at org.apache.spark.sql.execution.datasources.v2.TableWriteExecHelper.writeToTable(WriteToDataSourceV2Exec.scala:465)

at org.apache.spark.sql.execution.datasources.v2.TableWriteExecHelper.writeToTable$(WriteToDataSourceV2Exec.scala:460)

at org.apache.spark.sql.execution.datasources.v2.AtomicCreateTableAsSelectExec.writeToTable(WriteToDataSourceV2Exec.scala:97)

at org.apache.spark.sql.execution.datasources.v2.AtomicCreateTableAsSelectExec.run(WriteToDataSourceV2Exec.scala:118)

at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:40)

at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:40)

at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.doExecute(V2CommandExec.scala:55)

at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)

at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)

at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)

at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)

at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)

at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)

at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)

at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)

at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)

at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)

at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)

at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)

at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:686)

at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:619)

at <init>(<console>:26)

at <init>(<console>:30)

at <init>(<console>:32)

at <init>(<console>:34)

at <init>(<console>:36)

at <init>(<console>:38)

at <init>(<console>:40)

at <init>(<console>:42)

at <init>(<console>:44)

at .<init>(<console>:48)

at .<clinit>(<console>)

at .$print$lzycompute(<console>:7)

at .$print(<console>:6)

at $print(<console>)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:564)

at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)

at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1021)

at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:574)

at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:41)

at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)

at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)

at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)

at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:600)

at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:570)

at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:894)

at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:762)

at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:464)

at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:485)

at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:239)

at org.apache.spark.repl.Main$.doMain(Main.scala:78)

at org.apache.spark.repl.Main$.main(Main.scala:58)

at org.apache.spark.repl.Main.main(Main.scala)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:564)

at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)

at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)

at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)

at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)

at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)

at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

  at org.projectnessie.client.rest.ResponseCheckFilter.checkResponse(ResponseCheckFilter.java:55)

  at org.projectnessie.client.rest.NessieHttpResponseFilter.filter(NessieHttpResponseFilter.java:35)

  at org.projectnessie.client.http.HttpRequest.lambda$executeRequest$6(HttpRequest.java:139)

  at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

  at org.projectnessie.client.http.HttpRequest.executeRequest(HttpRequest.java:139)

  at org.projectnessie.client.http.HttpRequest.post(HttpRequest.java:178)

  at org.projectnessie.client.ClientTreeApi.commitMultipleOperations(ClientTreeApi.java:186)

  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.base/java.lang.reflect.Method.invoke(Method.java:564)

  at org.projectnessie.client.NessieHttpClient$ExceptionRewriter.invoke(NessieHttpClient.java:210)

  at com.sun.proxy.$Proxy36.commitMultipleOperations(Unknown Source)

  at org.apache.iceberg.nessie.NessieTableOperations.doCommit(NessieTableOperations.java:114)

  at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:126)

  at org.apache.iceberg.BaseTransaction.commitCreateTransaction(BaseTransaction.java:249)

  at org.apache.iceberg.BaseTransaction.commitTransaction(BaseTransaction.java:225)

  at org.apache.iceberg.spark.source.StagedSparkTable.commitStagedChanges(StagedSparkTable.java:35)

  at org.apache.spark.sql.execution.datasources.v2.TableWriteExecHelper.$anonfun$writeToTable$1(WriteToDataSourceV2Exec.scala:480)

  at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1473)

  at org.apache.spark.sql.execution.datasources.v2.TableWriteExecHelper.writeToTable(WriteToDataSourceV2Exec.scala:465)

  at org.apache.spark.sql.execution.datasources.v2.TableWriteExecHelper.writeToTable$(WriteToDataSourceV2Exec.scala:460)

  at org.apache.spark.sql.execution.datasources.v2.AtomicCreateTableAsSelectExec.writeToTable(WriteToDataSourceV2Exec.scala:97)

  at org.apache.spark.sql.execution.datasources.v2.AtomicCreateTableAsSelectExec.run(WriteToDataSourceV2Exec.scala:118)

  at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:40)

  at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:40)

  at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.doExecute(V2CommandExec.scala:55)

  at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)

  at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)

  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)

  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)

  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)

  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)

  at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)

  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)

  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)

  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)

  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)

  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)

  at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)

  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:686)

  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:619)

  ... 47 elided

if anyone encounter with this issue or can insight what am i missing it will be great.

Thanks,

[Eduard Tudenhoefner]

Hi Rei,

Which versions of Iceberg and Nessie are you using? In case you are using Iceberg 0.12.1, then you need to use Nessie 0.9.2 with it (instead of the latest Nessie version).

Once Iceberg 0.13.0 is out, it will contain Nessie 0.15.x, which includes backward compatible APIs. This will make it possible to use the latest Nessie versions with Iceberg, which wasn't the case for previous Iceberg/Nessie versions.

Regards

Eduard

--
You received this message because you are subscribed to the Google Groups "projectnessie" group.
To unsubscribe from this group and stop receiving emails from it, send an email to projectnessi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/projectnessie/82f15ed3-4c91-4397-a3cc-c27398bc7405n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Neelesh Salian]

Hi Rei,

Did you manage to solve this? Hit it while running Nessie as well.

[Neelesh Salian]

Hi Eduard,

Thanks for the response. Here are some more details of what I am seeing.

To reproduce:

//Spark-Shell Command:

spark-shell --packages "org.apache.iceberg:iceberg-spark3-runtime:0.12.1,
org.projectnessie:nessie-spark-extensions:0.9.2" \
--conf spark.sql.extensions="org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,
org.projectnessie.spark.extensions.NessieSparkSessionExtensions" \
--conf spark.sql.catalog.nessie.warehouse=s3://<bucket> \
--conf spark.sql.catalog.nessie.uri=http://0.0.0.0:19120 \
--conf spark.sql.catalog.nessie.ref=main \
--conf spark.sql.catalog.nessie.catalog-impl=org.apache.iceberg.nessie.NessieCatalog \
--conf spark.sql.catalog.nessie=org.apache.iceberg.spark.SparkCatalog \
--conf \
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions, \
org.projectnessie.spark.extensions.NessieSparkSessionExtensions

//Running Nessie on the machine via:

docker run -p 19120:19120 projectnessie/nessie \
-Dnessie.version.store.type=INMEMORY -Dquarkus.http.port=19120 \
-Dquarkus.log.level=DEBUG -Dquarkus.log.console.level=DEBUG \
-Dquarkus.log.category."io.netty".level=DEBUG \
-Dquarkus.log.category."io.quarkus.http.access-log".level=DEBUG

//Initializing the Nessie Catalog:

val nessieCatalog = new NessieCatalog()
nessieCatalog.setConf(spark.sparkContext.hadoopConfiguration)

val options1: Map[String, String] = Map[String, String](
("uri" -> "http://0.0.0.0:19120/api/v1"),
("ref" -> "main"),
("catalog-impl" -> "org.apache.iceberg.nessie.NessieCatalog"),
("authentication.type" -> "NONE"),
("warehouse" -> "s3://stitchfix.algo.infra/iceberg")
)
nessieCatalog.initialize("nessie_catalog",options1.asJava)

// Creating a table - where the failure occurs:

val regionSchema = new Schema(
Types.NestedField.optional(1, "R_REGIONKEY", Types.LongType.get()),
Types.NestedField.optional(2, "R_NAME", Types.StringType.get()),
Types.NestedField.optional(3, "R_COMMENT", Types.StringType.get()))

// and the partition
val regionSpec: PartitionSpec = PartitionSpec.unpartitioned()

// finally create the table
nessieCatalog.createTable(region_name, regionSchema, regionSpec)

// Error from the last statement

2022-02-02 02:54:47,138 DEBUG [org.pro.ser.res.BaseExceptionMapper] (executor-thread-0) Failure on server, propagated to client. Status: 400 Bad Request, Message: Cannot construct instance of `org.projectnessie.model.ImmutablePut`, problem: Cannot build Put, some of required attributes are not set [content]
 at [Source: (io.quarkus.vertx.http.runtime.VertxInputStream); line: 24, column: 3] (through reference chain: org.projectnessie.model.ImmutableOperations$Json["operations"]->java.util.ArrayList[0]).: com.fasterxml.jackson.databind.exc.ValueInstantiationException: Cannot construct instance of `org.projectnessie.model.ImmutablePut`, problem: Cannot build Put, some of required attributes are not set [content]
 at [Source: (io.quarkus.vertx.http.runtime.VertxInputStream); line: 24, column: 3] (through reference chain: org.projectnessie.model.ImmutableOperations$Json["operations"]->java.util.ArrayList[0])

[Neelesh Salian]

Will try with Iceberg 0.13.0 (recently released) and Nessie (0.18.0).

But doesn't really address the problem regarding why this is happening? Is it a bug?

[Eduard Tudenhoefner]

Sorry it took me a bit to spot this Neelesh. Essentially you need to run the same Nessie version (0.9.2) for the server. If you replace your Docker command with

docker run -p 19120:19120 projectnessie/nessie:0.9.2 \

-Dnessie.version.store.type=INMEMORY -Dquarkus.http.port=19120 \
-Dquarkus.log.level=DEBUG -Dquarkus.log.console.level=DEBUG \
-Dquarkus.log.category."io.netty".level=DEBUG \
-Dquarkus.log.category."io.quarkus.http.access-log".level=DEBUG

then it will work.

Previously your Nessie server was running with 0.18.0 but Iceberg 0.12.1 was using the Nessie 0.9.2 client and we had some breaking API changes in between those versions that we had to introduce before doing a backward-compatible GA release of Nessie.

Let me know if you have any other questions and if you're issue is resolved.

Best

Eduard

To view this discussion on the web visit https://groups.google.com/d/msgid/projectnessie/9a94e769-c2b8-4d35-83f4-63814b44f894n%40googlegroups.com.