Can't create artifacts in Nessie (argh!)

[Adam Morgan]

Hello, I am trying to use Nessie in a docker environment in conjunction with Iceberg, Trino,  Minio and Dremio.   In the docker config I did not create any network configurations as I wanted to keep things simple.  

I don't see to have any problems querying Nessie.  In both Trino and Dremio, I can create a connection to the Nessie endpoint and see the default artifacts.  From dremio, I can hit the Nessie endpoint to create tables - no problems.   I am not trying to create an iceberg catalog.  

I can access the endpoints in the browser to see the metadata:

For example:

/api/v2/config?warehouse=s3%3A%2F%2Fmy-bucket

{ "defaultBranch" : "main", "minSupportedApiVersion" : 1, "maxSupportedApiVersion" : 2, "actualApiVersion" : 2, "specVersion" : "2.1.0", "noAncestorHash" : "2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d", "repositoryCreationTimestamp" : "2025-02-12T16:43:25.535436114Z", "oldestPossibleCommitTimestamp" : "2025-02-12T16:43:25.535436114Z" }

What I can't do is create a catalog entry in python, pretty much anything for that matter.  I've tried numerous iterations of this script:

import requests

# Set up the Nessie server endpoint
nessie_url = "http://wash-nessie:19120/api/v2"

# Create a catalog entry
catalog_name = "my_catalog"
catalog_payload = {
    "type": "NESSIE",
    "uri": "http://wash-nessie:19120/api/v2",
    "warehouse": "s3://my-bucket",
    "ref": "main",
    "s3.access-key-id": "minio_user",
    "s3.secret-access-key": "minio_password",
    "s3.endpoint": "http://wash-minio:9000",
    "s3.region": "us-east-1"
}
response = requests.post(f"{nessie_url}/catalogs/{catalog_name}", json=catalog_payload)
if response.status_code == 201:
    print(f"Catalog '{catalog_name}' created successfully.")
else:
    print(f"Failed to create catalog '{catalog_name}':", response.status_code, response. Text)

this results in a 404 error.  I've had the same errors when using pyiceberg and load_catalog.  In those instances, I could see that it was messing with the end of the URI.   

The only thing of note in the logs are these:

2025-02-12 22:24:11,115 INFO  [io.qua.htt.access-log] (executor-thread-10) 172.19.0.5 - - [12/Feb/2025:22:24:11 +0000] "GET /api/v2/config HTTP/1.1" 200 374
2025-02-12 22:24:11,533 SEVERE [io.qua.ope.run.exp.otl.VertxGrpcExporter] (vert.x-eventloop-thread-3) Failed to export spans. The request could not be executed. Full error message: Connection refused: localhost/127.0.0.1:4317

from what I read, that shouldn't be causing my issue?  I tried to turn it off in my docker-compose - no luck.

speaking of Docker compose:

  nessie:
    image: projectnessie/nessie:latest
    container_name: wash-nessie
    ports:
      - "19120:19120"
    environment:
      - nessie.catalog.default-warehouse=warehouse
      - nessie.catalog.warehouses.warehouse.location=s3://my-bucket/
      - nessie.catalog.service.s3.default-options.endpoint=http://wash-minio:9000/
      - nessie.catalog.service.s3.default-options.access-key=urn:nessie-secret:quarkus:nessie.catalog.secrets.access-key
      - nessie.catalog.secrets.access-key.name=admin
      - nessie.catalog.secrets.access-key.secret=password
      - nessie.catalog.service.s3.default-options.region=us-east-1
      - nessie.server.authentication.enabled=false
      - NESSIE_DEFAULT_BRANCH=main
      - NESSIE_API_VERSION=2
      - QUARKUS_OPENTELEMETRY_ENABLED=false
      - QUARKUS_OIDC_ENABLED=false
      - QUARKUS_DATASOURCE_JDBC_URL=jdbc:postgresql://postgres:5432/iceberg
      - QUARKUS_DATASOURCE_USERNAME=iceberg
      - QUARKUS_DATASOURCE_PASSWORD=iceberg
    volumes:
      - /workspace/nessie:/workspace/nessie

Any help would be GREATLY appreciated.  I've spent days on this.  

[Dmitri Bourlatchkov]

Hi Adam,

> {nessie_url}/catalogs/{catalog_name}

What makes you believe that Nessie would accept a POST request at this URL?

Here's the full OpenAPI spec for Nessie API (including v2, which is based at /api/v2).

https://app.swaggerhub.com/apis/projectnessie/nessie/

For this kind of questions, please consider using Zulip if it's possible for you.

Cheers,

Dmitri.

--
You received this message because you are subscribed to the Google Groups "projectnessie" group.
To unsubscribe from this group and stop receiving emails from it, send an email to projectnessi...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/projectnessie/2774f8f3-7f35-4629-a286-b1187cd01c57n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adam Morgan]

Thank you for the response.  I had not found that page, and I think I was on the wrong api spec page before.  

So I have the basics working in postman.  Can you answer a question for me?  I am ultimately trying to use Nessie as the catalog service in my stack.   I want to user Superset to query it via trino.  In order for this to happen, I need to have schema created in the nessie catalog.  Is it possible to do this via the API?  I see the "schemas" reference at the bottom of the page that you mentioned, but it is not obviously clear how to use them or if they will even do what I want them to do.   My original attempts were using load_catalog in pyiceberg, but I couldn't get that to work.  Any help would be GREATLY appreciated!

[Adam Morgan]

I am putting a more detailed description on the other site - thank you again!