How to debug mlflow artificat connection?

[S Roychowdhury]

Hi, I am running into a problem where the mlflow server does not load the data from the artifact storage (which is a Minio for me) for a given run. Is there a way to debug mlflow side of communication with Minio server by enabling mlflow debug flags?

From the tables I know the artifact location and I am able to retrieve objects from the Minio server for that same artifact bucket and path. 

mlflow_db=# select * from experiments ;

 experiment_id |          name           | artifact_location | lifecycle_stage 

---------------+-------------------------+-------------------+-----------------

             1 | wine-pyfile-model       | s3://mlflow/1     | deleted

             2 | wine-pyfile-model-minio | s3://mlflow/2     | deleted

             3 | mlflow-wine             | s3://mlflow/3     | active

             4 | mlflow-wine-minio       | s3://mlflow/4     | active

             5 | rest-wine-1             | s3://mlflow/5     | active

             7 | rest-wine-2             | s3://mlflow/7     | active

             0 | rest-wine-2-updated     | s3://mlflow/0     | active

             8 | Default                 | s3://mlflow/8     | active

             9 | rest-wine-3             | s3://mlflow/9     | deleted

            13 | rest-wine-4             | s3://mlflow/13    | active

            14 | rest-wine-5             | s3://mlflow/14    | active

            15 | rest-wine-51            | s3://mlflow/15    | active

            16 | rest-wine-6             | s3://mlflow/16    | deleted

            17 | rest-wine-61            | s3://mlflow/17    | active

Any debugging ideas?

Thanks

Shounak   

 

[tkott]

I'm having a similar problem with my setup to minio. 

I'm starting my server with:

```

mlflow server \

    -p 80 \

    --host 0.0.0.0 \

    --backend-store-uri sqlite:///root/store.db \

    --serve-artifacts \

    --artifacts-destination s3://mlflow/ \

    --gunicorn-opts "--log-level debug"

```

In my case, when I run the `remote_server.py` example file and just passing the tracking URL, all the information is saved (including to the MinIO bucket) but MLFlow cannot show them in the UI. I'm trying to figure how to debug this as well. Starting with how to potentially save any logging information from the server itself.

I'd also appreciate debugging ideas.

[tkott]

Just to add to this, I can still grab the artifacts by setting the tracking server, then:

```

mlrun = mlflow.get_run('61788ca1775548ff83ceb9dcf23cb019')
client = mlflow.tracking.MlflowClient()

client.download_artifacts(mlrun.info.run_id, "another_dir")

```

That still works. 

So on my end it is only the UI that isn't able to show the artifacts.