Evaluating MLFlow on Kubernetes

[Zak Hassan]

Hi,

I meet Matei and Mani during spark summit. We are currently evaluating MLFlow on Kubernetes and had some questions about it. 

Scenario: We have MLFlow Tracking server deployed in Kubernetes we also have a Jupyter notebook to run MLFlow Training. 

However, if we don't provide s3 credentials in Jupyter Notebook container it sends an error. Is it required for both Jupyter Notebook and MLFlow Tracking Server to have s3 credentials in the container? 

Apache Spark allows for us to specify an non s3 endpoint other than aws. That way we can use systems like Ceph to store our models. Is this something that will work for MLFlow. 

Thanks, 

Zak Hassan

Engineer - Artificial Intelligence -  Center Of Excellence, CTO Office
http://radanalytics.io/ - Machine Learning On OpenShift

[Mani Parkhe]

Hi Zak,

Can you describe where and how these notebooks are hosted (locally or in a container) and how are the ACLs set up to talk to container hosting MLflow tracking server? Also can you share the error you are seeing in the notebooks?

If you would like to open an issue on github, we can help debug that. 

Thanks.

-- 

Mani Parkhe

ma...@databricks.com