Trying to use the mysql db with backend-store-uri

[Franklin Sarkett]

When I use a mysql url for the backend-store-uri flag, I cannot get the mlflow server to run. If I use a path and the filestore, it works fine.

 

I'm getting the standard error homepage: "Oops! Something went wrong"

Any suggestions?

[Franklin Sarkett]

Dockerfile and command:

FROM continuumio/anaconda3

RUN pip install --upgrade pip

RUN apt-get -y update && apt-get -y upgrade 

RUN apt-get install -y python3-dev default-libmysqlclient-dev

RUN pip install mysql-connector-python pymysql sqlalchemy mlflow

VOLUME ["/mlruns"]

WORKDIR /

CMD ["python", "--version"]

Run command:

sudo docker run -d \

  --name mlflow-backtest \

  -p 5000:5000 \

  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \

  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \

mlflow-server mlflow server \

  -h 0.0.0.0 -p 5000 \

  --default-artifact-root s3://... \

  --backend-store-uri mysql+pymysql://user:pass@host:3306/db

[Diego Serrano]

Hi, Franklin:

Were you able to find a fix?

We face a similar problem: We were trying to install mlflow service in a python3 environment using a mysql db to replace the local "file-store" (using the same backend-store-uri argument you specified); and we ran into the same issue, the package was working in the environment, we were able to start the service, but when going to the UI or trying to start and log any experiment, we got that "Oops..." message. 

Since we were redirecting the logs of the service to a file `nohup mlflow ... >  mlflow_server.out &`, we realized that the problem was when mlflow was trying to reach the database (there was a python error related to that). 

We might be using the wrong setup/requirements but we found out that the problem was that our environment was using python 3, and there might be some incompatibilities with the mysql packages. The fix that worked for us was using python 2.7, since then the requirements looked aligned (the mysql ones).

Good news is that you only need to have python 2.7 in the mlflow service's environment; you can use python3+mlflow in your running app and it works. Only thing to take into account is some dependencies like sklearn might not like python 2.7 (specially after the 05/10 release).

Please let us know if you find a better solution; or if what I just said is wrong please (anyone) correct me so we can also configure mlflow properly.

Thanks!

[Siddharth Murching]

Hi all,

This seems related to https://github.com/mlflow/mlflow/issues/1165 - in particular there is a bug in MLflow 0.9.1 in which deserializing runs from the database fails. The bug has been fixed in master & we plan to make an MLflow release in the near future containing the fix. For now, one possible workaround is to downgrade your MLflow server to 0.9.0.

Thanks,
Sid

--
You received this message because you are subscribed to the Google Groups "mlflow-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlflow-users...@googlegroups.com.
To post to this group, send email to mlflow...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mlflow-users/7e3f96d0-e732-4bb6-919c-ce8b5c5e01c3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.