Deploying a model in scale

[Lee Dongjin]

Hello. I am developing a system which uses predictions with stored ML model, like *.pkl from sklearn.

I'd like to use mlflow as a prediction service server but still doubt about its scalability. AFAIU, mlflow is providing Rest API only with flask. Although it will work well in dev or test environments, but I can't sure that it can handle millions of requests per second, which is so common nowadays.

In short, I have following questions to the main developers:

• Do you have any plan to support other types of RPC call in the future? I found that you are using protobuf to generate requests/responses. IMHO, it would be not so difficult to extend it to support gRPC on top of those definitions.
• Is there any plan to build a dedicated model service entity for scalability?

Thanks in advance. All kinds of opinions are welcome.

Best,

Dongjin

[Matei Zaharia]

Hi Lee,

The current model server (what you get with mlflow pyfunc serve and mlflow sagemaker and such) is meant to be used behind a load balancer and autoscaling to scale further. Although changing the RPC interface might help for some models, in others we’ll probably be bottlenecked by the cost of applying the model (especially for various Python ML libraries) so there’s not much we can do about it. The other thing we do plan to do though is to support Java-based serving as well, which you can see starting to be added with Mleap here: https://github.com/mlflow/mlflow/pull/278.

Please note that the MLflow UI itself (with experiment tracking) is *not* what does model serving. The MLflow UI uses Protobufs, but it won’t be involved in live serving in any way.

Hope this helps,

Matei

> --
> You received this message because you are subscribed to the Google Groups "mlflow-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mlflow-users...@googlegroups.com.
> To post to this group, send email to mlflow...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/mlflow-users/01b7ef82-6436-43fd-a4d2-7d1a4f8ce653%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Lee Dongjin]

Hi prof. Zaharia,

Understood! Thank you for your kind explanation and correcting my misunderstanding.

- Dongjin