Legitimate method to run quantized model on server?

[t kevin]

hi guys,

I’m trying to optimize my model with 8bit integer quantization for performance.
From what I learned from Post-training quantization  |  TensorFlow Model Optimization 1
the only way for TF to run a integer quantized model is through the tflite runtime.
I’m trying to deploy the service on the cloud with a powerful CPU server and a bunch of HW accelerators.
Right now we are running with native TF runtime and tfserving. it’s working well.
It sounds that the tflite is not designed for this scenario. also in some article it says the tflite implementation of cpu kernels are not best fit for server.
Please let me know what is the legitimate method to run quantized model on cloud.

Thank you very much.

Kevin

[Hyeonjong Ryu]

Hi Kevin,

We've recently worked on the x86 acceleration issue, I believe now many of the CPU kernel issues are fixed. If you still have the performance issue with your model please share more details with your model and running environment so we can investigate.

Thanks,

Hyeonjong

[T.J. Alumbaugh]

Hi Kevin,

You are correct that the mobile platform continues to be our primary use case. However, as Hyeonjong mentions, we have improved performance here recently. One important note about getting good performance on x86: not every compiler has good support for the necessary AVX intrinsics on x86. I would advise you to use as follows:

Clang: version 8 or above

GCC: version 9 or higher

MSVC: _MSC_VER 1920 or higher (Visual Studio 2019 Version 16.0)

If you compile TF Lite with any of the above, you will likely get the best performance on x86. Good luck!

-T.J.

[t kevin]

Hi Hyeonjong and T.J.

Thanks for you reply.
I have no problem with "that the mobile platform continues to be
our(tflite) primary use case."

Probably my question is more about design perspective.
Cloud inference service could also benefit from quantization (both
performance and power consumption) but why TF choose to bind
quantization inference to tflite and not supporting it with native
runtime.

I'll see what I can do with tflite.
Thanks again for your clarification.

Kevin

T.J. Alumbaugh <talu...@google.com> 于2021年5月26日周三 上午8:19写道：