"1 ms/image" for inference in the "Why Caffe?"statement

[Juanjo Lopez-Villarejo]

I have a doubt on a statement made in the intro page

Caffe can process over 60M images per day with a single NVIDIA K40 GPU*. That’s 1 ms/image for inference and 4 ms/image for learning and more recent library versions and hardware are faster still. 

Is it really true that Caffe+hardware can process an image (inference) in just 1ms? Or are we talking that we have many, many images being processed in parallel and so on average each one would correspond to 1ms ?

I have in mind industrial applications for visual detection of defects. If we talk of an industrial chain process, sometimes we need a speed of the order of miliseconds or less in order to take a decision for a given piece/object.



 

[Juanjo Lopez-Villarejo]

In this article it would seem that the total process of visual recognition take around 30s
https://software.intel.com/en-us/articles/caffe-optimized-for-intel-architecture-applying-modern-code-techniques#Fig4

[Przemek D]

That 1ms number is for a heavily parallel inference, I think it's possible for a batch of 128 or something like this. In an industrial setting, where you care about latency and thus can't wait to acquire another image (and your batch size is 1), it will be hard to get as low as that. For example (my own work, coincidentally an industrial application), a Titan Z (single core) needs about 10ms to process a slightly optimized AlexNet on a single image.