Caffe performance w/ docker images

[Anand Thakker]

Hi all,

I've been running into some pretty significant differences in performance using caffe from a docker image.

Timing `net.forward()` on a net slightly larger than the VGG_ILSVRC_16_layer, I get:

 - Mac OS X, gpu mode: 500-1000 ms

 - Mac OS X, cpu mode: 1000-2000 ms

 - Ubuntu on AWS g2.2xlarge, gpu mode: 100-750 ms

 - Ubuntu on AWS g2.2xlarge, cpu mode: 5000 ms (???) 

 - Docker image, based on nvidia/cuda:7.0-cudnn2-devel-ubuntu14.04, running with nvidia-docker on the g2 instance: 5000 ms

 - Same docker image, running with normal docker client in CPU mode on OS X machine: 5000 ms

Any ideas why these last three would show such poor performance?

Given the fact that CPU mode on the ubuntu box WITHOUT docker is performing quite badly, I was wondering if it might have something to do with the linux versions of some dependency like BLAS or something... are there any known issues of that sort?

Thanks in advance,

Anand

[Felix Abecassis]

Why are you using a Docker image based on cuDNN v2?
You should use cuDNN v5, you can use the official Docker image:
https://github.com/BVLC/caffe/blob/master/docker/standalone/gpu/Dockerfile
https://github.com/BVLC/caffe/tree/master/docker

[Anand Thakker]

Ah, I should have mentioned: I'm actually using https://github.com/alexgkendall/caffe-segnet, a fork that was forked a while back. 

I suspect that this is *not* the real problem, though.  Even if I run this test in CPU mode on my Mac, it is ~ 1000-2000ms, whereas through the docker container (again in CPU mode) it's ~5000ms.

[Felix Abecassis]

On Mac OS, Docker container runs inside a xhyve VM, there might be a performance overhead.
On a g2 instance, I haven't noticed any performance difference in Docker, so you probably have a different environment and that's all. Be also sure to mount volumes for your input and output folders of Caffe, you don't want to be doing I/O inside the Docker filesystem directly.

[Anand Thakker]

Just to note: starting again from scratch using the default docker-machine AMI, this problem seems to be resolved.  Looks like Felix's guess that it was something wrong w/ the environment was correct.  Thanks for the suggestions.