By default, Caffe uses CUDA for its GPU computations. If your Android device had an Nvidia GPU, you could use it straightforwardly, but I don't suppose this is the case.
You might be able to run an
OpenCL branch however, so do research this possibility. I think there used to be a port somewhere, but can't find it at the moment.