Check failed: error == cudaSuccess (8 vs. 0) invalid device function.got this issue when I run this ' ./examples/mnist/train_lenet.sh' in Official Caffe images

[Q Li]

I pull official caffe using "docker pull bvlc/caffe:gpu".

Starting a container by "nvidia-docker run -tdi --name=bvlcgpucaffe -v /home/ps/gpuworkir/:/workir/ bvlc/caffe:gpu"

Then I try to test the caffe image by minist, using './examples/mnist/create_mnist.sh' without changing any configurations for caffe container.

I got a error:

18 10:30:36.783571    76 caffe.cpp:248] Starting Optimization

I0618 10:30:36.783586    76 solver.cpp:272] Solving LeNet

I0618 10:30:36.783592    76 solver.cpp:273] Learning Rate Policy: inv

I0618 10:30:36.784211    76 solver.cpp:330] Iteration 0, Testing net (#0)

F0618 10:30:36.786037    76 pooling_layer.cu:212] Check failed: error == cudaSuccess (8 vs. 0)  invalid device function

*** Check failure stack trace: ***

    @     0x7f0ac77875cd  google::LogMessage::Fail()

    @     0x7f0ac7789433  google::LogMessage::SendToLog()

    @     0x7f0ac778715b  google::LogMessage::Flush()

    @     0x7f0ac7789e1e  google::LogMessageFatal::~LogMessageFatal()

    @     0x7f0ac7f1ed51  caffe::PoolingLayer<>::Forward_gpu()

    @     0x7f0ac7d1a9b1  caffe::Net<>::ForwardFromTo()

    @     0x7f0ac7d1aab7  caffe::Net<>::Forward()

    @     0x7f0ac7d0dbaa  caffe::Solver<>::Test()

    @     0x7f0ac7d0e82e  caffe::Solver<>::TestAll()

    @     0x7f0ac7d115d2  caffe::Solver<>::Step()

    @     0x7f0ac7d123ea  caffe::Solver<>::Solve()

    @           0x40ee7a  train()

    @           0x40b8a3  main

    @     0x7f0ac6473830  __libc_start_main

    @           0x40c249  _start

    @              (nil)  (unknown)

Environment:

cpu: Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz

MemTotal:  65832340 kB=62.78GB

disk: 3.7T*4

gpu: nvidia GeForce RTX 2080ti, 11016MiB*4=11GB*4

Driver Version: 430.34       CUDA Version: 10.1     

In docker container:

CUDA Version: 8.0

help:

• I thought the gpu caffe images can run in any nvidia gpu hardware, is it possible?I need to change some configuration?
• Is it possibel to use different cuda version in docker container or host?
• what should I do now for to use gpu caffe?