CUBLAS STATUS INTERNAL ERROR

[sharath s]

I am trying to train FCN's using KITTI Road dataset, i have 3 output classes.
Input images are in png and ground truth are in png. i convert both of them to grayscale/Channel=1 = CxHxW 1x160x640.
i use a script from evan to convert them to lmdb and generate mean file. 

My ground truth labels/pixels are in integer.
Input images are in integer.

i get the following error

F0519 14:59:46.156497  1213 math_functions.cu:121] Check failed: status == CUBLAS_STATUS_SUCCESS (14 vs. 0)  CUBLAS_STATUS_INTERNAL_ERROR

*** Check failure stack trace: ***

    @     0x7fe912179daa  (unknown)

    @     0x7fe912179ce4  (unknown)

    @     0x7fe9121796e6  (unknown)

    @     0x7fe91217c687  (unknown)

    @     0x7fe912add27f  caffe::caffe_gpu_asum<>()

    @     0x7fe912ab879e  caffe::SoftmaxWithLossLayer<>::Forward_gpu()

    @           0x41c815  caffe::Layer<>::Forward()

    @     0x7fe912968b96  caffe::Net<>::ForwardFromTo()

    @     0x7fe912968893  caffe::Net<>::Forward()

    @     0x7fe91296952b  caffe::Net<>::ForwardBackward()

    @     0x7fe912992064  caffe::Solver<>::Step()

    @     0x7fe912991a48  caffe::Solver<>::Solve()

    @           0x417448  train()

    @           0x4196b8  main

    @     0x7fe910cc7ec5  (unknown)

    @           0x416019  (unknown)

    @              (nil)  (unknown)

Aborted (core dumped)

can anyone please help.

[sharath s]

My ground truth Labels were something like 27, 127 and 0 for three classes.

when i changed the labels to 0, 1 and 2..i dont hve the error anymore.

[Clint Sebastian]

Hi,

Thanks for this. I was having the same issue.

Do you know why the labels don't work with larger values?

Is it because of the different distances between the pixel values?

Clint

[6440...@qq.com]

Hi

I have the same issue.

I have a data set from the internet and want to ask how you change your label to a small value ? 

在 2016年5月19日星期四 UTC+8下午9:05:54，sharath s写道：

[sharath s]

if your dataset is having 3-4 classes for example.

if you know the ground truth pixel values for the object. 

for example 4 classes are Road, cyclist, sky, human. you should know what are the ground truth pixel value assigned to this.

then its simple. just check for object pixel value and replace it with 0, 1,2,3.

--
You received this message because you are subscribed to a topic in the Google Groups "Caffe Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/caffe-users/L0hFbFNPFcc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/b0e8a19e-109e-4289-b7a5-b80b6900b618%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Best Regards
Sharath Naidu

[Jianzhong He]

Hi @sharath s,
have you solved your problem and make it run,i have encountered some similar problems,could you give some advice
Thanks

在 2016年5月19日星期四 UTC+8下午9:05:54，sharath s写道：

I am trying to train FCN's using KITTI Road dataset, i have 3 output classes.

[Steve Lee]

I had the same issue and it was label problem. In my case, there was unexpected class name and label in my data. 

[Skand Vishwanath Peri]

Hey, 

Even I faced the same problem. The error is quite misleading. (Should have a proper error!) 

The problem is with the ground truth that I was using. I was doing image segmentation into foreground and background and there were 0, 1, and 255 values in the gt.

I had to remove those extra stray 255 pixel values.

Here is a better reference.

https://github.com/alexgkendall/caffe-segnet/issues/44