IoU threshold problem

[Alex Ter-Sarkisov]

I got a bit confused on the computation of IoU reported by semantic segmentation papers. Pascal VOC leaderboard reports mean AP for IoU threshold (usually 50%), which is averaged for each class (20 classes) over the whole database. But, at the same time, 50% is a cut-off point, i.e. all intersections below it are considered failed. If, for example, I have 3 observations belonging to a particular class, and the model returns IoUs for them: 0.1, 0.6, 0.9, (and, let's say, corresponding precisions of 0.7, 0.6, 0.3) how do I compute mean IoU? Do I discard the first observation and get mean IoU of 0.75 for this class? In such case it is probably necessary to report something like percentage of failures, but I don't recall seeing it in papers.

For example, if we have 100 observations for some class in the dataset, and 99 of them have IoU of 0.01 and one 0.99?

Another thing I don't understand is how mAP per class is calculated. In the case above with 3 observations, for IoU>0.5, do I get 0.45? And, for IoU>0.75 0.3?

Sorry if the question is a bit confusing, I'll be happy to reformulate it.

[Evan Shelhamer]

It seems that you might be confusing *instance* segmentation with *semantic* segmentation. For semantic segmentation, these are common metrics: 

as summarized in https://arxiv.org/abs/1605.06211

For the PASCAL VOC leaderboard, the IU of each class is scored over all pixels and then all the classes (+ background, for 21 total) are averaged together. This scoring utility illustrates the general approach to the metrics https://github.com/shelhamer/fcn.berkeleyvision.org/blob/master/score.py#L19-L23 by first accumulating the histogram over all output and ground truth pixels.

For instance segmentation, refer to https://arxiv.org/abs/1407.1808 pg. 3 for instance or check out these slides on the COCO challenge http://image-net.org/challenges/talks/2016/ECCV2016_workshop_presentation_detection_segmentation.pdf

Evan Shelhamer

This email originated from DIT. If you received this email in error, please delete it from your system. Please note that if you are not the named addressee, disclosing, copying, distributing or taking any action based on the contents of this email or attachments is prohibited. www.dit.ie

Is ó ITBÁC a tháinig an ríomhphost seo. Má fuair tú an ríomhphost seo trí earráid, scrios de do chóras é le do thoil. Tabhair ar aird, mura tú an seolaí ainmnithe, go bhfuil dianchosc ar aon nochtadh, aon chóipeáil, aon dáileadh nó ar aon ghníomh a dhéanfar bunaithe ar an ábhar atá sa ríomhphost nó sna hiatáin seo. www.dit.ie

Tá ITBÁC ag aistriú go Gráinseach Ghormáin – DIT is on the move to Grangegorman

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/439d45c3-81bb-447f-a945-65384fdfc925%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Ter-Sarkisov]

Thanks Evan, so just to make sure I got you right:

if I have 1 class with 3 instances, IoUs with gt masks are 0.1,0.6,0.7, and corresponding precision of 0.7,0.5,0.2, and I use a threshold of IoU>0.5, what would be the precision in this case? Is it 2/3 (share of predictions with IoU>0.5) or 0.35 (0.2+0.5)/2? 

Alex

Evan Shelhamer

To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.