Confidence Score for predictions in linear chain CRF

208 views
Skip to first unread message

namrata ghadi

unread,
Jul 12, 2017, 10:49:21 PM7/12/17
to Factorie
Hello,

I would like to know if there is any kind of confidence score metric that can tell us how confident we are about the predicted Label for each token.
For example, Token = Namrata, Predicted = Name with 90% confidence.  

namrata ghadi

unread,
Jul 19, 2017, 12:25:38 PM7/19/17
to Factorie
Hello,

Any update or info on this will highly help.
Thanks

Emma Strubell

unread,
Jul 19, 2017, 12:34:11 PM7/19/17
to Factorie

--
--
Factorie Discuss group.
To post, email: dis...@factorie.cs.umass.edu
To unsubscribe, email: discuss+u...@factorie.cs.umass.edu

namrata ghadi

unread,
Jul 19, 2017, 1:20:06 PM7/19/17
to Factorie
I looked at it and tried to do what is being suggested in the post.
I have created a summary variable of type DiscreteSummary1.
Now, after the inference step, when I loop throuhg my labels and try to get the marginal proportions for each label as follows:
After the inference steps, I printout the following:
unseenLabels.foreach { l =>
println(l.value)
println(l.caseFactorProportions(model))
println("Summary proportions = " + summary.marginal(l).proportions)
}
And I see, equal proportions for Summary marginal proportions.
To be clear, the output looks something like below, for example :
name-beg
Proportions(0.6282789584735522,0.19915230341850754,0.0575229127026467,0.0575229127026467,0.0575229127026467)
Summary proportions = Proportions(0.2,0.2,0.2,0.2,0.2)

My question is, are the caseFactorProportions, the probability / confidence score of different Label assignments to a particular token?
For example, in above case, say if the token was Namrata, and Label Domain = [name-beg, name-int, company-beg,company-int,company-int]
After inference, 
the probability of Namrata being a name-beg = 62.82%
probability of Namrata being a name-int = 19.91%
probability of Namrata being company-beg = 5.75%
so on...

??


On Wednesday, July 12, 2017 at 7:49:21 PM UTC-7, namrata ghadi wrote:

Luke Vilnis

unread,
Jul 19, 2017, 2:29:46 PM7/19/17
to dis...@factorie.cs.umass.edu
Yes, those are the probabilities.

--
--
Factorie Discuss group.
To post, email: dis...@factorie.cs.umass.edu
To unsubscribe, email: discuss+unsubscribe@factorie.cs.umass.edu

---
You received this message because you are subscribed to the Google Groups "Factorie" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@factorie.cs.umass.edu.

Luke Vilnis

unread,
Jul 19, 2017, 2:31:01 PM7/19/17
to dis...@factorie.cs.umass.edu
I'm not sure why the marginal function is returning uniform probabilities, though?

Emma Strubell

unread,
Jul 19, 2017, 2:38:08 PM7/19/17
to dis...@factorie.cs.umass.edu
My guess re: summary.marginal(l).proportions returning uniform proportions is that the summary isn't getting initialized and/or used correctly during inference, since l.caseFactorProportions(model) looks reasonable.

To unsubscribe, email: discuss+u...@factorie.cs.umass.edu


---
You received this message because you are subscribed to the Google Groups "Factorie" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@factorie.cs.umass.edu.


--
--
Factorie Discuss group.
To post, email: dis...@factorie.cs.umass.edu
To unsubscribe, email: discuss+u...@factorie.cs.umass.edu

namrata ghadi

unread,
Jul 19, 2017, 3:27:38 PM7/19/17
to dis...@factorie.cs.umass.edu
excellent. Thank Much @Luke @Emma.



---
You received this message because you are subscribed to the Google Groups "Factorie" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@factorie.cs.umass.edu.


--
--
Factorie Discuss group.
To post, email: dis...@factorie.cs.umass.edu

--
--
Factorie Discuss group.
To post, email: dis...@factorie.cs.umass.edu


---
You received this message because you are subscribed to the Google Groups "Factorie" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@factorie.cs.umass.edu.



--
It is only impossible until it’s done.
-Nelson Mandela
Reply all
Reply to author
Forward
0 new messages