CRF Segment Level Confidence Score

[namrata ghadi]

Hello,

I have following set up:

1] Template Model for Linear Chain CRF with Bias, Evidence and Transition parameters (similar to https://github.com/factorie/factorie/blob/master/src/main/scala/cc/factorie/tutorial/ChainNERDemo.scala)

2] I am using batch training on Likelihood examples with Viterbi inference i.e  MaximizeByBPChain

3] And for inferencing the Test Data, I use the InferByBPChainMax

4] Finally, I get the label proportions distribution for each Token and pick the probability frm the distribution for the predicted / infered label in step 3

So, I basically get the marginal probabilities for each Token's predicted label.

What I am interested in is a Viterbi probability of the whole Sequence and the Viterbi Probability of segments within the sequence.

For example:

If my query is = Namrata Ghadi Software Engineer

And If my Viterbi score for each token is as follows:

Namrata is a Name-Beg with score = 0.99

Ghadi is a Name-Int with  score = 0.97

Software is a Jobtitle-Beg with score = 0.89

Engineer is a Jobtitle-Int with score = 0.90

Then, I would like to know if there is way to get following scores (not the individual level scores that we got above):

Score for most probable viterbi path through the linear chain.. (this is the overall probability of the sequence of labels that we got)

probability of the subpart of the path (for example, the viterbi probability of Segment = Namrata Ghadi , and viterbi probability of another segment (software engineer)

[namrata ghadi]

Hello,

Any inputs on this?

[namrata ghadi]

Hello,

So, I did some more research  on calculating segment level confidence scores and found the following paper:

https://people.cs.umass.edu/~mccallum/papers/crfcp-hlt04.pdf

This paper talks about Constrained Forward Backward algorithm that can give confidence scores of segments and not just token level scores.

Can someone please point me to an example that uses this to get segment level scores.

Thanks

Namrata

On Wednesday, July 26, 2017 at 3:44:56 PM UTC-7, namrata ghadi wrote:

[Luke Vilnis]

This is not implemented out of the box in factorie, but you would want to feed in custom weights / potentials that gave very low negative scores to all but the segment you would like to evaluate the score of, and regular potentials for the other segments, and then compute the likelihood using standard inference with the modified potentials.

This isn't builtin as far as a I know so you would have to build a bit of machinery, and the example you gave uses SampleRank I believe to train so it cannot give true probabilities. You could put something like this together by hacking around with ChainModel and its helpers (and LiteChainModel) which use exact forward-backward inference and have some helper methods to do inference with custom potentials.

--
--
Factorie Discuss group.
To post, email: dis...@factorie.cs.umass.edu
To unsubscribe, email: discuss+unsubscribe@factorie.cs.umass.edu

---
You received this message because you are subscribed to the Google Groups "Factorie" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@factorie.cs.umass.edu.