Questions for paper "Frame-Semantic Role Labeling with Heterogeneous Annotations"

[Bishan Yang]

Hi,

I am a postdoc working on semantic frame extraction. I emailed Meghana and she suggested to post my questions here. TheyM are mainly about the experiment setup in the paper "Frame-Semantic Role Labeling with Heterogeneous Annotations". 

1. Table 1 lists the train/test split of the exemplars in FrameNet1.5, but I didn't find descriptions in the paper about how this is done. Is it possible to share this data split with me? I hope to run experiments using the same setup and compare the results.

2. The beginning of page 4 mentions "training SEMAFOR using the exemplars". Did you retrain SEMAFOR using the exemplars instead of the full text data? 

3. Did you evaluate SEMAFOR on the test set of the exemplars? If yes, were the argument identification results similar to the results on the fulltext test set shown in Table 2?

I would really appreciate your help. Thanks!

Best,

Bishan

[Sam Thomson]

Hi Bishan,

The splits are available here: http://www.cs.cmu.edu/~ark/SEMAFOR/naacl2012_splits.tar.gz

In our best-performing model, we train on the full text sentences *plus* the exemplar sentences (and use hierarchy features, bottom row in table 2).

I think we did evaluate on exemplars, but chose not to report it because we believe the full-text annotations are more realistic. I don't remember the results exactly, but training on exemplars gives a *drastic* improvement when testing on exemplars. Maybe Meghana remembers the exact numbers?

Cheers,

-Sam

[Nathan Schneider]

I found a supplementary document with extra results that we decided not to publish. It includes a table of results on the exemplars. See attached.

Nathan

--
You received this message because you are subscribed to the Google Groups "semafor-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semafor-user...@googlegroups.com.
To post to this group, send email to semafo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semafor-users/e700c3fb-a32a-4aae-9673-e222a85b5f64%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Meghana]

Thanks Nathan, Sam.

Q3 is answered by the supplement that Nathan shared.

Q2: for the paper, like Sam said the results that are reported used both.
However, we also have the results where we trained using the exemplars only and evaluated it on both the fulltext test set and the exemplars test set. Here are the results:

Train on Exemplars only
Test on fulltext:           Recall=0.49049 (3094.5/6309.0)  Precision=0.61084 (3094.5/5066.0)  Fscore=0.54409
Test on exemplars:     Recall=0.65958 (4961.0/7521.5)  Precision=0.77010 (4961.0/6442.0)  Fscore=0.71057

The exemplar annotations have a different distribution as compared to the fulltext annotations w.r.t both the density of annotations and the role types. We think that is the reason why the performance on FT-test is worse that what the train-on-fulltext model gets (which gets an Fscore of ≈0.59).


Does this answer all your questions?

[Bishan Yang]

Thanks, Meghana, Nathan, and Sam. Sorry I just saw your messages, somehow I thought I would get notified...

I have some additional questions. For Meghana's answer to Q2, does "Train on Exemplars only" mean training on all exemplars or a subset of the examplars? Is it possible to share the prediction output for the setting "Train on Exemplars only, Test on fulltext"? 

I am getting results on the fulltext test set by runing Semafor from the github repository https://github.com/Noahs-ARK/semafor-semantic-parser. I am wondering whether this is the newest version of Semafor. If not, could you point me to the newest version or the version that is used in your paper if it is available?

Thanks!

Best,

Bishan

[Meghana Kshirsagar]

Hi Bishan,

I have some additional questions. For Meghana's answer to Q2, does "Train on Exemplars only" mean training on all exemplars or a subset of the examplars?

It means train on all exemplars (except of course the small test set of ≈4000 sentences/annotations). Here is the size of the exemplars train/test data:

 

Is it possible to share the prediction output for the setting "Train on Exemplars only, Test on fulltext"? 

Find attached.
Btw, the exemplars data is here: http://www.cs.cmu.edu/~mkshirsa/data/exemplars_data_for_swabha.tgz
 

I am getting results on the fulltext test set by runing Semafor from the github repository https://github.com/Noahs-ARK/semafor-semantic-parser. I am wondering whether this is the newest version of Semafor. If not, could you point me to the newest version or the version that is used in your paper if it is available?

The version with the hierarchical features is here: https://github.com/Noahs-ARK/semafor/tree/acl2015
The others are here (checkout the various branches): https://github.com/meghana-kshirsagar/semafor

 

[Meghana Kshirsagar]

(Removing all the other inline text so it's easier to read)

Hi Bishan,

I have some additional questions. For Meghana's answer to Q2, does "Train on Exemplars only" mean training on all exemplars or a subset of the examplars?

It means train on all exemplars (except of course the small test set of ≈4000 sentences/annotations). Here is the size of the exemplars train/test data:

 

Is it possible to share the prediction output for the setting "Train on Exemplars only, Test on fulltext"? 

Find attached.
Btw, the exemplars data is here: http://www.cs.cmu.edu/~mkshirsa/data/exemplars_data_for_swabha.tgz
 

I am getting results on the fulltext test set by runing Semafor from the github repository https://github.com/Noahs-ARK/semafor-semantic-parser. I am wondering whether this is the newest version of Semafor. If not, could you point me to the newest version or the version that is used in your paper if it is available?

The version with the hierarchical features is here: https://github.com/Noahs-ARK/semafor/tree/acl2015
The others are here (checkout the various branches): https://github.com/meghana-kshirsagar/semafor

Hope this helps,
Meghana.

[Bishan Yang]

Thanks a lot for the prompt reply!

Best,

Bishan

--

Bishan Yang

Postdoctoral Fellow 

Machine Learning Department

Carnegie Mellon University

5000 Forbes Avenue, Pittsburgh, PA, 15213

[eme...@gmail.com]

Hello, 

I am a PhD student at Purdue interested in this dataset and I want to work with the same splits. The link for the exemplars data is broken, would it be possible to get access to it?

Thank you very much.