TREC KBA queries for CCR and SSF

[John R. Frank]

KBAers,

Please read and respond.

We will hold open discussion on the SSF target slots until next Friday
April 19th. Please give us reactions and any suggested changes.

If you want to see another slot type added, speak up now.

Tomorrow morning, the query entities and target slots will be posted to
the Active Participants section of trec.nist.gov

Look for this tarball:
trec-kba-ccr-and-ssf-2013-04-08.d41d8cd98f00b204e9800998ecf8427e.tar.gz

Here is the README from that tarball.


Regards,
The KBA Organizers



KBA 2013
========

The list of KBA 2013 entities and target slots to fill is enclosed in
this tarball. The format is:
http://trec-kba.org/schemas/v1.1/filter-topics.json

We will hold open discussion on the SSF target slots until next Friday
April 19th. Please give us reactions and any suggested changes.

If you want to see another slot type added, speak up now.


Entities for both CCR and SSF
-----------------------------

Entities come from both Wikipedia and Twitter.

Since Twitter does not offer a name-expansion API, it is acceptable to
manually examine the twitter profile page to identify alternate names
for these entities. This is still considered "run_type": "automatic",
because a human entering this entity as a query could easily be asked
to examine the twitter profile page (and no other texts).

The training time range (TTR) runs through the end of February 2012.

Some entities have no vital examples in the TTR.
For example, some entities only appear in the corpus when they die in
the second half of 2012.

You will notice that the proportion of judged documents increases
signficantly some time after the TTR, and yet, the number of vitals
does not. This is because the spinn3r portion of the corpus has more
re-visits to certain URLs, so we enabled the assessors to assign
ratings to these pages in bulk. We also filtered out many of these
re-visits.

To optimize assessor resources in the first assessment period (March
2013), we did not judge every document fo four of the larger new
entities not any of the entities from KBA 2012. If time permits
during the June/July assessor window, we will conduct more vital
judging on pooled assertions from run submissions for all entities.


Streaming Slot Filling
----------------------

Each entity is one of these three types:

Facility (FAC)
Person (PER)
Organization (ORG)


The entity_type determines the target slots to fill:

type PER: Affiliate, Contact_Meet_PlaceTime, AwardsWon, DateOfDeath, CauseOfDeath, Titles, FounderOf, EmployeeOf

type FAC: Affiliate, Contact_Meet_Entity

type ORG: Affiliate, TopMembers, FoundedBy


Three of these slots require special explanation (below). The other
values are directly from TAC-KBP or ACE definitions:

http://www.nist.gov/tac/2012/KBP/task_guidelines/TAC_KBP_Slots_V2.4.pdf

http://projects.ldc.upenn.edu/ace/docs/English-Events-Guidelines_v5.4.3.pdf


Since this is *streaming* slot filling, we are only interested in new
slot values that were not substantiated earlier in the streamcorpus,
which ranges from October 2011 to February 2013. In examining the
Vital-rated documents identified by assessors, we observed that many
of the interesting events in the lives of these people and
organizations, did not easily fit existing slots in KBP or ACE.
Rather than invent specific new slots, we propose these three
generalized slots classes.


1) "Affiliate" is any type of relation that *directly* connects the
target entity to another entity of any type. This is the union of all
close relations, such as StudentOf, EmployeeOf, MemberOf, and their
inverses from ACE and KBP, and similar relations in which the relation
is of a simple unambiguous type. We propose to use this generalized
notion of "close" relation as the base class for all the PER-PER and
PER-FAC and PER-ORG relations. Instead of enumerating all possible
such close relations, we intend to allow KBA systems to generate many
such examples, which assessors will then judge in the post-hoc pooled
assessment scheduled for June 17-July 17 2013.

While this is somewhat in the spirit of open IE, our aim is
specifically to enable upstream filtering systems to down-select the
stream for feeding KB population systems operating with fixed
inventories of slot types, which will typically be more specific than
Affiliate.

Since little annotation has been performed for "Affiliate", we are
concerned that it could generate a lot of noise. We welcome feedback
and ideas on how to refine this.


Example 1: "Matthew DeLorenzo and Josiah Vega, both 14 years old and
students at Elysian Charter School, were honored Friday morning by
C-SPAN and received $1,500 as well as an iPod Touch after winning a
nationwide video contest."

target_id: http://en.wikipedia.org/wiki/Elysian_Charter_School

Affiliate: "Matthew DeLorenzo"
Affiliate: "Josiah Vega"

NOT an affiliate: "C-SPAN"
NOT an affiliate: "iPod Touch"


Example 2: "Allen Curtis, administrator, Citrus Health and Rehab, and
Inverness Mayor Bob Plaisted, standing from left in rear, joined the
Joyner family in celebrating the 104th birthday of Edith Joyner on
Wednesday at Citrus Health and Rehab."

target_id: https://twitter.com/bobplaisted

Affiliate: "Allen Curtis"
Affiliate: "the Joyner family"
Affiliate: "Edith Joyner"

NOT an Affiliate: "Citrus Health and Rehab"

That is, merely being present at a place once does not substantiate an
Affiliation. Affiliation is closer than Meeting(place-time).


Example 3: "Veteran songwriters and performers Ben Mason, Jeff
Severson and Jeff Smith will perform on Saturday, April 14 at 7:30 pm
at Creative Cauldron at ArtSpace, 410 S. Maple Avenue."

target_id: http://en.wikipedia.org/wiki/Jeff_Severson

Affiliate: "Ben Mason"
Affiliate: "Jeff Severson"
Affiliate: "Jeff Smith"

NOT an Affiliate: "Creative Caldron"
NOT an Affiliate: "Art Space"


Example 4: "Driftwagon has teamed up with Peruvian designer
Dunkelvolk."

target_id: http://en.wikipedia.org/wiki/Dunkelvolk

Affliate: "Driftwagon"


2) Contact_Meet_Entity is a catch-all slot for Facilities. It is a super
set including any event in which one or more entities (of any type) are present
at the target facility.

Example: "The Senior Wellness Coalition of Fargo-Moorhead will host a
wellness seminar from 1 to 3 p.m. March 28 at the Hjemkomst Center,
202 1st Ave."

target_id: http://en.wikipedia.org/wiki/Hjemkomst_Center

Contact_Meet_Entity: "The Senior Wellness Coalition of Fargo-Moorhead"


3) Contact_Meet_PlaceTime is a catch-all slot for Persons. It is a super
set including any event in which the target entity is present at a
particular place at a particular time. For SSF, we want short
passages that a human or downstream algorithm can digest to generate
structured location/date-time values.

The place or time might not specified in the text. When this happens,
just emit the best available string.

Example: "Lt. Gov. Drew Wrigley and Robert Wefald, a retired North Dakota
district judge and former state attorney general, unveiled the crest Friday
during a ceremony at the North Dakota Capitol."

target_id: http://en.wikipedia.org/wiki/Hjemkomst_Center

Contact_Meet_PlaceTime: "Friday during a ceremony at the North Dakota Capitol"



Output Format
-------------

We have added an optional component to the run submission format
described here:

http://trec-kba.org/trec-kba-2013.shtml

Specfically:

ninth column: slot name from the TAC KBP slot ontology. Used in
SSF. Runs for CCR should use 'NULL' in this field. This field most
have a string from the list below. Optionally, this field may contain
a second string separated from the first by a colon ":", where the
second string is a system-selected name for a sub-type or variant of
the target slot. This will not be used in scoring and is provided
solely for the purpose of allowing systems to output more information
about the algorithm's perspective on the slot. This field must not
contain any spaces.

[ps]

The KBA 2013 queries, training data, and inter-assessor agreement matrix are posted here:

      http://trec.nist.gov/act_part/tracks/kba/2013/trec-kba-ccr-and-ssf-2013-04-08.d41d8cd98f00b204e9800998ecf8427e.tar.gz

To get a username password, you simply need to submit this registration form to NIST:

       http://trec.nist.gov/data/kba.html

Regards,

The KBA Organizers

[John R. Frank]

KBAers,

We are making one important change to the SSF slots, described below.

Summary:

Affiliate(PER, PER) --> AssociateOf

The updated (and final) tarball of training data and queries will appear
here as soon as NIST staff can post it (hopefully next couple hours):

http://trec.nist.gov/act_part/tracks/kba/2013/trec-kba-ccr-and-ssf-2013-04-22.a27da77716822ff420cbd6f1f104fa25.tar.gz

Note that this training data now has the four entities that contained \"
and _, characters in their target_id -- they are interesting entities :-)

I will post a diff of this README against the previous in just a moment.

jrf

KBA 2013
========

The list of KBA 2013 entities and target slots to fill is enclosed in
this tarball. The format is:
http://trec-kba.org/schemas/v1.1/filter-topics.json

Entities for both CCR and SSF
-----------------------------

Entities come from both Wikipedia and Twitter.

Since Twitter does not offer a name-expansion API, it is acceptable to
manually examine the twitter profile page to identify alternate names
for these entities. This is still considered "run_type": "automatic",
because a human entering this entity as a query could easily be asked
to examine the twitter profile page (and no other texts).

The training time range (TTR) runs through the end of February 2012.

Some entities have no vital examples in the TTR.
For example, some entities only appear in the corpus when they die in
the second half of 2012.

You will notice that the proportion of judged documents increases
signficantly some time after the TTR, and yet, the number of vitals
does not. This is because the spinn3r portion of the corpus has more
re-visits to certain URLs, so we enabled the assessors to assign
ratings to these pages in bulk. We also filtered out many of these
re-visits.

To optimize assessor resources in the first assessment period (March

2013), we did not judge every document for four of the larger new

entities not any of the entities from KBA 2012. If time permits
during the June/July assessor window, we will conduct more vital
judging on pooled assertions from run submissions for all entities.


Streaming Slot Filling
----------------------

Each entity is one of these three types:

Facility (FAC)
Person (PER)
Organization (ORG)


The entity_type determines the target slots to fill:

type PER: Affiliate, AssociateOf, Contact_Meet_PlaceTime, AwardsWon, DateOfDeath, CauseOfDeath, Titles, FounderOf, EmployeeOf

type FAC: Affiliate, Contact_Meet_Entity

type ORG: Affiliate, TopMembers, FoundedBy

Four of these slots require special explanation (below). The other

values are directly from TAC-KBP or ACE definitions:

http://www.nist.gov/tac/2012/KBP/task_guidelines/TAC_KBP_Slots_V2.4.pdf

http://projects.ldc.upenn.edu/ace/docs/English-Events-Guidelines_v5.4.3.pdf


Since this is *streaming* slot filling, we are only interested in new
slot values that were not substantiated earlier in the streamcorpus,
which ranges from October 2011 to February 2013. In examining the
Vital-rated documents identified by assessors, we observed that many
of the interesting events in the lives of these people and
organizations, did not easily fit existing slots in KBP or ACE.
Rather than invent specific new slots, we propose these three
generalized slots classes.

1.1) "Affiliate" is any type of relation that *directly* connects the
target entity to another entity. Affiliate must have a FAC or ORG on
at least one side of the relation. Affiliate is its own inverse.

1.2) "AssociateOf" is the PER-PER analog of Affiliate. Both sides of
this relation must have a person-type entity.

The challenge with both Affiliate and AssociateOf is judging what
level of "closeness" distinguishes these relations from two entities
that merely co-occurring in the same sentence or passage. By
attempting these new slots in KBA 2013, we will be able to measure
inter-assessor agreement and work toward more refined guidelines than
the few examples below.

Affiliate is the union of all

close relations, such as StudentOf, EmployeeOf, MemberOf, and their
inverses from ACE and KBP, and similar relations in which the relation
is of a simple unambiguous type. We propose to use this generalized

notion of "close" relation as the base class for all the ORG-ORG, ORG-FAC,

PER-FAC and PER-ORG relations. Instead of enumerating all possible
such close relations, we intend to allow KBA systems to generate many
such examples, which assessors will then judge in the post-hoc pooled
assessment scheduled for June 17-July 17 2013.

While this is somewhat in the spirit of open IE, our aim is
specifically to enable upstream filtering systems to down-select the
stream for feeding KB population systems operating with fixed
inventories of slot types, which will typically be more specific than

Affiliate or AssociateOf. By filtering the stream, KBA systems allow
such KBP systems to focus compute power on a smaller substream.
Further, the less strongly typed graph of affiliated/associated
entities may serve valuable purposes for humans directly exploring the
data.

AssociateOf: "Ben Mason"
AssociateOf: "Jeff Severson"
AssociateOf: "Jeff Smith"

NOT an Affiliate: "Creative Caldron"
NOT an Affiliate: "Art Space"


Example 4: "Driftwagon has teamed up with Peruvian designer
Dunkelvolk."

target_id: http://en.wikipedia.org/wiki/Dunkelvolk

Affiliate: "Driftwagon"

[John R. Frank]

For those of you who prefer to read diffs:


diff --git a/2013/entities/README.rst b/2013/entities/README.rst
index 9ee3c99..6c81ab4 100644
--- a/2013/entities/README.rst
+++ b/2013/entities/README.rst
@@ -49,14 +49,14 @@ Each entity is one of these three types:

The entity_type determines the target slots to fill:

-type PER: Affiliate, Contact_Meet_PlaceTime, AwardsWon, DateOfDeath, CauseOfDeath, Titles, FounderOf, EmployeeOf
+type PER: Affiliate, AssociateOf, Contact_Meet_PlaceTime, AwardsWon, DateOfDeath, CauseOfDeath, Titles, FounderOf, EmployeeOf

type FAC: Affiliate, Contact_Meet_Entity

type ORG: Affiliate, TopMembers, FoundedBy

-Three of these slots require special explanation (below). The other
+Four of these slots require special explanation (below). The other

values are directly from TAC-KBP or ACE definitions:

http://www.nist.gov/tac/2012/KBP/task_guidelines/TAC_KBP_Slots_V2.4.pdf

@@ -74,12 +74,28 @@ Rather than invent specific new slots, we propose these three
generalized slots classes.


-1) "Affiliate" is any type of relation that *directly* connects the
-target entity to another entity of any type. This is the union of all
+
+
+
+1.1) "Affiliate" is any type of relation that *directly* connects the
+target entity to another entity. Affiliate must have a FAC or ORG on
+at least one side of the relation. Affiliate is its own inverse.
+
+1.2) "AssociateOf" is the PER-PER analog of Affiliate. Both sides of
+this relation must have a person-type entity.
+
+The challenge with both Affiliate and AssociateOf is judging what
+level of "closeness" distinguishes these relations from two entities
+that merely co-occurring in the same sentence or passage. By
+attempting these new slots in KBA 2013, we will be able to measure
+inter-assessor agreement and work toward more refined guidelines than
+the few examples below.
+
+Affiliate is the union of all

close relations, such as StudentOf, EmployeeOf, MemberOf, and their
inverses from ACE and KBP, and similar relations in which the relation
is of a simple unambiguous type. We propose to use this generalized

-notion of "close" relation as the base class for all the PER-PER and
+notion of "close" relation as the base class for all the ORG-ORG, ORG-FAC,

PER-FAC and PER-ORG relations. Instead of enumerating all possible
such close relations, we intend to allow KBA systems to generate many
such examples, which assessors will then judge in the post-hoc pooled

@@ -89,7 +105,11 @@ While this is somewhat in the spirit of open IE, our aim is

specifically to enable upstream filtering systems to down-select the
stream for feeding KB population systems operating with fixed
inventories of slot types, which will typically be more specific than

-Affiliate.
+Affiliate or AssociateOf. By filtering the stream, KBA systems allow
+such KBP systems to focus compute power on a smaller substream.
+Further, the less strongly typed graph of affiliated/associated
+entities may serve valuable purposes for humans directly exploring the
+data.

Since little annotation has been performed for "Affiliate", we are
concerned that it could generate a lot of noise. We welcome feedback

@@ -133,9 +153,9 @@ at Creative Cauldron at ArtSpace, 410 S. Maple Avenue."

target_id: http://en.wikipedia.org/wiki/Jeff_Severson

-Affiliate: "Ben Mason"
-Affiliate: "Jeff Severson"
-Affiliate: "Jeff Smith"
+AssociateOf: "Ben Mason"
+AssociateOf: "Jeff Severson"
+AssociateOf: "Jeff Smith"

[miles]

Sorry if this was addressed already.  But the TREC current participant website links to a different tar ball than the one mentioned in the organizers' email below.  Specifically, NIST directs to:

  trec-kba-ccr-and-ssf-2013-04-22.a27da77716822ff420cbd6f1f104fa25.tar.gz

I count 170 target_id's in trec-kba-ccr-and-ssf-query-topics-2013-04-08.json.  The KBA website mentioned ~500 entities for this year.  So I just wanted to confirm, is this the right set of entities for us to work with this year?

Thanks,

-Miles 

[John R. Frank]

>�Specifically, NIST directs to:
> ��trec-kba-ccr-and-ssf-2013-04-22.a27da77716822ff420cbd6f1f104fa25.tar.gz


Yes, that is the correct (final) query release tarball.

This post describes the changes from previous -- see "AssociateOf":

https://groups.google.com/d/msg/trec-kba/utOe7Lz1RZ0/t9--G1zf_SMJ

I will update the trec-kba.org -- thanks for the prompt!

> I count 170 target_id's in
> trec-kba-ccr-and-ssf-query-topics-2013-04-08.json. �The KBA website
> mentioned ~500 entities for this year. �So I just wanted to confirm, is
> this the right set of entities for us to work with this year?

Yes, 170 is the correct number.

We started with 600 and winnowed down to 170.



jrf

[Anshul Mittal]

Hi John,

In the list of entities, Fargo_Moorhead_Derby_Girls has a type PER, is that a typo?

Anshul

[John R. Frank]

> Hi John,In the list of entities,ï¿œFargo_Moorhead_Derby_Girls has a type

> PER, is that a typo?

Yes, that's an error. Thanks for catching it.

We will probably discard that entity, because we should not change the
topics file this close to the first deadline and it's not possible for the
assessors to judge slot fills for the wrong entity type.

You are welcome to submit ORG slot fills for that entity, but it won't be
used in computing official scores.


jrf

[Kevin Chen]

Hi John,

I have a little confusion regarding the relevance judgments for CCR 2013.

wc -l trec-kba-ccr-judgments-2013-04-08.before-cutoff.filter-run.txt

7930 trec-kba-ccr-judgments-2013-04-08.before-cutoff.filter-run.txt

There are 7930 lines in the file, so less than 7930 relevance judgments are available. As discussed above, the number of target ids is 170. Therefore, on average, only 7930 / 170 ~ 46 relevance judgments are available for each target id. 

On the other hand, the number of relevance judgments in the 2012 CCR task was a lot more than 7000, as shown on the task description page (copied and pasted below), and the number of entities was much fewer (~25). Hence, there is a big gap between [judgments per entity] in 2012 and 2013.

Counts from KBA 2012 annotation:

Mentions	7991	3862	1397  1	7806
Zero Mentions	15367	163	61	0
	garbage	neutral	useful	vital

Do I understand anything wrong here, or is it just what we are supposed to work on?

I'm working on the 2013 CCR task and planning to train a regressor for each target entity, but it seems to me that this strategy might not perform well given ~50 positive training instances. 

Thanks in advance for any help you are able to provide.

Kevin Chen

[ps]

There are 7930 lines in the file, so less than 7930 relevance judgments are available. As discussed above, the number of target ids is 170. Therefore, on average, only 7930 / 170 ~ 46 relevance judgments are available for each target id. 

Correct, and the variance is high.  Some entities have no examples in the TTR.

 

On the other hand, the number of relevance judgments in the 2012 CCR task was a lot more than 7000, as shown on the task description page (copied and pasted below), and the number of entities was much fewer (~25). Hence, there is a big gap between [judgments per entity] in 2012 and 2013.

Counts from KBA 2012 annotation:

Mentions	7991	3862	1397  1	7806
Zero Mentions	15367	163	61	0
	garbage	neutral	useful	vital

It is true that 2012 had more training examples, however it is not as dramatic as you are thinking.  The table above shows *all* of the judgments, both training and evaluation.

 

I'm working on the 2013 CCR task and planning to train a regressor for each target entity, but it seems to me that this strategy might not perform well given ~50 positive training instances.  

It will probably work better on some entities and less well on others.  The 2012 data should work well for training data on recognizing vital+useful, so that should help on those particular entities.

One of the key reasons we revised the definition of "central" --> "vital" was to make such training more viable.  It reduced inter-assessor disagreement by 20%, and I believe that distance supervision probably got much easier.

Happy to discuss more --- any questions?

John

[Praveen Kumar]

Kevin Chen,

   I just wanted to point out that amongst 7930 stream-ids, only 7103 are unique I think!

Praveen

[Tom Kenter]

Hi all,

There are 7930 lines in the file, so less than 7930 relevance judgments are available. As discussed above, the number of target ids is 170. Therefore, on average, only 7930 / 170 ~ 46 relevance judgments are available for each target id. 

Correct, and the variance is high.  Some entities have no examples in the TTR.

In fact, if look at the number of entities that were judged, I get 132.

I am doing:

$ cut -f4 trec-kba-ccr-judgments-2013-04-08.before-cutoff.filter-run.txt | head -7910 | sort | uniq | wc -l

132

So of the 170 target entities, 38 have no training examples.

Is this correct or am I misunderstanding things?

Thanks!

Tom

[John R. Frank]

> In fact, if look at the number of entities that were judged, I get 132.
>
> I am doing:
>
> $ cut -f4 trec-kba-ccr-judgments-2013-04-08.before-cutoff.filter-run.txt | head -7910 | sort | uniq | wc -l
> 132
>
> So of the 170 target entities, 38 have no training examples.
> Is this correct or am I misunderstanding things?

29 of those 38 are from KBA 2012, and we are about to drop them from the
official query list for this year because there is insufficient rating
data for them -- they are too large.

For the other 9, yes, you are right, they have no training examples.
These particular entities are small because they are "normal" people or
new organizations, like startups, that emerge on the public scene during
the time frame of the corpus either as a result of being new-to-the-world
or experiencing dramatic events.

The SSF assessing finished yesterday, and we'll have an updated tarball
with stats and assessing guidelines and more info later today. We're
pretty excited about how it is all fitting together. More soon.


John