Meta information for each run
Laura Dietz
Dear all,
I encourage you to take a look at the TREC CAR submission form
before you prepare your runs. You will have to fill in some meta
data about each of your runs, such as which external resources or
methods you used. We will use this information in the overview
report to study which kinds of methods/data have a positive
impact.
Each team can submit up to 3 automatic entity rankings and up to
3 automatic passage rankings. I anticipate most of your runs will
be automatic.
TREC's concept of a manunal run is one where the ranking is
created by a human, or with the help of a human. If you have a run
which would "cheat" in some way or another but you would like to
see how it fares in comparison, please upload it as a manual run.
We will probably not be able to include manual runs in the
assessment (but we will try our best).
As always, feel free to ask questions.
Laura
Here the submission form:
https://ir.nist.gov/trecsubmit/car.html (The login/password was sent to you by Ellen Voorhees a couple of months ago)
It looks like this:
| REPLY EMAIL ADDRESS: na...@internet.address |
|
|---|---|
| ORGANIZATION: | |
| RUN IDENTIFICATION: (case sensitive) |
|
|---|---|
| SUBMISSION FILE: | File can be compressed using either gzip or bzip2, but
you cannot use archive files (e.g., zip or tar) |
| RUN DETAILS | |
| TASK | Passage
Ranking Entity Ranking |
| RUN TYPE: |
Automatic Manual |
| RUN FEATURES: | Please mark any of the following features as they apply
to this run. Please clarify any checked feature in the
External Resources field below." method uses a linguistic knowledge graph such as Wordnet method uses a knowledge graph such as DBpedia, Freebase, etc. method uses entity linking method uses a dump of Wikipedia method uses data v2.0, v1.5 or v1.4 data provided as "unprocessedAllButBenchmark", unprocessedtrain",or "halfwiki" method uses data v2.1 data provided as "unprocessedAllButBenchmark" method uses another external document corpus (e.g., ClueWeb, CommonCrawl, ...) method uses pre-trained word embeddings method uses neural network technology method uses learning-to-rank method uses clustering, topic models, or summarization method uses other supervised machine learning method method does not make use of any training data (unsupervised) method is trained with data provided as train-v2.0 method is trained with data provided as train-v1.5 method is trained with data provided as train-v1.4 method is trained with data provided as test200 method is trained with data provided as benchmarkY1train method is trained with data provided as benchmarkY1test automatic qrels method is trained with data provided as benchmarkY1test manual qrels method uses other training data |
| EXTERNAL RESOURCES | Please describe the features checked above here. |
| RANKING | Please list the retrieval models used as part of the
pipeline (e.g., BM25, query likelihood). |
| DESCRIPTION | Please give a short description of the techniques used to
produce the run. |
| CITATION | If this method is already published, please give the
citation to the publication here. |
| PRIORITY FOR ASSESSMENT: | |
(If the same precedence is given to different runs, NIST will arbitrarily choose.)
Rodrigo Nogueira
Rodrigo
Laura Dietz
If you would want to submit a run created with a human in the loop, you could do this by submitting a "manual" run.
Also if you want to submit a run that is not 100% in line with the tasks requirements (i.e you looked up the page on Wikipedia) you may submit it as manual.
Best,
Laura
--
You received this message because you are subscribed to the Google Groups "TREC Car" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-car+u...@googlegroups.com.
To post to this group, send email to trec...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trec-car/abb3726d-e542-4b35-b673-879ce855fc2f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Laura Dietz
I wish you all a Happy TREC CAR submission day!
To be safe, submit by tonight midnight --- but practically the submission form will be open all night until Ellen Voorhees comes in the office tomorrow morning and closes it.
Looking forward to your submitted runs!
Best,
Laura
