TREC CAR 2018 submission deadline: August 1 / pre-announcement of passage ordering task.

[Laura Dietz]

Dear all,

Sorry for being silent for so long. We are working on new topics, a new task description, and new evaluation metrics.

1. You find results from last year's evaluation here: http://trec-car.cs.unh.edu/results/

2. Block your calendars. The TREC CAR 2018 submission deadline will be on August 1.

3. In addition to the retrieval tasks, we will offer a "passage ordering" task.

passage ordering task: We will give you an outline (yes, I caved) and a budget of paragraphs.  Your task is to provide a sequence of paragraphs that  would represent the article in reading order. Your article should cover all topics in the outline, and address topics coherently (i.e., all paragraphs for heading 1 should be right next to one another, and not interleaved with paragraphs for other headings).

You will provide one sequence per outline (not per heading).

You will provide a sequence of paragraph ids (you don't get to modify the paragraph).

We will give tell you how long the sequence should be (i.e., a budget $k$ of paragraphs).

The passage ordering task is different from the passage ranking task, is that a ranking is sorted by relevance. For the ordering, you should consider the flow of arguments, coherence of topics, and maximize coverage.

Alternatively, we offer a passage ranking task as in previous years, but it will be evaluated through the passage ordering task as follows. We will take the top ranked paragraphs for each heading in order of relevance.

You are allowed to submit a mix of results for the passage ordering and ranking task -- but in total only three passage runs will be considered.

For the evaluation, each passage will be assigned to the most relevant top-level heading, i.e., query-facet, in the outline. We will use this to assess relevance for the article, coverage of query facets, topical coherence.

Please stay tuned for more details on the submission format, training data, and test topics.

If you have any suggestion for evaluation metrics, please reach out to us.

Best,
Laura Dietz, Ben Gamari, John Foley

[Ryan]

Hi, our team want to attend the "passage ordering" task. But we didn't take part in Y1 or Y2 before. Does it matter? Could we have the opportunity to attend the "ordering" task?

Another problem is that what the timeline of "passage ordering" task is. Is its deadline the same as TREC CAR 2018, August 1?

Thanks!

[Laura Dietz]

Ryan,

Thanks for your interest, and: Welcome!

You don't have to participate in previous years.

There is only one task this year, and it is passage ordering.

However: Part of the submission is to also select a subset of passages that will be re-ordered in the next step.

Nevertheless: We plan on providing a set of simple rankings from which you can construct your candidate set.

Let me know if you have any questions.

Best,

Laura

--
You received this message because you are subscribed to the Google Groups "TREC Car" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-car+u...@googlegroups.com.
To post to this group, send email to trec...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trec-car/94155e6d-4d89-4e92-a64f-0bfdd0676acc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ryan]

Laura, 

Thanks for replying. We still want to know more about the TREC CAR. We will be grateful if you could answer the following questions.

Is registration for TREC this year closed? If not, could you please tell me where to sign up for it?

By the way, has the dataset for the task "passage ordering" task been published? I look through the home page and dataset page, but can't find the corpus or dataset especially for Y3. Could you please tell me which dataset and corpus should I use for this year?

Thanks!

Ryan

To unsubscribe from this group and stop receiving emails from it, send an email to trec-car+unsubscribe@googlegroups.com.

[Laura Dietz]

Ryan,

I am the wrong person to talk to with respect to registrations/signup. Please get in touch with Ellen Voorhees, she organizes TREC.

The previous releases include articles from Wikipedia, the idea is that you use those to train how a good article looks like.

We will release test topics with a small amount of additional training data soon.

Best,

Laura

To unsubscribe from this group and stop receiving emails from it, send an email to trec-car+u...@googlegroups.com.

To post to this group, send email to trec...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trec-car/3412d818-036f-4e84-817c-ed17c071e89e%40googlegroups.com.

[Ryan]

Laura, 

Thanks for the information. I will contact the organizer. 

[Yutao Zeng]

Hi, our team have registered for TREC CAR 2019. From the task description, we find there are three tasks: passage ranking task, entity ranking task and passage ordering task. But according to the previous posts and replies,

There is only one task this year, and it is passage ordering.

 

Alternatively, we offer a passage ranking task as in previous years, but it will be evaluated through the passage ordering task as follows. We will take the top ranked paragraphs for each heading in order of relevance.

You are allowed to submit a mix of results for the passage ordering and ranking task -- but in total only three passage runs will be considered.

 

Does this mean that we only need to attend the passage ordering task (and we still can submit passage ranking results) ?

And will the entity ranking task be evaluated?

Thanks!

[Laura Dietz]

Yutao,

I think you might have read information about the TREC CAR Y2 evaluation which ran in 2018. This information does apply to this year's evaluation.

This year we offer only the passage ordering task.

We will provide some basic code for converting passage ranking data into a naive paragraph ordering. So if you only want to work on passage ranking, this code will convert your ranking outputs into a passage ordering submission. 

We will offer several evaluation metrics, some of which measure the quality of paragraphs in your submitted ordering/ranking.

We do not officially host an entity ranking evaluation. However, we will likely provide an automatic entity benchmark after the submission, to encourage research in that area.

I hope this helps --- please keep on asking more questions!

Best,
Laura

--
You received this message because you are subscribed to the Google Groups "TREC Car" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trec-car+u...@googlegroups.com.

To post to this group, send email to trec...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trec-car/6c2d683c-9070-4a06-8303-3d3b4779d9d0%40googlegroups.com.

[Yutao Zeng]

Laura,

Thanks for your reply. It helps a lot!

Best,

Yutao