CFP: Unshared Task at LENLS 13, Theory and System analysis with FraCaS, MultiFraCaS and JSeM Test Suites
14 views
Skip to first unread message
Daisuke Bekki
Jul 6, 2016, 1:11:29 AM7/6/16
to linguistics-jp, logic-ml, sono...@googlegroups.com
（重複して受け取られた場合はご容赦ください）
今年の１１月に東京で開催される Unshared Task at LENLS 13 の
ご案内をお送り致します。皆様のご参加をお待ちしております。
戸次大介（お茶の水女子大学）
[Apologies for multiple copies]
=================================================================
今年の１１月に東京で開催される Unshared Task at LENLS 13 の
ご案内をお送り致します。皆様のご参加をお待ちしております。
戸次大介（お茶の水女子大学）
[Apologies for multiple copies]
=================================================================
Call for Papers:
Unshared Task at LENLS 13
Theory and System analysis with FraCaS, MultiFraCaS and JSeM Test Suites
Workshop Site: National Institute for Japanese Language and Linguistics
10-2 Midori-cho, Tachikawa City, Tokyo, 190-8561, Japan
http://www.ninjal.ac.jp/english/utility/access
Date: November 13th, 2016
Contact Person: Alastair Butler (National Institute for Japanese Language and Linguistics)
Contact Email: lenls13[[at]]easychair.org
LENLS 13 website: http://www.is.ocha.ac.jp/~bekki/lenls
Unshared Task website: http://www.compling.jp/fracas_task/index.html
Introduction
This one day shared task focused on undertaking Theory and System
analysis with FraCaS and FraCaS inspired Test Suites is to be held as
part of Logic and Engineering of Natural Language Semantics 13 (LENLS
13). See http://www.is.ocha.ac.jp/~bekki/lenls/ for full information
about LENLS 13, which is to be held on November 13-15, 2016. LENLS is an
annual international workshop on formal syntax, semantics and pragmatics.
The FraCaS test suite was created by the FraCaS Consortium as a benchmark
for measuring and comparing the competence of semantic theories and
semantic processing systems. It contains inference problems that
collectively demonstrate basic linguistic phenomena that a semantic
theory has to account for; including quantification, plurality, anaphora,
ellipsis, tense, comparatives, and propositional attitudes. Each problem
has the form: there is some natural language input T, then there is a
natural language claim H, giving the task to determine whether H follows
from T. Problems are designed to include exactly one target phenomenon,
to exclude other phenomena, and to be independent of background knowledge.
Following FraCaS, overlapping test suites are now available for a number
of languages (notably in addition to the original English: Farsi, German,
Greek, Japanese, and Mandarin), which together cover both universal
semantic phenomena as well as language-specific phenomena. With the
problem sets categorised according to the semantic phenomena they involve,
it is possible to focus on obtaining results for specific phenomena
(within a language or cross-linguistically), as well as strive for
wide coverage.
Goals
Shared tasks typically provide "gold" analysed data with clear evaluation
criteria for competing systems and have become popular within NLP
fields. The concept of a so-called "unshared task" is an alternative
to shared tasks. In an unshared task, there are neither quantitative
performance measures nor set problems that have to be solved. Instead,
participants are given a common ground (e.g., data) and an open-ended
prompt.
With the availability of FraCaS, MultiFraCaS and JSeM Test Suites, the
aim of this unshared task is for participants to put these resources to
work as the basis for inspiring analysis, e.g., for showcasing a semantic
theory or semantic processing system, or syntactic annotation model for
the data.
We would also be interested to hear about the creation of complementary
data for other languages not yet represented by the existing test suites,
or with work concerning properties of the existing test suites, or with
cross-linguistic comparisons using the test suites, etc.
Being an unshared task, use made of the datasets is up to the authors. Any
of the data sets might serve as a benchmark for testing the approach
taken (or even a computational model, for participants who go that far)
and reporting success levels on the problems (if applicable).
Submissions
Papers should be submitted via the LENLS 13 easychair site as "Unshared
Task" papers, and should mention "Unshared Task" in their paper title to
further distinguish themselves from other workshop submissions. Papers
must be anonymous, up to 4 pages, including figures and references,
A4 or letter size, with 12 point font, and submitted electronically in
PDF format at:
https://easychair.org/conferences/?conf=lenls13
Submissions will be reviewed by the LENLS 13 program committee.
We plan for all accepted unshared task papers to be orally presented as
30 minute talks at the workshop, followed by an open discussion.
The acceptance of submissions will be based on reviewers' assessment
with a focus on ambition, thoroughness, and the overall quality.
When the abstract is accepted, the author is expected to submit a full
paper (10-14 pages) before the workshop for inclusion in the LENLS 13
workshop proceedings. The online proceedings of the LENLS 13 workshop
will be available at the conference site.
Where applicable, we encourage submission along with the paper of
data analysis (e.g., syntactic/semantic annotations) as supplementary
material, so that the community has a chance to explore in-depth how the
datasets have been used, rather than the few samples most likely shown
in the paper.
Data
We invite papers to apply either theoretical or computational analyses
or other ideas to any of the following datasets, or subsets thereof,
and describe findings:
FraCaS textual inference test suite (English)
Download machine readable version: http://www-nlp.stanford.edu/~wcmac/downloads
For the original: ftp://ftp.cogsci.ed.ac.uk/pub/FRACAS/del16.ps.gz
MultiFraCaS (Farsi, German, Greek, Mandarin)
Download: http://www.ling.gu.se/~cooper/multifracas
Japanese Semantics Test Suite (JSeM)
Download JSeM_beta.zip from: http://researchmap.jp/community-inf/JSeM
Dates
Same as LENLS 13 workshop paper submissions, see http://www.is.ocha.ac.jp/~bekki/lenls
Organisers
Alastair Butler (National Institute for Japanese Language and Linguistics (NINJAL))
Koji Mineshima (Ochanomizu University/JST CREST)
Ai Kubota (National Institute for Japanese Language and Linguistics (NINJAL))
Yusuke Kubota (University of Tsukuba)
Program Committee
http://www.is.ocha.ac.jp/~bekki/lenls/#committee
References
Bos, J. 2008. "Let's not argue about semantics." Proceedings of the
6th Language Resources and Evaluation Conference (LREC 2008):2835-2840,
Marrakech, Morocco.
Kawazoe, A., Tanaka, R., Mineshima, K., Bekki, B. 2015. "An Inference
Problem Set for Evaluating Semantic Theories and Semantic Processing
Systems for Japanese," Proceedings of the Twelfth International Workshop
on Logic and Engineering of Natural Language Semantics (LENLS12).
Cooper, R., Crouch, D., van Eijck, J., Fox, C., van Genabith, J., Jan, J.,
Kamp, H., Milward, D., Pinkal, M., Poesio, M., Pulman, S., Briscoe, T.,
Maier, H., and Konrad, K. 1996. "Using the framework." Technical report,
FraCaS: A Framework for Computational Semantics. FraCaS deliverable D16.
MacCartney, B., and Manning, C. D. 2007. "Natural logic for textual
inference." In Proceedings of the ACL-PASCAL Workshop on Textual
Entailment and Paraphrasing, 193-200.
Unshared Task at LENLS 13
Theory and System analysis with FraCaS, MultiFraCaS and JSeM Test Suites
Workshop Site: National Institute for Japanese Language and Linguistics
10-2 Midori-cho, Tachikawa City, Tokyo, 190-8561, Japan
http://www.ninjal.ac.jp/english/utility/access
Date: November 13th, 2016
Contact Person: Alastair Butler (National Institute for Japanese Language and Linguistics)
Contact Email: lenls13[[at]]easychair.org
LENLS 13 website: http://www.is.ocha.ac.jp/~bekki/lenls
Unshared Task website: http://www.compling.jp/fracas_task/index.html
Introduction
This one day shared task focused on undertaking Theory and System
analysis with FraCaS and FraCaS inspired Test Suites is to be held as
part of Logic and Engineering of Natural Language Semantics 13 (LENLS
13). See http://www.is.ocha.ac.jp/~bekki/lenls/ for full information
about LENLS 13, which is to be held on November 13-15, 2016. LENLS is an
annual international workshop on formal syntax, semantics and pragmatics.
The FraCaS test suite was created by the FraCaS Consortium as a benchmark
for measuring and comparing the competence of semantic theories and
semantic processing systems. It contains inference problems that
collectively demonstrate basic linguistic phenomena that a semantic
theory has to account for; including quantification, plurality, anaphora,
ellipsis, tense, comparatives, and propositional attitudes. Each problem
has the form: there is some natural language input T, then there is a
natural language claim H, giving the task to determine whether H follows
from T. Problems are designed to include exactly one target phenomenon,
to exclude other phenomena, and to be independent of background knowledge.
Following FraCaS, overlapping test suites are now available for a number
of languages (notably in addition to the original English: Farsi, German,
Greek, Japanese, and Mandarin), which together cover both universal
semantic phenomena as well as language-specific phenomena. With the
problem sets categorised according to the semantic phenomena they involve,
it is possible to focus on obtaining results for specific phenomena
(within a language or cross-linguistically), as well as strive for
wide coverage.
Goals
Shared tasks typically provide "gold" analysed data with clear evaluation
criteria for competing systems and have become popular within NLP
fields. The concept of a so-called "unshared task" is an alternative
to shared tasks. In an unshared task, there are neither quantitative
performance measures nor set problems that have to be solved. Instead,
participants are given a common ground (e.g., data) and an open-ended
prompt.
With the availability of FraCaS, MultiFraCaS and JSeM Test Suites, the
aim of this unshared task is for participants to put these resources to
work as the basis for inspiring analysis, e.g., for showcasing a semantic
theory or semantic processing system, or syntactic annotation model for
the data.
We would also be interested to hear about the creation of complementary
data for other languages not yet represented by the existing test suites,
or with work concerning properties of the existing test suites, or with
cross-linguistic comparisons using the test suites, etc.
Being an unshared task, use made of the datasets is up to the authors. Any
of the data sets might serve as a benchmark for testing the approach
taken (or even a computational model, for participants who go that far)
and reporting success levels on the problems (if applicable).
Submissions
Papers should be submitted via the LENLS 13 easychair site as "Unshared
Task" papers, and should mention "Unshared Task" in their paper title to
further distinguish themselves from other workshop submissions. Papers
must be anonymous, up to 4 pages, including figures and references,
A4 or letter size, with 12 point font, and submitted electronically in
PDF format at:
https://easychair.org/conferences/?conf=lenls13
Submissions will be reviewed by the LENLS 13 program committee.
We plan for all accepted unshared task papers to be orally presented as
30 minute talks at the workshop, followed by an open discussion.
The acceptance of submissions will be based on reviewers' assessment
with a focus on ambition, thoroughness, and the overall quality.
When the abstract is accepted, the author is expected to submit a full
paper (10-14 pages) before the workshop for inclusion in the LENLS 13
workshop proceedings. The online proceedings of the LENLS 13 workshop
will be available at the conference site.
Where applicable, we encourage submission along with the paper of
data analysis (e.g., syntactic/semantic annotations) as supplementary
material, so that the community has a chance to explore in-depth how the
datasets have been used, rather than the few samples most likely shown
in the paper.
Data
We invite papers to apply either theoretical or computational analyses
or other ideas to any of the following datasets, or subsets thereof,
and describe findings:
FraCaS textual inference test suite (English)
Download machine readable version: http://www-nlp.stanford.edu/~wcmac/downloads
For the original: ftp://ftp.cogsci.ed.ac.uk/pub/FRACAS/del16.ps.gz
MultiFraCaS (Farsi, German, Greek, Mandarin)
Download: http://www.ling.gu.se/~cooper/multifracas
Japanese Semantics Test Suite (JSeM)
Download JSeM_beta.zip from: http://researchmap.jp/community-inf/JSeM
Dates
Same as LENLS 13 workshop paper submissions, see http://www.is.ocha.ac.jp/~bekki/lenls
Organisers
Alastair Butler (National Institute for Japanese Language and Linguistics (NINJAL))
Koji Mineshima (Ochanomizu University/JST CREST)
Ai Kubota (National Institute for Japanese Language and Linguistics (NINJAL))
Yusuke Kubota (University of Tsukuba)
Program Committee
http://www.is.ocha.ac.jp/~bekki/lenls/#committee
References
Bos, J. 2008. "Let's not argue about semantics." Proceedings of the
6th Language Resources and Evaluation Conference (LREC 2008):2835-2840,
Marrakech, Morocco.
Kawazoe, A., Tanaka, R., Mineshima, K., Bekki, B. 2015. "An Inference
Problem Set for Evaluating Semantic Theories and Semantic Processing
Systems for Japanese," Proceedings of the Twelfth International Workshop
on Logic and Engineering of Natural Language Semantics (LENLS12).
Cooper, R., Crouch, D., van Eijck, J., Fox, C., van Genabith, J., Jan, J.,
Kamp, H., Milward, D., Pinkal, M., Poesio, M., Pulman, S., Briscoe, T.,
Maier, H., and Konrad, K. 1996. "Using the framework." Technical report,
FraCaS: A Framework for Computational Semantics. FraCaS deliverable D16.
MacCartney, B., and Manning, C. D. 2007. "Natural logic for textual
inference." In Proceedings of the ACL-PASCAL Workshop on Textual
Entailment and Paraphrasing, 193-200.
Reply all
Reply to author
Forward
0 new messages