P1EDA 3-way classification

[Tom Bosc]

Hello,

I am using EOP 1.2.3. I am trying to compare EDAs. At the time, I use the SICK dataset.

AdArte works very well and gives very good results. I'd like to compare with P1EDA but I cannot get a working config file.

I use the following:

<?xml version="1.0" encoding="utf-8"?>

<configuration>
<section name="PlatformConfiguration">
            <!-- <property name="activatedEDA">eu.excitementproject.eop.alignmentedas.p1eda.instances.SimpleWordCoverageEN</property> -->
            <property name="activatedEDA">eu.excitementproject.eop.alignmentedas.p1eda.sandbox.FNR_EN</property>
            <property name="language">EN</property>
            <property name="activatedLAP">eu.excitementproject.eop.lap.dkpro.TreeTaggerEN</property>
</section>

<section name="eu.excitementproject.eop.alignmentedas.p1eda.sandbox.FNR_EN">
    <property name="modelFile">/tmp//P1EDA_Base_EN.model</property>
    <property name="trainDir">/tmp/EN/dev/</property>
    <property name="testDir">/tmp/EN/test/</property>
</section>

</configuration>

And here are the results (SICK):

<?xml version="1.0" encoding="UTF-8"?>
<results>
  <label id="ENTAILMENT">
    <Accuracy>36.282</Accuracy>
    <Precision>30.943</Precision>
    <Recall>99.573</Recall>
    <F1>47.214</F1>
    <ContingencyTable FN="6" FP="3120" TN="382" TP="1398"/>
  </label>
  <label id="UNKNOWN">
    <Accuracy>43.131</Accuracy>
    <Precision>1.000</Precision>
    <Recall>0.000</Recall>
    <F1>0.000</F1>
    <ContingencyTable FN="2790" FP="0" TN="2116" TP="0"/>
  </label>
  <label id="CONTRADICTION">
    <Accuracy>85.487</Accuracy>
    <Precision>1.000</Precision>
    <Recall>0.000</Recall>
    <F1>0.000</F1>
    <ContingencyTable FN="712" FP="0" TN="4194" TP="0"/>
  </label>
  <Accuracy>54.967</Accuracy>
  <Precision>30.943</Precision>
  <Recall>28.496</Recall>
  <F1micro>29.669</F1micro>
  <ContingencyTable FN="3508" FP="3120" TN="6692" TP="1398"/>
</results>

The results are bad, so I suspect P1EDA doesn't do 3-way classification? (entailment, non-entailment, neutral)

If it does, could you give me a good config file?

Thank you! and best wishes for 2016.

Tom

[Tae-Gil Noh]

Hello Tom,
As you have correctly guessed, P1EDA does not support 3-way classification yet. So what you see is actually a correct result. Maybe we will have 3 way classification for P1EDA later this year when I get back to a NLP start-up... But as of now, no 3 way support yet. :-)

All the best,
Gil