Get confusion matrix for single-label classification and strange error

[Nicolai Erbs]

Hi,

I haven't used DKPro TC in a while and I'm happy to see that you have recently released a new version. Many thanks for the effort.

When running the example WekaTwentyNewsgroupsDemo, I noticed that I don't get a confusion matrix. I changed the experiment to the following: 

WekaTwentyNewsgroupsDemo experiment = new WekaTwentyNewsgroupsDemo();
experiment.runCrossValidationWithStatsEval(pSpace);

Then the following error occurs:

Exception in thread "main" org.dkpro.lab.engine.LifeCycleException: org.dkpro.tc.api.exception.TextClassificationException: baselineClassificationArgs not found in discriminators set.
at org.dkpro.lab.engine.impl.DefaultLifeCycleManager.complete(DefaultLifeCycleManager.java:144)
at org.dkpro.lab.engine.impl.BatchTaskEngine.run(BatchTaskEngine.java:159)
at org.dkpro.lab.engine.impl.BatchTaskEngine.runNewExecution(BatchTaskEngine.java:341)
at org.dkpro.lab.engine.impl.BatchTaskEngine.executeConfiguration(BatchTaskEngine.java:235)
at org.dkpro.lab.engine.impl.BatchTaskEngine.run(BatchTaskEngine.java:133)
at org.dkpro.lab.engine.impl.DefaultTaskExecutionService.run(DefaultTaskExecutionService.java:52)
at org.dkpro.lab.Lab.run(Lab.java:113)
at org.dkpro.tc.examples.single.document.WekaTwentyNewsgroupsDemo.runCrossValidationWithStatsEval(WekaTwentyNewsgroupsDemo.java:188)
at org.dkpro.tc.examples.single.document.WekaTwentyNewsgroupsDemo.main(WekaTwentyNewsgroupsDemo.java:92)
Caused by: org.dkpro.tc.api.exception.TextClassificationException: baselineClassificationArgs not found in discriminators set.
at org.dkpro.tc.core.util.ReportUtils.getDiscriminatorValue(ReportUtils.java:347)
at org.dkpro.tc.ml.report.BatchStatisticsTrainTestReport.execute(BatchStatisticsTrainTestReport.java:74)
at org.dkpro.lab.engine.impl.DefaultLifeCycleManager.complete(DefaultLifeCycleManager.java:137)

Is there anything wrong with the example? I checked that the discriminator baselineClassificationArgs is added in the parameter space.

 Dimension<List<String>> dimBaselineClassificationArgs = Dimension.create(
                DIM_BASELINE_CLASSIFICATION_ARGS,
                Arrays.asList(new String[] { NaiveBayes.class.getName() }));

Any help is appreciated!

Kind regards,

Nicolai 

         

[Johannes Daxenberger]

Hi Nico,

welcome back!
I can reproduce this error on the DKPro TC master branch. I assume that the reports that are executed in the “runCrossValidationWithStatsEval” have not been tested with the version. The problem is that some dimensions which are required for the reports are not declared (as discriminators) by the TestTask – and thus not recognized by the reports although they are declared in the parameter space. Would you mind opening an issue to reflect this problem?

About the confusion matrix: as far as I can see you are right, they are not created using the default reports. However, you should be able to produce them yourself easily, by adding up over the id2outcome files of the individual test tasks. We had this functionality before but it got lost somehow while restructuring the evaluation. I assume it should be added to EvaluatorFactory? This again is something that should probably be addressed globally (and worth another issue…).

Best,
Johannes

--
You received this message because you are subscribed to the Google Groups "dkpro-tc-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dkpro-tc-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.