Running dkpro from command line

[Johannes P]

Hello everyone, 

I'm using dkpro 1.6.2 with additional own annotations (some text features as new types) as well as the standard annotations (DocumentMetadata, Token). Everything works, when I run it using eclipse IDE. 
But experiments I need to run use a lot of memory and disk space (and time), so I have to be able to run them on the server and not my local machine. So I figured I need to (a) export the project as fat jar or (b) build the maven project on the server and run it there. 

The first approach led to the (apparently well-known) error about types not being declared in XML type descriptor: 

Caused by: org.apache.uima.cas.CASRuntimeException: JCas type "com.example.types.MyNewType" used in Java code,  but was not declared in the XML type descriptor.
at org.apache.uima.jcas.impl.JCasImpl.getType(JCasImpl.java:414)

I searched the forum and found several threads discussing the problem: 

https://groups.google.com/forum/#!msg/dkpro-core-user/Gl8ty8ThiSg/BfeUGSs7EGgJ but that wasn't it, I seem to have access to the dkpro types, as the type that's not "being declared" is being annotated after the dkpro types. 

I then git-cloned the project to the server and build it there with mvn compile / mvn exec:java - compiling works, but the exec:java results in the same error. 

The discussion about fat jars here (https://groups.google.com/d/topic/dkpro-core-user/ZIG_NAj0a-E/discussion) finally brought me to this page https://issues.apache.org/jira/browse/UIMA-3385 where I found out I need  to change my eclipse generated pom.xml to handle the build process - so I included the maven shade plugin - but with no effects. Exporting the "runnable jar" from eclipse in the jar file there is the META-INF\org.apache.uima.fit\types.txt with my types, so it should be there, but it's not being loaded. Building on the server I had a look at the target folder and there this file was missing. So I guess here still lies the problem. Eclipse is doing something different than my pom.file at the moment suggests. 

Here's the <build> part, I'm omitting all the <dependencies>

<build>

      <resources>

     <resource>

       <directory>META-INF</directory>

     </resource>

   </resources>

               <plugins>

                       <plugin>

                               <groupId>org.apache.maven.plugins</groupId>

                               <artifactId>maven-compiler-plugin</artifactId>

                               <version>2.5.1</version>

                               <configuration>

                                       <source>1.5</source>

                                       <target>1.5</target>

                               </configuration>

                       </plugin>

                       <plugin>

     <groupId>org.apache.maven.plugins</groupId>

     <artifactId>maven-shade-plugin</artifactId>

     <version>2.2</version>

     <executions>

       <execution>

         <phase>package</phase>

         <goals><goal>shade</goal></goals>

         <configuration>

           <transformers>

             <!-- Set the main class of the executable JAR -->

             <transformer

               implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">

               <mainClass>com.example.CmdLineInterface</mainClass>

             </transformer>

             <!-- Merge the uimaFIT configuration files -->

             <transformer

               implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">

               <resource>

                 META-INF/org.apache.uima.fit/fsindexes.txt

               </resource>

             </transformer>

             <transformer

               implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">

               <resource>

                 META-INF/org.apache.uima.fit/types.txt

               </resource>

             </transformer>

             <transformer

               implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">

               <resource>

                 META-INF/org.apache.uima.fit/typepriorities.txt

               </resource>

             </transformer>

             <!--

               Prevent huge shaded artifacts from being deployed

               to a Maven repository (remove if not desired)

             -->

             <outputFile>${project.build.directory}/${project.artifactId}-${project.version}-standalone.jar</outputFile>

           </transformers>

         </configuration>

       </execution>

     </executions>

   </plugin>

               </plugins>

       </build>

I now just tried a

mvn package

and got the following error 

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:2.2:shade (default) on project MyProj-UIMA: Unable to parse configuration of mojo org.apache.maven.plugins:maven-shade-plugin:2.2:shade for parameter outputFile: Abstract class or interface 'org.apache.maven.plugins.shade.resource.ResourceTransformer' cannot be instantiated

Am I missing dependencies here? 

I also tried this here to explicitly use the types.txt resource in the code - as suggested 

TypeSystemDescription tsd = TypeSystemDescriptionFactory.createTypeSystemDescription();

              JCas jCas = JCasFactory.createJCas("com.example.types.myTypes"); // without .xml extension

                MyAnnotation t = new MyAnnotation(jCas); //that's a myTypes.MyAnnotation

               t.setCoveredText("this is a debug test");

               logger.log(Level.INFO, "ANNOT TEXT: "+t.getCoveredText());

All not that easy, I'm sorry - I've been trying to get this run for several days/weeks now and still didn't find a solution that works for me, so I'd appreciate all hints that might help me continue and finally run my experiments on the server. 

Thank you for patiently reading through!

Best,

Johannes

[Richard Eckart de Castilho]

Hi Johannes,

> Caused by: org.apache.uima.cas.CASRuntimeException: JCas type "com.example.types.MyNewType" used in Java code, but was not declared in the XML type descriptor.
> at org.apache.uima.jcas.impl.JCasImpl.getType(JCasImpl.java:414)
>
> I searched the forum and found several threads discussing the problem:
> https://groups.google.com/forum/#!msg/dkpro-core-user/Gl8ty8ThiSg/BfeUGSs7EGgJ but that wasn't it, I seem to have access to the dkpro types, as the type that's not "being declared" is being annotated after the dkpro types.

That sounds like you may not have set up a types.txt telling uimaFIT about your own types:

http://uima.apache.org/d/uimafit-current/tools.uimafit.book.html#ugr.tools.uimafit.typesystem

> I then git-cloned the project to the server and build it there with mvn compile / mvn exec:java - compiling works, but the exec:java results in the same error.

In contrast to fat-jars from Eclipse, I would suspect that exec:java should in principle work. This is another hint to me that you might be missing a types.txt file for your type descriptors.

> The discussion about fat jars here (https://groups.google.com/d/topic/dkpro-core-user/ZIG_NAj0a-E/discussion) finally brought me to this page https://issues.apache.org/jira/browse/UIMA-3385 where I found out I need to change my eclipse generated pom.xml to handle the build process - so I included the maven shade plugin - but with no effects.

The official documentation on this is here:

http://uima.apache.org/d/uimafit-current/tools.uimafit.book.html#ugr.tools.uimafit.packaging

The documentation might be buggy - see below.

> Exporting the "runnable jar" from eclipse in the jar file there is the META-INF\org.apache.uima.fit\types.txt with my types, so it should be there, but it's not being loaded. Building on the server I had a look at the target folder and there this file was missing. So I guess here still lies the problem. Eclipse is doing something different than my pom.file at the moment suggests.

Adding this configuration to the POM has no effect on the fat-jar export of Eclipse. It works only when you run "mvn clean package" on the command line.

> Here's the <build> part, I'm omitting all the <dependencies>
>
> <build>
>
> <resources>
>
> <resource>
>
> <directory>META-INF</directory>
>
> </resource>
>
> </resources>

Changing the default Maven resources directory doesn't sound good. If you actually do have a file META-INF/org.apache.uima.fit/types.txt, then this manual override of the resource directory will cause it to be placed into "org.apache.uima.fit/types.txt" in you JAR and not into "META-INF/org.apache.uima.fit/types.txt" where it belongs.
Move your META-INF folder to src/main/resources. So the relative path of the types.txt should be "src/main/resources/META-INF/org.apache.uima.fit/types.txt"

> <plugins>
>
> <plugin>
>
> <groupId>org.apache.maven.plugins</groupId>
>
> <artifactId>maven-compiler-plugin</artifactId>
>
> <version>2.5.1</version>
>
> <configuration>
>
> <source>1.5</source>
>
> <target>1.5</target>
>
> </configuration>
>
> </plugin>
>
> <plugin>

I'm pretty sure that DKPro Core 1.6.x is already on Java 6. You should probably also configure your project for Java 1.6

> I now just tried a
> mvn package
>
> and got the following error
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:2.2:shade (default) on project MyProj-UIMA: Unable to parse configuration of mojo org.apache.maven.plugins:maven-shade-plugin:2.2:shade for parameter outputFile: Abstract class or interface 'org.apache.maven.plugins.shade.resource.ResourceTransformer' cannot be instantiated

According to a post on Stackoverflow, the "<outputFile>" section should be moved out of the <transformers> section in the shade plugin configuration.

http://stackoverflow.com/questions/25936424/maven-shade-plugin-cannot-create-instance-of

Please try that. Looks like the uimaFIT documentation may have a bug here.

> I also tried this here to explicitly use the types.txt resource in the code - as suggested
> TypeSystemDescription tsd = TypeSystemDescriptionFactory.createTypeSystemDescription();
>
> JCas jCas = JCasFactory.createJCas("com.example.types.myTypes"); // without .xml extension
>
> MyAnnotation t = new MyAnnotation(jCas); //that's a myTypes.MyAnnotation
>
> t.setCoveredText("this is a debug test");
>
> logger.log(Level.INFO, "ANNOT TEXT: "+t.getCoveredText());

That invocation of createJCas doesn't use the types.txt. Instead, it loads only the type descriptor for your own types,
assuming that the location was given correctly. Doing this will give you access to your types, but not to the DKPro Core
types unless your type descriptor imports them by name. Better use the types.txt approach mentioned above and stick
with the no-args variant of createJCas.

> All not that easy, I'm sorry - I've been trying to get this run for several days/weeks now and still didn't find a solution that works for me, so I'd appreciate all hints that might help me continue and finally run my experiments on the server.

I'm sure we can work this out together :)

Cheers,

-- Richard

[Johannes P]

Hello Richard, 

thank you for your quick reply! 

First, I have a types.txt file for my own types (see below). 

After changing the lines about outputFile and deleting those about resources in the pom.xml I got rid of that instantiation error when running :)

mvn clean package

I also moved my META-INF directory containing my types.txt from src/main/java to src/main/resources. The types.txt looks like this

classpath*:com/example/types/aType.xml

classpath*:com/example/types/anotherType.xml

classpath*:com/example/types/subdir/myTypes.xml

When I'm running

mvn exec:java -Dexec.mainClass="com.example.CmdLineInterface" -Dexec.args="-a"

or 

java -jar target/MyProj-0.0.1-SNAPSHOT-standalone.jar -a

I still get the error about undeclared JCas type. I also can't see the respective xml files being created inside the target folder. 

Best,
Johannes

[Johannes P]

forgot to mention: I also changed the instantiation code of my Token to 

TypeSystemDescription tsd = TypeSystemDescriptionFactory.createTypeSystemDescription();

               JCas jCas = JCasFactory.createJCas();

               Token t = new Token(jCas);

               t.setCoveredText("this is a debug test");

               logger.log(Level.INFO, "TOKEN TEXT: "+t.getCoveredText());

And the path to my types.txt is src/main/resources/META-INF/org.apache.uima.fit/types.txt - just to make sure, that this should be correct. 

[Richard Eckart de Castilho]

And the path to your XML descriptors is e.g.

src/main/resources/com/example/types/subdir/myTypes.xml

?

Did you remove the <resources> section from your pom?

> <resources>
> <resource>
> <directory>META-INF</directory>
> </resource>
> </resources>

-- Richard

[Jose Luis Vieyra Sagaon]

I don't know if it is the same problem but sounds like a problem I had, we fixed adding to:

 src/main/resources/META-INF/org.apache.uima.fit/types.txt

the lines: 

classpath*:gex/ds/nlp/dkpro/types/desc/MacaTypes.xml

classpath*:desc/type/**/*.xml

Hope it helps

--
You received this message because you are subscribed to the Google Groups "dkpro-core-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dkpro-core-us...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mto. Ing. José Luis Vieyra S.

Este correo y sus archivos adjuntos son personales, privilegiados y confidenciales y están destinados exclusivamente para el uso de la persona a la que están dirigidos. Si usted recibió este correo por error, le agradeceremos regresarlo al remitente notificando dicho hecho y borre el presente y sus anexos de su sistema sin conservar copia de los mismos. Si usted no es el destinatario al que este correo está dirigido, queda usted notificado que la difusión, distribución o el copiado de este correo y de sus archivos adjuntos está prohibido.

   

 AVISO: Hemos tomado las precauciones razonables para prevenir que este correo esté infectado por virus. No aceptamos responsabilidad alguna por daños o pérdidas causadas por el uso de este correo o de los archivos adjuntos al mismo.

 This e-mail and any attachments transmitted with it are personal, privileged and confidential and solely for the use of the individual to whom they are addressed and intended. If you have received this e-mail in error, please notify the sender by return e-mail and delete this message and its attachments from your system without keeping a copy. If you are not the intended recipient, you are hereby notified that the dissemination, distribution or copying of this e-mail and attachments transmitted with it is strictly prohibited.

   

 NOTICE: We have taken reasonable precautions to prevent viruses from being present in this e-mail. We do not accept responsibility for any loss or damage arising from the use of this e-mail or attachments.

[Johannes P]

Aha! Well, no - the xml files are in the same place as the types' java files (as discribed in the uimaFIT guide book):

Assume that a type org.apache.uima.fit.type.Token is specified in the TSD org/apache/uima/fit/type/Token.xml, then the file should have the following contents:
classpath*:org/apache/uima/fit/type/Token.xml

But maybe I misunderstood it - at least I see your point, so they should also be in the resources directory - I just moved them to the resource directory, and now after 

mvn clean package

they are in the target/classes at the correct position! So far so good, but I still receive the same error when running mvn exec:java. 

And yes, I removed the <resources> section from the pom. 

Johannes

[Jose Luis Vieyra Sagaon]

Mmm maybe then you are not running the maven plugin for generating annotations (jcasgen) ??

If its that maybe this can help, this goes in my pom.xml.

I don't recall where I found the plugin, but it works and fixed my issues.

To fix it I had to:

1) add the lines I told you about in last email in the file:

src/main/resources/META-INF/org.apache.uima.fit/types.txt

2) mvn clean install with the jcasgen plugin on your pom:

<plugin>

<groupId>org.apache.uima</groupId>

<artifactId>jcasgen-maven-plugin</artifactId>

<configuration>

<typeSystemIncludes>

<include>src/main/resources/gex/ds/nlp/dkpro/types/desc/MacaTypes.xml</include>

</typeSystemIncludes>

</configuration>

<executions>

<execution>

<phase>generate-sources</phase>

<goals>

<goal>generate</goal>

</goals>

</execution>

</executions>

</plugin>

Hope it helps

--

You received this message because you are subscribed to the Google Groups "dkpro-core-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dkpro-core-us...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mto. Ing. José Luis Vieyra S.

[Richard Eckart de Castilho]

On 10.04.2015, at 20:20, Johannes P <de.pietsc...@gmail.com> wrote:

> Aha! Well, no - the xml files are in the same place as the types' java files (as discribed in the uimaFIT guide book):
>
> Assume that a type org.apache.uima.fit.type.Token is specified in the TSD org/apache/uima/fit/type/Token.xml, then the file should have the following contents:
> classpath*:org/apache/uima/fit/type/Token.xml

I see how that can be confusing, in particular to people that are rather new to Maven. The guide refers to the location of the file in the classpath. As you have noticed, Maven will not copy the descriptors over to the target/classes folder (which becomes part of the classpath) unless these files are under src/main/resources.

> But maybe I misunderstood it - at least I see your point, so they should also be in the resources directory - I just moved them to the resource directory, and now after
> mvn clean package
> they are in the target/classes at the correct position! So far so good, but I still receive the same error when running mvn exec:java.

I have to admit, that I never tried exec:java (at least I cannot remember it). I believe it should work, but I'm not sure.

Did you try again building a standalone JAR with the shade plugin given the advice from Stackoverflow.

Cheers,

-- Richard

[Johannes Pietsch]

That's true, I'm not using this maven plugin yet. 

When I use it, the myType.java and myType_Type.java files are being generated accordingly in target/generated-resources/com.example.types/... - additionally to the ones already present in the src/main/java/com/example/types... directory, which leads to compile errors "The type .. is already defined" - so I delete the old files.

But still I get the same error - and when changing myType.xml in eclipse the myType.java and myType_Type.java files are being generated in their old location again. Leading to the error above. So I think my uima eclipse plugin and the maven plugin use different destinations and are thus not compatible. 

You received this message because you are subscribed to a topic in the Google Groups "dkpro-core-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dkpro-core-user/YKQnvNJHaLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dkpro-core-us...@googlegroups.com.

[Richard Eckart de Castilho]

On 10.04.2015, at 20:50, Johannes Pietsch <de.pietsc...@gmail.com> wrote:

> That's true, I'm not using this maven plugin yet.
> When I use it, the myType.java and myType_Type.java files are being generated accordingly in target/generated-resources/com.example.types/... - additionally to the ones already present in the src/main/java/com/example/types... directory, which leads to compile errors "The type .. is already defined" - so I delete the old files.
>
> But still I get the same error - and when changing myType.xml in eclipse the myType.java and myType_Type.java files are being generated in their old location again. Leading to the error above. So I think my uima eclipse plugin and the maven plugin use different destinations and are thus not compatible.

When you get the message

> JCas type "com.example.types.MyNewType" used in Java code, but was not declared in the XML type descriptor.

That means that your the classes generated by JCasGen have been generated and have been found in the correct location. If they had not been found, you'd get a compiler error whenever your try to use the JCas classes in your code.

Whether you generate these classes using the Maven plugin or the type-system editor in Eclipse doesn't make much of a difference. The one in Eclipse should generate the Java classes to src/main/java, while the Maven plugin puts them under target/generated-classes, but both end up on the classpath.

The error clearly indicates that the XML type descriptors have not been found.

Please try adding the following dependency to your Maven project:

---

<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.15</version>
<exclusions>
<exclusion>
<artifactId>jmxtools</artifactId>
<groupId>com.sun.jdmk</groupId>
</exclusion>
<exclusion>
<artifactId>jmxri</artifactId>
<groupId>com.sun.jmx</groupId>
</exclusion>
<exclusion>
<artifactId>jms</artifactId>
<groupId>javax.jms</groupId>
</exclusion>
<exclusion>
<artifactId>mail</artifactId>
<groupId>javax.mail</groupId>
</exclusion>
</exclusions>
</dependency>

---

And add a file called "log4j.properties" to your src/main/resources folder with the contents:

---

log4j.rootLogger=DEBUG,development

log4j.appender.development=org.apache.log4j.ConsoleAppender
log4j.appender.development.layout=org.apache.log4j.PatternLayout
log4j.appender.development.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %5p [%t] (%C{1}) - %m%n

---

Then try to run your test code. You should see something like this on the console:

---

2015-04-10 21:15:01 DEBUG [main] (PathMatchingResourcePatternResolver) - Looking for matching resources in directory tree [.../uimafit-trunk/uimafit-core/target/test-classes/org/apache/uima/fit/type]
2015-04-10 21:15:01 DEBUG [main] (PathMatchingResourcePatternResolver) - Searching directory [.../uimafit-trunk/uimafit-core/target/test-classes/org/apache/uima/fit/type] for files matching pattern [.../uimafit-trunk/uimafit-core/target/test-classes/org/apache/uima/fit/type/**/*.xml]
2015-04-10 21:15:01 DEBUG [main] (PathMatchingResourcePatternResolver) - Resolved location pattern [classpath*:org/apache/uima/fit/type/**/*.xml] to resources [file [.../uimafit-trunk/uimafit-core/target/test-classes/org/apache/uima/fit/type/AnalyzedText.xml], file [.../uimafit-trunk/uimafit-core/target/test-classes/org/apache/uima/fit/type/Sentence.xml], file [.../uimafit-trunk/uimafit-core/target/test-classes/org/apache/uima/fit/type/Token.xml]]
2015-04-10 21:15:02 DEBUG [main] (TypeSystemDescriptionFactory) - Detected type system at [file:.../uimafit-trunk/uimafit-core/target/test-classes/org/apache/uima/fit/type/AnalyzedText.xml]
2015-04-10 21:15:02 DEBUG [main] (TypeSystemDescriptionFactory) - Detected type system at [file:.../uimafit-trunk/uimafit-core/target/test-classes/org/apache/uima/fit/type/Sentence.xml]
2015-04-10 21:15:02 DEBUG [main] (TypeSystemDescriptionFactory) - Detected type system at [file:.../uimafit-trunk/uimafit-core/target/test-classes/org/apache/uima/fit/type/Token.xml]

---

Maybe that will help us see in more detail where uimaFIT is looking, what it is finding and what not.

Cheers,

-- Richard

[Jose Luis Vieyra Sagaon]

In addition to what Richard says, you need to tell maven and eclipse where are these sources generated. If I recall correctly I still had the same error as you did until I did something like this on the pom:

<plugin>

<groupId>org.codehaus.mojo</groupId>

<artifactId>build-helper-maven-plugin</artifactId>

<executions>

<execution>

<id>addToSourceFolder</id>

<goals>

<goal>add-source</goal>

</goals>

<phase>process-sources</phase>

<configuration>

<sources>

<!--default path to generated sources-->

<source>${project.build.directory}/generated-sources/jcasgen</source>

</sources>

</configuration>

</execution>

</executions>

</plugin>

-- Richard

--
You received this message because you are subscribed to the Google Groups "dkpro-core-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dkpro-core-us...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mto. Ing. José Luis Vieyra S.

[Richard Eckart de Castilho]

Hi Jose,

if the JCas cover classes are not generated or the generated Java files are not in a source folder, you'll get a compiler error when trying to use them

Johannes gets an error at runtime.

Mind that it is totally possible to use UIMA without JCas cover classes if you stick to using only the CAS API instead of the JCas API.

Cheers,

-- Richard

[Johannes P]

Thanks for the tip with the log4j, 

when I run it now in eclipse I get several (34) messages like this:

2015-04-10 22:01:58 DEBUG [main] (TypeSystemDescriptionFactory) - Detected type system at [jar:file:/Users/jpietsch/.m2/repository/de/tudarmstadt/ukp/dkpro/core/de.tudarmstadt.ukp.dkpro.core.api.lexmorph-asl/1.6.2/de.tudarmstadt.ukp.dkpro.core.api.lexmorph-asl-1.6.2.jar!/desc/type/Morpheme.xml]

among them also my own types

2015-04-10 22:01:58 DEBUG [main] (TypeSystemDescriptionFactory) - Detected type system at [file:/Users/jpietsch/Documents/.../MyProj/target/classes/com/example/types/MyType.xml]

But running on the server I just get 26 messages with only the dkpro types. 

[Richard Eckart de Castilho]

Hi,

great you got it sorted out! :)

Cheers,

-- Richard

On 11.04.2015, at 20:35, Johannes Pietsch <de.pietsc...@gmail.com> wrote:

> Hello Richard,
>
> It works now! :) Thanks to your last comment I noticed my types-txt was still not in the resources folder on the server (but in eclipse) - I simple forgot to move it. Now all types are being found!
>
> Thank you so much for your assistance!
>
> All the Best,
> Johannes