cleartk questions
Philip Ogren
I am reposting your email to the new google group I created yesterday. I apologize for not making time to answer your email. You are asking great questions about tasks that we had in mind when we built ClearTK and there is a lot of infrastructure for speeding you on your way with these tasks. I don't have time to give a detailed response before the weekend - but I wanted to repost your inquiry before it gets lost in the abyss that is my inbox.
Philip
-------- Original Message --------
| Subject: | Re: cleartk questions |
|---|---|
| Date: | Thu, 19 Feb 2009 17:44:18 +0100 (CET) |
| From: | Olivier Grisel <ogr...@nuxeo.com> |
| To: | Philip Ogren <phi...@ogren.info> |
| CC: | \"steven bethard \ <steven.bethard @gmail.com>, phwe...@gmail.com |
----- "Philip Ogren" <phi...@ogren.info> a écrit : > Olivier, > > Thank you for your interest in ClearTK. Please let us know if there > are specific questions that we can answer with respect to the two > tasks you will be working on as mentioned below. Since you nicely offer help, here are more details: I am currently reading the source code and the tests to get to get an overview on what is available. My high level goal is to develop tools to: 1- semantically annotate documents (news from news agencies, internal corporate reports, whatever a CMS / ECM user might produce) with named entities occurence and their type. For instance I would like to obtain results similar to what the opencalais project provides using the gnosis plugin for firefox for instance: https://addons.mozilla.org/en-US/firefox/addon/3999 2- to extract facts / assertions from the unstructured documents, probably expressed as RDF triples using some kind of ontology (probably based on wikipedia exports such as http://freebase.com or http://dbpedia.org). Then a SPARQL engine with a (user-friendly?) frontend would allow the users to perform smart queries on the database such as who has won Rolland Garros tournament between 1995 and 1999. The query part is planned for later. I want to focus on the structured knowledge extraction part for now. Could you please which algorithms / classifiers provided by cleartk and wrapped projects are the most efficient for each task? Do you have sample chains of AE that would offer prototype start I could then refine on my own? Which corpora should I use to train my models based on my high level goals? For the SRL (goal number 2) part I wanted to focus on the Mallet CRFs engine by the sample found on the test suite is used for biological sequence labeling. Are CRLs no so suited for SRL? Are there better machine learning engines for this task? Are there engines that perform both NE and SRL at the same time or is it better to perform each task separately or one after the other? > We have not set up a project mailing list yet - but this is an > excellent (and quite obvious) suggestion. At the moment all > communication is accomplished via email between the three of us > developers. We, ran into this yesterday because we had a long and > thoughtful discussion about feature extraction that was doomed for > loss - so I posted it to a googlecode issue. But this is no > replacement for a proper email list. I am new to google code - it > doesn't appear that there is an analogy to sourceforge forums. Do you > have a recommendation for a user-list service or something like a > forum that we could use? >From my experience with google code hosted projects, they usually setup a google group dedicated to the project. Sometimes two of them for instance: - cleartk-dev: for developer discussions on the coding of the cleartk project itself - cleartk-users: for discussion on how to develop application with cleartk A single cleartk google groups is probably enough in the first place. http://groups.google.com/ Don't forget to add a link to the group on the frontpage of the project. > In the past we have used ant for building ClearTK. However, I think we > have let our build.xml scripts collect dust as all three of us have > been developing in Eclipse and our subversion repository is simply a > checked in Eclipse project. When I am performing many experiments I > generally run them on a linux node via ant tasks. I have never looked > into Maven and don't have an intuition as to why it is better. If you > would send me a link that would have a succinct comparison, then I > would be happy to take a look and consider it. Does it make running > stuff any easier? Or, is it just for building? > Yes sure. Compared to ant, maven mainly add the dependency management: you no longer need to include the jars of the dependencies and then fight build.xml and eclipse build path to make them appear in the right place. All you need to do is setup a pom.xml file in your project specifying the list of projects (maven artifacts you depend on). Then by typing: $ mvn install It will: 1- download any dependency from standard maven repositories on the internet and place a local copy of the jars in ~/.m2/repostory folder for later reuse. 2- build the jars of the project from the source code 3- run the tests (you can skip that by using 'mvn install -Dmaven.test.skip=true' instead) 4- install a local copy of the cleartk jars in your ~/.m2/repository so that other maven based projects can have it in their classpath Also by typing: $ mvn eclipse:eclipse maven will generate to the eclipse project files (.project and .classpath) so pointing to the .m2/repostory to find the build dependencies. So you only need to maintain one file (the pom.xml file) to have both the maven eclipse command line and the eclipse build work the same way. You can also run "mvn deploy" to push the released jars of your project to a public maven repository to make it easier for other opensource project to depend on publicly released jars of cleartk. There are also a bunch of other maven plugins to perform a variety of tasks (like building pdf documentation from dockbook for instance). As maven2 is used for the UIMA project it will make it much easier to build cleartk from the source just by adding the UIMA dependency in the pom.xml file of the cleartk project. I can help you setting that up in a svn branch if you want. As it requires some directory reorg to meet the maven conventions I cannot contribute this work easily using only a simple patch / diff. I think maven is becoming the de facto standard for java frameworks that need dependency management so it is probably a good war to make cleark a nice player with the rest of the java open source community. Regards, -- Olivier
ogrisel
> Olivier,
>
> I am reposting your email to the new google group I created yesterday.
> I apologize for not making time to answer your email. You are asking
> great questions about tasks that we had in mind when we built ClearTK
> and there is a lot of infrastructure for speeding you on your way with
> these tasks. I don't have time to give a detailed response before the
> weekend - but I wanted to repost your inquiry before it gets lost in the
> abyss that is my inbox.
Just a message to let you know that I have joined both groups so that
we can
now go on with that thread on this group only. I am looking forward to
reading
your insights on ClearTK usage best practices.
--
Olivier
Philip Ogren
such as that performed by the Gnosis plugin. There is a quite a lot of
literature on named entity recognition. I would start by looking at some
of the shared tasks such as ACE 2005, CoNLL 2003 (see
http://www.cnts.ua.ac.be/conll2003/proceedings.html), and BioCreative
(see http://biocreative.sourceforge.net/evaluations.html - also
http://genomebiology.com/2008/9/s2/S2). Generally, having a complete
list of the things you are looking for is not feasible because there are
too many items of interest to list and because of name collisions with
things you want to ignore. However, many people use approaches where
lexicons / word clusters are derived from a large corpus. The most
straightforward way performing named entity recognition with ClearTK
would be to treat it as a sequential tagging task where each token is
classified with a BIO-style labeling scheme and then train up a
sequential classifier such as the Mallet CRF. The chunk package in
ClearTK provides some core infrastructure for performing this kind of
chunking. For feature extraction, some features you might want to use
include:
word
Word – lowercase
Capital type (mixed, initial cap, all caps)
Numeric type
Contains hyphen
Char n-grams, prefixes and suffixes, sizes 3-5
3 tokens to left, right (plus n-grams)
pos tag (plus n-grams)
stem
Gazetteer features - using TermFinderAnnotator
ClearTK provides feature extractors for all of these. There are also
collection readers for the ACE 2005 data and CoNLL 2003 data sets which
can be used for training. There is generally no succinct recipe that
will guarantee great performance on a named entity task. Many of your
decisions will depend specifically on your task definition, your target
data, and speed requirements. Still, ClearTK could do more to make some
of the basic approaches much easier. I had done some experimentation on
ACE and CoNLL data early on when we were first getting ClearTK going -
but that code is not currently available because of neglect. I really
should put some cpe's together that call everything in a reasonable
order to provide a baseline system.
I am not as familiar with semantic role labeling as Philipp - so I will
defer to him. Often when folks refer to SRL - they mean performing the
task of "propbanking" - i.e. identifying predicates and their arguments.
ClearTK provides a collection reader for PropBank which is otherwise
non-trivial to parse. The 'srl' package has a lot of the infrastructure
to perform semantic role labeling - but it is not yet a polished tool.
There should be some CPE's in the cpe folder that should give you a
rough outline of what should get run and in what order. We have not,
however, made the effort to get the SRL stuff working at
state-of-the-art levels yet.
I hope this helps!
Philip
Olivier Grisel
- Afficher le texte des messages précédents -
Thanks for the insight. I was starting to go that way. Having a base list of
pertinent feature extractors can save me some time!
Do you think NE performance could be greatly improved by tagging tokens or
chunks that match the textual name of famous persons, places or organizations
found in wikipedia using DB lookups in dbpedia or freebase? Have you already
done so?
> There is generally no succinct recipe that
> will guarantee great performance on a named entity task. Many of your
> decisions will depend specifically on your task definition, your target
> data, and speed requirements. Still, ClearTK could do more to make some
> of the basic approaches much easier. I had done some experimentation on
> ACE and CoNLL data early on when we were first getting ClearTK going -
> but that code is not currently available because of neglect. I really
> should put some cpe's together that call everything in a reasonable
> order to provide a baseline system.
That would indeed be great to have sample CPEs to achieve near state
of the art performance on benchmark corpora. I discovered the cpe/ folder
in the source tree after posting my initial question email. I think the source
tree should come with something like a QUICKSTART.txt file giving the
the minimal command lines to run a CPE to achieve a common task such as NE
on a corpus which is either found by default in the source tree of to be
downloaded freely on a public website.
Also, a README.txt file in the cpe folder giving an overview of the available
sample CPE would be great.
BTW: the online tutorial does not match the current state of the trunk source
tree but I guess this is already known to you.
> I am not as familiar with semantic role labeling as Philipp - so I will
> defer to him. Often when folks refer to SRL - they mean performing the
> task of "propbanking" - i.e. identifying predicates and their arguments.
> ClearTK provides a collection reader for PropBank which is otherwise
> non-trivial to parse. The 'srl' package has a lot of the infrastructure
> to perform semantic role labeling - but it is not yet a polished tool.
> There should be some CPE's in the cpe folder that should give you a
> rough outline of what should get run and in what order. We have not,
> however, made the effort to get the SRL stuff working at
> state-of-the-art levels yet.
Ok thanks. Do you know if there are research results on experiments
to match SRL tagged chunks (e.g. A0 - REL - A1) to terms found in a preexisting
ontology so as to populate a knowledge base (a RDF triple store) from raw
unstructured text?
Regards,
--
Olivier