[ANNOUNCE] RelEx relationship extractor
Linas Vepstas
I'd like to announce a new project that I think will be of interest to
everyone here; it is a semantic relationship extractor built on top
of the link grammar parser. It works with the link structure coming
out from link grammar, and presents it in a direct fashion. It includes
support for using GATE as an entity detector front-end; it also has
a basic implementation of the Hobbs algorithm for anaphora resolution.
The URL:
https://launchpad.net/relex
Perhaps a picture is worth a thousand words; here's a parse of
"Alice looked at the cover of Shonen Jump. She decided to buy it."
Alice looked at the cover of Shonen Jump.
====
Parse 1 of 2
====
(S (NP Alice) (VP looked (PP at (NP (NP the cover) (PP of (NP Shonen
Jump))))) .)
+--------------------------------Xp-------------------------------+
| +---Js---+ +--------Jp-------+ |
+----Wd----+----Ss---+-MVp-+ +--Ds-+--Mp-+ +-----A----+ |
| | | | | | | | | |
LEFT-WALL Alice[?].n looked.v at the cover.n of Shonen[?].a Jump[?].n .
======
at(look, cover)
_subj(look, Alice )
tense(look, past)
of(cover, Jump)
DEFINITE-FLAG(cover, T)
noun_number(cover, singular)
_amod(Shonen, Jump)
DEFINITE-FLAG(Shonen, T)
noun_number(Shonen, singular)
DEFINITE-FLAG(Jump, T)
noun_number(Jump, singular)
DEFINITE-FLAG(Alice , T)
gender(Alice , feminine)
noun_number(Alice , singular)
person-FLAG(Alice , T)
======
She decided to buy it.
====
Parse 1 of 2
====
(S (NP She) (VP decided (S (VP to (VP buy (NP it))))) .)
+---------------Xp--------------+
+--Wd--+--Ss--+--TO--+-I-+-Ox-+ |
| | | | | | |
LEFT-WALL she decided.v to buy.v it .
======
_subj(decide, she_2)
_to-do(decide, buy)
tense(decide, past)
_subj(buy, she_2)
_obj(buy, it_1)
tense(buy, infinitive)
HYP(buy, T)
DEFINITE-FLAG(it_1, T)
gender(it_1, neuter)
PRONOUN-FLAG(it_1, T)
DEFINITE-FLAG(she_2, T)
gender(she_2, feminine)
noun_number(she_2, singular)
PRONOUN-FLAG(she_2, T)
======
Antecedent candidates:
_ante_candidate(it_1, cover) {0}
_ante_candidate(it_1, Jump) {1}
_ante_candidate(she_2, Alice ) {0}
Genta
What a good news!
BTW, what is GATE??
Genta
Linas Vepstas
"General Architecture for Text Engineering"
see below ... describes what GATE is and how its used.
I extended the README for the future version 0.8.1 ---
OpenRelEx Semantic Relation Extractor
-------------------------------------
RelEx is a syntactic relationship extractor; it will parse
English language sentences and return the relationships
between different parts of the sentence.
There are two parts: the command-line version, and an
graphical visualizer. Build and install for the command
line version is discussed first.
Dependencies
-------------
The following packages are required to be installed:
- libgetopt-java
- LinkParser
- Wordnet 2.0 or later
- JWNL Java wordnet library
- OpenNLP tools
- GATE 4.0 or later (optional)
Building command-line RelEx
---------------------------
- Link Grammar Parser
XXX this is wrong, since version 4.3.2 or later is needed. XXXX
Ubunut/Debian users can install the linkparser with apt-get:
apt-get install liblink-grammar4 link-grammar-dictionaries-en
Other users:
Compile and install the Link Grammar Parser. This parser is
described at http://www.link.cs.cmu.edu/link/, and sources
are available for download at
http://www.abisource.com/projects/link-grammar/#download
The Link Grammar Parser is the underlying engine, providing
the core sentence parsing ability.
If the link-grammar is installed in an unusual location,
be sure to modify -Djava.library.path appropriately in
relation-extractor.sh.
- Wordnet
Wordnet is used by RelEx to provide basic English morphology,
such as singular versions of (plural) nouns, base forms (lemmas)
of adjectives, adverbs and infinitve forms of verbs.
Download, unpack and install WordNet 2.0. The install directory
then needs to be specified in data/wordnet/file_properties-linux.xml,
with the name="dictionary_path" property in this file.
Some typical install locations are:
/opt/WordNet-2.0/data for RedHat and SuSE
/usr/share/wordnet for Ubuntu and Debian
C:\Program Files\WordNet\2.0\data for Windows
The relex/Morphy/Morphy.java class provides a simple, easy-to-use
wrapper around wordnet, providing the needed word morphology info.
- didion.jwnl
The didion JWNL is the "Java WordNet Libary", and provides the
Java programming API to access the wordnet data files.
Its home page is at http://sourceforge.net/projects/jwordnet
and can be downloaded from
http://sourceforge.net/project/showfiles.php?group_id=33824
Verify that the final installed location of jwnl.jar is correctly
specified in the build.xml file. Note that GATE (below) also
provides a jwnl.jar, but the GATE version of jwnl.jar is not
compatible (welcome to java DLL hell).
- GATE
GATE, the "General Architecture for Text Engineering",
http://www.gate.ac.uk/ provides a large framework which
RelEx does not use. However, it does provide a good entity
detector, which RelEx does employ. "Entities" are the
names of people, corporations and institutions, as well
as time, date and money expressions. Factoring these out
explicitly makes parsing and relation extraction simpler.
Download GATE 4.0 from http://gate.ac.uk/download/index.html
Install it at /opt/GATE-4.0 . If you change this location,
please modify the system property -Dgate.home=/opt/GATE-4.0
in relation-extractor.sh. Modify build.xml to point at the
correct location of gate-4.0.jar.
GATE is used solely to perform entity resolution; that is,
to determine substrings that correspond to names, dates,
addresses, money amounts, etc. The use of GATE is optional,
however, without it, the parser has trouble with dates,
money amounts etc. GATE is also needed to identify the gender
(male, female) of people names; the gender is needed for
anaphora resolution (so that "she" is matched nly to female
names).
Alternatives to GATE may be used by providing a replacement
for the relex/corpus/GateEntityMaintainer.java file.
- OpenNLP
OpenNLP provides a number of natural language processing tools.
RelEx uses OpenNLP for sentence detection, giving RelEx the
ability to input texts containing multiple sentences.
The OpenNLP home page is at http://opennlp.sourceforge.net/
Download and install OpenNLP tools, and verify that the
installed files are correctly identified in both build.xml
and in relation-extractor.sh.
The OpenNLP pacake is used solely in corpus/DocSplitter.java,
which provides a simple, easy-to-use wrapper for splitting a
document into sentences. Replace this file if an alternate
sentence detector is desired.
Building
--------
After the above are installed, the relex java code can be built.
The build system uses "ant", and the ant build specifictions
are in "build.xml". Simply saying "ant" at the command line
should be enough to build.
Running Relex
-------------
Several example shell scripts (MS Windows batch files) are included
to show sample usage. These files (*.sh in unix, or *.bat, in Windows)
define the required system properties, classpath and JVM options.
If there are any ClassNotFound exceptions, please verify the paths
and values in these files. An important property is relex.algpath;
it defines the semantic algorithms used by RelEx. The default file
is data/relex-semantic-algs.txt.
1) Simple single-sentence parser
The "relation-extractor.sh" file is a simple shell script illustrating
the parse of single sentence. Shows typical RelEx raw output.
Evgenii Philippov
- First, I had a lot of hassle with Cygwin compiling lg, but via dirty
hacking got it to compile.
- Second, I had to tweak link-grammar sources a bit, so that it would
find a dictionary on my cyrillic system.
a) api.c:
Dictionary dictionary_create(...) {
return internal_dictionary_create(dict_name, pp_name, cons_name,
affix_name, DICTIONARY_DIR);
}
b) api.h, at the very end, before endif:
#define DICTIONARY_DIR "C:\\link-grammar\\4.3.2\\data\\"
//i.e. hardcoded DICTIONARY_DIR
c) jni-client.c:
static void init(void){
...
dict = dictionary_create_lang("en");
test();
}
//since it does not make sense to check machine's default locale.
Third, it runs and parses OK. Excellent work Linas!
I currently want to first add a useless trick: add a Swing semantic
graph + anaphora resolution visualization.
Then, I'll choose something saner to do with RelEx.
-e
Evgenii Philippov
I currently want to first add a useless trick: add a Swing semantic graph + anaphora resolution visualization.
-e
Evgenii Philippov
http://efilippov.googlepages.com/relex001.JPG
http://efilippov.googlepages.com/relex002.JPG
http://efilippov.googlepages.com/relex003.JPG
The above images are public domain.
-e
Linas Vepstas
>
> - First, I had a lot of hassle with Cygwin compiling lg, but via dirty
> hacking got it to compile.
If you have any non-dirty patches, let me know.
> - Second, I had to tweak link-grammar sources a bit, so that it would
> find a dictionary on my cyrillic system.
>
> a) api.c:
The dictionary open code was insane. I cleaned it up a bit in
the source tree.
> dict = dictionary_create_lang("en");
> test();
> }
>
> //since it does not make sense to check machine's default locale.
Yes; I changed this, it will have to be revisted if/when there are more
dictionaries.
> Third, it runs and parses OK. Excellent work Linas!
Thanks, although I'm not the only one ...
Linas Vepstas
> I currently want to first add a useless trick: add a Swing semantic
> graph + anaphora resolution visualization.
>
> Implemented a visualizer using jung2 library. Results are less than
> satisfactory (bad graph layout algorithms of jung2), but I'll use the
> visualizer for myself.
Nice!
At one point, we had code that used graphviz, but it got broken
or lost or something.
> My code can be published though.
I'll include it, if you help me with one thing: I'd like to make installing
swing optional. I don't want to force users to install a zillion packages
just to get he basics working.
Problem is, I don't know how to make things optional in Java. There
are no #defines or conditional compiles -- it seems like everything
is always just on ...
--linas
Evgenii Philippov
On 16/02/2008, Evgenii Philippov <efil...@gmail.com> wrote:Evgenii Philippov wrote: I currently want to first add a useless trick: add a Swing semantic graph + anaphora resolution visualization. Implemented a visualizer using jung2 library. Results are less than satisfactory (bad graph layout algorithms of jung2), but I'll use the visualizer for myself.Nice! At one point, we had code that used graphviz, but it got broken or lost or something.
I'll include it, if you help me with one thing: I'd like to make installing swing optional.
The answer is that complete Swing is included in any JRE since the ancient JRE 1.2 or 1.3 or so. So you do not need to bother with this, users will not need to install anything except for the vis. library JAR files.
Problem is, I don't know how to make things optional in Java. There are no #defines or conditional compiles -- it seems like everything is always just on ...
I will move the visualizer into a separate JAR.
-- For conditional compiles of some sort, Java folks use final static boolean variables (but with visualizer, we can use the above mechanism since it is simpler for use).
final static boolean DEBUG=true;
if(DEBUG)code1();
else code2();
//compiled class files will have only code1 compiled into a class file, and not code2.
-e
Evgenii Philippov
I wrote:
I am retrying my visualizer implementation --- will try to find a nicer graph layout library.
Evgenii Philippov
With enabled GATE entity extractor:
http://efilippov.googlepages.com/relex004.JPG
Linas Vepstas
> Oops! It seems that you are relatively new to Java. I have 10 years of
> experience in Java,
I've 10 years of using java and disliking it; I've mostly run away given the
opportunity. I admit that its fun to program in, now that I'm doing it again,
but its terrible for large, complex systems; it seems immature and not yet
grown-up. I'm maintainer of the current code, rather than its original
developer.
> And I am extremely interested in
> RelEx and link grammar development.
Excellent!
> The answer is that complete Swing is included in any JRE since the ancient
> JRE 1.2 or 1.3 or so. So you do not need to bother with this, users will not
> need to install anything except for the vis. library JAR files.
I'm not sure that there is a concept of "JRE" in linux. Instead of installing a
gigabyte of everything and the kitchen sink, linux distros tend to partition
things up into small pieces, and install only that which is actually needed.
Since java is generically unpopular on Linux, most java packages
are not available on linux. In particular, none of the packages that relex
depends on are in linux(debian/ubuntu/redhat/suse) So that is a major
hassle, since that means linux users have to go to some damned website
somewhere and download and read install instructions and follow them,
etc.etc.
Since a linux relex user will have to manually install 4 or 5 packages,
that seems like too much complexity already, and I want to avoid
making dependencies on yet more packages. Its basically a near total
lack of automation of java on linux.
> -- The only way I know of is moving stuff into separate JAR files... and
> then checking if ClassNotFoundException is thrown at some point.
Ohh, I like that! I will experiment with this, both for GATE and for the
senetence detector,
--linas
Evgenii Philippov
Linas wrote:
Ohh, I like that! I will experiment with this, both for GATE and for the senetence detector
-e
Evgenii Philippov
Can I ask: what work on LG/Relex output disambiguation is planned for short term and long term?
I am currently reading papers on parse ranking and verb/arguments subcategorization classes.
-e
Linas Vepstas
> Can I ask: what work on LG/Relex output disambiguation is planned for short
> term and long term?
There's some code that has been written for parse ranking, but I haven't
looked at it carefully yet; its sitting inside a strange infrastructure that is
related to relex, but not directly integrated; I have to figure out if it can
be integrated.
> I am currently reading papers on parse ranking and verb/arguments
> subcategorization classes.
I am planning on merging in code for providing semantic frames based
on relex output. It is inspired by framenet. (google framenet) I have not
yet thought about whether/how the frames can be used to help with parse
ranking.
--linas
Linas Vepstas
>>> -- The only way I know of is moving stuff into separate JAR files... and
>>> then checking if ClassNotFoundException is thrown at some point.
>>
I tried that, and multiple variants thereof, including e.g.
ClassLoader ld = ClassLoader.getSystemClassLoader();
ld.loadClass("opennlp.tools.sentdetect.EnglishSentenceDetectorME");
at first, it seemed to work, but as I went back to double check everything,
it stopped working, and now, no matter what, it seems I cannot catch
the throwable/error. I'm mystified.
--linas
Evgenii Philippov
VisualizerLoader works ok with and without libraries. Maybe later I'll do this myself, to satisfy your needs with opennlp separation etc. What components do you want to separate? EnglishSentenceDetectorME, what else?-e
Linas Vepstas
GATE, for entity detection.
Please wait a few days; I am trying to set up a public svn repository;
if you modify the code now, it will collide with some changes I made.
--linas
Evgenii Philippov
Linas Vepstas wrote:
On 17/02/2008, Evgenii Philippov <efil...@gmail.com> wrote:Can I ask: what work on LG/Relex output disambiguation is planned for short term and long term?There's some code that has been written for parse ranking, but I haven't looked at it carefully yet; its sitting inside a strange infrastructure that is related to relex, but not directly integrated; I have to figure out if it can be integrated.
I am currently reading papers on parse ranking and verb/arguments subcategorization classes.I am planning on merging in code for providing semantic frames based on relex output. It is inspired by framenet. (google framenet) I have not yet thought about whether/how the frames can be used to help with parse ranking.
--linas
Evgenii Philippov
Any plans for discourse analysis? Dialogue maintenance?
-e
Linas Vepstas
>
>
>
> Linas Vepstas wrote:
>
> Any plans for language generation?
Actually, yes; and some work has started. Plans are to use
link-grammar so as to construct grammatically correct
sentences.
However, no one at all seems to be thinking about literary style.
I'm afraid that system will write as third graders:
Abraham Lincoln was a president. He was tall. He wore a hat.
as opposed to the more complex, information-full:
Lincoln was the tallest president of the United States,
he is typically pictured with a stove-pipe hat.
I don't know how to define and control style like that, except
possibly by emulation?
--linas
muriloq
>
> > Evgenii Philippov wrote:
> > I currently want to first add a useless trick: add a Swing semantic
> > graph + anaphora resolution visualization.
>
> > Implemented a visualizer using jung2 library. Results are less than
> > satisfactory (bad graph layout algorithms of jung2), but I'll use the
> > visualizer for myself.
>
> Nice!
>
> At one point, we had code that used graphviz, but it got broken
> or lost or something.
RelEx output. It still being used in our project (which includes a web
interface to RelEx and nice graphs, which multiple layout algorithms)
and it was working fine with our internal branch of link-grammar.
Here's an example output:
http://img169.imageshack.us/img169/1978/relexgraphuh1.png
It still doesn't work with the current link-grammar because I'm having
problems with the JNI interface (see my other message).
Best,
muriloq
Linas Vepstas
> On Feb 17, 2:45 am, "Linas Vepstas" <linasveps...@gmail.com> wrote:
>
> > At one point, we had code that used graphviz, but it got broken
> > or lost or something.
>
> I'm the author of the graphviz-based code for generating graphs from
> RelEx output. It still being used in our project (which includes a web
> interface to RelEx and nice graphs, which multiple layout algorithms)
The current relex is a subset of totality of disco code; it didn't seem
appropriate to publish all of disco.
--linas