A question about parsers
Alex Sirota
I'm looking for some good open source, generic parser software. Is Lucene considered a parser? From what I've read it is a high quality search engine, but can it be programmed to parse specific grammars? I'm looking at parsing curriculum vitae documents -- a much harder version of a resume.
Thanks for any info you can lend.
--
Alex Sirota
Michael Reid
Are these CV documents really regular grammars? If they don't have a
well defined structure then I would suppose you would need something
like data mining or maybe some really rudimentary/lightweight natural
language processing? I'm an expert in neither of these areas so my
terminology may be way off or I may be simply wrong.
However, if you really do have a regular grammar then there are simply
reams and reams of parser generators/libraries available for almost
any (computer) language you care to work in.
Two packages available for Java are:
* JParsec - http://jparsec.codehaus.org/
* ANTLR - http://www.antlr.org/
I choose these as examples because they represent the two major
approaches you'll encounter. The JParsec style is more like a library
of small parsers that you can combine to compose larger more complex
parsers. With the ANTLR approach, you precisely specify your grammar
to the parser software, and the tool spits out a program that can
parse that grammar.
Personally, I tend to favour the first approach since it lets you work
more informally and generally I find they are easier to integrate into
existing code. However the ANTLR approach has been around since the
70s so it is well proven.
If you can clarify what kind of data you are looking to extract, as
well as some details about the grammar I'm sure a lot of options will
present themselves.
/mike.
> --
> You received this message because you are subscribed to the Google Groups
> "TorCamp" group.
> To post to this group, send email to tor...@googlegroups.com.
> To unsubscribe from this group, send email to
> torcamp+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/torcamp?hl=en.
>
Alex Sirota
Thank you for the detailed set of options. Much appreciated. CVs are somewhat complex form of a resume. Although it would be great to present a grammar there is unfortunately not a single grammar that works. There are a multitude of resume formats and CVs unfortunately share very little with resumes, primarily work history and demographic information.
Here's a sample CV format from University of Toronto - Medicine Department.
http://www.deptmedicine.utoronto.ca/Faculty/Promotion_CV/Curriculum_Vitae.htm
CVs can be many pages long and can have as many as 100 different data types.
Here's a standard that is being proposed that would form a grammar, but how do you get from unstructured CVs to a proposed standard without going through a lengthy adoption and conversion procedure (many years?)
http://casrai.org/data-standards/cv-standard
I'm thinking of a multiphase process that includes
a) a standard best effort parse to get items like work history and demographics, honors and awards
b) a custom parse that looks at bibliographic and funded projects as well as research teams and publications
c) a system that allows one to fill in the gaps visually after a) and b) have completed to correct mistakes.
d) conversion to XML
e) a Lucene-style search engine on top to analyze the results
The main issue is that while there are only about 40,000 such CVs in Canada, the length and depth of each CV is quite complex. There are probably then 40,000 different formats and only a small subset of them would share any formatting structure (due to a shared nature in how they were created).
While resume parsers seem to do a decent enough jobs on a well formatted resume, seems like the CV parser is a potentially intractable problem, or so I've been told.
But surely it can't be.
Thanks for replying. Looking forward to any other input.
Yanni Chiu
>
> I'm thinking of a multiphase process that includes
>
> a) a standard best effort parse to get items like work history and
> demographics, honors and awards
>
> b) a custom parse that looks at bibliographic and funded projects as
> well as research teams and publications
>
> c) a system that allows one to fill in the gaps visually after a) and b)
> have completed to correct mistakes.
>
> d) conversion to XML
>
> e) a Lucene-style search engine on top to analyze the results
>
> The main issue is that while there are only about 40,000 such CVs in
> Canada, the length and depth of each CV is quite complex. There are
> probably then 40,000 different formats and only a small subset of them
> would share any formatting structure (due to a shared nature in how they
> were created).
>
> While resume parsers seem to do a decent enough jobs on a well formatted
> resume, seems like the CV parser is a potentially intractable problem,
> or so I've been told.
>
> But surely it can't be.
>
> Thanks for replying. Looking forward to any other input.
I see a solution like this...
- develop a model of CV metadata (your link to
http://casrai.org/data-standards/cv-standard looks like a good place to
start)
- build your CV database
- analyze each CV, and record the metadata
- a manual interface to fill in the metadata, while simultaneously
viewing the original CV would be a useful escape hatch
- a search interface that can look at the raw CV (i.e. free-text search)
and/or the metadata (i.e. tags or keywords in particular metadata fields)
The technical unknown is "analyze each CV". You could have a set of CV
parsers available (and more developed as needed). A heuristic (or manual
selection) could be used to choose which parser to use. Maybe multiple
parsers, each targeted at gathering specific metadata, could be used. A
final validation heuristic could be used to determine if all the useful
text has been extracted into metadata. A manual interface could be
available to fine tune the metatdata of any particular CV.
As for technology, have a look at the hot off the press PhD work and
open-source implementation at:
http://www.lukas-renggli.ch/blog/petitparser-1
http://www.lukas-renggli.ch/blog/petitparser-2
In this system, the parsers are just a part of your development
environment (Smalltalk). You can modify them, create new ones, and
compose them - which, IMHO, is the kind of parsing system you'll need to
solve your problem.
Hope that helps.
--
Yanni Chiu
Alex Sirota
It does seem that different parts of the CV need to have different parsers suited to them, as some are very consistent (bibliography) but many formats are quite different from each other.
If you come across any more parsers please let me know.
Alex
--
You received this message because you are subscribed to the Google Groups "TorCamp" group.
To post to this group, send email to tor...@googlegroups.com.
To unsubscribe from this group, send email to torcamp+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/torcamp?hl=en.
Yanni Chiu
> Smalltalk is an unfortunate
> technology choice, but is probably very well suited to the task.
Not sure whether there's a link off the blog, but there's a paper that
lists the author's reasons for choosing Smalltalk for the task of
"language engineering". The reasons are:
- minimal syntax
- dynamic semantics
- reflective facilities
- homogeneous language (Smalltalk is implemented in Smalltalk)
- homogeneous tools (developer tools & env implemented in Smalltalk)
- on-the-fly programming (no compile/link, changes are immediate to the
live system)
I'd like to understand why you chose the words "unfortunate technology
choice". What impediment do you foresee due to the language choice?
Within the Smalltalk community this is a difficult question to answer,
due to familiarity with can be done with Smalltalk - "it makes simple
things simple, and complicated things possible".
Alex Sirota
But that's the breaks... Parsers are somewhat an esoteric field I guess.
Alex
--
You received this message because you are subscribed to the Google Groups "TorCamp" group.
To post to this group, send email to tor...@googlegroups.com.
To unsubscribe from this group, send email to torcamp+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/torcamp?hl=en.
Rick Innis
Prolog and natural language can be a petty good fit. There's a list of
Prolog natural language tools at http://www.ai.uga.edu/mc/pronto/
Unfortunately, as with smalltalk, finding a good Prolog developer can
be challenging. (OK, I have to ask: hands up, anyone else here who's
programmed in Prolog and Smalltalk outside of coursework. Anyone?
Bueller?)
R.
Yanni Chiu
> Unfortunate because the system should be open sourced and
> unfortunately there is a very small Smalltalk community compared to
> other languages.
The Squeak/Pharo variant of Smalltalk, on which the PetitParser work is
built on, as well as the parser stuff itself, is all MIT licensed. It
took many years of community effort to move from the original "custom
open source" license used by Apple for Squeak's release.
As for small community, how big does it need to be to provide you a
solution? I recently learned that Ryerson U. is actually using Smalltalk
as the main teaching language in one of its core courses. So there are
more Smalltalk knowledgeable people out there than you might think.
Also, there's:
"I always knew that one day Smalltalk would replace Java. I just didn't
know it would be called Ruby." -- Kent Beck
> And the installation profile is probably more complex and getting
> support/help is either more expensive and/or harder to do.
It's not complex to install. Download the one-click image, execute 5
lines of installation code, and you should have the parser system ready
to evaluate. Most things seem complicated, when you've not seen it before.
Getting support/help on someone's fresh PhD work could be hard in
general, but I've seen the author respond to email and blog questions
about his work.
As for Squeak/Pharo suport/help being expensive or hard to obtain, I'm
not sure what your expectations are. Generally the mailing list is the
place to ask, as with any open-source project - support there is free
and easy to get. There are several commercial Smalltalk vendors - so
that support is not free, but I doubt it'd out of line with industry
pricing. If you're looking for a highly visible ecosystems of small
companies, then that's not here in North America, but it's developing in
Europe and South America.
> But that's the breaks... Parsers are somewhat an esoteric field I
> guess.
For your parsing problem, it seems the evaluation parameters need to be
wider.
Yanni Chiu
> Might I suggest there's some prolog apps out there that'll solve this?
> Prolog and natural language can be a petty good fit. There's a list of
> Prolog natural language tools at http://www.ai.uga.edu/mc/pronto/
>
> Unfortunately, as with smalltalk, finding a good Prolog developer can
> be challenging. (OK, I have to ask: hands up, anyone else here who's
> programmed in Prolog and Smalltalk outside of coursework. Anyone?
> Bueller?)
Can I raise my hand up halfway, if part of my job years ago, was to
install and support an experimental Prolog implementation? I only had to
learn enough Prolog to know that it was working.
Alex Sirota
I'm learning Ruby now so maybe this will be a side project to explore how parsers work along side.
Seems like a steep learning curve, but I agree everything does until you actually do it.
Alex
--
You received this message because you are subscribed to the Google Groups "TorCamp" group.
To post to this group, send email to tor...@googlegroups.com.
To unsubscribe from this group, send email to torcamp+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/torcamp?hl=en.
Aaron Wieland
> Unfortunately, as with smalltalk, finding a good Prolog developer can
> be challenging. (OK, I have to ask: hands up, anyone else here who's
> programmed in Prolog and Smalltalk outside of coursework. Anyone?
> Bueller?)
I moved to Toronto in 2001 for a Smalltalk position, where I remained
for a few years. Previously, I had used Smalltalk semi-covertly for
several months at a federal job.
-- Aaron
Bob Hutchison
All right -- you make some great points. Last time I looked at Smalltalk was on a Mac in the 80s. Lots has evolved since then. I like the Ruby quote -- I was hoping this system was somehow available in Ruby but maybe it's early days still.
I'm learning Ruby now so maybe this will be a side project to explore how parsers work along side.
Seems like a steep learning curve, but I agree everything does until you actually do it.