NLP with lisp: Naturally manipulating the view on a tree and automatically reflecting the changes to the inner structure.

[pushpendre rastogi]

Assume you have an English sentence that has been encoded to represent the structure and grammar of a sentence.

Peter likes surfb0ards.
Subject Verb Object
(Verb likes
(Subject Peter)
(Object surfb0ards))

Many times during NLP we need to clean the input data, e.g in the above example we need to correct the spelling. The problem is that the spell corrector works on a different representation of this tree (An inorder traversal view) and it works in a way that naturally affects that view of the tree (look at word surfb0ard and its surrounding words , then change the word). What I want to do is to perform operations on a particular view of the tree automatically and correctly change the inner structure of the tree, so that you don't have to keep track of what changed how. This is a huge problem when you have an annotated data set and you want to do some cleaning or pre-processing of it. Have you ever built/seen/used something like this before? Do you think that there's a simpler way to tackle this problem?

p.s. for nlp folks - The immediate use case I have is for learner-error tagged english corpora. It's impossible to keep track of which section and words were affected by a particular type of error once you do pre-processing, since the number of words may increase or decrease, even sentences may change after you correct the punctuation, spelling etc. in a learner error corpus.

[kent...@gmail.com]

On Thursday, June 6, 2013 8:10:01 AM UTC-4, pushpendre rastogi wrote:
> Assume you have an English sentence that has been encoded to represent the structure and grammar of a sentence.
>
>
>
> Peter likes surfb0ards.
>
> Subject Verb Object
>
> (Verb likes
>
> (Subject Peter)
>
> (Object surfb0ards))
>
>
>
> Many times during NLP we need to clean the input data, e.g in the above example we need to correct the spelling. The problem is that the spell corrector works on a different representation of this tree (An inorder traversal view) and it works in a way that naturally affects that view of the tree (look at word surfb0ard and its surrounding words , then change the word). What I want to do is to perform operations on a particular view of the tree automatically and correctly change the inner structure of the tree, so that you don't have to keep track of what changed how.

Cells (isn't it always Cells?):

http://smuglispweeny.blogspot.com/2008/02/cells-manifesto.html
https://github.com/kennytilton/cells

-hk

[tar...@google.com]

| Assume you have an English sentence that has been encoded to represent the
| structure and grammar of a sentence.
|
| Peter likes surfb0ards.
| Subject Verb Object
|
| (Verb likes
| (Subject Peter)
| (Object surfb0ards))
|
| Many times during NLP we need to clean the input data, e.g in the above
| example we need to correct the spelling. The problem is that the spell
| corrector works on a different representation of this tree (An inorder
| traversal view) and it works in a way that naturally affects that view of the
| tree (look at word surfb0ard and its surrounding words , then change the
| word). What I want to do is to perform operations on a particular view of the
| tree automatically and correctly change the inner structure of the tree, so
| that you don't have to keep track of what changed how.

Up until this point I would have thought that using shared structure would get you what you wanted. As long at it was something that has an extremely local change like changing the spelling without changing any of the additional structure or meaning of the words.

So in the simple example of "surfb0ard" => "surfboard" you could get away with introducing a token and changing the string that represents the external name of that token.

But reading more of what you write, I fear that this is not really possible.

| This is a huge problem
| when you have an annotated data set and you want to do some cleaning or
| pre-processing of it. Have you ever built/seen/used something like this
| before? Do you think that there's a simpler way to tackle this problem?
|
| p.s. for nlp folks - The immediate use case I have is for learner-error tagged
| english corpora. It's impossible to keep track of which section and words were
| affected by a particular type of error once you do pre-processing, since the
| number of words may increase or decrease, even sentences may change after you
| correct the punctuation, spelling etc. in a learner error corpus.

So with this elaboration, it seems that you would also want to be able to handle something like

I have sand.
(Verb have
(Subject I)
(Object have))

transformed into

I have sinned.
(Verb sinned
(Object I)
(Aux have))

In that case I don't think you have any real recourse except to recompute the structure whenever you make these clean ups.

Perhaps if you wanted to get really fancy, you could try a technique like one of the old Truth Maintenance Systems (TMS) to trace the dependencies and then update just the affected parts. But I think keeping a notion of the original input and recomputing would be a much simpler approach.