Custom vocabulary?

[Ben Ryan]

Excited to see this new version of Stanford's NLP facilities for python! 

My team is trying to parse a sizable corpus of sentences into dependency trees, but we're having a problem. Some of the "words" in these texts are technical/non-English tokens. Specifically, we've pulled out in-line LaTeX equations and symbols and replaced them with dummy tokens like "LatexEquation123" in order to parse these sentences. We've done this instead of just using a generic "equation" token because we need to slot them back in at a later point in the pipeline. Our problem is that the tokenization/dependency parsing is (inconsistently) splitting these dummy tokens into multiple tokens (e.g. "Latex Equation 123" or even "Late xEquation 123") and then parsing the dependencies. 

We'd like to either turn off that token-splitting (leaving mwt out of the pipeline instantiation doesn't seem to do it) or add those dummy tokens as a custom vocabulary/dictionary so that they are alone and dependencies are parsed as if they're each one word. Is there a way to do that with this system?

Thanks,

Ben

[Peng Qi]

Hi Ben,

If you have your own tokenizer, one option is to construct the pipeline with "tokenize_pretokenized" to True, in which case the tokenizer will treat each line in the input as a sentence, and the sentence simply as a space-separated string of words.

If you do want to use the StanfordNLP tokenizer, there's a workaround: you could probably first run the tokenizer only, postprocess the tokenizer output to concatenate these dummy words back together, and finally use toe pretokenized option.

Hope this helps!

[Ben Ryan]

Thanks! This does seem to work (using the StanfordNLP tokenizer first and then the tokenize_pretokenized=True flag second), and for the posterity our post-processing is below. Note that while this involves looping over the tokens in each sentence 3 times, it adds negligible time to the overall process compared to the nlp() calls themselves (with two pipelines it's now 35% more total time). I'm sure there are ways to batch the data for processing better, but I think this gets us where we need to be for now. (we also found that replacing the tokens with unique random long alphabetical strings prevented them from being split up as well, allowing us to use just one nlp pipeline, though I can't guarantee that works all the time)

Post-processing on an example 'doc':

nlp = stanfordnlp.Pipeline(processors='tokenize,lemma,pos')

deps = stanfordnlp.Pipeline(tokenize_pretokenized=True)

nlpifiedDoc = nlp(doc)

# Post-processing on tokenization:

#

# 1) collapse any LateXEquation123 tokens that have been split

#

for sent in nlpifiedDoc.sentences:

    for i, w in enumerate(sent.words): 

         if 'LateXEquation' in w.text and not w.text[-1].isdigit(): 

             w.text += sent.words[i+1].text 

             w.lemma += sent.words[i+1].lemma 

    #

    # 2) Remove the numeric remnants

    # 

    

    for i, w in enumerate(sent.words):

         if w.text.isnumeric() and sent.words[i-1].text == 'LateXEquation'+w.text: 

             del sent.words[i]

    

    # 

    # 3) Re-index the tokens in the sentence

    # 

    for i, w in enumerate(sent.words): 

         w.index = i+1

# Then parse the dependencies

depped_doc = deps('\n'.join([' '.join([w.text for w in sent.words]) for sent in nlpifiedDoc.sentences]))