Training Pipeline for StanfordNLP

[manas.s...@gmail.com]

Great work with the implementation and versatility of the library. Just wanted to understand the time when we can expect a training Pipeline for this library without the need to use the source code? This would help us in creating a production ready solution using StanfordNLP itself without much hassle.

Thanks.

[Yuhao Zhang]

Hi,

Can you be more specific on what you meant by "without the need to use the source code"? The current library already supports training of models using Python. Entry points for training different models can be found in `stanfordnlp.models`. For example, you can train your own tagger with:

import stanfordnlp
from stanfordnlp.models import tagger

# get default arguments
args = tagger.parse_args()
opt = vars(args)

# set treebank, file paths, parameters, etc.
opt['lang'] = 'en'
opt['shorthand'] = 'en_gsd'
opt['batch_size'] = 2000
opt['gold_file'] = 'your_training_data'
opt['eval_file'] = 'your_validation_data'
opt['save_dir'] = 'your_directory_for_model'

# start training
tagger.train(opt)

The training examples that we provided are in bash, but you can certainly do this in a more Pythonic way. You'll just need to make sure you input training data is in the CoNLL-U format.

Is there any specific need that you hope we should support?

[manas.s...@gmail.com]

Hi Yuhao,

This is extremely helpful. At you official website model training and evaluation it states that "Currently, we do not support model training via the Pipeline interface. Therefore, to train your own models, you will need to clone the source code from the git repository and follow the procedures below". That is where I misunderstood and stated "without the need to clone the source code from your github repo". I will definitely look at stanfordnlp.models. 

My specific problem is that - I need to train my own Named Entity Recognizer based on a new label ("adverse event" label in pharmacovigilance context)

If you have any generic code like above that can help me do that. It would be extremely helpful.

Thanks in advance.

[Yuhao Zhang]

Sorry to hear that the training guide on the website is somehow misleading - we will try to make it more clear that training can also be done in Python. For NER, unfortunately the stanfordnlp neural pipeline does not support NER yet. We will very likely support it in the next release, but we do not have an ETA at this point. Have you looked at the NER model in CoreNLP? If that satisfies your need, you can certainly train an NER model with CoreNLP and deploy it with the Python CoreNLP client interface.