[ANN] Sentences: A multilingual command line sentence tokenizer
90 views
Skip to first unread message
Eric B
unread,
Jan 6, 2016, 11:27:15 AM1/6/16
Reply to author
Sign in to reply to author
Forward
Sign in to forward
Delete
You do not have permission to delete messages in this group
Copy link
Report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to golang-nuts
Sentences (https://github.com/neurosnap/sentences) is a multilingual command line sentence tokenizer. This golang package converts a blob of text into a list of sentences. The ultimate goal is to become one of the fastest and accurate sentence tokenizers as well as support for multiple languages with enough composition to be easily adaptable to any developers specific needs.
Any feedback is greatly appreciated.
This project started out as a straight port from NLTK's punkt sentence tokenizer but since the original migration I have made substantial updates to how the tokenizer functions and its performance. I'd be happy to discuss with anyone interested in the struggles with porting a python package to golang.