Location of sent_tokenize

32 views
Skip to first unread message

Julius Hamilton

unread,
Feb 6, 2022, 7:23:28 AMFeb 6
to nltk-...@googlegroups.com
Hey,

In which file in the source code is sent_tokenize defined?

Thanks very much,
Julius

Julius Hamilton

unread,
Feb 6, 2022, 7:23:34 AMFeb 6
to nltk-...@googlegroups.com
I found it in __init__.py.

I see it loads the “Punkt” sentence tokenizer from a pickle file.

Why is the tokenizer a downloaded pickle file and not just part of the NLTK module internally?

Thanks very much,
Julius
Reply all
Reply to author
Forward
0 new messages