Stopwords
15 views
Skip to first unread message
Andrew Nystrom
Jun 27, 2014, 11:14:18 AM6/27/14
to berkeleyl...@googlegroups.com
Do you suggest removing stop words, or leaving them in the text when using the language models?
Andrew Nystrom
Jun 27, 2014, 11:14:37 AM6/27/14
to berkeleyl...@googlegroups.com
And by using, I mean both training and inferring :)
Adam Pauls
Jun 30, 2014, 12:15:22 PM6/30/14
to berkeleyl...@googlegroups.com
Definitely not. Language models are designed to use the whole text.
Andrew Nystrom
Jun 30, 2014, 3:47:54 PM6/30/14
to berkeleyl...@googlegroups.com
OK great. Does the same apply to all punctuation?
Andrew Nystrom
Jun 30, 2014, 3:52:39 PM6/30/14
to berkeleyl...@googlegroups.com
Including quotes, semicolons, dashes, etc (not just sentence markers).
Reply all
Reply to author
Forward
0 new messages