How to use n-gram by document matrix instead of word by document matrix ?

[ChamingaD]

I went through a research paper. They mention that accuracy of LSA can be increase by using n-gram by document matrix instead of word by document matrix.

How can i do that with Gensim ? Is it possible ?

[Fil]

Would you maybe rewrite your input texts into a set of (say) tri-grams.

e.g. the previous sentence you would rewrite as

"wou oul uld you mayb ayb ybe rew ewr wri rit ite …"

then feed to gensim.

--
Fil

[ChamingaD]

Hmm. I didn't get what you said :/

[Fil]

Just compute the trigrams for each of your texts and make that the
text you analyze with gensim? or maybe I didn't understand your
question.

--
Fil

[IKS Public]

I just did it the hard way by adding w(i-1) and w(i) as the string w(i-1)+w(i) - i use "*" as begin end. Then i weed out infrequent cases with the gensim command

[IKS Public]

Oops i put an otherwise illegal char in between like

W(i)+'-'+w(i+1)

This avoids single words that look like bigrams

On Apr 22, 2012, at 5:17 AM, ChamingaD <chami...@gmail.com> wrote: