Re: help regarding running genism code

[Radim Rehurek]

Hello Arvind (cc mailing list),

> I know that your code does a fast SVD (as shown in the LSI example), and
> but the problem is that my input is a full matrix not a sparse. I was
> wondering if your code could do it for the dense matrix as well.

You can pass in dense vectors as well, gensim doesn't care. There is no hardcoded threshold on what's "sparse enough" or "too dense".

Only beware of your memory footprint -- with sparse vectors, you can typically pass in tens of thousands of documents at a time (10k documents and 100k features at 0.01% density is still only 10M non-zeroes). With 100k features and vectors at 100% density, to get the same 10M non-zeroes, you can only pass in 100 document vectors!

So set the `chunksize` parameter accordingly.

Best,
Radim

[Radim Rehurek]

> Thanks for your quick reply.
>
> I understand what you are saying but what I meant to ask is something
> different.
>
> Actually, my input file is in a different format, and there is no
> word-id mapping. I have a file where there are d columns and n rows,
> each row has d entries seperated by tab. I can do the preprocessing, and
> convert it in the format of LSI example (sparse), but I would rather not
> do it if can be avoided.
>
> so what I would like to do is:
> lsi = gensim.models.lsimodel.LsiModel(corpus=mm, id2word=id2word,
> num_topics=400)
>
> where there is no id2word mapping; and if nm is something that can be
> created from the input file that I described above?

You don't need the id2word mapping. It's there mostly for debugging models -- you can just pass `id2word=gensim.utils.FakeDict(d)` in there, if `d` is the total number of features.

The models only accept vectors in the [(feature_id, feature_weight)] sparse format. So even if your data format is dense, you'd still need to pass `zip(range(d), row)` as a document.

HTH,
Radim


>
>
> Arvind

> --
> Arvind Agarwal
> PhD Student, Computer Science
> University of Maryland, College Park
> http://www.umiacs.umd.edu/~arvinda/
>
>
>
>
>

[Radim Rehurek]

> Hi Radim,
> Thanks for your help.
>
> I have resolved the input format problem, and been able to run your
> code.. yayyyy!
>
> I have two mode followup questions.
>
> 1. I need to do the PCA, do you think I can do it using your code? For
> PCA, I need two things. First I need to subtract the mean; but I guess
> this would require another pass over the data, Is there any way to avoid
> that?

PCA works over a covariance matrix, which indeed means centering the data matrix (or whitening). You can avoid this centering step, as many people do, but then it won't be PCA :)

Of course, centering produces a dense matrix, even where the original was sparse, so it's computationally a very heavy step.

> 2. I would need the right singular vectors; or more specifically, I need
> the S(:,:k)*V(:k,:) if USV is your SVD decomposition of the input
> matrix. Is there any way of doing it without doing an additional pass
> over the data?

Yes, pass in the input matrix transposed. Then U becomes V and vice versa (SVD is symmetrical that way). If you need *both* U and V, you'll need an additional pass though, yes.

It's best to compute U over whichever matrix dimension is smaller (usually the number of features, `m`), and compute V (~number of observations, `n`, the bigger number) in the extra pass, for memory reasons. That is, if `m`>>`n`, just compute SVD of X^T instead of X. The memory footprint is constant in `n`, but linear in `m`.

HTH,
Radim