How to get the cooccurrence matrix from cooccurrence.bin?

[applebas...@gmail.com]

Hi,

I would like to see the cooccurrence matrix in text.

How can I read the cooccurrence.bin from the binary structure?

Seems that it's not provided with both .txt and .bin like vector does but only .bin.

Thanks for reading!

Best,

James

[cannb...@gmail.com]

Hey,

I also need the co-occurrence matrix of words in txt format. Did you find any solution to that? If so, can you share, please.

Thank you in advance.

Best,

Berkay

4 Ocak 2017 Çarşamba 07:59:55 UTC+3 tarihinde applebas...@gmail.com yazdı:

[Marc]

Hi all,

I actually implemented a python wrapper to train, test and analyze GloVe models.

There: https://github.com/marcevrard/GloVe/blob/master/scripts/run_glove.py#L260

you can find the `read_cooccurrences()` method that allows just this.

Hope this helps!

Marc

[cannb...@gmail.com]

Hi Marc,

Actually, I don't understand how to get 'just' co-occurrence matrix from your code. I only need the global co-occurrence matrix, not the whole Glove training. After that I will just use this matrix for Pointwise Mutual Information calculation. Thus, can you share a main method with me just for calculating the matrix and writing it to a .txt file by using your script?

Best regards.

13 Mayıs 2017 Cumartesi 03:35:50 UTC+3 tarihinde Marc yazdı:

[aklouch...@gmail.com]

Hi,

Did you get an answer or find a way to do that ?

Regards 

[Guglielmo Reggio]

Hi,

Has there been any updates on this? 

Thank you,

Guglielmo

[Guglielmo Reggio]

Hi,

I've found a solution:

import struct

file = open("/path/to/cooccurrence.bin", "rb")

byte = file.read(16)

while byte:

cooccur_data = struct.unpack('iid',byte)

print(cooccur_data)

byte = file.read(16)

file.close()

Hope this is useful!

Best,

Guglielmo

[Sal Aguiñaga]

Just to clarify: this output "(1, 2, 409566.0412529093)"

means that words 1 & 2 occur this many times on average if a given corpus --hence the fraction?

[Murat Aydogdu]

I am not sure about this solution. I would expect the last number should be a (large) integer, the co-occurence count for the pair.
I tried to use a code converter (https://www.codeconvert.ai/java-to-python-converter). According to that, the file structure may be:

    def __init__(self, word1, word2, val, id):
        self.word1 = word1
        self.word2 = word2
        self.val = val
        self.id = id

So maybe there is a fourth number, an identifier for each pair or words.

[Sal Aguiñaga]

Looking the .c source (I too expect that 3rd element to be an int) but they apply a distance weighting that I'm still trying to figure out. 
It's not a specific count; do correct me if I'm wrong. 

The cooccur value is stored into a ''real" bigram_table and into a variable r of type "real"
... 
    real *bigram_table = NULL, r;
...
            if ((r = bigram_table[lookup[x-1] - 2 + y]) != 0) {
....
                fwrite(&x, sizeof(int), 1, fid);
                fwrite(&y, sizeof(int), 1, fid);
                fwrite(&r, sizeof(real), 1, fid);

[Murat Aydogdu]

OK, then why are the precisions have so many digits? I wonder if this has anything to do with how Python represents floats.