How to write a binary data file? (Matlab to Julia translation)

[Dennis Eckmeier]

Hi,

I'm translating matlab code (a wrapper for Laurens van der Maaten's Barnes-Hut t-sne code) in order to learn Julia.

Within the code, data are saved as .dat file to then be read by a windows executable.

The Matlab code (which works on my system) writes the data this way:

fwrite(h, n, 'integer*4');

fwrite(h, d, 'integer*4');

        fwrite(h, theta, 'double');

        fwrite(h, perplexity, 'double');

fwrite(h, no_dims, 'integer*4');

        fwrite(h, X', 'double');

So, it writes every variable and determines it's datatype. Because I didn't find a way to specify the type, I simply translated this to:
        write(h, n, d, theta, perplexity, no_dims, X')

I successfully start the program with
 run(`bh_tsne`)

However, the file doesn't seem to be read correctly by the executable. I know that the 1000x32 array X is read as 1000x0. The error message simply says 'Process exited(3221225477)'

How can I replicate the data file as it is saved by matlab? And related: how would I read such file in Julia (the program saves the results in another dat file)?

thanks! :)

[Dennis Eckmeier]

* sorry, I forgot to mention: in the Matlab code the file is opened and closed with 'big-endian ordering'.

[Dennis Eckmeier]

Update: Currently trying hton() to write in big endian, but apparently it doesn't have a method for arrays?

[Steven G. Johnson]

On Sunday, September 11, 2016 at 3:10:22 PM UTC-4, Dennis Eckmeier wrote:

Update: Currently trying hton() to write in big endian, but apparently it doesn't have a method for arrays?

Just call hton in a loop. 

In Julia 0.5, you could call hton.(a) on an array, but this kind of usage is pretty inefficient: doing write(io, hton.(a)) first calls hton.(a) to make a new array that is bigendian, then writes the new array to the stream.   If you instead just do

    for x in a; write(io, hton(a)); end

then no temporary copy is made.

[Dennis Eckmeier]

So, I tried this, but the data are still not stored as the Matlab code (first post) would do it...

function write_data(X, no_dims, theta, perplexity)

  n, d = size(X)

  h = open("data.dat","w+")

  A = write(h, hton(n), hton(d), hton(theta), hton(perplexity), hton(no_dims))

  # X=X'

  for i in eachindex(X)

      write(h,hton(X[i]))

  end

    close(h)

end

The output is "Read the 0 x -402456576 data matrix successfully!" - which, obviously doesn't make sense...

D

[Steven G. Johnson]

On Tuesday, September 20, 2016 at 3:46:47 PM UTC-4, Dennis Eckmeier wrote:

So, I tried this, but the data are still not stored as the Matlab code (first post) would do it...

function write_data(X, no_dims, theta, perplexity)

  n, d = size(X)

  h = open("data.dat","w+")

w+ means that you will append to data.dat if it already exists.  I think you just want "w" to create a new file and overwrite any data.dat that might exist.

[Steven G. Johnson]

On Tuesday, September 20, 2016 at 3:46:47 PM UTC-4, Dennis Eckmeier wrote:

So, I tried this, but the data are still not stored as the Matlab code (first post) would do it...

function write_data(X, no_dims, theta, perplexity)

  n, d = size(X)

  A = write(h, hton(n), hton(d), hton(theta), hton(perplexity), hton(no_dims))

Note that integers are 64 bit by default on Julia, so these are writing 64-bit values.  From your Matlab code, it seems like you need 32-bit (4-byte) values for your integers.  Just convert them to Int32 or UInt32 first, e.g. hton(UInt32(n)) 

[Dennis Eckmeier]

Hi,

thanks for the help! Coming from Matlab data types and binary files had never been a concern... it was about time I learned how to do this!

I was able to complete the Julia wrapper for Barnes Huts t-sne, and I will put it on github :)

Next up: plotting the results ;)

cheers,

Dennis

[Steven G. Johnson]

On Wednesday, September 21, 2016 at 5:23:24 AM UTC-4, Dennis Eckmeier wrote:

thanks for the help! Coming from Matlab data types and binary files had never been a concern...

You had to explicitly pass 'integer*4' in Matlab, so they were a concern there too, it is just that you express that concern differently in Matlab (by passing a format parameter rather than by converting the argument).