How to save variables to disk

[Aaron Ponti]

Hi,

I got Julia code from a colleague to run on one of our machines. The result of the run is just one variable in the workspace. How can I save that variable to disk? Is there an equivalent of the save() command from MATLAB?

Thanks and sorry for the silly question,
Aaron

[nathan hodas]

Hi Aaron,

I've struggled with the same thing.  You can try the HDF5 package to write structured data, or you can try using mmap to just dump an array to the disk.

Nathan

[Aaron Ponti]

Hi,

would this work?

f = open("file", "w")
write(f, my_var)
close(f)

Thanks,
a2

[John Myles White]

Not unless you define a write method for your type.

What sort of data structure do you have? And what programs are you hoping will be able to read it?

 -- John

[Aaron Ponti]

I guess what I suggested will just dump my data into a vector of some type default type. I need to by able to write a 3-D array of Float64 to disk preserving its dimensions. I was looking into the HDF5 module now (https://github.com/timholy/HDF5.jl) and I cried a bit since I need to run it on Windows... but I guess I will give it a try. It really amazes me, though, that there is no native method in Julia to read and write variables from and to disk.

[nathan hodas]

I agree that the lack of a quick "pickle" ability can be frustrating. Maybe we can use PyCall to write data to the disk using cPickle?

[Aaron Ponti]

Now I am giving the HDF5 module a try...

[nathan hodas]

If you can't get HDF5 to work, you can try to serialize the data.

out = open("blah","w")

serialize(out,mydata);

close(out)

in = open("blah","r")

mydata2 = deserialize(in)

However, the help on serialize gives the following disclaimers:

The read-back value will be as identical as possible to the original. In general, this process will not work if the reading and writing are done by different versions of Julia, or an instance of Julia with a different system image.

[Aaron Ponti]

It seems to work. I will be writing some HUGE file soon...

[Cassio Martini M. P.]

I also think that a simple built-in function for saving and reading arrays (with compression) would be nice, like R's save/load combo.

HDF5 is great, but it's a hassle to just save some simple variables.

best,

Cássio

[Tim Holy]

On Tuesday, August 20, 2013 04:50:34 PM Aaron Ponti wrote:
> I guess what I suggested will just dump my data into a vector of some type
> default type. I need to by able to write a 3-D array of Float64 to disk
> preserving its dimensions. I was looking into the HDF5 module now (
> https://github.com/timholy/HDF5.jl) and I cried a bit since I need to run
> it on Windows...

Out of curiosity, why did you cry? The installation instructions? I didn't
think they were that complicated?

> but I guess I will give it a try. It really amazes me,
> though, that there is no native method in Julia to read and write variables
> from and to disk.

MATLAB uses HDF5 also, it's just that they bundle it with Matlab. I suppose we
could do that too, and save people the trouble of installing it.

In my experience, JLD really is just like matlab's save/load commands; I've
been using it a lot lately, with nary a hiccup. One advantage over Matlab is
that it is much more performant for things like large arrays. (My work on
Julia's HDF5 inspired this submission:
http://www.mathworks.com/matlabcentral/fileexchange/39721)

--Tim

[Tim Holy]

On Tuesday, August 20, 2013 09:48:50 AM Cassio Martini M. P. wrote:
> HDF5 is great, but it's a hassle to just save some simple variables.

What would it take to make it less of a hassle? E.g.,

@save "myfile.jld" var1 var2 var3

or something? Those kinds of things are really only a few lines of code, it
would be a trivial addition on top of the hard part (which is doing the type-
preservation correctly).

--Tim

[Cassio Martini M. P.]

Exactly. With R I can just do save(a, b, c, file = "data.RData") and have a, b and c be saved and compressed with gzip, bzip2 or xz. I can choose an ascii or binary representation, etc.

Having the save macro you mentioned would be already a step forward.

I think that the decision to push a lot of features to packages is good in a way, but I feel julia would benefit from having some basic things, such as this, built-in.

I really feel HDF5 should come standard. If julia is meant for technical computing, being able to save and load data is a bare minimum.

I also think that plotting should be built-in, at least basic capabilities. Just issuing "using Winston" on a file takes me 

Cassios-iMac:~ cassio$ time julia test.jl

real 0m10.279s

user 0m8.825s

sys 0m0.341s

which, to me, is absurd.

Expecting users to issue 

p = FramedPlot()

add(p, Curve(x, y)) 

Winston.display(p)

 to get a basic x-y plot is another thing that baffles me. With R i can just plot(x,y) and get a basic feel for the data. If I need more details I can go through the ggplot2 route, but it's not a necessity for the most basic plotting.

But this is starting to hijack the OP thread...

[Tim Holy]

On Tuesday, August 20, 2013 10:26:34 AM Cassio Martini M. P. wrote:
> Having the save macro you mentioned would be already a step forward.

Done.

@save "/tmp/myfile.jld" x y
@load "/tmp/myfile.jld" x y

You need to say Pkg.checkout("HDF5", "master") to get it (I'll make a new
release when I've got a little more time).

--Tim

[Cássio M. M. Pereira]

Great work Tim.

Any chance @load could load all the variables of the file into the
global namespace without having to specify x and y?

If you save a file to work on a few days later, sometimes it's hard to
remember all the variables names...

Thanks,
Cássio

--
Cássio M. M. Pereira

[ggggg]

This was one of my favorite features in MATLAB.  Also in MATLAB simply 'save(filename)' would save every variable in the local namespace. So 'save(filename)' 'load(filename)' was enough to recover a state. Is there a function like MATLAB's 'who' that simply lists all the variables in the local namespace?

[Cassio Martini M. P.]

yes, whos()

[Tim Holy]

On Tuesday, August 20, 2013 03:46:43 PM Cássio M. M. Pereira wrote:
> Any chance @load could load all the variables of the file into the
> global namespace without having to specify x and y?

I knew you were going to ask for that :-)

Done. There are even docs in the main README.

If you switched over to master, you can do a Pkg.release("HDF5") and go back
to tracking the officially-released version (I just updated it).

--Tim

[Cássio M. M. Pereira]

Awesome!

Thanks!
Cássio

[Aaron Ponti]

Out of curiosity, why did you cry? The installation instructions? I didn't
think they were that complicated?

Sorry, no, they were not complicated at all. I just quickly glanced at them and saw the Windows part was a bit longer than for the other platforms and just generalized. When I actually _tried_ installing the module, it turned out to be straightforward. Thanks!

 

[David van Leeuwen]

Hello, I am interested in the same things.  I find that for my today-compiled julia / HDF5 @save and @load are missing

On Tuesday, August 20, 2013 8:32:06 PM UTC+2, Tim Holy wrote:

On Tuesday, August 20, 2013 10:26:34 AM Cassio Martini M. P. wrote:
> Having the save macro you mentioned would be already a step forward.

Done.

@save "/tmp/myfile.jld" x y
@load "/tmp/myfile.jld" x y

You need to say Pkg.checkout("HDF5", "master") to get it (I'll make a new

which for me gives:

Pkg.checkout("HDF5", "master")

INFO: Checking out HDF5 master...

 ...

 in requirements at pkg/query.jl:10

ERROR: cannot resize array with shared data

 in splice! at array.jl:818

...

any idea what this can be?

---david

[Tim Holy]

The part below seems like a Pkg bug. Can you please file an issue?

FYI you don't need to check out the "master" branch to get this functionality,
it's in the latest release (and maybe even an earlier one, didn't check).

--Tim

[program...@gmail.com]

nevvar=rand(1)
writecsv("newfile.txt",nawvar)