passing large spec

[Marc Jäger]

Hi, 

what is the most memory saving way to pass a numpy array between fireworks? I am running into the 16mb limit of MongoDB when trying to update a spec with a large matrix.

Is there any way to circumvent that problem? Is it better to pass the information through files in that case?

Cheers

Marc

[Alex Dunn]

Hi Marc,

If your matrices are sparse, you might look into the matrices in scipy.sparse, which maybe are a more memory friendly way to hold big matrices.

Otherwise, I’d probably recommend just using files to hold them, and having the spec entry be the path to the file. Your Firetask should then load the file from this path.

A third option is storing your matrices in GridFS with mongo. Your Firetask could then look up the GridFS entry and get the matrices from that.

Thanks,

Alex

--
You received this message because you are subscribed to the Google Groups "fireworkflows" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fireworkflow...@googlegroups.com.
To post to this group, send email to firewo...@googlegroups.com.
Visit this group at https://groups.google.com/group/fireworkflows.
For more options, visit https://groups.google.com/d/optout.