Writing to EDN

[je...@thinktopic.com]

Hi,

  A couple quick questions.  

First, we are caching feature vectors on disk using EDN at the moment because it makes it dirt simple to write maps to a file.  To do that we are using coerce like this:   

    (mat/coerce [] (mat/array [1 2])) ; => [1.0 2.0]

Is that how you are supposed to convert core.matrix arrays to clojure data?

Second, this doesn't seem like the right response given the doc string: 

    (mat/vec? (mat/array? [1 2])) ; => false    (using Vectorz)

A bug or a misunderstanding on my part?

Last, I was reading a few threads on serialization, but I didn't see any mention of Fressian (https://github.com/clojure/data.fressian).  How about using its custom tag and handler support to read/write core.matrix arrays?  This would get us a compact binary format that also allows for arbitrary clojure data to be mixed with numerical data, rather than just writing out matrices.

Cheers,

Jeff

[Mike Anderson]

On Thursday, 15 January 2015 07:14:30 UTC+8, je...@thinktopic.com wrote:

Hi,

  A couple quick questions.  

First, we are caching feature vectors on disk using EDN at the moment because it makes it dirt simple to write maps to a file.  To do that we are using coerce like this:   

    (mat/coerce [] (mat/array [1 2])) ; => [1.0 2.0]

Is that how you are supposed to convert core.matrix arrays to clojure data?

I tend to use "to-nested-vectors" rather than "coerce" - they effectively do the same thing.

 

Second, this doesn't seem like the right response given the doc string: 

    (mat/vec? (mat/array? [1 2])) ; => false    (using Vectorz)

A bug or a misunderstanding on my part?

The boolean result of the "array?" predicate isn't a vector:

(vec? (array? [1 2]))

=> false

The result of the "array" function however is:

(vec? (array [1 2]))

=> true

;-)

 

Last, I was reading a few threads on serialization, but I didn't see any mention of Fressian (https://github.com/clojure/data.fressian).  How about using its custom tag and handler support to read/write core.matrix arrays?  This would get us a compact binary format that also allows for arbitrary clojure data to be mixed with numerical data, rather than just writing out matrices.

This sounds like a good idea! The only thing I am unsure about is how easy/efficient it is to implement a tag handler that understands arbitrary core.matrix arrays? It would need to test something like the "array?" predicate, you can't get away with using class based dispatch because core.matrix is (deliberately!) open to extension with arbitrary classes. 

[Jeff Rose]

On Wed, Jan 14, 2015 at 5:49 PM, Mike Anderson <mike.r.an...@gmail.com> wrote:

On Thursday, 15 January 2015 07:14:30 UTC+8, je...@thinktopic.com wrote:

Hi,

  A couple quick questions.  

First, we are caching feature vectors on disk using EDN at the moment because it makes it dirt simple to write maps to a file.  To do that we are using coerce like this:   

    (mat/coerce [] (mat/array [1 2])) ; => [1.0 2.0]

Is that how you are supposed to convert core.matrix arrays to clojure data?

I tend to use "to-nested-vectors" rather than "coerce" - they effectively do the same thing.

Ahhh, nice. Thanks.

 

A bug or a misunderstanding on my part?

The boolean result of the "array?" predicate isn't a vector:

(vec? (array? [1 2]))

=> false

 

Doh!

 

Last, I was reading a few threads on serialization, but I didn't see any mention of Fressian (https://github.com/clojure/data.fressian).  How about using its custom tag and handler support to read/write core.matrix arrays?  This would get us a compact binary format that also allows for arbitrary clojure data to be mixed with numerical data, rather than just writing out matrices.

This sounds like a good idea! The only thing I am unsure about is how easy/efficient it is to implement a tag handler that understands arbitrary core.matrix arrays? It would need to test something like the "array?" predicate, you can't get away with using class based dispatch because core.matrix is (deliberately!) open to extension with arbitrary classes. 

Here's a first shot at it:

https://github.com/rosejn/clojure.core.matrix.fressian 

This only works with vectorz as is, but to add additional implementations we'd only need to know their base classes.  At first I thought we could use current-implementation-object, but that's not quite the same thing.

Already saving my disk space. :-)

-Jeff