problem with edn

[Jim - FooBar();]

Hi everyone,

I am trying to serialise a record with edn. I think I am using all the good practices but I get a weird error...I am using these simple functions:

(defn data->string
"Writes the object b on a file f on disk as a string."
[b f]
(io!
(with-open [w (clojure.java.io/writer f)]
  (binding [*print-dup* true
            *out* w]  (prn b)))))

(defn string->data
"Read the file f back on memory safely. Contents of f should be a clojure data-structure."
[f]
(io!
 (edn/read-string (slurp f)))) 

and I am getting this error:

RuntimeException No dispatch macro for: =  clojure.lang.Util.runtimeException (Util.java:219)

Out of curiosity I tried something simpler like this:

Clondie24.games.chess=> (defrecord FOO [a b c])
Clondie24.games.chess.FOO
Clondie24.games.chess=> (ut/data->string (FOO. :a :b :C) "FOO")
nil
Clondie24.games.chess=> (ut/string->data "FOO")

RuntimeException No reader function for tag Clondie24.games.chess.FOO  clojure.lang.EdnReader$TaggedReader.readTagged (EdnReader.java:739)


What I am I missing? any suggestions?

Jim
                         
 

[kawas]

Hi,

Maybe you should provide a custom reader for your record.

See answer on this question

http://stackoverflow.com/questions/17991565/clojure-defrecord-serialization-classnotfoundexception

regards,

[kawas]

By the way *print-dup* is the problem, maybe you should not use it :)

Spot the difference :

  user=> (binding [*print-dup* true] (prn (->Foo 1 2 3)))

  #user.Foo[1, 2, 3]

  user=> (binding [*print-dup* false] (prn (->Foo 1 2 3)))

  #user.Foo{:a 1, :b 2, :c 3}

cheers

Le mercredi 21 août 2013 19:55:59 UTC+2, Jim foo.bar a écrit :

[kawas]

Never had time to play with edn and custom readers but they are funny :)

Use *print-dup* if you need to... just know how to custom read it :

  user=> (edn/read-string {:readers {'user.Foo map->Foo}} "#user.Foo{:a 1 :b 2 :c 3}")

  #user.Foo{:a 1, :b 2, :c 3}

  user=> (edn/read-string {:readers {'user.Foo #(apply ->Foo %)}} "#user.Foo[1, 2, 3]")

  #user.Foo{:a 1, :b 2, :c 3}

Nice

[kawas]

In fact never user *print-dup* when using edn to read back data

*print-dup* will output type and meta information that will not play well with edn

  user=> (binding [*print-dup* true] (pr-str (sorted-map :z 5 :a 1)))

  "#=(clojure.lang.PersistentTreeMap/create {:a 1, :z 5})"

This "#=" with raise a RuntimeException because there is no dispatch on "=" in edn

Not using *print-dup* may lose some important information on data

  user=> (binding [*print-dup* false] (pr-str (sorted-map :z 5 :a 1)))

  "{:a 1, :z 5}"  ;; this is a plain hash map

Just take care of providing your own custom tags to be able to read data correctly

  ;; clumsy example

  user=> (binding [*print-dup* false] (str "#treemap " (pr-str (sorted-map :z 5 :a 1))))

  "#treemap {:a 1, :z 5}"

You can then read it back with your custom reader

  user=> (edn/read-string {:readers {'treemap #(merge (sorted-map) %)}}

                 (binding [*print-dup* false] (str "#treemap " (pr-str (sorted-map :z 5 :a 1)))))

  {:a 1, :z 5}   ;; this is a sorted map

Seems like it should be possible to easily provide default printers and readers for custom tags & data...

Cheers

[Jim]

Oh wow! I shouldn't have turned my computer off yesterday evening!!! there were many suggestions to try out... :)

Ok let's see...I've got a record with meta-data let's call it FOO and its instance MFOO:

user=> (defrecord FOO [a b c])
user.FOO
user=>  (def MFOO (with-meta (FOO. 1 2 3) {:x false :y true}))
#'user/MFOO

Now I want to serialise it but keep the meta-data when reading it back in. pint-dup seems  like my only option as it preserves metadata. As you say though, there is no dispatch for '=' so it seems like I'm stuck with standard java serialisation. Even i I convert it to a map with metadata the same thing will happen (#=)...

Is there no way to keep the metadata?

many many thanks :)

Jim

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Meikel Brandmeyer (kotarak)]

Hi,

Am Donnerstag, 22. August 2013 11:32:37 UTC+2 schrieb Jim foo.bar:

Oh wow! I shouldn't have turned my computer off yesterday evening!!! there were many suggestions to try out... :)

Ok let's see...I've got a record with meta-data let's call it FOO and its instance MFOO:

user=> (defrecord FOO [a b c])
user.FOO
user=>  (def MFOO (with-meta (FOO. 1 2 3) {:x false :y true}))
#'user/MFOO

Now I want to serialise it but keep the meta-data when reading it back in. pint-dup seems  like my only option as it preserves metadata. As you say though, there is no dispatch for '=' so it seems like I'm stuck with standard java serialisation. Even i I convert it to a map with metadata the same thing will happen (#=)...

Is there no way to keep the metadata?

many many thanks :)

(defmethod print-method Foo
  [foo ^Writer w]
  (.write w "#my/foo ")
  (print-method {:a (:a foo) :b (:b foo) :c (:c foo) :meta (meta foo)} w))

(defn foo-reader
  [foo-data]
  (with-meta (map->Foo (dissoc foo-data :meta)) (:meta foo-data)))

Read with:

(edn/read {'my/foo foo-reader} ...)

Printing might be optimised a bit. And the :meta key could be made more robust. (records may contain arbitrary keys.)

Kind regards
Meikel

[Jim]

Hi Meikel,

this is funny! I thought about this approach but I originally considered it to be a clever hack rather than the official way to do this...Since I can't test it yet with my record , I hope you don't mind me asking another question...

there is no need for print-dup bound to  true here, right?

thanks a lot for your time :)

Jim

On 22/08/13 12:00, Meikel Brandmeyer (kotarak) wrote:

[John D. Hume]

On Aug 22, 2013 6:25 AM, "Jim" <jimpi...@gmail.com> wrote:
> this is funny! I thought about this approach but I originally considered it to be a clever hack rather than the official way to do this...

If you need some data persisted or sent over the wire, then it should probably be considered part of a value, and maybe metadata isn't an appropriate place to store it. It could go into the record, as in Meikel's workaround, or in some wrapper structure, if you need to maintain the behavior that records with distinct metadata can compare =. (Or you could embed the data in the record but `#(dissoc % :data-fka-meta)` only when you want to compare while disregarding that data.)

[kawas]

Jim,

This is indeed a hack and not a best practice, maybe you're not using the right tool for your problem...

- If you want to exchange data (think values), you should not be in need of keeping types and meta data
  when you exchange data in json, for example, you're not providing object class in the stream

- If you just want serialization to reload hot data or use some kind of RPC mechanism over the wire, than I think edn is not the right tool.


Cheers

[Jim - FooBar();]

all I'm trying to do is to provide a uniform mechanism to save/load game-states across my 2D board-games. Everything works just fine without that trick for most games but chess is peculiar (or my implementation is peculiar if you like) because in order to implement castling/enpassant moves without global nastiness I had to attach some metadata to every piece that moved.The notion of 'having-moved' though is not necessarily part of a piece itself and that is why I thought meta-data would be appropriate and wouldn't affect equality between pieces that have moved and ones that haven't. The same principal I think applies to being alive or dead...logically speaking, a bishop is always a bishop, it's only when the game starts that suddenly 'alive' and 'dead' matter...

anyway, the way I've currently set it up it works for all games...my chess does exactly what Meikel showed earlier but the other 2 games work without that...btw, java serialisation always worked but I can't recall whether it preserves metadata...

thanks everyone - I can't believe how much I learnt through this post and some poking around on the web :)

clearly, YOU ROCK!   

Jim

ps: now that I've explained the use-case can anyone recommend any alternatives? 

[Softaddicts]

> Jim,
> > This is indeed a hack and not a best practice, maybe you're not using the > right tool for your problem...
> > - If you want to exchange data (think values), you should not be in need of > keeping types and meta data

Metadata is part of the Clojure environment and part of the value domain it handles.
Why should it not be transmitted along with the value ?
If the receiver is not written in Clojure it may be questionable an probably not
very useful to transmit it but otherwise ?

> when you exchange data in json, for example, you're not providing object > class in the stream

What's an array then in Json ? Somehow it has to end up in a concrete type.
If both ends are written in Clojure, why not convey the type ? What harm can it do ?

Records are essentially immutable maps. I would carry the type of record as
metadata and let the other end decide to instantiate it or not under the given
record type. This is a personal choice based on the context (same app at both
ends or maybe not, ...).

Again if the receiver is not written in Clojure obviously it does not make sense but if read the OP post correctly, it's not the case.

> > - If you just want serialization to reload hot data or use some kind of RPC > mechanism over the wire, than I think edn is not the right tool.
>

Uh ? "hot data" ? Here we exchange thousands of messages per hour using
Edn. We care little that the data is cold, medium or hot. It simply works..
Configuration data, live data,..
We added a few extensions to support some objects as is but very little.

RPC stuff is hard to work with and couples both ends a lot. Edn is much more
flexible by allowing you to decide when and how you want to instantiate
concrete implementations.

Any (easier) alternative you can come with ?

Luc P.

> > Cheers
> > Le jeudi 22 août 2013 13:24:33 UTC+2, Jim foo.bar a écrit :
> >
> > Hi Meikel,
> >
> > this is funny! I thought about this approach but I originally considered > > it to be a clever hack rather than the official way to do this...Since I > > can't test it yet with my record , I hope you don't mind me asking another > > question...
> >
> > there is no need for print-dup bound to true here, right?
> >
> > thanks a lot for your time :)
> >
> > Jim
> >
> >
> > On 22/08/13 12:00, Meikel Brandmeyer (kotarak) wrote:
> > > > (defmethod print-method Foo
> > [foo ^Writer w]
> > (.write w "#my/foo ")
> > (print-method {:a (:a foo) :b (:b foo) :c (:c foo) :meta (meta foo)} w))
> >
> > (defn foo-reader
> > [foo-data]
> > (with-meta (map->Foo (dissoc foo-data :meta)) (:meta foo-data)))
> >
> > Read with:
> >
> > (edn/read {'my/foo foo-reader} ...)
> >
> > Printing might be optimised a bit. And the :meta key could be made more > > robust. (records may contain arbitrary keys.)
> >
> >

> > > > -- > -- > You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- > You received this message because you are subscribed to the Google Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

> --
Softaddicts<lprefo...@softaddicts.ca> sent by ibisMail from my ipad!

[John D. Hume]

On Aug 22, 2013 2:19 PM, "Softaddicts" <lprefo...@softaddicts.ca> wrote:
>
>
>
>
> > Jim,
> > > This is indeed a hack and not a best practice, maybe you're not using the > right tool for your problem...
> > > - If you want to exchange data (think values), you should not be in need of > keeping types and meta data
>
> Metadata is part of the Clojure environment and part of the value domain it handles.
> Why should it not be transmitted along with the value ?
> If the receiver is not written in Clojure it may be questionable an probably not
> very useful to transmit it but otherwise ?

I don't think anyone suggested the type of a record should not be part of its edn representation. Here we're talking about arbitrary metadata. While it is "part of the Clojure environment," the way in which it's "part of the value domain it handles" is... subtle.

"An important thing to understand about metadata is that it is not considered to be part of the value of an object. As such, metadata does not impact equality (or hash codes). Two objects that differ only in metadata are equal."
http://clojure.org/metadata

[Softaddicts]

I disagree... strongly.

The value domain I am referring to is the one covered by your application,
customized metadata is of little value elsewhere obviously.

In typed languages the container defines a lot about what can be done with a
value and what is forbidden.
Nobody would try to rely on the raw value to guess these things in a typed language.
The container is used for this purpose. Raw values are useless without their
container even more when it is composed to create complex types.

Data types are defined against your app. domain to carry values.

When you serialize in Java and send data on the wire, the data type is included,
and you need to have the same version available on both ends. Using RPCs,
you end up with the same requirement whether in C, Java, .... because data
has to be serialized against it's container definition which have to be agreed to
by both ends. No mismatches are tolerated.

The fact that many other languages do not support something
as dynamic as metadata does not mean that metadata is not an integral part of
your domain values.

It merely says that these typed languages rely a lot on the data
containers to carry similar information that could be also carried by metadata
and have no or little need for dynamic metadata.

They however lack the flexibility that metadata offers. It can be generated
dynamically, it can evolve without impacting other processing steps.

Metadata has no impact on values, which allows values to be exchanged
even when the other party is unaware that additions have been made to the
metadata for other purposes.

Contrary to RPCs, native serialization, ... where a single slight change requires you
to upgrade everyone to the new definition.

Saying that metadata should not be transmitted over the wire is restrictive and
unjustified when you compare with the alternatives.

I suspect that too few people have been using meta data customized to their app
and that many see it has an internal feature restricted to the Clojure runtime and
other exotic uses.

Luc P.

[Cedric Greevey]

::meta would be better, to ensure against clashing with a key that a Foo might happen to use.

[Meikel Brandmeyer (kotarak)]

Hi,

Am Donnerstag, 22. August 2013 17:46:42 UTC+2 schrieb kawas:

This is indeed a hack and not a best practice

I disagree. As long as you own the type (or write an application which is self-contained) you are free to define how it is printed.

user=> (java.util.Date.)
#inst "2013-08-23T07:07:35.536-00:00"

Kind regards
Meikel

[kawas]

Hi Luc,

Le jeudi 22 août 2013 21:19:00 UTC+2, Luc a écrit :

Metadata is part of the Clojure environment and part of the value domain it handles.
Why should it not be transmitted along with the value ?
If the receiver is not written in Clojure it may be questionable an probably not
very useful to transmit it but otherwise ?

All you're saying is right, as long as you own your data and communicate between clojure programs, any "hack" is just particular design solution.

 

>   when you exchange data in json, for example, you're not providing object > class in the stream

What's an array then in Json ? Somehow it has to end up in a concrete type.
If both ends are written in Clojure, why not convey the type ? What harm can it do ?

Records are essentially immutable maps. I would carry the type of record as
metadata and let the other end decide to instantiate it or not under the given
record type. This is a personal choice based on the context (same app at both
ends or maybe not, ...).

Again if the receiver is not written in Clojure obviously it does not make sense but if read the OP post correctly, it's not the case.

I am not advocating for no types and meta in edn, in fact edn provides custom tags to do just that : read what you want from data and map it to your design.
As you said, you end up with concrete types : string values map to your language types (long, arraylist, hashmap, etc).

When I was saying, that we should not need to convey types and meta, it was more in a language agnostic situation (ex: client agnostic API)
 

> > - If you just want serialization to reload hot data or use some kind of RPC > mechanism over the wire, than I think edn is not the right tool.
>
Uh ? "hot data" ? Here we exchange thousands of messages per hour using
Edn. We care little that the data is cold, medium or hot. It simply works..
Configuration data, live data,..
 We added a few extensions to support some objects as is but very little.

I used the term "hot data" to mean data with all your meta and states, like Jim is doing for his chess game. He persists a game state to reload it fast without too much reprocessing.

Don't get me wrong, I'am not against any use of edn, in any situation. If it solves you're problems, than you're good to go.

Regards
Kawas
 

[kawas]

You're right Meikel,

I should have not used the term "hack". It is indeed just a solution to serialize data and meta data to edn.
As long as you need it, and know how to read it back, I see no misuse.

My point was rather on meta data that are not part of the solution domain. If I keep technical details in meta data (ex: locks) I see no need to serialize it to share it over the wire.

Kind regards
Kawas

[kawas]

Hello Jim,

As we know now, that you need data serialization to freeze your chess game and you're not writing any client API :)

Have you consider libraries like :
  - Carbonite
  - Deep Freeze
  - Nippy

I have no experience with them, just sharing

Regards
Kawas

[Jim]

Hi Kawas,

well it is in a sense a client API as I've developed a mini-library for writing board-games and I am using it myself for my chess, checkers, tictactoe etc. The serialization bit is part of the library and not of the games themselves and that's why it should be uniform across games. I haven't looked at any of these libraries but ideally, I'd like to avoid adding dependencies. I can do what I want with Java serialization and after this thread with edn as well...

I do regret not having checked whether Java serialization preserves the metadata or not .I plan to do it this afternoon. If it does ,I've got 2 perfectly working ways to achieve what I want, with no extra dependencies added...

many thanks again, :)

Jim

On 23/08/13 10:26, kawas wrote:

[Softaddicts]

This brings an interesting twist...

I am used to build polyglot solutions, our project has been a mix of Java, Ruby and
Clojure for a year and a half and prior to that I would mix languages in a project
on the fly.

However I am not in a mood these days to sacrifice features available in Clojure to
bend to other languages to comply with their poor feature set given the challenges
we are facing in our domain.

If you can own both ends, do it in Clojure exploiting all it's features and provide poor
man's API to other languages as required. This may require implementing some
custom front end logic within these APIs that have little to do with the internal
representation used elsewhere.

Being agnostic to do a "favor" to other languages maybe unavoidable
in some situations but it may impact your design decisions, you could cripple your
mental processes by avoiding using features not available in every language involved.

Being polyglot was a necessity to me before Clojure/ClojureScript got most of
the platforms covered.

It's not to me an advantage anymore to write polyglot solutions. Reducing to
common language features to me = dragging your feet on the rug while you could be running faster than Usain Bolt.

Luc

[Jim - FooBar();]

On 23/08/13 10:49, Jim wrote:
> I do regret not having checked whether Java serialization preserves
> the metadata or not .I plan to do it this afternoon. If it does ,I've
> got 2 perfectly working ways to achieve what I want, with no extra
> dependencies added...

I finally got the chance to check whether standard java serialisation
preserves meta-data and actually it does...so from the looks of it, it
seems like that would be more appropriate for my use-case, especially if
I had any specific reasons against the hackish approach demonstrated by
Meikel. In any case, this can only be good news...both approaches work :)

thanks again :)

Jim