High-volume JSON serialization and validation into structured format?

[Joel Meyer]

Apologies if this has been addressed elsewhere, I saw a few posts that looked like they might apply (using Ecto / Vex for validation of JSON) but I have a use case that might be a little different and would like some feedback.

I'm contemplating creating an OpenRTB Bidder in Elixir, which means high-volume JSON serialization with some validation on the parsing side. I'm currently looking at Poison as the serialization library of choice and I see that it allows me to create my own decode function to handle creating nested structs [1]. I could implement the parsing validation logic at that layer, but I'm wondering if there is a better way. Assuming that is the best way, it leads to my next question: is it possible to have a multi-line defstruct? OpenRTB has a multitude of fields and having to cram them all on a single line is very unappealing. Or perhaps the answer is that I should implement the validation but not bother with converting to a structured format (beyond the result that Poison returns).

Elixir newbie here, so please forgive me if the answers are obvious.

Thanks,

Joel

[1] http://stackoverflow.com/questions/30855638/elixir-nested-json-parsing-to-structs

[Jason M Barnes]

I don’t know about the JSON serialization part, but defstructs can be multi-lined:

defstruct a: 0,

               b: 1,

               c: 2

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/640aa99a-7e6e-403c-8aed-f59804959e0a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stian Håklev]

Depending what you are doing with the struct, it might be faster to keep it as a simple map. I think structs are particularly good for validating user input, providing defaults, and defining protocols etc. If you're just receiving some data from the server, processing it, maybe sending stuff out etc, it might be fine to keep it as a map. 

(I'd love to hear others' take on this - I've so far not created any structs in my own code, but I've of course interfaced with structs from some libraries I've used).

Stian

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAEx7qBMBXTwJfFT0Z%2B3mqAdCHcK_fwuxtrEe4zDAiAywgB995Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

http://reganmian.net/blog -- Random Stuff that Matters

[Joel Meyer]

Thanks for the quick reply, Jason. Turns out I made a noob mistake - I wasn't supplying default values for the struct fields and consequently the compiler was failing because it couldn't find a function named defstruct with arity 3 (the number of fields in my test struct). I took this to mean that my multi-line definition wasn't valid, but your example showed me my error.

[Joel Meyer]

Thanks for the thoughts, Stian. Leaving the data in map form without explicit validation may be the right answer - if necessary fields aren't there processing will fail at some other point in the code and that would be in line with the erlang ethos of "let it crash". Definitely interested in hearing from others as well.

[Dan Swain]

Hi Joel,

I've been working on something fairly similar to this.  High volume JSON processing.

Jiffy is still significantly faster than Poison.  Like several times faster.  I really wanted to use Poison but the difference is just too drastic to ignore and that was by far the slowest part of my processing.

I've had good luck converting JSON into structs.  It took a bit of thought.  You don't want to convert directly to symbolic keys because there's a huge risk of memory leak there (there's a good explanation of this in the Poison docs).  What I do is generate a list of the struct keys converted to strings at compile time.  Then, I have a method that takes the parser output (which has string keys) and maps those into a prototype struct (which has symbol keys).  Past that layer, it's all structs with all of beautiful type-checking that comes with them :)  It's plenty fast enough for my use (> 1k per second per process).

Best,
 - Dan

[Joel Meyer]

Dan,

Apologies for the delay - thanks very much for the thoughtful reply. I'm currently playing with using Jiffy for conversion from json to maps, and then Poison from maps to structs. I verified that decoding from string -> map is much faster with Jiffy, but haven't looked at the overhead of using Poison for the map -> struct modification. After a bit of a hiatus I'm going to resume work on this and will post here if I discover anything interesting. Thanks again for sharing your approach.

Best,

Joel