Deserialisation into Dart classes? JSON? ProtoBuffers? Is there really no solution?

[Danny Tuppeny]

The dev teams at my company are currently evaluating a bunch of new technologies for a rewrite of our core applications. Our applications are fairly large (> 1m LoC) and we need our new stack to last 5-10 years without needing significant rewriting (the current codebase is too tightly coupled for us to do a huge amount with).

Today I finally got around to starting a Dart prototype and I hit a big stumbling block! I couldn't find a way to deserialise JSON from our server into Dart classes! I spent some time investigating, and everyone seems to be using 3rd party libraries, and none of those I could find looked ideal for our use; and we're trying to steer clear of 3rd party components as much as possible (especially for such basic stuff) because we're bitten by poor testing, breaking changes and abandoned projects so much in our current codebase.

It feels like I must be missing something - for a language that's aiming to be an alternative to JavaScript in the browser and that has support for types; it seems like parsing JSON (or similar) from a server into types should be a rather simple task. I understand minification makes this complicated and it's likely to need reflection/mirrors (which apparently isn't great for production code) but this seems like a rather poor reason to not support something that seems to be so basic.

Have I overlooked something, or is there no good first-party way to do this? How is everyone else getting data from their server into their client code?

Note: I'm not married to JSON (though it'd be convenient); I just need a stable format I can deserialise into, that I could wedge into C# for WebAPI input/output (though it seems even Dart client + Dart server will have this issue?).

Any pointers? Is there any built-in (or Dart-Team-owned) code for this purpose? I was so sure there must be some, I started reading through the Dart repo code; but much of it seems to use maps. Not only do maps lose many of the benefits of Types, DartPolymer doesn't seem to want to let you do obj.X on a map even though Dart seems to be fine with it in Dart code :(

Some of the team are suggesting we should discount Dart on the basis of this; so I reeeeally want to find a nice solution! I don't mind writing code, but it does need to be pretty dynamic; since we have a ton of entities (deep object graphs) that will go back and forth in some parts of our apps, so manually maintaining JSON mapping code isn't an option.

(If not, I'm probably going to try doing Dart code-gen from our C# classes with a T4 template or similar, but it'll make Dart a harder sell than if I can do this nicely natively).

[Günter Zöchbauer]

I like protocol buffers for that use case a lot.

see also

https://github.com/dart-lang/dart-protoc-plugin

http://pub.dartlang.org/packages/protobuf

https://code.google.com/p/protobuf-net/

You define the data structure and protocol buffers generates classes and (de)serialization code for many common languages. This way you write you data structure once and have the classes generated for server and client automatically.

It is also very efficient because it doesn't include the field names in the payload.

PB also supports API changes and includes some basic validation rules like 'required'.

What do you mean with `obj.X on a map even though Dart seems to be fine ...`
You can't access an element in map using obj.X in Dart, you have to use obj['X'].

[Filipe Morgado]

All the serialization stuff is currently missing from the language, which is surprising considering how pervasive it is, specially on server-side.

Protobuf may be a way to go but it's far from convenient and doesn't seem to be update-friendly, which is problematic when a given service is accessed by multiple apps.

Also there's no way to efficiently deserialize const classes. We always have to allocate a temporary Map and use a custom constructor. Good bye performance.

The current trend is to use basic Json types for every services arguments/responses, the JS way, which I despise.

I already mentioned in this group that the serialization stuff is missing from Dart, that we need language support, but replies were everything but friendly.

They told me there were libraries to do everything, but they come with a 5x performance penalty, produce a lot of runtime garbage, don't support const classes, union types, default property values, etc

I wouldn't expect anything different now.

I'm not sure what the perfect solution would be. I just haven't seen anything close to being good. That led me to write custom codecs.

Language-level inspectors and builders (with custom validation) would be great, would support const classes, union types and other corner cases, and would allow custom codecs, as Json is very limited for a lot of cases.

My 2 cts, as I'm preparing to get bashed.

[Alex Tatumizer]

Filipe,

pigeon_map library didn't come with any performance/memory penalty. It's very fast. Even after compilation to javascript, binary serialization format was faster than any competition I'm aware of. 

I didn't update it to use transformers, so it's dated. Also, because no one was willing to discuss it, the syntax of annotations leaves something to be desired.

I thought - no one is interested, why bother?

If you or someone believe the library can be helpful, you are welcome to participate, take ownership etc - I'm fine with that.

I agree this was supposed to come supported by language natively (e.g. via "struct" construct, automatically serializable), but it didn't, and I don't think we will get it anytime soon.

Take a look, it's in pub and github. 

[Justin Fagnani]

Danny,

Serialization is a much more complicated topic, even in JavaScript, than just having "native" JSON support.

I think it's a big myth that you can deserialize JSON into useful JavaScript in the first place. What would you deserialize JSON to in JavaScript? By default, generic objects and arrays that have no special prototype chains. If you want to deserialize into interesting objects with methods and inheritance setup you'll need a custom deserializer. This is basically the same as in Dart. Decoding JSON will get you Maps and Lists, and if you want different classes you'll need a custom deserializer.

Once you're dealing with real object more complexity creeps in, and you have to answer questions like how do you know the right constructor to call, and how do you read and write private data (this comes up in JS too if you use the private/privileged members pattern). Then there's all the questions about the wire format - do you need to read and write to something specific? What about versioning?

There are _many_ different choices to be made. It's much better if serialization is not built in to the language, so that different packages can choose differently. There are a few serialization packages out there:

http://pub.dartlang.org/packages/serialization

http://pub.dartlang.org/packages/streamy

https://github.com/dart-lang/dart-protobuf

Or search "serialization" on pub.

There's no one best answer yet, and it's still evolving, so I feel your pain, but I don't think it's something inherently missing in the platform.

Cheers,

  Justin

--
For other discussions, see https://groups.google.com/a/dartlang.org/
 
For HOWTO questions, visit http://stackoverflow.com/tags/dart
 
To file a bug report or feature request, go to http://www.dartbug.com/new

To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

[Danny Tuppeny]

On 3 August 2014 16:35, 'Justin Fagnani' via Dart Misc <mi...@dartlang.org> wrote:

Serialization is a much more complicated topic, even in JavaScript, than just having "native" JSON support.

I think it's a big myth that you can deserialize JSON into useful JavaScript in the first place. What would you deserialize JSON to in JavaScript? By default, generic objects and arrays that have no special prototype chains. If you want to deserialize into interesting objects with methods and inheritance setup you'll need a custom deserializer. This is basically the same as in Dart. Decoding JSON will get you Maps and Lists, and if you want different classes you'll need a custom deserializer.

I don't buy this. JSON works fine for my C#-to-C# code with classes (not maps). Comparing to JavaScript is pointless; JavaScript is shit.

Besides, I said I'm not really bothered if it's JSON. We need an over-the-wire protocol that just works. Dart is intended to be an alternative to JS for writing web apps and it adds classes. Web apps almost all need to send data back and forth to a server; so how did Dart get so far without meeting a fundamental requirement that almost every web app has - receiving data from a server and putting it into a class? :-/

 

Once you're dealing with real object more complexity creeps in, and you have to answer questions like how do you know the right constructor to call, and how do you read and write private data (this comes up in JS too if you use the private/privileged members pattern).

This is complexity being invented as an excuse not to solve the problem. Every other language seems to be handling it ok. I'm fine if it only supports simple classes (structs) with default constructors and no logic. One of the big selling points of having classes in Dart is that I can get statement-completion and static analysis on my code. Maps totally destroy all of that. Communicating with a server is an incredibly common requirement; and if we lose half of the benefit of using Dart, then what's the point?

Thenthere's all the questions about the wire format - do you need to read and write to something specific? What about versioning?

It doesn't matter. Anything is better than nothing. If people have complex requirements, they can extend or write their own code. Something simple that lets me get "{ id: 1; name: 'Danny' }" into an equivalent class (with nested object) is all I want. Everybody else is managing this :-/

 

It's much better if serialization is not built in to the language

I don't think it's something inherently missing in the platform 

 

I have to totally disagree. 90% of peoples requirements are simple; we shouldn't need to pick between a bunch of third party libraries to do something that 99% of web applications need to do. It doesn't matter how restricted it is, or what the over-the-wire protocol looks like; but the requirement is fundamental; people need to get data in and out of Dart.

[Alex Tatumizer]

>  It's much better if serialization is not built in to the language, so that different packages can choose differently.

I agree with that. On condition that language/runtime provides enough information to make efficient implementation of libraries possible. But it doesn't. 

There's not so much needed for it: if you implement "struct" in the language, it can come with minimal information about types, which can still be sufficient.

I know that because I implemented pigeon_map, which is very straightforward, very fast, and covers absolute majority of practical scenarios.

Compare it with "serialization" package you cited. The whole web programming revolves around data exchange between server and client.  It should be trivial to do. Instead, out of the blue, it's a rocket science.

What is the purpose of the language, again? Make complicated things simple, or the opposite?

PS.Complexity aside, I can't find performance characteristics of recommended libraries. Did anyone benchmark them? 

[Danny Tuppeny]

On 3 August 2014 12:45, Günter Zöchbauer <gzo...@gmail.com> wrote:

I like protocol buffers for that use case a lot.

see also

https://github.com/dart-lang/dart-protoc-plugin

http://pub.dartlang.org/packages/protobuf

https://code.google.com/p/protobuf-net/

I've no problem with using protobuffers (anything I can produce easily in C# and consume easily in Dart), but it seems even they are not straight forward...

https://github.com/dart-lang/dart-protoc-plugin/issues/10

The Dart protoc plug-in is not supported on Windows

Currently the protoc plug-in is only tested on Linux

It's going to be hard to sell Dart to the rest of our dev teams when something so trivial is so complicated; it'll make us wonder how hard it's going to be to solve the hard problems we might hit when we're 500,000 lines into our products. At least with TypeScript if it doesn't work out we can throw it away and keep the already-created JavaScript.

I'm trying really hard to make Dart a real option for us (it's where almost all of my motivation for DartVS comes from); but how many more trivial requirements are we going to find are overly complicated? :(

[alberto.ags]

+1 for Danny

--

[Rich E]

On Sun, Aug 3, 2014 at 8:22 PM, Danny Tuppeny <da...@tuppeny.com> wrote:

It's going to be hard to sell Dart to the rest of our dev teams when something so trivial is so complicated;

Maybe I come from a different planet, but I never thought of deserialization of dynamic data into type safe objects to be a trvial problem.  Though I would agree that it is important that even a solution that handles POD objects would be better than nothing.

Is the plan to support deserialization with the mirrors API?  Forgive my ignorance, but I saw ways to create objects by string name using this, but no way to get a static type of it (since there is no typecasting)...

[Alex Tatumizer]

 > Though I would agree that it is important that even a solution that handles POD objects would be better than nothing.

My conjecture is that it's necessary and sufficient to have PODs efficiently serializable with no extra boilerplate (no matter through library of natively). These PODs are allowed to contain: primitive data types and Date, other PODs, List<POD> and Map<String,POD>, -recursively.

Do you have a counterexample showing that it's essential to have MORE than that?

[Filipe Morgado]

Binary data (don't mention base64, ugh), union types (list of photo album object, are they images, shapes, cliparts, videos, text or something else? Each type requiring its own validation), custom types (such as MongoDB ID and geolocation), Date with no time, Time with no date, Duration, References (for cyclic data?), typed arrays ... there are probably others.

I like to support NaN and infinities as well.

I like to omit properties with default values too. It greatly reduces my network activity.

Oh, and const classes are great but there's no way to efficiently deserialize them (2-3x slower because of temporary Map, despite of validation overhead).

Protobuf, pigeon, Json, etc work great, although with their own limitations. But as soon as we choose to go with one of them, good luck supporting another one in parallel in your webservice without ugly hacks. And then, oh crap, our database driver doesn't work with our objects, gotta write a custom too.

...

A generic, efficient way to inspect and build/validate our PODOs (including const classes) could be used seamlessly by any communication protocol or database driver.

Pluggable codecs should be a key concept in the language.

[Alex Tatumizer]

Filipe,

the way pigeon_map is handling the stuff can be easily extended to cover pluggable types. 

For typed data - I just forgot to mention it, it's supported. Pigeon tries to keep things simple, with just thin preprocessor generating metadata - all based (mostly) on plain dart declarations.. 

Your list of requirements is good. I didn't get that far because there was no interest from others, so there was no incentive to continue.

That's the right way to proceed: formulate requirements, discuss them, discuss how to implement (in plain English) then just sit down and write the code, no big deal.

(Due to pigeon, I at least know how to serialize to binary efficiently, can help).

If we have public discussion, there's a good chance that we come up with something useful, instead of waiting for someone else to provide no one knows what.

[mythz]

Have to agree. Dart's serialization support is even worse than in static languages. Is the recommended option for everyone to seriously hand-roll their own serialization strategies? Given Dart's Web Focus this seems like a major shortcoming to me and I'm surprised the big Apps that are using Dart internally aren't flagging this as a major friction point.

It doesn't have to be in the language itself but there should be first-class JSON support in the core libraries that works by default. 

It doesn't even need to serialize every class just PODO's and even if only a subset was supported that would still be OK, just provide a guideline on what's supported and some extensibility hooks falling outside the beaten path. Maybe the solution ends up being a 2 step process, i.e. the JSON is parsed into a generic collections but then we have an easy way to hydrate an instance of a class from a Dictionary<string,object>. 

So far the best option with the least effort I've found is to use a Mixin that depends on Mirrors: http://stackoverflow.com/a/20035072/85785

But that doesn't work well too well on the client due to flaky Mirror support, also a Mixin means you have to own the class you want to serialize which isn't ideal for serializing PODO's you don't own. 

- Demis

[Alex Tatumizer]

Mirrors are out of the question, they will never perform well enough for serialization, even if supported, and they are not.

Which brings us to code generation via preprocessing.

One obvious thing (based on pigeon experience) is that PODs (except those defined as "primitive") should implement Map. There's at least 2 ways to do that (both via Map mixin).

Pigeon explores one way, but there's another - probably better one (not quite clear). Anyway, it's simple: if class looks like

class Foo {

   int x;

   double y;

}. 

then we can generate maps:

var fooSetters= {

   x: (Foo foo, int value) => foo.x = value,

   y: (Foo foo, double value) => foo.y = value

}

and similarly - fooGetters, plus list of fooKeys=["x", "y"] and convert Foo into

class Foo extends Object with MapMixin {

   int x;

   double y;

   // plus a couple of methods that make it a map, based upon the above setters & getters

}

This is not difficult at all, given the will to do so.

[Danny Tuppeny]

On 4 Aug 2014 02:21, "Rich E" <reaki...@gmail.com> wrote:
>> It's going to be hard to sell Dart to the rest of our dev teams when something so trivial is so complicated;
>
> Maybe I come from a different planet, but I never thought of deserialization of dynamic data into type safe objects to be a trvial problem.

The C# to do this is just one line of code, and all built-in. Sure, there are restrictions, but it works fine for 95% of our needs.

> Is the plan to support deserialization with the mirrors API?  Forgive my ignorance, but I saw ways to create objects by string name using this, but no way to get a static type of it (since there is no typecasting)...

Mirrors are one way to do this, but I got the impression they're not a great thing to use for production js code (not sure if there are bugs, or it's just bloated; seems like MirrorsUsed should help, but I haven't investigated yet).

[Filipe Morgado]

Hey Alex,

I don't think it's a good idea to provide support in the PODOs themselves via mixins, as we may need to support native classes or classes from libraries we don't control (I use Point and Rectangle from dart:math quite a lot). And the memory footprint of the classes can remain unchanged.

What I did is, for every class I need to serialize/deserialize, I create a companion constant (or final) schema instance which will provide support solely for that class.

For example, this class that I don't control:

class Point {
  num x;
  num y;
}

I create (currently manually) a schema used solely for that class:

// Schema for a mutable class. Schemas are always immutable.
// They're final (not const but could be) so I can inline getters and setters
// (and it seems finals are a little more performant than consts)
final pointSchema = new ObjectSchema.mutable(
  // Provide the list of fields
  [
    // The 'x' field
    new ClassField(
      // The name of the field
      'x',
      // The schema of the value, could be any type or class, not nullable (by default)
      const IsNum(nullable: false),
      // A function used to get the value (getter)
      (Point p) => p.x,
      // A function used to set the value (setter)
      (Point p, num value) { p.y = value; },
      // Optionally, the default value of the field, so it may be omitted from the output
      0),
    // The 'y' field
    new ClassField(
      'y',
      const IsNum(),
      (Point p) => p.y,
      (Point p, num value) { p.y = value; },
      0)
  ],
  // The default constructor of the class.
  () => new Point()
  // A custom validation function that is called after all fields were initialized
  (Point p) { /* Does nothing. */ }
  // ... other optional (less relevant) fields ...
);

Yes, this is very verbose. Looking forward to auto-generate them but I haven't had enough time to dig into transformers.

No mirrors needed at all, not even Symbol. This allows for the fastest (de)serialization I could come up with (considering it supports PODOs and pluggable encoders/decoders).

I support 2 forms of union types as well (using the 'String type' of onObjectBegin, see below).

Const classes are supported but deserialization requires a temporary Map (2-3x slower than mutable classes).

If a custom schema is there but not used, it gets tree-shaken out, if needed.

Additionally, I use an interface similar to the old dart:json

abstract class CodecListener {
  // A basic value was found
  void onValue(value);
  // A reference was found
  void onReference(int reference);
  // The start of an object was found
  // Must be followed by 0+ properties and onObjectEnd
  void onObjectBegin([String type]);
  // An object property was found
  // Must be followed by onValue, onObjectBegin or onListBegin
  void onProperty(String name);
  // The end of the current object was found
  void onObjectEnd();
  // The start of a list was found
  // Must be followed by 0+ onValue, onObjectBegin or onListBegin, and onListEnd
  void onListStart();
  // The end of the current list was found
  void onListEnd();
}

A class inspector, using a provided schema, inspects my PODOs and feeds any CodecListener instance of my choice.

A class builder, using a provided schema, implements CodecListener, validates incoming data and builds my PODOs.

Using this interface, I can wire up things as I see fit:

PODOs inspect  ------->  Json encode

PODOs inspect  ------->  Bson encode

PODOs inspect  ------->  ... any format ...

Json decode  ------->  validation/build  ------->  PODOs

Bson decode  ------->  validation/build  ------->  PODOs

Json decode  ------->  validation  ------->  Bson encode (without instantiating runtime types)

Bson decode  ------->  Json encode   (without instantiating runtime types)

A codec (such as Json, Bson, etc, I wrote my own binary format, 50% faster than Json and 40% smaller data) only needs to implement 2 classes, a decoder and an encoder.

The decoder only cares about feeding a CodecListener (could be validator/builder that builds runtime instances, or any other encoder which implements CodecListener)

The encoder implements CodecListener and doesn't need to care about where the data comes from (from PODOs or from the decoder of another codec).

When (de)serializing, I must always provide the schema to use, so inspectors/builders/codecs know how to inspect/validate/build the PODOs.

My method still has some raw edges but it fits my needs, both in terms of performance and in terms of expressiveness, with (optional) full data validation out of the box. My method may seem a little complex at first, but I do think it's the way to go. I cannot consider another serialization library if it conflicts with my database drivers or if prevents me to support other encoding formats.

I may get involved in a community project regarding (de)serialization if it follows the same guidelines and if performance is a primary focus.

The packages I use the most are all in one single library, that includes of course serialization.

If interest is shown, I may try to isolate serialization into a single package or share the whole thing on github.

Haven't seen much interest last time I shared this architecture, though.

I would love to see some kind of language support for this kind of things (transformers may do the trick though).

[Alex Tatumizer]

Filipe: anything will do, as long as all internal scaffolding is autogenerated. And it can be done easily, with a bit of annotations + transformer. There's, of course, a million ways to do it.

The "special" classes you mention (Point, Rectangle) - for them, metadata can be written by hand (they are like primitive types, there's very few of them), the rest - autogenerated. If you find a way to do it, problem solved. As example, look at annotations in pigeon_map - you will see it's almost boilerplate-free.

For performance: pigeonson is significantly faster that your binary format (by a factor of 3 or 4)  - maybe you used not the most efficient methods from typed_data library? The secret is simple: don't rely on ByteData class, it's slow.

[Alex Tatumizer]

Forgot to ask: what do you need References for? I don't remember I ever needed them for any practical purpose (though java serialization supports them).

References may come into play if you want to persist some of your (complex) data structures, but this might be a wrong design, reminds ancient pre-relational database ideas (e.g. this one) which were a terrible mess.

[Nigel Magnay]

We've exactly the same issue.

My route is to use AMF3. I've been noodling with this (https://github.com/magnayn/dart_amf/). I will probably apply more effort to it in the coming months, but I was learning dart on the fly too.

FWIW: AMF3 comes from flex. It's a binary wire protocol. It's dead-simple to use, has robust server-side implementations that allow you to either do a 'default' serialization, or control it with your standard serialization language features (Externalizable, in the case of Java). And there are known solutions to the usual crap (it's an object graph. What if it's a hibernate proxy. Etc., etc). There's also libraries for things like iOS. So we have our same back-end talking to different front-ends.

It's not perfect. But it's easy. Protocol buffers looked like too much work for me. I just want to define domain objects in Java, and have them 'just work'.

--

[Filipe Morgado]

Hey :)

On Monday, 4 August 2014 14:50:25 UTC+1, Alex Tatumizer wrote:

Filipe: anything will do, as long as all internal scaffolding is autogenerated. And it can be done easily, with a bit of annotations + transformer. There's, of course, a million ways to do it.

Gotta find time :P 

On Monday, 4 August 2014 14:50:25 UTC+1, Alex Tatumizer wrote:

For performance: pigeonson is significantly faster that your binary format (by a factor of 3 or 4)  - maybe you used not the most efficient methods from typed_data library? The secret is simple: don't rely on ByteData class, it's slow.

I benchmarked PigeonMap against my codecs and dart:convert JSON. Performance-wise, it was on par with JSON.

Gotta run the benchmarks again, but I didn't see anything close to what you claim.

Maybe you're talking about your custom structs? I think my requirements are quite different.

I think I do rely on ByteData for float/double.

Are Float32List and Float64List the way to go?

Feels awkward if I only need to write 1 value.

On Monday, 4 August 2014 16:27:01 UTC+1, Alex Tatumizer wrote:

Forgot to ask: what do you need References for? I don't remember I ever needed them for any practical purpose (though java serialization supports them).

References may come into play if you want to persist some of your (complex) data structures, but this might be a wrong design, reminds ancient pre-relational database ideas (e.g. this one) which were a terrible mess.

It's not about persistence. It's about cyclic data or objects referenced multiple times in a single payload.

I'm supporting it although I'm not using it at all, since most data comes from/goes to MongoDB.

[Alex Tatumizer]

1. When I mentioned x3-4 performance, I meant pigeonson binary format, not JSON. JSON serialization for PigeonMap has the same performance as normal JSON, and it cannot be made much faster obviously. Disclaimer: I didn't benchmark it recently, but I monitor performance of dart core classes from time to time - nothing dramatic happened there for a long time (except that read/write from/to file is faster now)

2. For ByteData: use typed data instead directly! E.g. when you want to write int into buffer, use

buf[n]=myInt&0xFF;;

buf[n+1]=(myint>>8)&0xFF;

etc.

3. I think a better way to structure implementation would be: for each POD like Foo, generate another class that provides "map view" on Foo, e.g

class Foo {

   int x;

   double y;

}. 

class FooMapView extends Object mixin MapMixin {

  FooMapView(Foo foo);

  // add methods that makes it look like a map, but read/write to underlying foo instance

}

I think it provides a better "wording" (concept becomes much clearer) - and you can write by hand MapViews for classes you cannot autogenerate (like Point).

What do you think?

4. Even if  references are supported by some db, I, for one, would be very reluctant to use them (unless I see a clear use case). So if reference is returned, generating runtime error would be fine, no?

These references make things complicated without clear benefit.

[Filipe Morgado]

Alex,

I pushed my benchmarks to GitHub.

These are my old versions, before I merged everything into one single package, which I haven't made public yet.

Need to inquire some people to do so.

I did did quite a few changes since then, but performance should be rawly the same.

The file benchmarks/dynamic_all.dart runs benchmarks for dynamic basic types (no PODOs, no schema).

It imports the file benchmarks/_bench_dynamic.dart which contains the actual benchmarks.

Could you provide a benchmark implementation which uses Pigeonson, similar to the following?

So I can integrate it in the benchmark sets in dynamic_all.dart?

import 'dart:convert' as convert;

class ConvertEncodeBench extends BenchmarkBase {

  final             object;

  List<int>         result;

  ConvertEncodeBench(String name, this.object): super(name);

  void run() {

    result = convert.UTF8.encode(convert.JSON.encode(object));

  }

}

class ConvertDecodeBench extends BenchmarkBase {

  final List<int> bytes;

  dynamic         result;

  ConvertDecodeBench(String name, object)

      : bytes = convert.UTF8.encode(convert.JSON.encode(object)),

        super(name);

  void run() {

    result = convert.JSON.decode(convert.UTF8.decode(bytes));

  }

}

Thanks :)

[Alex Tatumizer]

Filiple, I can't find definitions of your classes used for test in pure dart. Where are they? Could you just copy them inline here? I mean, pure dart like

class Foo {

   int x;

   double y;

   String z;

}

[Filipe Morgado]

On Monday, 4 August 2014 17:14:56 UTC+1, Alex Tatumizer wrote:

1. When I mentioned x3-4 performance, I meant pigeonson binary format, not JSON. JSON serialization for PigeonMap has the same performance as normal JSON, and it cannot be made much faster obviously. Disclaimer: I didn't benchmark it recently, but I monitor performance of dart core classes from time to time - nothing dramatic happened there for a long time (except that read/write from/to file is faster now)

A little below my requirements.

2. For ByteData: use typed data instead directly! E.g. when you want to write int into buffer, use

buf[n]=myInt&0xFF;;

buf[n+1]=(myint>>8)&0xFF;

etc.

I'm only using ByteData for float/doubles.

Everything else is byte-packed. 

 

3. I think a better way to structure implementation would be: for each POD like Foo, generate another class that provides "map view" on Foo, e.g

class Foo {

   int x;

   double y;

}. 

class FooMapView extends Object mixin MapMixin {

  FooMapView(Foo foo);

  // add methods that makes it look like a map, but read/write to underlying foo instance

}

I think it provides a better "wording" (concept becomes much clearer) - and you can write by hand MapViews for classes you cannot autogenerate (like Point).

What do you think?

That's what I'm doing, except I'm not auto-generating "views" yet.

I call them "schema" because I use them to inspect but also to validate/build instances.

And I usually use "View" in a UI context :P

4. Even if  references are supported by some db, I, for one, would be very reluctant to use them (unless I see a clear use case). So if reference is returned, generating runtime error would be fine, no?

These references make things complicated without clear benefit.

I'm gonna keep the support.

I did use them once, in our flash photo album app, where each page had a list of elements that could be referenced from the global album configuration. The album Json was stored as-is so database was not a problem and was still able properly process them server-side.

There were arguments that is was generally a bad-design. I had mixed feelings.

[Filipe Morgado]

dynamic_all.dart runs only on dynamic Json data (Map, List and basic types). Data defined in _objects_dynamic.dart.

schema_decode.dart deserializes PODOs with schema validation. Data defined in _objects_schema.dart.

schema_encode.dart serializes PODOs using schemas. Data defined in _objects_schema.dart.

For example, in dynamic_all.dart, I run a set of benchmarks like this:

class EncodeBenchSet extends BenchmarkSet {
  EncodeBenchSet(String name, object): super('encode $name') {
    add(new PixmEncodeBench('pixm', object));
    add(new JsonEncodeBench('json', object));
    add(new Bson1EncodeBench('bson', object));
    add(new Bson2EncodeBench('bson2', object));
    add(new ConvertEncodeBench('convert', object));
  }
}

Which runs all the added benchmarks and orders the results.

The idea would be to add a pigeonson benchmark to the set above that runs on the same given objects, so the results are actually meaningful.

[Filipe Morgado]

A little side-note ...

An encoding that simply dumps C-like structs to a byte array will be, of course, a lot faster than dynamic full-featured encodings.

But such an encoding is not usable in my context.

[Danny Tuppeny]

Oh man...

If all the code and discussion in this thread is not proof that a) we need to be able to deserialise data of some known protocol into Dart types and b) there is clearly no good way to do it "from the outside"; then I don't know what is!

I'm really disappointed that Google / Dart Team think that it's an acceptable for a web language to have such a gaping hole and no plans to fill it. If something as basic as getting data from a server into a class that is only one line of code in C# turns out to be hundreds in Dart; something is really messed up.

I really hoped I'd overlooked something obvious :-/

[alberto.ags]

Again +1 for danny

--

[Filipe Morgado]

+1 too (although I'm somewhat happy with my lib) ... and -1 for the Dart team for supporting the toJson/fromJson aberration.

[Rich E]

On Mon, Aug 4, 2014 at 8:05 PM, Danny Tuppeny <da...@tuppeny.com> wrote:

Oh man...

If all the code and discussion in this thread is not proof that a) we need to be able to deserialise data of some known protocol into Dart types and b) there is clearly no good way to do it "from the outside"; then I don't know what is!

I have to agree. Even if writing a transformer is the best solution, then official support from the dart team that covers the '95% of cases' seems warranted.  Instead, there are at least two 'official' options (toJson/fromJson and serialization package) that are not adequate in production.

Is there no feature request open for this?  If there is I'll add my star.

[Alex Tatumizer]

Filipe, thanks, I will try to run benchmarks later this week.

Most likely, you do everything right, but you handle more general/complicated problem, which accounts for some necessary extra overhead.

[Filipe Morgado]

On Monday, 4 August 2014 20:53:25 UTC+1, Alex Tatumizer wrote:

Filipe, thanks, I will try to run benchmarks later this week.

Most likely, you do everything right, but you handle more general/complicated problem, which accounts for some necessary extra overhead.

Haha, that's very unlikely!!!

I almost didn't comment the code, didn't provide examples, didn't even put a license.

So it will be very hard to get any kind of code review from more experienced developers.

But the code could probably be improved.

(I try to avoid allocations, so I usually use one single global parser/builder/buffer/inspector/etc,

That kind of coding is really not popular. It works well IMO but I doubt it would pass reviews. )

Overall, it was a huge performance improvement over the Java/Hibernate implementation we previously had.

And, that I know of, we haven't yet hit any significant bug. So people were kinda pleased around here.

:)

[Alan Knight]

There is not a single way, and generally the Dart project's emphasis has been on keeping the things in the core Dart minimal and supporting a rich package ecosystem. 

I'm aware of three options, all already mentioned here, that are at least partly written by the Dart team, if you are looking for extremely simple there is a JSONCodec in dart:convert. If you are looking for something very general that can with minimal configuration (minimal configuration requires mirrors, so it will have overhead)  there is the serialization package. If you are looking for high performance but need some configuration then protocol buffers are probably your best bet, and there are certainly things that support using protocol buffers in C#. There are various other options people have written for their specific requirements, as seen elsewhere in this thread.

Even when people's requirements are simple, they are often different. For example, when people talk about JSON serialization they often mean interoperating with existing services, which is quite a different use case from wanting to serialize objects between your own client and server. And serializing occasional objects or small parts of rich object graphs is also quite a different problem from large volumes of data where performance is a major factor.

On Sun, Aug 3, 2014 at 3:58 AM, Danny Tuppeny <da...@tuppeny.com> wrote:

The dev teams at my company are currently evaluating a bunch of new technologies for a rewrite of our core applications. Our applications are fairly large (> 1m LoC) and we need our new stack to last 5-10 years without needing significant rewriting (the current codebase is too tightly coupled for us to do a huge amount with).

Today I finally got around to starting a Dart prototype and I hit a big stumbling block! I couldn't find a way to deserialise JSON from our server into Dart classes! I spent some time investigating, and everyone seems to be using 3rd party libraries, and none of those I could find looked ideal for our use; and we're trying to steer clear of 3rd party components as much as possible (especially for such basic stuff) because we're bitten by poor testing, breaking changes and abandoned projects so much in our current codebase.

It feels like I must be missing something - for a language that's aiming to be an alternative to JavaScript in the browser and that has support for types; it seems like parsing JSON (or similar) from a server into types should be a rather simple task. I understand minification makes this complicated and it's likely to need reflection/mirrors (which apparently isn't great for production code) but this seems like a rather poor reason to not support something that seems to be so basic.

Have I overlooked something, or is there no good first-party way to do this? How is everyone else getting data from their server into their client code?

Note: I'm not married to JSON (though it'd be convenient); I just need a stable format I can deserialise into, that I could wedge into C# for WebAPI input/output (though it seems even Dart client + Dart server will have this issue?).

Any pointers? Is there any built-in (or Dart-Team-owned) code for this purpose? I was so sure there must be some, I started reading through the Dart repo code; but much of it seems to use maps. Not only do maps lose many of the benefits of Types, DartPolymer doesn't seem to want to let you do obj.X on a map even though Dart seems to be fine with it in Dart code :(

Some of the team are suggesting we should discount Dart on the basis of this; so I reeeeally want to find a nice solution! I don't mind writing code, but it does need to be pretty dynamic; since we have a ton of entities (deep object graphs) that will go back and forth in some parts of our apps, so manually maintaining JSON mapping code isn't an option.

(If not, I'm probably going to try doing Dart code-gen from our C# classes with a T4 template or similar, but it'll make Dart a harder sell than if I can do this nicely natively).

[Ruud Poutsma]

My experience with my own little 20K+ pet project went as follows:

I Used JsonObject library. This used noSuchMethod and thus slow, also all properties are not type-safe. This lead me to remove it all completely when I saw the first version of the dart serialization package.

The serialization package seems to solve mange problems of serialization. The immaturity comes mainly from not supporting serialized CustomRules and having vague error messages about (de) serializing going haywire. Troubleshooting those was very timeconsuming and frustrating and I am currently right in the middle of it. Overall, I was impressed by the way dart serialization handles the core concepts of converting between a serialized flat structure and an object tree. I have not seen many libraries who try to solve these problems. 

I have looked to various other solutions. Dart protobuf has had a serious try, but the lack of Windows support was a dealbreaker for me as I temporarily had no access to Linux VMs. Also, my main work manchines run Windows and as such didn't help in the appeal. Other libraries where mirror-based or way too young/immature to be considered as an option.

This being said, I think the way forward is to

• Have simple serialization. Json-like structures map to struct-like objects. References are set manually. Possibly by transformers?
• Solve the problems in dart-serialization:
• debuggability. Somehow my code won't serialize, what is the problem? The devver should be aided in resolving these issues. For instance, I tried serializing Set<T>'s. After some weird, non-descriptive error messages I found out by tedious elimination process.
• CustomRule serilization with selfDescribing = true
• Better example CustomRule serialization code
• More examples as given on the package page
• More complex examples, e.g. a complex object/class tree

[Alan Knight]

Comments inline.

On Mon, Aug 4, 2014 at 2:02 PM, Ruud Poutsma <rti...@gmail.com> wrote:

My experience with my own little 20K+ pet project went as follows:

I Used JsonObject library. This used noSuchMethod and thus slow, also all properties are not type-safe. This lead me to remove it all completely when I saw the first version of the dart serialization package.

The serialization package seems to solve mange problems of serialization. The immaturity comes mainly from not supporting serialized CustomRules and having vague error messages about (de) serializing going haywire. Troubleshooting those was very timeconsuming and frustrating and I am currently right in the middle of it. Overall, I was impressed by the way dart serialization handles the core concepts of converting between a serialized flat structure and an object tree. I have not seen many libraries who try to solve these problems. 

I have looked to various other solutions. Dart protobuf has had a serious try, but the lack of Windows support was a dealbreaker for me as I temporarily had no access to Linux VMs. Also, my main work manchines run Windows and as such didn't help in the appeal. Other libraries where mirror-based or way too young/immature to be considered as an option.

This being said, I think the way forward is to

• Have simple serialization. Json-like structures map to struct-like objects. References are set manually. Possibly by transformers?
• Solve the problems in dart-serialization:
• debuggability. Somehow my code won't serialize, what is the problem? The devver should be aided in resolving these issues. For instance, I tried serializing Set<T>'s. After some weird, non-descriptive error messages I found out by tedious elimination process.

Yes, debugging and error messages could use considerable improvement. One approach would be a way of visualizing the partially serialized tree. Because it goes breadth first it can be non-trivial to see how you got somewhere, and not having to manually decode the output format would also be helpful.

• CustomRule serilization with selfDescribing = true

That would be great, but how would you do it? You could serialize the name of the class, possibly with the name of the containing library as well. But then you need to use mirrors to create it, and if you're writing CustomRule it's often because you don't want to use mirrors. Alternatively, the receiving end could provide a mapping from names to classes, or better, to closures for constructing them. But that's a lot of configuration overhead and could be unpleasant if you get it wrong. So probably the best thing would be to have a transformer that generates that for CustomRule classes. And you'd want to be careful about having CustomRule classes that you weren't using keeping code from being tree-shaken.

• Better example CustomRule serialization code
• More examples as given on the package page
• More complex examples, e.g. a complex object/class tree

Yes, those would be very good.

 

On Sunday, August 3, 2014 12:58:00 PM UTC+2, Danny Tuppeny wrote:

The dev teams at my company are currently evaluating a bunch of new technologies for a rewrite of our core applications. Our applications are fairly large (> 1m LoC) and we need our new stack to last 5-10 years without needing significant rewriting (the current codebase is too tightly coupled for us to do a huge amount with).

Today I finally got around to starting a Dart prototype and I hit a big stumbling block! I couldn't find a way to deserialise JSON from our server into Dart classes! I spent some time investigating, and everyone seems to be using 3rd party libraries, and none of those I could find looked ideal for our use; and we're trying to steer clear of 3rd party components as much as possible (especially for such basic stuff) because we're bitten by poor testing, breaking changes and abandoned projects so much in our current codebase.

It feels like I must be missing something - for a language that's aiming to be an alternative to JavaScript in the browser and that has support for types; it seems like parsing JSON (or similar) from a server into types should be a rather simple task. I understand minification makes this complicated and it's likely to need reflection/mirrors (which apparently isn't great for production code) but this seems like a rather poor reason to not support something that seems to be so basic.

Have I overlooked something, or is there no good first-party way to do this? How is everyone else getting data from their server into their client code?

Note: I'm not married to JSON (though it'd be convenient); I just need a stable format I can deserialise into, that I could wedge into C# for WebAPI input/output (though it seems even Dart client + Dart server will have this issue?).

Any pointers? Is there any built-in (or Dart-Team-owned) code for this purpose? I was so sure there must be some, I started reading through the Dart repo code; but much of it seems to use maps. Not only do maps lose many of the benefits of Types, DartPolymer doesn't seem to want to let you do obj.X on a map even though Dart seems to be fine with it in Dart code :(

Some of the team are suggesting we should discount Dart on the basis of this; so I reeeeally want to find a nice solution! I don't mind writing code, but it does need to be pretty dynamic; since we have a ton of entities (deep object graphs) that will go back and forth in some parts of our apps, so manually maintaining JSON mapping code isn't an option.

(If not, I'm probably going to try doing Dart code-gen from our C# classes with a T4 template or similar, but it'll make Dart a harder sell than if I can do this nicely natively).

--

[Alex Tatumizer]

@Alan,

To make serialization efficient, we need code generation. (Doing everything through mirrors is expensive).

I'm curious whether the following approach is feasible:

1) we don't generate code by preprocessing. Instead, when program starts, we call function generateSerializationCode(arrayOfClassNames).

2) this call generates dart source containing metadata and some necessary code (not much is needed). This is done through mirrors, but because it's on server side only, that's OK.

3) the file can be imported using deferred import - both on server and client.

4) the rest is straightforward. 

There are some details involved, but they are purely technical. In principle, can it work?

[Alan Knight]

If you're using dart2js, deferred import still requires that all the code was available at compile time. So no, at won't work as stated. Even on the VM it might be rather dodgy to have code running which then specifies a deferred library to be created, but it might work.

However, a system that generated code for serialization ahead of time would work fine. Or, as in the earlier message, if the serialization code existed, enough information to allow it to be found dynamically.

[Anders Holmgren]

I agree that it shouldn't be in the core library as it's an area that is useful to have alternatives in. Even on java you don't have json support built in. It's left to third party libs like gson and Jackson.

At the moment I've been living with hand written toJson / fromJson using the core dart:convert lib. It's more verbose than it might be but thanks to dart at least it is pretty terse.

I would favour a third party lib that does code generation based on conventions and works with dart:convert.

For example it could look at your constructors and generate a fromJson ctr from it following conventions. Where you want to override default behavior you could have annotations or even hand write your own fromJson ctr for full control.

Similarly for toJson.

This library could start small as you just need to handle the simple common stuff first. That would already get you 80% there.

I'm tempted to write one but I'm trying to resist creating any new projects :/

[Alex Tatumizer]

> However, a system that generated code for serialization ahead of time would work fine

That's what I'm doing now, but it's messy. Complexity of solution by far exceeds complexity of the problem.

In fact, code generation is needed just to make efficient calls to setters/getters and constructor.

E.g., if dart could provide, outside of mirror system, implementations of

Function mySetter=generateSetter(class, attribute) - and simllarly, getters and constructor,

this would be enough for serialization.

This is, of course, possible to do dynamically - javascript has no problem generating functions on the fly, so it's just a matter of adding support in the runtime.

[Danny Tuppeny]

On Monday, 4 August 2014 23:54:40 UTC+1, Anders Holmgren wrote:

I agree that it shouldn't be in the core library as it's an area that is useful to have alternatives in

Having something in core doesn't mean there can't be alternatives; C# has a built-in serialiser, but there are many third party ones (in fact, some ASP.NET from MS uses JSON.NET over the built-in one) .

However, I'm not really bothered whether the support is in core, or a package. The problem here is that the language/SDK doesn't provide the capabilities required to implement serialisation in a reasonable way. This entire thread is a discussion of "the best of the worst ways to implement serialisation". This isn't an obscure requirement that a few web apps have; it's a fundamental issue that almost every Dart developer in the world needs to solve. I've seen no good reason for why it's this complicated, just ideas for writing tons of code and excuses for why deserialisation is non-trivial.

I'm trying to convince developers across multiple teams here that Dart is what we should be investing in for our large product rewrites which will need to be good for the next 5-10 years. It's really hard to have faith in a platform if "keeping core small" or technical implementation details are favoured over providing functionality for basic real-world needs. How can we write stable code quickly if we're writing hundreds of lines of boilerplate deserialisation code or using third party libraries for something so trivial?

I'm really trying hard to like Dart; I've poured many hours into a Visual Studio extension trying to bring decent Dart support to VS so it's a more viable option for .NET shops like ours; but I find this issue (and lack of input from the Dart Team) really frustrating. How many more issues are there like this? Am I going to find Dart doesn't support HTML encoding next? :-/

[Anders Holmgren]

Look I am also surprised and disappointed that dart doesn't have better json support.

And you will certainly hit other features that are missing compared to more mature stacks. That's simply the state of play with dart today. It is still a relatively young platform. It takes a while to build it out.

Is dart the best platform for you today? Not sure. Will it be the best platform for you a year or two from now? I believe so but that's based on what I value in a platform and my crystal ball gazing.

[Filipe Morgado]

I don't know how things are right now, but when I tried the serialization package (after many battles to understand how it actually works), it was at least 5x slower than anything else (and produced a lot of garbage). The certainly makes it unusable for 90% of the cases. What happens to #perfmatters?

My opinion hasn't changed regarding toJson/fromJson. So auto-generating them is making the rope on our neck a little more tight, IMO.

We don't need more Json support as much as we don't need more XML support, ... What we need, IMO, is a common efficient infrastructure to be used by any encoding, be it Json, Bson, Myson, Yaml, MySQL, PostgreSQL, or any other (non-)existing encoding/protocol.

On the server, it's a lot worse because we have to deal with at least 2 different encodings (Json and DB driver, I gotta deal with at least 4) which often don't interoperate at all. Good bye PODOs, if you don't write your drivers yourself.

+1 for Alex

One of Dart's selling point is performance. It seems only people outside of the Dart team realize that includes serialization (of PODOs).

[Matthew Butler]

On Tuesday, August 5, 2014 4:52:46 AM UTC-3, Danny Tuppeny wrote:

On Monday, 4 August 2014 23:54:40 UTC+1, Anders Holmgren wrote:

I agree that it shouldn't be in the core library as it's an area that is useful to have alternatives in

Having something in core doesn't mean there can't be alternatives; C# has a built-in serialiser, but there are many third party ones (in fact, some ASP.NET from MS uses JSON.NET over the built-in one) .

However, I'm not really bothered whether the support is in core, or a package. The problem here is that the language/SDK doesn't provide the capabilities required to implement serialisation in a reasonable way. This entire thread is a discussion of "the best of the worst ways to implement serialisation". This isn't an obscure requirement that a few web apps have; it's a fundamental issue that almost every Dart developer in the world needs to solve. I've seen no good reason for why it's this complicated, just ideas for writing tons of code and excuses for why deserialisation is non-trivial.

Because it is a complicated topic which you want to brush off as easy because you can do it in C#. In the past for instance the Editor had a feature to allow you to automagically create to/fromJson functions but it had to be stripped out because if it wasn't used extremely carefully, and ran it on say something which extended PolymerElements then every single DOM property was also passed to the Json functions, sometimes recursively. You also need to keep in mind that any solution that is developed in Dart, needs to be converted to run in Javascript in a performant way. Anything the Dart language does, has to be achievable in JavaScript as well. As such numerous language features have been excluded because they could not reliably ensure that they could be maintained in JavaScript.

 

I'm trying to convince developers across multiple teams here that Dart is what we should be investing in for our large product rewrites which will need to be good for the next 5-10 years. It's really hard to have faith in a platform if "keeping core small" or technical implementation details are favoured over providing functionality for basic real-world needs. How can we write stable code quickly if we're writing hundreds of lines of boilerplate deserialisation code or using third party libraries for something so trivial?

Already in the Dart platform the developers have broken off numerous packages out of the core platform into 3rd party packages in their attempts to keep the core libraries smaller. An example is the crypto package. There are numerous 3rd party packages, created by the Dart team which are not part of the official core framework. This enables the developers to iterate these packages at a different pace than the internal core project. This fear/aversion of 3rd party packages will be more detrimental to your project than how many language features the core language and libraries have. 

I'm really trying hard to like Dart; I've poured many hours into a Visual Studio extension trying to bring decent Dart support to VS so it's a more viable option for .NET shops like ours; but I find this issue (and lack of input from the Dart Team) really frustrating. How many more issues are there like this? Am I going to find Dart doesn't support HTML encoding next? :-/

Matt 

[Yegor Jbanov]

Hi, Danny,

I maintain the Streamy package. I think it fits your use-case pretty well (although it is not maintained by the Dart team). We are seeing up-and-to-the-right adoption and are planning to be relevant for a long time.

As many have mentioned there is no one good solution when it comes to (de)serialization. What you risk ending up with is a set of (de)serializable classes and a separate set of classes that are actually suitable to use to implement your application logic. Streamy provides a flexible fully type annotated object model and API generated from a discovery document (we are also experimenting with protobuf support).

All generated objects come with JSON serializers and deserializers, and it is tree-shakeable. The API comes with translation to REST-over-HTTP. Streamy comes with a lot of runtime components that allow you to build a robust RPC system, including caching, request deduplication, performance tracing, transactional API and batching. It is designed via a command pattern that makes it very easy to add custom pieces into the pipeline.

Some unique features of Streamy compared to other solutions:

• Command pattern (allows swapping runtime, e.g. IndexedDB in place of HTTP)
• Stream-based API (very powerful in our experience)
• Observable objects (good for UI data binding)
• Flexible schema (entities have operator[] in addition to type-annotated getters/setters, and (are about to) implement Map)

Streamy does not (yet) support some of Discovery features (no nested resources, no file upload, enums are strings), but it proved to have enough for pretty large and complex applications.

Cheers,

Yegor

[Alan Knight]

Ahead of time would include doing it in a transformer, so it's not that different from what you were proposing, just that you'd have to specify which things you wanted to generate code for in a way that the transformer could read, not by running initial startup code.

The problem is that you can't really provide, outside of the mirror system, a way to dynamically invoke setters, getters, and constructors. Not because it's technically not doable, but that as soon as you do it, you've effectively provided a large chunk of the mirror system and re-introduced a big chunk of the problems you were trying to avoid by not using mirrors, both with difficulty of tree-shaking and with performance. Once you can dynamically construct instances of classes, particularly if you can look the classes up by name, you've made tree-shaking very difficult. Once you can construct a setter on the fly the compiler can't inline or remove things. You say that Javascript has no problem generating functions on the fly. That's true, it has no problems, except performance. Changing the running program is a bad thing for optimizations and something Dart works to avoid.

I think a better approach than generating setters is generating serializers/deserializers for a specific class, roughly the CustomRules in the serialization package. That has a much more limited scope (e.g. the compiler can tell that those setters are only ever invoked on the instance of Foo that you just constructed) so is easier to optimize.

[Danny Tuppeny]

On Tuesday, 5 August 2014 13:54:50 UTC+1, Matthew Butler wrote:

Because it is a complicated topic which you want to brush off as easy because you can do it in C#. In the past for instance the Editor had a feature to allow you to automagically create to/fromJson functions but it had to be stripped out because if it wasn't used extremely carefully, and ran it on say something which extended PolymerElements then every single DOM property was also passed to the Json functions, sometimes recursively.

Once again; this is because it's trying to solve a more complicated problem than I'm asking for. You shouldn't be trying to serialise random classes you don't own that have huge graphs. Simply adding restrictions for anything complicated would solve the simple problem. Eg.; only support serialising/deserialising things that have an annotation (must go all the way down to all base classes).

Given we're all pulling data from servers and that data has to be a string, just being able to deserialise "{ id: 1, name: 'Danny' }" into "class Person { int id; string name }" would be a HUGE improvement over nothing.

 

You also need to keep in mind that any solution that is developed in Dart, needs to be converted to run in Javascript in a performant way. Anything the Dart language does, has to be achievable in JavaScript as well. As such numerous language features have been excluded because they could not reliably ensure that they could be maintained in JavaScript.

I don't see anything being requested stopping this from being possible. The JavaScript is generated, so mapping code can be created from the Dart classes by dart2js.

 

This fear/aversion of 3rd party packages will be more detrimental to your project than how many language features the core language and libraries have. 

I don't have a fear or 3rd party packages; but I maintain an enormous legacy project and we're constantly bitten by third party code. e're not going to stop using it; we're just going to be far more careful when we introduce it. Something as fundamental as deserialisation that will be used for every single server call needs to be well tested and maintained; and open source programmers haven't exactly convinced they're great at this over the years.

[Danny Tuppeny]

On Tuesday, 5 August 2014 17:50:02 UTC+1, Yegor Jbanov wrote:

I maintain the Streamy package. I think it fits your use-case pretty well (although it is not maintained by the Dart team). We are seeing up-and-to-the-right adoption and are planning to be relevant for a long time.

I had a quick look at this; and it looks like this package involves up-front code-gen. If we're forced to go that route (which is seems is the only reasonable option) we'd probably do code-gen from C# using reflection over the server-side types we're sending back and forth.

 

What you risk ending up with is a set of (de)serializable classes and a separate set of classes that are actually suitable to use to implement your application logic.

This isn't just a risk; this is expected. We simply want the data from our server mapped out to some PODOs that represent the transmitted data. We don't want to mix our logic into these classes; we think OO is bollocks and FP and mutability is a better way ;D

To be honest; many of my colleagues have been following this thread; and it seems like it's going to be a tough job to sell them on Dart. It feels like the switch to Dart is far too big when there is so little guidance on (what we consider) fundamentals like this; and if our expected use of Dart is so different to everybody elses; I think we'd just hit frustration after frustration. It's a shame, because Dart really looks like it'd solve a lot of our gripes with web development today.

Thanks to everyone that understood the needs and tried to suggest solutions; and sorry if I started a riot. My intention was only ever to sell Dart to my colleagues for our rewrite; but I couldn't do that without answers to the questions I knew they'd ask.

[Justin Fagnani]

I'm curious - since you can't even do what you're asking in JavaScript without a separate serialization library - what is your other option? 

[Alex Tatumizer]

@Alan,

then, is the following feasible:

Provide standard transformer that generates standard serialization metadata?

Possible objection might be that different serializers need different metadata.

Maybe. But had we had JSON serializer and, say, bson serializer, this would cover a majority of use cases. And metadata is the practically the same. All we need is MapView generated for each class to be serialized.

Transformer can recognize a limited number of annotations, e.g.

@Serializable

class Person {

   String firstName;

   String lastName;

   int yearOfBirth;

   // etc.

}

will not need any extra rules, or annotations, or anything. This is just POD after all. Data types supported by such POD can be predefined  (e.g. primitive types, Maps<String.POD>, List<POD>, typed data, Date - nothing fancy). If adapters are needed for some other types of data, they can be easily configured.

Some protocol-specific annotations, in principle, can be just copied to generated metadata.

Do you think this is feasible? The advantage is that unless you make your classes complicated, everything will work just out of box, with no hassle. And if you want to make your classes complicated ... just don't. :-)

[Danny Tuppeny]

On 5 August 2014 18:20, 'Justin Fagnani' via Dart Misc <mi...@dartlang.org> wrote:

I'm curious - since you can't even do what you're asking in JavaScript without a separate serialization library - what is your other option? 

That's a tough one! JavaScript sucks, which is why I wanted to use Dart. I'd like to move more of our code into the client for the new version of our software (better user experience), but others want to move more to the server (more type checking, less JS). I hoped Dart would give us the best of both; but now we have a dilemma!

If Dart is a non-starter; React is looking most promising so far; and since some recent stuff from Facebook has included TypeScript bindings; we're hoping we could possibly go with React+TypeScript; but it's still early days for our prototypes.

[Alan Knight]

I'm not quite sure what you're suggesting, and particularly what you mean by the word "standard". If you mean, should something be included with the base system as suitable for all uses, then I think telling people they're not allowed to make complicated classes (e.g. ones without default constructors) is a problem. So is saying that your objects can't have references to anything that you can't modify.

If you don't want it to be part of the basic system and are willing to live with those limitations, then it's certainly feasible and not particularly difficult to write.

--

[Alex Tatumizer]

> I'm not quite sure what you're suggesting, and particularly what you mean by the word "standard".

We can discuss what we mean by "standard", but initially, "standard" can be defined informally as this:

Minimal amount of metadata to serialize/deserialize between PODs and JSON (which is pretty standard). You just use PODs (basically structs) instead of ad-hoc Maps there. JSON is the key word, we take it as starting point.

Actually, I already did it for pigeon (generated metadata such that I can convert between classes and JSONs), then it turned out (predictably) that same is enough for improvised binary serialization, and same is enough for bson etc.

And no, this solution will not try to cover infinite number of possible protocols. Let more fancy protocols (e.g. protobuf) care about their own transformers, BUT: if we have this minimalistic transformer, it would be easy for others to extend it.

Otherwise, writing preprocessor from scratch is really a big deal, it's like writing a mini-compiler.

Maybe I should better put it like this: what we mean by "standard metadata" is exactly what we should discuss, but my feeling is that we can find some acceptable definition. What do you think?

[Justin Fagnani]

But if you go with React and/or TypeScript you'll still have the same issue: JSON deserialization give you plain Objects, not anything with an interesting prototype chain. Given that Dart is certainly no worse here, and IMO better because Lists and Maps behave better in Dart, I can't see any gain. Unless TypeScript has something different than JS to say about JSON serialization, and /happens/ to make all the exact choices that you prefer for each decision that needs to be made. If that's the case, I'm certainly interesting in hearing what their solution is.

I think it'd be enlightening to see what you want to behavior this basic deserialization to be. I'd be very interested in what you want to happen in these cases:

• A field is not type annotated, possibly because it true type is not representable in Dart (ie, a union type)
• Private fields (they couldn't have been read by a generic serializer, should they be written?)
• The class has multiple constructors
  
• The class has a public getter that returns a private field, but no public setter (a common immutability pattern)
• A field's type annotation is for an abstract classes that doesn't have factory constructors
• A field is type annotated with a common type unrepresentable in JSON, like DateTime, or Int64
• A field's type is a supported collection, like Map or List, but is missing a generic, or the generic is abstract.
• A field doesn't have corresponding property in the JSON
• The JSON has properties that aren't present in the class.
• A field is final, and set in the initializer list, but also has data in the JSON
• The class seems "flat" to the user, but actually isn't - I've seen this confusion with trees that have parent-references.
• A field name has changed over time, or the JSON key is not a valid Dart identifier

You can definitely choose what to have happen in all of those cases, but you necessarily end up with either complexity or severe restrictions. Are the trade-offs you make suitable for everyone else? How quickly do you fall off the easy path, and how steep is the fall of?

Calling this inherent complexity in deserialization "invented" as an excuse to not solve the problem honestly shows that either you're not aware of the true complexity, or you think that the exact set of choices that you would make are the same that everyone else would. I can tell you from experience with our customers who are doing serialization, that this is not the case. This is the core of my assertion that the type of deserialization that you appear to want is not "missing" from the platform in any way. Because of the necessary decisions and tradeoffs it absolutely should be external to the core.

We can do more here to make things easier, and I'm sure we will, but the platform is still growing and some of the serialization libraries started before we had things like the analyzer, the pub transformer framework, JS mirrors, etc. I think given the improvement in the infrastructure, we can improve serialization a lot. One obvious help would be a pub transformer that generates package:serialization rules, so that you can use mirrors in development and not in production. 

Also, like Alan says, for a very restrictive set of choices, writing a deserializer is not particularly difficult. We could put out a very basic and very limited serialization library, (provided we find the most precious resource of all - time), but it would be very... limited.

-Justin

[Danny Tuppeny]

On 5 August 2014 19:09, 'Justin Fagnani' via Dart Misc <mi...@dartlang.org> wrote:

But if you go with React and/or TypeScript you'll still have the same issue: JSON deserialization give you plain Objects, not anything with an interesting prototype chain. Given that Dart is certainly no worse here, and IMO better because Lists and Maps behave better in Dart, I can't see any gain. Unless TypeScript has something different than JS to say about JSON serialization, and /happens/ to make all the exact choices that you prefer for each decision that needs to be made. If that's the case, I'm certainly interesting in hearing what their solution is.

I wasn't suggesting we'd go with React/TS instead of Dart because of serialisation; but if we were to go with Dart, we need to be convinced the extra effort (it's a bigger shift than React/TS and the other things we're evaluating) will be worthwhile. If we think we'll hit stumbling blocks like this one; it might not be worthwhile. If the reason we disagree on this is because Dart isn't intended to be used as we were planning/hoping to use it; then that increases the chances of friction elsewhere. JavaScript, we're all very (way too) familiar with.

 

I think it'd be enlightening to see what you want to behaviour this basic deserialisation to be. I'd be very interested in what you want to happen in these cases:

I've answered these; but I have little Dart experience, so some of them might not make sense. The main gist is that if something is not obvious, it should error. Nobody expects the serialise to be magic; and it should be free to add as many constraints as are required. A simple, restricted serialiser is better than nothing.

 

• A field is not type annotated, possibly because it true type is not representable in Dart (ie, a union type)

If it's something that can be deserialised, it should be. If not, it should error. If it looks like an object, it could be converted into a Map, or an error raised because there was no type annotation. I have no real preference how edge cases are handled; errors are fine - it's totally fine for the serialiser to put restrictions on classes it can serialise/deserialise.

 

• Private fields (they couldn't have been read by a generic serialiser, should they be written?)

No real opinion, but if they can't be read, they probably shouldn't be written. I'd love to be able to deserialise into mutable type, but it's no deal breaker. Whatever is most consistent would make sense; it's totally fine IMO for the serialiser to dictate things about the classes it can deserialise into/serialise.

 

• The class has multiple constructors
  

Always call the normal default parameterless constructor.

 

• The class has a public getter that returns a private field, but no public setter (a common immutability pattern)

Again; no real preference (I don't really know enough Dart to know the ins and outs). Supporting immutability would be great; but I don't think not being able to support it should be a reason not to support mutable classes.

 

• A field's type annotation is for an abstract classes that doesn't have factory constructors

I guess this should be a runtime error (or whatever normally happens if I so something silly, like try to instantiate an abstract class).

 

• A field is type annotated with a common type unrepresentable in JSON, like DateTime, or Int64

If there's no way to convert, this should be a runtime error. However in some cases it might be possible eg. you could deserialise a number into the Int64; though it might error when serialising if it's too big; but again, I have no real preference; I'm happy for serialiser to dictate I can't do this.

(Note: This is assuming JSON; as I said before; I don't care if it's JSON, I just want a "static" format I can implement in C#/WebAPI, it could be JSON, ProtoBuffers, even something custom; as long as the protocol is defined and won't change randomly on a pub update).

 

• A field's type is a supported collection, like Map or List, but is missing a generic, or the generic is abstract.

If the generic is abstract, I guess this is similar to above, and it'd have to error. Without a type, I guess it should be possible to deserialise literals, error on anything non-obvious. (Again, there may be complications here; just outright reject it if it's non-trivial).

 

• A field doesn't have corresponding property in the JSON

Generally, I would expect this to just be silent; though I wouldn't be upset if it was an error.

 

• The JSON has properties that aren't present in the class.

This should be an error; otherwise the data is being lost (this seems more likely to be accidental than just having additional fields on your class).

 

• A field is final, and set in the initializer list, but also has data in the JSON

Runtime error (I presume this happens if I try to set it in code?)

 

• The class seems "flat" to the user, but actually isn't - I've seen this confusion with trees that have parent-references.

I presume you mean circular references to objects? Deserialising should have no knowledge of it (I don't believe JSON supports references like this?) and might create duplicated data; serialisation should error (again, because JSON won't support it) unless there's some annotation to exclude a field from serialisation/deserialisation; which would allow you to skip it.

 

• A field name has changed over time, or the JSON key is not a valid Dart identifier

Runtime error.

 

You can definitely choose what to have happen in all of those cases, but you necessarily end up with either complexity or severe restrictions. Are the trade-offs you make suitable for everyone else? How quickly do you fall off the easy path, and how steep is the fall of?

Severely restricted is better than no option. The problem here, is that if you don't provide it's incredibly difficult for us to do something reasonable (this entire thread is full of drawbacks of mirrors, code-gen, etc.). It seems like there are features/concepts missing from Dart that would allow any sensible deserialisation; the problem is not just about making implementation decisions.

 

Calling this inherent complexity in deserialization "invented" as an excuse to not solve the problem honestly shows that either you're not aware of the true complexity, or you think that the exact set of choices that you would make are the same that everyone else would.

I keep hearing this; yet many other languages have serialisers that are used lots. Third parties can fill in the gaps; the problem is that Dart doesn't even have an *incredibly restrictive* option out of the box. As someone new to Dart; it feels really silly to be told to go and download a third party library to do something as trivial as pull JSON from a server and put it into the basic Class you just told me to create! (trivial == the problem; I'm not suggesting the implementation for you is trivial).

One of Dart's huge selling points (the most important one, IMO) is types. Having a tutorial for AJAX that pulls data into a map and makes no mention of Types seems crazy; it's like thee two features were implemented independently with no communication; why can't we have both?

 

Because of the necessary decisions and tradeoffs it absolutely should be external to the core.

Again, I'm not all that fussed if it's core or a pub package. I simply believe it should be owned by the Dart Team; and that it should be reasonable. It seems to me (based on this thread) that currently Dart simply lacks the features necessary to implement this in a reasonable way. Given my Dart code is shipped, and not some compiled version, why is it impossible to set a property of a class by name without all of the mirrors overhead? This seems like a massive overhead for something trivial that the interpreter can already do, because it does it with my raw code?

 

We can do more here to make things easier, and I'm sure we will, but the platform is still growing and some of the serialization libraries started before we had things like the analyzer, the pub transformer framework, JS mirrors, etc. I think given the improvement in the infrastructure, we can improve serialization a lot. One obvious help would be a pub transformer that generates package:serialization rules, so that you can use mirrors in development and not in production. 

That would be a great start. Why has it taken so much arguing before any suggestion that it could be improved? :-(

 

Also, like Alan says, for a very restrictive set of choices, writing a deserializer is not particularly difficult. We could put out a very basic and very limited serialization library, (provided we find the most precious resource of all - time), but it would be very... limited.

Open more offices, hire more devs ;P

Seriously though; nobody is expecting the equivilent of JSON.NET built in to Dart core; but having something simple that can decode "{ id: 1; name: 'Danny' }" as a Dart Team-owned package doesn't seem like a huge ask given Dart's focus/goals. But again; I don't think the problem here is just writing code; the messages in this thread seem to suggest it's not just possible in a reasonable way given the current capabilities.

[Yegor Jbanov]

I agree we should have annotation-driven serialization in Dart. I wanted to build one for Streamy, but never got to it. I don't think we should rush something into the core SDK though. A good start would be to build a separate package, test out the concepts and if find that it works well for most applications, move into the SDK.

 

Severely restricted is better than no option. The problem here, is that if you don't provide it's incredibly difficult for us to do something reasonable (this entire thread is full of drawbacks of mirrors, code-gen, etc.).

Dart as a platform and ecosystem is far from no option. For one you have a JSON encoder/decoder that lets you do stuff. There is a good collection of 3rd-party serialization libraries making various reasonable tradeoffs.

As for serialization, mirrors vs code-gen is pretty much the same tradeoff other languages have to make. In Java/C# if you want to use reflection you need to keep your symbols intact when compiling, which is the same as using mirrors in Dart. Since those generally don't deploy to the web, they don't have to worry about code size. Otherwise you do code generation. There's nothing new in Dart in this respect.

I keep hearing this; yet many other languages have serialisers that are used lots.

I'd say all these built-in serializers become pretty useless pretty quickly, which is why we have things like protobufs and thrift for production systems. Java RMI? Nobody cared for years now. .NET data contracts? Better, but quickly lose robustness as soon as your application grows (and notably, didn't appear until 3.0 or so). JavaScript? No such thing as serialization at all (as Justin points out, might as well use maps and lists). There, I think we covered 99% of all code written for the web. I don't suggest that it is not possible to have a good solution come out-of-the-box, I just don't think it happened yet on any platform. Let's not repeat it in Dart.

As someone new to Dart; it feels really silly to be told to go and download a third party library to do something as trivial as pull JSON from a server and put it into the basic Class you just told me to create! (trivial == the problem; I'm not suggesting the implementation for you is trivial).

The implementation will still be either mirrors or code-gen. Something has to be able to map from "firstName" in JSON to "xy3" in compiled minified application. Mirrors is a form of code-gen anyway, except it's supported by the SDK.

[Alex Tatumizer]

I think everybody is severely overestimating the complexity of solution.

To be a bit more specific: the idea is that to support almost every  kind of serialization, it's enough to generate approx. the following MapView for every class:

class PersonMapView extends MapView {

  static List<Metadata> attributes = [

      new AttributeMetadata(name: "firstName", type: "String", setter : (obj, value) => obj.firstName=value,

         getter: (obj) => obj.firstName),

      //etc.

  ];

 

  //  the rest is as straightforward

 

}

On top of that, there must be a map between class and map view class (in pigeon map, it's called pigeonTypeCatalog).

The only difference between trivial protocols and non-trivial ones (like protobuf) is that AttributeMetadata may need extra parameters, like max. number of bits or something, which probably can be added generically and transparently for preprocessor (by just copying some extra annotations from source to metadata).

When we have this intermediate level ready, it solves the hardest part of the problem; after that, writing serialization proper is much easier exercise, and you can write 10 different serializers based on same metadata.

[Justin Fagnani]

On Tue, Aug 5, 2014 at 1:10 PM, 'Yegor Jbanov' via Dart Misc <mi...@dartlang.org> wrote:

I agree we should have annotation-driven serialization in Dart. I wanted to build one for Streamy, but never got to it. I don't think we should rush something into the core SDK though. A good start would be to build a separate package, test out the concepts and if find that it works well for most applications, move into the SDK.

 

Severely restricted is better than no option. The problem here, is that if you don't provide it's incredibly difficult for us to do something reasonable (this entire thread is full of drawbacks of mirrors, code-gen, etc.).

Dart as a platform and ecosystem is far from no option. For one you have a JSON encoder/decoder that lets you do stuff. There is a good collection of 3rd-party serialization libraries making various reasonable tradeoffs.

As for serialization, mirrors vs code-gen is pretty much the same tradeoff other languages have to make. In Java/C# if you want to use reflection you need to keep your symbols intact when compiling, which is the same as using mirrors in Dart. Since those generally don't deploy to the web, they don't have to worry about code size. Otherwise you do code generation. There's nothing new in Dart in this respect.

I keep hearing this; yet many other languages have serialisers that are used lots.

I'd say all these built-in serializers become pretty useless pretty quickly, which is why we have things like protobufs and thrift for production systems. Java RMI? Nobody cared for years now.

Not only that, but many members of the Dart team worked on Java and learned their lesson about how useless and downright dangerous Java serialization is. The only thing Java serialization is remotely useful for is extremely short lived caches - I've used it with memcache, and just evict the key if the value doesn't deserialize. In olden days before better release management was more wide spread, I've even seen data trapped in Java serialization written to disk with an unreproducible build. Oops.

GWT serialization seems to work fairly well, as long as you only care about GWT client to Java server transport. If you recompile your app you need to redeploy for the server and client, and existing clients need to restart. You'd never write GWT serialization to disk. Good luck if you want an alternate client or want to load test. I've done those things and it's awful.
 

.NET data contracts? Better, but quickly lose robustness as soon as your application grows (and notably, didn't appear until 3.0 or so).

Looks like DataContractJsonSerializer appeared in .NET 3.5, and it also had a fair bit of configuration options and control via annotations. It's not simple. It also could easily be an external library, it's placement under .NET seems just to signify it's officialness, a practice we shy away from on Dart (especially because packages can release independently of the SDK). It's also not the only choice of serializers, combating the idea that C# has a universal solution to this problem. I just found this blog post from someone trying to decide between it and JavaScriptSerializer ( http://publicityson.blogspot.com/2010/06/datacontractjsonserializer-versus.html ) That doesn't seem so different from choosing between serialization packages in Dart.

JavaScript? No such thing as serialization at all (as Justin points out, might as well use maps and lists). There, I think we covered 99% of all code written for the web. I don't suggest that it is not possible to have a good solution come out-of-the-box, I just don't think it happened yet on any platform. Let's not repeat it in Dart.

As someone new to Dart; it feels really silly to be told to go and download a third party library

If you don't want to use pub packages, you probably shouldn't use Dart. The core SDK isn't going to grow much, and shouldn't. Packages are a much, much better place to put code that absolutely can't go in the core SDK for some reason. There's also no reason the community can't create very high-quality packages, none of us want to assume that the only code you can trust has to be authored by the Dart team itself. That's not a good way to grow an ecosystem.

Specifically for serialization, package:serialization is authored by the Dart team (Alan Knight, specifically, who's on this thread), as well as protobufs and we contribute to discovery_api_client_generator. Yegor, a Googler, maintains Streamy.

Danny, I've seen some others say that your problem can't be solved well, and I actually disagree with that. _Your_ problem can be solved in the way _you_ want. The question is is how general is that solution, and how well does it work for other people and their problems.

You made many choices to the questions I posted that may not work for very many people. It seems to me like an extremely limited serialization package that only works with a small subset of the language is exactly the type of thing a package is for, I just don't think it should be surprising that given the huge variety of valid approaches to serialization that we don't have a one-size-fits-most, yet easy-to-use, solution yet.

I'll also add that I think our metaprogramming capabilities aren't strong enough yet, so it's harder than I would ideally like (though totally possible) to write that package in our recommended way (mirrors for development, codegen for production). The pub, library and dart2js teams are hard at work on this though, and it's going to be getting much better over the coming quarters.

-Justin

to do something as trivial as pull JSON from a server and put it into the basic Class you just told me to create! (trivial == the problem; I'm not suggesting the implementation for you is trivial).

The implementation will still be either mirrors or code-gen. Something has to be able to map from "firstName" in JSON to "xy3" in compiled minified application. Mirrors is a form of code-gen anyway, except it's supported by the SDK.

[Seth Ladd]

Here's two examples: https://gist.github.com/sethladd/34b0d812b29c3bfd9529  I am using your example of: "{ id: 1; name: 'Danny' }"

In the first code (class Simple) we hand generate the mapping. Note that I need both a toJson method a fromJson constructor. For a small number of classes, and a small number of fields, this is probably fine. However, I don't know what scale Danny is operating at. I assume "big" because you want automatic mapping. So I only provide this for illustration.

class Simple {

  int id;

  String name;

  Simple();

  Simple.fromJson(Map json)

    : id = json['id'],

      name = json['name'];

  toJson() => {'id': id, 'name': name};

}

  var simple = new Simple()

    ..id = 1

    ..name = 'simple';

  var jsonString = JSON.encode(simple);

  print(jsonString);

  var s2 = new Simple.fromJson(JSON.decode(jsonString));

  print(s2);

In the second, I use the smoke library for easy access to the mirrors API. I wrote a mixin that uses reflection. Note I needed @MirrorsUsed to tell dart2js how to ignore everything else when doing tree shaking.

import 'package:smoke/smoke.dart' as smoke;

@MirrorsUsed(targets: const [SimpleWithMirrors, Simple], override: '*')

import 'dart:mirrors';

abstract class Serializable {

  static fromJson(object, Map json) {

    json.forEach((k, v) {

      smoke.write(object, smoke.nameToSymbol(k), v);

    });

    return object;

  }

  Map toJson() {

    var options = new smoke.QueryOptions(includeProperties: false);

    var res = smoke.query(runtimeType, options);

    var map = {};

    res.forEach((r) => map[smoke.symbolToName(r.name)] = smoke.read(this, r.name));

    return map;

  }

}

class SimpleWithMirrors extends Object with Serializable {

  int id;

  String name;

}

  var s3 = new SimpleWithMirrors()

    ..id = 1

    ..name = 'simpleWithMirrors';

  var map = s3.toJson();

  print(map);

  var newJson = JSON.encode(map);

  var s4 = Serializable.fromJson(new SimpleWithMirrors(), JSON.decode(newJson));

  print(s4);

Note: this demonstrates it's easy to do the easy thing. However, every application grows its complexity and, at least for the apps I've worked on, either we needed to bust out of the default serialization or we needed to constrain our object designs to work within the constraints of serialization. Although, from what I read of this thread, Danny is simply looking for {simple: map, of: numbers, and: strings} => simple Dart class of fields and methods.

In no way am I saying "this is exactly what Danny wants", because I didn't parse the whole thread, but hopefully it shows that the very simple thing is, in fact, simple to do. But there are always trade-offs, and no perfect solution for everyone, unfortunately.

--

[Alan Knight]

I think that would simplify the situation of classes with only public fields containing simple types and default constructors with no arguments. Consider

class Person {

   final String name;

   String gender;

   List<Person> _parents;

   List<Person> get mother => parents.firstWhere((p) => p.gender == 'f', orElse: () => null);

   List<Person> _children = new ObservableList();

   void addChild(Person child) => _children.add(child);

   Stream childrenChanges => _children.changes;

   Person(Person mother, Person father, this.name) {

     parents.add(mother);

     parents.add(father);

   };

   Person.parents(this._parents, this.name);

}

   

[Alex Tatumizer]

@Seth: you sacrifice performance in either case, and require user to write a TON of boilerplate code. Just imagine that one object contains a list of other objects, which, in turn ... You need to write a lot of fromJson,toJson calls inside other fromJson/toJson calls, and modify your code whenever you add/remove anything.

With mirrors, you throw away all performance completely, plus it won't work on client.

Actually, your examples demonstrate the exact opposite of what you intended.

"To define" always means "to restrict". Java serialization is is trying to be general without limits. No one uses it, and can't even reuse anything from it if he needs a sane serialization like JSON. The whole thing was a waste of time.

Introduce restrictions - problem will solve itself. No extra code or rules are necessary to convert PODO into JSON and vice versa.

[Alex Tatumizer]

> I think that would simplify the situation of classes with only public fields containing simple types and default constructors with no arguments.

YES!!! If dart provides this, it will be priceless! That's POD (or PODO), this concept exists for a reason, and has many advantages. This discussion would never take place if it was possible to do it right out of box.

[Seth Ladd]

On Tue, Aug 5, 2014 at 11:37 PM, Alex Tatumizer <tatu...@gmail.com> wrote:

@Seth: you sacrifice performance in either case, and require user to write a TON of boilerplate code. Just imagine that one object contains a list of other objects, which, in turn ... You need to write a lot of fromJson,toJson calls inside other fromJson/toJson calls, and modify your code whenever you add/remove anything.

Of course. The first example was only to set a baseline. I am not recommending any particular example in my gist.

 

With mirrors, you throw away all performance completely, plus it won't work on client.

It works on the client. I ran it on v8.

 

Actually, your examples demonstrate the exact opposite of what you intended.

"To define" always means "to restrict". Java serialization is is trying to be general without limits. No one uses it, and can't even reuse anything from it if he needs a sane serialization like JSON. The whole thing was a waste of time.

Introduce restrictions - problem will solve itself. No extra code or rules are necessary to convert PODO into JSON and vice versa.

[Alan Knight]

That's not a PODO, more like PODS. http://en.wikipedia.org/wiki/Plain_old_data_structure

Things can be plain ordinary objects and still have constructors. Even for data structures lots of people prefer to have them immutable.

And it comes back to the earlier point that with tree-shaking, you need a mechanism to map between strings and setters, which either means reflection or code generation.

On Tue, Aug 5, 2014 at 2:43 PM, Alex Tatumizer <tatu...@gmail.com> wrote:

> I think that would simplify the situation of classes with only public fields containing simple types and default constructors with no arguments.

YES!!! If dart provides this, it will be priceless! That's POD (or PODO), this concept exists for a reason, and has many advantages. This discussion would never take place if it was possible to do it right out of box.

--

[Alex Tatumizer]

OK, PODO or PODS - it used to be called struct, and still exists in different languages (even brand-new swift has it).

BTW, Alan calls "simple" types are not completely simple - there can be Lists and Maps, too (in pigeon_map, I added also DateTime - somebody requested it).

My point is that there's a reason for this concept to exist, which is: make thinking about the program simpler!

And should be an ultimate goal of any language or any library IMO

[Frank Pepermans]

Wow, a lot of talk going on, and some very valid points.

My own experience, I wrote my own serialization lib as well, it's on pub too and being used in a very large app.

The thing is, each implementation does bring its own level of complexity,

for instance in my case, I have a lot of circular references, plus I need the generated instances to be unique, i.e. Person id:100 can only exist once, and should not be recreated in a next call where it is again transferred.

Then again there might be objects with combined PKs, and not necessarily of type INT too.

Stuff like this also requires a lot of tinkering on the server, standard JSON serializers don't support it out of the box. For instance, to tackle cyclic, I encode pointers into the JSON string.

My lib requires that the server generates the dart classes, and they get annotated with metadata. Mirrors is not used, but I do need a transformer to parse the annotations to an object that can be read without Mirrors.

@Justin

"We could put out a very basic and very limited serialization library, (provided we find the most precious resource of all - time), but it would be very... limited."

=> That is a bummer, Dart rocks IMO, if anything Google has resources, I'm hoping for not only more Dart libs and tooling to really create a good sell on *any* platform, be it gaming or enterprise, pure web or mobile app.

And not just that, but also some marketing push to spread the word. I'm doing Dart now and want to do it again in the future, but not just @home after office hours for fun :)

[Justin Fagnani]

On Tue, Aug 5, 2014 at 3:11 PM, Frank Pepermans <fr...@igindo.com> wrote:

Wow, a lot of talk going on, and some very valid points.

My own experience, I wrote my own serialization lib as well, it's on pub too and being used in a very large app.

The thing is, each implementation does bring its own level of complexity,

for instance in my case, I have a lot of circular references, plus I need the generated instances to be unique, i.e. Person id:100 can only exist once, and should not be recreated in a next call where it is again transferred.

Then again there might be objects with combined PKs, and not necessarily of type INT too.

Stuff like this also requires a lot of tinkering on the server, standard JSON serializers don't support it out of the box. For instance, to tackle cyclic, I encode pointers into the JSON string.

My lib requires that the server generates the dart classes, and they get annotated with metadata. Mirrors is not used, but I do need a transformer to parse the annotations to an object that can be read without Mirrors.

@Justin

"We could put out a very basic and very limited serialization library, (provided we find the most precious resource of all - time), but it would be very... limited."

=> That is a bummer, Dart rocks IMO, if anything Google has resources,

We have many resources, but more than many things to work on. The type of serialization library that seems to be what Danny wants would indeed be very limited, and none of the customers that we work with so far would be able to use it. So it's an issue of resource prioritization, and an issue of making sure that we didn't make something so limited that no one could use it outside of examples.

I think that if there's a clear indication that a specific set of limitations and tradeoffs is very commonly acceptable to a large set of users, then that would help us design and prioritize. From what I've seen though, when you get into details requirements quickly diverge, which is why we end up with flexible, but (appropriately) complex, solutions like package:serialization.

Of course, we can and will do better, but I still disagree that it's even close to a simple problem, or that it's even simple to agree what the problem is.

[Alex Tatumizer]

> Of course, we can and will do better, but I still disagree that it's even close to a simple problem, or that it's even simple to agree what the problem is.

That might be true, but I think it's very easy to agree on what "simple form" of the problem is, especially considering that it has a name already: "struct" (aka POD aka PODS). Is it that unusual to have simple tool for a simple purpose, and complicated tool (or set of tools of different caliber) for other purposes?

[Rich E]

On Tue, Aug 5, 2014 at 8:09 PM, 'Justin Fagnani' via Dart Misc <mi...@dartlang.org> wrote:

I think given the improvement in the infrastructure, we can improve serialization a lot. One obvious help would be a pub transformer that generates package:serialization rules, so that you can use mirrors in development and not in production. 

This seems like a promising solution to me.  People in this post keeping dogging on mirrors because they are currently slow and not complete in design, don't work with dart2js etc, but that is now, I look forward to seeing the progress on this front.

In relation, reflection is moving much slower on the C++ front (though there are active committees working through the problems) and it is the single largest roadblock towards efficient and easy to maintain serialization there as well.

[Greg Lowe]

Since you've already got C# POCOs on the server side, it sounds like one solution is to write C# code that reads the POCOs AST, and generates equivalent Dart PODOs, including toJson(), and fromJson() implementations. Given your simple requirements you could whip something up quickly. If your requirements become more complex it would make sense to switch to protobufs, which would do this work for you.

[Anders Holmgren]

Another approach that I'd like to see (and could use at work) is a
code generator for JsonSchema. Then you could go C# classes -> Json
Schema -> Dart classes with json serialisation / deserialisation built
in (toJson / fromJson)

[mythz]

Justin,

Just being able to serialize plain PODO/PODS (whatever people are calling it), i.e. public readable fields using the parameter-less constructor would be a major win and cover a lot of use-cases.

It would enable the use PODOs as Data Transfer Objects (DTOs) - a common Services best-practices pattern, which one of the benefits is that it takes care of the wire-format and serialization. 

Solutions that require the burden of manually parsing/setting of properties is really defeating the purpose and benefits of a serialization library.

The answers Danny gave are reasonable defaults that anyone would expect, and even if Dart chose different defaults it still doesn't matter as everyone would just learn to live whatever they were. 

We just want a solution for PODO's and it's infinitely better to have something that removes the burden, than nothing.

As for JSON Serialization in .NET - it's a very solved problem, and something I haven't had to think about in .NET for 4 years (and have been using heavily in my projects ever since). 

Its extremely rare for any .NET developer to ever consider parsing/emitting JSON manually, that's what the serialization libraries take care of. 

Whilst JSON.NET wasn't developed by Microsoft, they've adopted it as their default which is now automatically registered in most of their starter VS.NET templates, that's now the most popular NuGet package at 6.7M+ downloads, so I'd say that it does qualify as a universal solution.

I'm the author of ServiceStack.Text JSON/JSV/CSV Text serializers (2nd most popular JSON Serializer outside of .NET), and it was pretty straight forward to get a good functional serialization story that works as expected.

Where for most use-cases using extension methods on any object is all that's needed, e.g:

var json = new Poco { ... }.ToJson(); //serialize

var dto = json.FromJson<Poco>();   //deserialize

Over time I have ended up adding a number of config options and hooks to cater for different edge-cases and the DataContract attributes are there to provide some level of (opt-in) customization, but in most cases everyone just uses the defaults. 

.NET does benefit from having a rich meta programming library that provide a number of different ways to populate objects using Reflection, Compiled Expressions, Reflection.Emit, CodeDom, etc. (Most are cacheable, so they're fast at runtime).

I personally dislike code-gen myself, it adds dependencies on tools, increases compile times, generates code-bloat and artificial artefacts that can easily get out-of-sync, so I'd tend to prefer a Mirror-based approach.

But it wont be useful if it's too slow to use or disables tree-shaking, so I'd be happy with using annotations if that's all it takes.

Also JSON Serialization isn't a problem with JavaScript either, as you can trivially populate an object from a Dictionary or use it as-is in substitute as another object with the same shape, given property access remains the same.

- Demis

[Danny Tuppeny]

On 5 August 2014 22:13, 'Justin Fagnani' via Dart Misc <mi...@dartlang.org> wrote:

If you don't want to use pub packages, you probably shouldn't use Dart.

You took my comment out of context. I have no problem with pub packages (either from the Dart team or 3rd parties). However given the problems we face on an weekly basis from poor quality third party code; for our new codebase we're going to be extremely careful when we do take dependencies to minimise the chances of us being bitten in the ass. There's a lot of great 3rd party code out there, but there is orders-of-magnitude more poor code that will be abandoned, poorly written/tested and introduce breaking changes along with their bug fixes.

Anyway, my comment was about the new-user-experience. It's totally lame if within the first 5 minutes of using Dart you need to be evaluating and comparing 3rd party packages when you're just trying to build a very quick prototype doing something incredibly basic.

On 5 August 2014 22:16, 'Seth Ladd' via Dart Misc <mi...@dartlang.org> wrote:

Here's two examples: https://gist.github.com/sethladd/34b0d812b29c3bfd9529  I am using your example of: "{ id: 1; name: 'Danny' }"

This is pretty much what we (and others in this thread) want; but your code illustrates the problem perfectly. All of these solutions have pretty bad downsides (the first requires every property name is written *five* times; two of them in strings that have no ability to be checked or "refactor->renamed"; mirrors I keep being told are no good after dart2js and in production (and frankly seem like overkill nothing more than setting properties like this; and code-gen/transformers introduce another step; not to mention the protobuffer plugin not even being supported on Windows).

I checked the Analyzer source (since I was aware it used JSON and written by Google), but that's all written using maps. Maybe I'm alone in wanting to spend my time focussing on my apps and not writing mundane mapping code for every piece of data I ever want to send across the wire.

(FWIW, even plain JavaScript and TypeScript save me from this!).

Anyway; I don't want to stoke the fire any further; I'm throwing in the towel. It's clear my definition of what a web language/framework should do is different to the Dart teams; and this is their project, not mine. I'm unsubscribing from this thread because it's rather noisy and I've taken enough of the Dart team's time. Feel free to CC/email me if you want a response; but I don't think I can really add anything that hasn't already been said.

[jm]

Em terça-feira, 5 de agosto de 2014 18h14min18s UTC-3, Justin Fagnani escreveu:

There's also no reason the community can't create very high-quality packages, none of us want to assume that the only code you can trust has to be authored by the Dart team itself. That's not a good way to grow an ecosystem.

You need to have a community to have an ecosystem. To have a community we need ways to highlight quality packages (voting, curating). Otherwise pub is just a wasteland where command-line-goblins like to dwell and scare the others with their scars. 

I haven't looked at pub in a while, but responded to Bob a while ago, and my impression was that community support was out of the question. If it had, and if there was a good lib out there, this question/debate would not have come up in the first place. Now the good stuff goes undiscovered. 

 

[Daniel Davidson]

I'm not sure how you could go from C# classes to Json Schema as json schema allows for constraints that I imagine are not described by the C# language. If the starting point is a json schema I could see how you can easily generate C#, Dart classes as well as serialization support in json and many others. I think most developers naturally want to think about hierarchies in terms data structures in their language of choice and have it automatically double as schema but that is missing the constraint side of the schema definitions.

My solution so-far is to store models in json schema and use it to generate the dart. It's not terribly clean or fun.

For using json schema I use: https://github.com/patefacio/json_schema
For generating the Dart I use: https://github.com/patefacio/ebisu

There is a third piece that allows me to use json schema and call out to the code generation which I'm too embarrassed to share :-)

I agree with the original sentiment of the OP in that it is too easy for this to become way overcomplicated and an out-of-the-box solution would be awesome.

[Anders Holmgren]

Typically additional constraints are described by metadata (annotations). In Java you use something like http://beanvalidation.org/

I've written a constraint library in Dart called constrain. I've started to implement some core constraints largely for this purpose. I've cataloged the constraints in Json Schema and XML Schema so I can ensure I create equivalents

So far I've added

• Min & Max which correspond to Json Schema's minimum, exclusiveMinimum, maximum, exclusiveMaximum
• Pattern which corresponds to Pattern

I'm keen to integrate these with code generators for Json & XML Schema (and the reverse - schema from code). Later I will also look to integrate these with Polymer input fields. For example

<core-input type="number" min="5" placeholder="enter a number greater than or equal to 5"></core-input>

The min would bound to the @Min constraint on the corresponding model 

cheers

Anders

[Alan Knight]

Here's a sketch. I started out with the idea I could finish something up in a day, but it's not quite that simple and I'm on vacation after today, so this is what exists now. It's a transformer that for each class in the file it's run on generates a CustomRule subclass that gets and sets all public fields and uses the default constructor. It's using the serialization library underneath, because that was easiest. As a result it will, unfortunately, also handle object references with cycles and other things you might not want, and is probably slower than if it had a simpler infrastructure underneath. But it should serve as an illustration of how a transformer might be done on some other foundation.

In writing this I realized I'm still not clear on the use case, if there's a single one. For example, none of the examples I saw here included enough information to identify the type of the object, so I assume that we're expecting the user to know and decode it appropriately. Which also suggests a model with no subclassing, which I guess also fits with structs. Not having additional information can be important if you're trying to talk to an existing JSON API, but if you just want to talk back and forth between client and server it seems like you might want that information. This does include that because of the way serialization works, and as a result you can read any arbitrary object from a generic read() call.

Right now it treats each file independently, so you'd write something like

transformers:

   - minimal_serialization :

       $include: lib/test_models.dart

and it will generate a separate file for each. That doesn't scale, so there should be something like an aggregate transformer ( https://www.dartlang.org/tools/pub/transformers/aggregate.html ) that creates a single file that imports the others and makes a global list of rules available. Or maybe run it globally instead and filter out classes that have methods or constructors with arguments. So then something like this works

  var s = new Serialization();

  rules.values.forEach(s.addRule);

  var output = s.write(thing, format: new SimpleJsonFormat(storeRoundTripInfo: true)); 

You'd have to included the same library on both the client and server side and add the rules the same way. And it might be a good idea to sort the rules somehow in case JS hashtables have a different order than in the VM. But a better answer would be to have the rules map on the client side and allow meta-serialization of custom rules.

This is set up for the moment as two repos, one with the transformer and a very simple example/test.

https://github.com/alan-knight/minimal_serialization

https://github.com/alan-knight/minimal_serialization_example

Comments, pull requests etc. welcomed, but with slow response as I'll be on vacation without access to a development machine until the 20th.

[Alex Tatumizer]

I like it. Except that these 2 lines

  var serialization2 = new Serialization();
  rules.values.forEach(serialization2.addRule);


can be combined into one, e.g. by new Serialization()..addAll(rules.values);

Not clear why "rules.values", and not just "rules", but ... never mind. Also, I don't understand why it's called Serialization, and not Serializer. Again, not very important. 
If we had this from day 1, there won't be much incentive for anyone to work on his own serializer :-)
("Missing features" can always be discussed and added to transformer incrementally).
.

[Frank Pepermans]

Here's my take on the the problem, based upon Flash/Flex remoting.

I use it @work with Java/Hibernate but it would work best with a Dart server app.

- Generation is needed, I do it automatically from Hibernate classes, you can use inheritage if needed, for example: https://github.com/frankpepermans/dorm/tree/master/example/orm_domain

- In setup, you *can* define specific serialization rules, i.e. to deal with DateTime:

serializer.addRule(

  DateTime,

  (int value) => new DateTime.fromMillisecondsSinceEpoch(value),

  (DateTime value) => (value != null) ? value.millisecondsSinceEpoch : null

);

=> First method is for decoding, second for encoding

=> This rule will then run for all occasions where the target entity is using the first param *Type* as type

- Init the serializer, default JSON is included, e.g. Serializer serializer = new SerializerJson(); (global)

- now call serializer.incoming(jsonString); and serializer.outgoing(array or single entities);

- You can annotate entities, the entity factory will never create 2 persons with ID=2 for example, if one exists, then that one is always returned.

=> e.g. jobsArray[0].job.personsWithThisJobArray[0].person.colleaguesArray[0].job ....... Can have cyclic Person id=2 occurances

=> to support cyclic, the serializer.outgoing will encode cyclic occurances as pointers

=> an entity can have any number of ID fields, of any type

- Entities extend Observable and have dirty management

=> // test

Person person = serialize.incoming(jsonDataOfPerson);

print(person.isDirty()); //false

person.name = 'test'; 

print(person.isDirty()); //true

person.revertChanges(); // or just give back his original name

print(person.isDirty()); //false

=> there's also a manager class which auto-registers all entities, calling manager.flush() will return a list of all dirty entities.

There's much more, but not very well documented at this point. Not perfect by all means, but it does work for my situation, which needed a replacement for Flash remoting/blazeDS

--

[Alex Tatumizer]

I opened an "issue" in Alan's project on github, where I tried to comment and discuss alternative design (based on MapView) in some detail. Github is more convenient for this, because it supports markdown.:

https://github.com/alan-knight/minimal_serialization/issues/1

[Don Olmstead]

Its relatively trivial to do an annotation based serialization library using mirrors for POD types. You can add annotations, or really just use the key of a map, to figure out what to serialize. It would look something like this.

  class Foo {

    @SerializationProperty('bar')

     int bar;

  }

You can extend that however you like. I have a library lying around at work that even allows you to invoke a method for custom parsing when the JSON object doesn't ideally fit into how I would model it.

Speed is an issue with mirrors but having reflection always comes with a cost. Lets be realistic though. You should really profile before you declare something as a bad decision. When you're in the browser something like a paint or a layout will likely take longer than parsing some JSON into a dart object.

If you have a schema you could write a transformer that takes that schema and then generates the code around it.

Using the smoke library is almost an option as well but there are a few pieces missing there. You can't get at a constructor and you can't determine something generic. You also cant get info on a library which could be used to register what classes are being serialized. It does make the mirrors API a lot nicer but as its not on Github its much harder to contribute to it. This article is a good primer on smoke http://dovdev.com/smoke-and-mirrors/ to see what it does.

On Sun, Aug 10, 2014 at 9:24 PM, Alex Tatumizer <tatu...@gmail.com> wrote:

I opened an "issue" in Alan's project on github, where I tried to comment and discuss alternative design (based on MapView) in some detail. Github is more convenient for this, because it supports markdown.:

https://github.com/alan-knight/minimal_serialization/issues/1

[Frank Pepermans]

I've looked at smoke to parse annotations but could find a how-to. So I ended up removing mirrors in favour of a transformer, which parses annotations to a generated init method.

However, for me the main bottleneck is json convert, very problematic when I try to load huge amounts of data.

[Don Olmstead]

This in the DartVM or V8? https://code.google.com/p/dart/issues/detail?id=6037 says V8 is still faster in this department.

[Vadim Tsushko]

Hi Alex.

I think your MapView idea is very interesting - simple and powerful. And probably such MapView can be used not only for serialisation - for visual data inspectors for example. 

Regarding code generation - I think it could be mirror based (so just transformer use mirrors). Mirrors on code generation phase do not have their negative implications.

I've recently implemented mirrors based code generation in objectory, so from class

    class Foo {

      String bar;

      int baz;

    }

generated class 

    class Foo extends PersistentObject {

      String get bar => getProperty('bar');

      set bar (String value) => setProperty('bar',value);

      int get baz => getProperty('baz');

      set baz (int value) => setProperty('baz',value);

    }

https://github.com/vadimtsushko/objectory/blob/master/test/generator_test.dart

IMHO schema generation for MapView could have similiar implementation. 

понедельник, 11 августа 2014 г., 10:24:41 UTC+6 пользователь Alex Tatumizer написал:

[Robert Åkerblom-Andersson]

Hi Don and others in this mega thread,

It was a nice article you did on smoke Don. In part because of this thread and in part because I feel writing toJson and fromJson for all objects can slow down the development flow, I started to working on a model object that combines both automatic Json encoding/decoding and constraints checking with the help of Anders constrain package (http://pub.dartlang.org/packages/constrain).

For Json encoding/decoding, from my point of view I had three different options:

- The serialization lib discussed here by Alan

- Smoke

- Mirrors

Initially I thought the serialization lib would be the best pick, but after some testing I found that it might not be the best pick even if it seems very powerful. I initially thought that it could be used to generate json directly, like JSON.encode/decode, but more complete, but I don't think it works, I could not get it to work at least. It might be at better use inside a transformer generating json functions or for serialization of objects to a database. Because of the reasons Don mentioned in the blog post I decided to go with Smoke. 

So I chose to go with smoke, my experience with smoke was that it works for very simple objects, something like this:

class MyData extends X {

 String name;

 String age;

}

But as soon as you want something more complex it simply lacks in functionally (as was also mentioned in the blog post). So, I decided to get some help from mirrors, keeping smoke for the parts it can do and maybe in the future the mirror parts could be replaced when smoke gets extended. Or a transformer is developed and all the models get faster that way, I have not really gotten into transformer development just yet but I'm quite sure it's technically possible for the model I've developed (and help or contributions are welcomed). 

 

I have not pushed it to Github yet since I want to do some more tests first, but so far it works on all our models that we use for DartVoid's management interface at least and  I got 17 unit tests that works. There are quite a lot of objects with other objects or list of objects inside our models and that was the kind of "complex" things that smoke alone could not get to work. I will probably upload it later today or tomorrow, if anyone would like to try it out or to help making it better you can send me a ping at google plus and will setup you up.

As discussed in this thread it of course have some limitations (any json encode/decode in Dart will), but I think it can fill the need for about 90-95% of most developers need. There are only two requirements for a model to be encoded/decoded to/from json:

1. Extends a class, currently called VaneModel

2. Implement an empty model constructor (this is needed to support lists of objects in a model, if someone is interested I can explain how and why it works (and don't work without it))

So a simple model class can look like this:

class SimpleModel extends VaneModel {

  String name;

  int age;

  SimpleModel(this.name, this.age);

  SimpleModel.model();

}

I've highlighted the required parts from the user, nothing else is needed and the model could be much bigger without anything more needed from the user. Here is the first unit test showing basically how it works:

  test('1. VaneModel.encode(new SimpleModel("Dart", 100))', () {

    SimpleModel a = new SimpleModel("Dart", 100);

    SimpleModel b = VaneModel.decode(new SimpleModel.model(), '{"name":"Dart","age":100}');

    expect(VaneModel.encode(a), equals('{"name":"Dart","age":100}'));

    expect(a.name, equals(b.name));

    expect(a.age, equals(b.age));

  });

The reason this works without annotating or adding a method for each member (which I feel would be almost as much work as writing the actual to/from functions) is that we pass in a new empty object during decode. This is of course a compromise, but considering the gain I think it is well worth it (not needed for encode since the object is passed anyway). With the help of the passed in object, we can use that object as a base to populate and then also read out information about other objects or lists of objects inside said object. VaneModel also supports top level lists of objects, something I felt was a bit of a pain when decoding with JSON.decode when you first had to decode to a list and then go over that list to create each object. Here is a sample of the before and after for decode on the client side:

Before:

        List<GalleryTemplate> templates = new List<GalleryTemplate>();

        // Decode json string to list of json objects

        List jsonList = JSON.decode(response.responseText);

        // Decode each template in list of json object

        jsonList.forEach((t) {

          templates.add(new GalleryTemplate.fromJson(t));

        });

After:

        // Decode json templates

        List<GalleryTemplate> templates =

            VaneModel.decode(new GalleryTemplate.model(), response.responseText);

Here you can see how we before had to do this in two steps, now it can be done with one. But the biggest difference is of course on the model class, on this GalleryTemplate for example we could remove: 

55+10+16 = 81 rows 

And replace them with 

1+1+1 = 3 rows (plus 3 extend statements)

(The reason for the 10+16 is that they from two other objects that the GalleryTemplate object has lists of. )

81 rows might sound a lot but I think the biggest advantage is the time saved, in part to write those to/from function and in part keeping them up to date as you models change during development. With VaneModel you only have add two small parts and then never edit them again and the bigger objects you have the bigger is the save in time. Removing to/from json methods simply makes development a lot faster and more fun to write models. If performance becomes an issue, I think a transformer can help in the near future. Plus, I think it also important to remember you can always add to/from json methods later if it is a big issue, the work needed is not that much bigger than it would have been to use them from the start (but yeah, I think a transformer could remove that need all together). 

Hope someone found this interesting! It was an interesting and quite challenging thing to implement, as I mentioned I will push my code to Github soon and if someone would like to help as a tester, writing more unit tests or improve the code I'm happy to get contributions as well. If someone knows of any other implementations similar to what I just described it would be interesting to know and checkout as well! And yeah, I almost forgot, except this json magic VaneModel will also check your objects constrains if you have added any on each encode/decode automatically. So if you get data from the client for example, and the client have put in some bad data, it will be caught automatically when you decode the json data making your application more save and reliable.

Regards, Robert 

[Alex Tatumizer]

Vadim: I used the same idea of generating getters/setters before in pigeon_map, exactly like you say. I was not using transformers (they were not invented yet at the point), but used build script, which didn't make it very easy to use regular mirrors. I still don't know how to use mirrors in transformer - for this, you need to compile your transformer together with class definitions somehow - that is, use "import" for class definition files. Maybe you know the trick to avoid this?

Anyway, this question is orthogonal to another one: generate these setters/getters by adding code to original definition of class, or generate completely different class which is a Map View to original class? I think the latter is preferable, for many reasons, and I consider it my mistake in pigeon_map to not follow this idea (which for some reason never came to my head at the time of writing pigeon_map). Original defintion of class is better to be preserved intact, without polluting them with artifacts needed only for serialization.

The uses of MapView, as you said, can be many. E.g. you get ResultSet from database, which is a bunch of name/value pairs per record. Having MapView, you can implement (in 10 lines) generic mapper that pushes these name/value pairs into MapView, and you get normal struct (POJO, PODO, you name it) from there. It's a very lightweight kind-of ORM with no frameworks and with no performance penalty whatsoever.

Anyway, my main question: HOW to use mirrors in transformer? (Alan uses analyzer in his example).

[Alex Tatumizer]

@Don, some comments on your post:

> You should really profile before you declare something as a bad decision. When you're in the browser something like a paint or a layout will likely take longer than parsing some JSON into a dart object.

In a browser, you can't easily have mirrors (at least, at the time of writing). It's a hassle you can live with, but you can't force users of some library to do it. Without mirrors your thing will not work in browser at all, no matter how slow.

But what about SERVER? You want it to be slow? Then your server under load will spend 50% of processing time on just parsing (I found this figure somewhere with respect to XML; article advocated using special hardware accelerators to just parse XML - and indeed, if you search google, you can find AND BUY these devices - assuming you really don't know a better way to waste your money, of course).

[Frank Pepermans]

Don't forget you do need some logic besides the pure deserialization,

Ideally, you want unique entities based on the identifiers, i.e. person.job.personsListWithThisJob.contains(person) == true

and identifiers can be a uuid, or a combined PK, ...

The above also requires to take care of cyclic references.

This applies to both client and server.

Then preferably also dirty management, so you only need to send back/commit dirty properties.

All this will affect performance,

in my lib 2/3 is JSON parsing, 1/3 is taking care of the above

--

[Alex Tatumizer]

Frank,

I think dirty management is orthogonal to serialization, may be implemented differently. E.g. by creating proxy (which might  be supported by language itself).

Cyclic references - not sure what particular problem you have. You want to identify cyclic references, and encode them in some ad-hoc manner? You can probably find some workaround for this without really involving serialization mechanism, otherwise said mechanism will soon become "feature-rich" (= "bloated") and inefficient. (Though ideally, you would be able to extend serializer to incorporate your custom logic) If you want to just protect against cycles, there's Floyd algorithm that costs  almost 0 in performance terms - the check can be supported in serialization with no penalty.

[Frank Pepermans]

I have no problem with cyclic, I wrote a lib that serializes entities from server to client, deals with cyclic and uses no mirrors or smoke but just a transformer. You do need to generate dart classes with annotations from somewhere. (be it Hibernate in my case, but could be anything and ideally a nosql source)

Floyd is to detect a cyclic reference, in my case a reference is just a pointer in json, as soon as it is encountered, it will just swap the pointer out with a previously generated entity from before in the json.

Anyway, different goals then, I was just shouting out that I had the same problem and wrote something to deal with it :)

--

[Don Olmstead]

Just a heads up that wasn't my post. That was something I used to get up and running with smoke but as mentioned there's a good amount of stuff missing especially the constructors. Since smoke was really built around Polymer it doesn't need that and from my talks with the dart team they seem willing to take patches to smoke to beef it up. I just personally haven't had the time and as the community has really mentioned the move away from Github for libraries such as Polymer and Smoke make it much harder to contribute.

[Vadim Tsushko]

Alex, I've looked at pigeon_map generators when implemented my own code generator.

Same as you I use script for code generation, but I use mirrors in that script not source code parsing. I'm not sure that script should be called build script - it put generated file along with source file (for example in lib/src directory, then generated file used in IDE. I believe build scripts and transformers put generated files under output directory and that files do not used in IDE (not sure about it)

So I think maybe transformers not so applicable in that case.

If transformer are needed maybe simple but no so elegant way to implement it is to create ad-hoq transformer for each project.

For example

 - in our project we have file/files with PODO data model classes.

 - we have import code generating library in dev section of pubspec 

 - our ad-hoc transformer () import code generating library, import files with PODO data model and and run some onliner (something like generateScheme()) , to create MapView/Schema classes for them.

[Samuel Hapak]

You might be interested in http://blog.cognitect.com/blog/2014/7/22/transit

My team from vacuumlabs is currently working on port for Dart.

[Jan Vladimir Mostert]

I know this topic is like 4+ months old, is there any news on a official working toJson / fromJson library / package for classes in Dart?

Currently calling JSON.encode on a class throws a bunch of exceptions:

  toString(){

    return JSON.encode(this);

  }

... it's kinda a showstopper when my whole frontend is driven by serializing / deserializing JSON

I was prototyping a new frontend for a Java-based API, in Java it uses Gson for which you call toJson and fromJson and it serializes / deserializes perfectly according to clients using PHP, C#, Java and Python. The way Gson does serialization / deserialization in Java is awesome, the way PHP does it using associative arrays (basically maps) is horrible.

If there's nothing official yet, what is the preferred 3rd party library that can serialize / deserialize simple classes in a similar way that Gson does it?

Gson docs in case unfamiliarity with the Gson library: https://sites.google.com/site/gson/gson-user-guide#TOC-Using-Gson

On Sunday, 3 August 2014 12:58:00 UTC+2, Danny Tuppeny wrote:

The dev teams at my company are currently evaluating a bunch of new technologies for a rewrite of our core applications. Our applications are fairly large (> 1m LoC) and we need our new stack to last 5-10 years without needing significant rewriting (the current codebase is too tightly coupled for us to do a huge amount with).

Today I finally got around to starting a Dart prototype and I hit a big stumbling block! I couldn't find a way to deserialise JSON from our server into Dart classes! I spent some time investigating, and everyone seems to be using 3rd party libraries, and none of those I could find looked ideal for our use; and we're trying to steer clear of 3rd party components as much as possible (especially for such basic stuff) because we're bitten by poor testing, breaking changes and abandoned projects so much in our current codebase.

It feels like I must be missing something - for a language that's aiming to be an alternative to JavaScript in the browser and that has support for types; it seems like parsing JSON (or similar) from a server into types should be a rather simple task. I understand minification makes this complicated and it's likely to need reflection/mirrors (which apparently isn't great for production code) but this seems like a rather poor reason to not support something that seems to be so basic.

Have I overlooked something, or is there no good first-party way to do this? How is everyone else getting data from their server into their client code?

Note: I'm not married to JSON (though it'd be convenient); I just need a stable format I can deserialise into, that I could wedge into C# for WebAPI input/output (though it seems even Dart client + Dart server will have this issue?).

Any pointers? Is there any built-in (or Dart-Team-owned) code for this purpose? I was so sure there must be some, I started reading through the Dart repo code; but much of it seems to use maps. Not only do maps lose many of the benefits of Types, DartPolymer doesn't seem to want to let you do obj.X on a map even though Dart seems to be fine with it in Dart code :(

Some of the team are suggesting we should discount Dart on the basis of this; so I reeeeally want to find a nice solution! I don't mind writing code, but it does need to be pretty dynamic; since we have a ton of entities (deep object graphs) that will go back and forth in some parts of our apps, so manually maintaining JSON mapping code isn't an option.

(If not, I'm probably going to try doing Dart code-gen from our C# classes with a T4 template or similar, but it'll make Dart a harder sell than if I can do this nicely natively).

[Bigluis Vargas]

There is:

1. Dson https://pub.dartlang.org/packages/dson (I forked from Dartson since it looks abandoned)

2. DartSon https://pub.dartlang.org/packages/dartson

3. JsonX https://pub.dartlang.org/packages/jsonx

4. Jsonify https://pub.dartlang.org/packages/jsonify

5. json_mapper https://pub.dartlang.org/packages/json_mapper

[Peter StJ]

I am with Danny here - coming from another language (well mine is JS) and looking for ways to compose data classes from over-the-wire data. It is a problem we solve in JS apps with code generation.

So is there already a reliable library that uses code generation and standard input format to generate file(s) with classes for Dart? 

I know about protocol buffer but we cannot use it on the server side for various reasons so I am looking for a solution that uses json as over-the-wire format. Could be anything that generates classes with fromJson constructor and toJson method generated correctly (possibly with some checks for validity but not required). 

The config file for the generation could be anything: json schema or yaml or really anything, even proto files if the generated code works with json texts.

Please recommend something.

Thanks.

[Jonas Kello]

I’m also coming from a large C# app on both client and server. My pattern of choice is DTO so my solutions are for that scenario only. In particular I have two requirement on the DTO classes:

1. The DTO classes have to be totally ignorant of the serialization. There should be no annotations, extra methods, mixins or other hard dependencies to any specific way of serialization. This allows me to keep the serialization options open to change in the future.

2. The DTO classes have to be immutable. I should be able to cache them knowing that no-one will ever be able to change the cached objects.

When I tried to apply my DTO pattern from C# to Dart I found some things out:

1. You cannot use mirrors on the client side (does not work properly when compiled to JS). I tried for the longest time with mirrors but finally gave it up as an option.

2. Given (1), unlike C#, Dart does not have an idea of a class structure. And this is probably why you can do it so easily in C# but it is hard in Dart (and any other language that do not have introspection, like JS). So you have to provide this information somehow as extra meta-data. There is simply no way around this. One you accept this fact, you can have many solutions for this. I explored a few:

2.1 Dart has the concept of transformers, which is basically hidden code-gen. Serialisation package has this. I tried this for the longest time but could not make it work well in practise. One big issue was that transformers does not run on the server-side (yes you can with pub run but Webstorm does not support that). Also the transformer did not work with “part of” files. I tried at one point with transformers for the client-side and mirrors for the server-side but in the end I wanted a unified client/server way of doing things. So transformers is out of the question (and I generally do not like code-gen and even less hidden code-gen like the transformers concept so I try to avoid it if possible :-)).

2.2 You can hard-code the required meta-data. Serialisation package has support for this through the use of “rules”. This is the way I currently do things.

2.3 Since my DTO classes are totally ignorant of serialisation, I could at any point adopt another solution which I might do in the future. Eg. I saw some work in Dart 1.9 about mirrors and dart2js. Maybe at some point in the future, mirrors support will finally work for the client-side and I can use that :-).

So my advice would be: Use the serialisation package because that is as close to an “official” package you can get (at least I think it is?). Accept that you have to hard-code (handwritten) meta-data for your DTOs today. But keep you options open and in the future you might be able to do this dynamically and can simply remove all the handwritten meta-data.

Just to give an idea of my current solution:

This is an ignorant DTO:

part of XXXX.dto;

class ProductDto {

  final String product_id;

  final String system_type_id;

  final String product_type;

  final String description;

  final int sort_no;

  ProductDto({this.product_id, this.system_type_id, this.product_type, this.description, this.sort_no});

}

This is a hand-written rule I hope to eliminate in the future (used from both client and server):

part of XXXX.z_rules;

class ProductDtoSerializationRule extends ImmutableSerializationRule {

  bool appliesTo(instance, _) => instance.runtimeType == ProductDto;

  create(state) => new ProductDto(

      description: state[0],

      product_id: state[1],

      product_type: state[2],

      sort_no: state[3],

      system_type_id: state[4]

  );

  getState(ProductDto instance) => [

      instance.description,

      instance.product_id,

      instance.product_type,

      instance.sort_no,

      instance.system_type_id];

}

/Jonas

[Don Olmstead]

Do you have some sort of schema for the classes you want to serialize? One thing I've thought of doing is having a transformer on a schema that then generates the dart classes. From there you could generate a encode/decode function that would not need mirrors.

--

[Jonas Kello]

In my case the schema is described by the class, so there is no schema other than the class itself. I have Dart at both ends and no requirement to integrate with other applications (for these DTO objects). 

The idea is that you hand-write one of these DTO classes following certain conventions (basically that it should be immutable with a constructor that can set all fields). Then the framework takes care of all the rest, like transfer it and re-create an object of the same class at the other end.

I generally don’t like code-gen either static or dynamic with transformers. If mirrors was working on the client-side with dart2js I think we would not need transformers for anything (exception of dart2js of course because for some reason that is implemented as a transformer).

The most promising idea I’ve seen in this thread is to use mirrors on the server-side to generate serialisation rules dynamically, which can then somehow be read in to the client-side. Admittedly this is similar to using transformers but to me it seems more debuggable and less messy than a transformer.

/Jonas