Mapping key-value pairs to newly-instantiated class?

[Sean McCleary]

Hi Dartisans,

I'm looking for a package to help me do the following, and can't find one.  Before I set off on writing one myself, can anyone jump in and tell me if this exists?

I'd like a package that lets me take a Map and/or List and use it to instantiate typed objects. i.e.  A really simple example would be classes like this:

class Person {
  int age;
  String name;
  Person boss;
  Person[] children;


  Person(this.age, this.name);
}

It could accept an input like the following and convert it into typed objects, or vice versa:

{
  "age": 50,
  "name": "Bob",
  "boss": {
    "age": 20,
    "name": "Sally",
    "children": [ { ... } ]
  },
  "children": [
    {
      "name": "Steve",
      "age": 30,
      "children": [ { .. } ],
    },
    {
      "name": "Jane",
      "age": 20,
      "children": [ { ... } ]
    }
}

Conditions:

• The class must not require a zero-argument constructor to accommodate the mapper, or require a special constructor which simply takes the collection of values as a parameter.  Invariants such as "a name and age will always be set" must be enforceable via construction.
  
• Annotations on the class and/or class members would be OK.
  
• Must support subclasses
  
• Must run on server and client
  

I had originally thought reflection would be the way to accomplish this but since that seems to be discouraged in the browser and unavailable in flutter, perhaps a transformer would be better?  One which can generate the required mapping code for each object type?

So, does anyone know, does this exist?  Or is it my next personal project?

Thanks!

Sean

  

[Dennis Kaselow]

dson is one package that comes close to your conditions, at least up to version 0.3.3 (I don't know about the enforcability of your example invariant, I don't think it's there). Since 0.4.0 it uses source_gen which requires meddling with the type hierarchy and will make it impossible to have serializable subclasses.

--
For other discussions, see https://groups.google.com/a/dartlang.org/
 
For HOWTO questions, visit http://stackoverflow.com/tags/dart
 
To file a bug report or feature request, go to http://www.dartbug.com/new
---
You received this message because you are subscribed to the Google Groups "Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

[Danny Tuppeny]

built_value also does something similar (created by someone from YouTube) using the build/source_gen packages (I would recommend avoiding both transformers and mirrors for new code and using these instead). However, I've been trying to update my code to use them and it's been way more effort than I expected (and your classes have to be immutable to use even the json parts).

Deserialising is one of my big gripes with Dart; it should be trivial (and not rely on third party code) in something that was built as a web programming language (and compiles to JS and almost certainly needs to communicate with a server) but it's still ridiculously complicated even with built_value.

I haven't tried dson, but it doesn't seem like it should be hard to just generate toJson/fromJson methods/constructors on classes using build/source_gen. I was hoping to try it this weekend, but I got distracted... there might be any more Dart for a while; at least until I can get the SDK running on it 😂

[David Morgan ☯]

I'm responsible for built_value; indeed, this is the approximate problem it's trying to solve.

built_value is intentionally strict, however: as Danny mentioned the types have to be immutable. And, you can't have subclasses. The problem with subclasses and data types is that it breaks equality; I recommend using interfaces instead (no such thing in Dart, but you know what I mean; 'implement' instead of 'extend').

FWIW, I don't work for YouTube any more; I'm on the Dart team now :)

Cheers

Morgan

[Danny Tuppeny]

On Mon, 20 Feb 2017 at 12:43 'David Morgan ☯' via Dart Misc <mi...@dartlang.org> wrote:

I'm responsible for built_value; indeed, this is the approximate problem it's trying to solve.

built_value is intentionally strict, however: as Danny mentioned the types have to be immutable. And, you can't have subclasses. The problem with subclasses and data types is that it breaks equality; I recommend using interfaces instead (no such thing in Dart, but you know what I mean; 'implement' instead of 'extend').

Have you thought about a version of built_json that doesn't require immutability (I don't know if that's what built_json was before it was folded in)? It's been a PITA trying to move my code over (and I never finished). It seems like I'm adding all sorts of crap to support immutability which in this case I really don't care about (or rather, I don't think is worth the "noise" trade-off because they're very verbose).

 

FWIW, I don't work for YouTube any more; I'm on the Dart team now :)

Even better! :-) 

[David "Morgan" Morgan]

built_json was the same before it was folded in.

I see that the more general version is useful for some cases -- but for built_value it's an explicit non-goal.

This is because I've worked with code that used general serialization (think Java serialization) and had some serious problems with inconsistent data types. Some mutable, some immutable, some with equals/hashCode, some without, ... small projects can afford to give developers freedom, in large projects though it's a disaster waiting to happen.

BTW, built_value shouldn't be _too_ verbose -- did you discover that you don't have to write the Builder class if you don't want to? It can be fully generated. With that, and by adding extra factories by hand where you need them for convenience, it should be quite close. If there are any awkward cases I'd love to hear about them.

 

FWIW, I don't work for YouTube any more; I'm on the Dart team now :)

Even better! :-) 

--

For other discussions, see https://groups.google.com/a/dartlang.org/
 
For HOWTO questions, visit http://stackoverflow.com/tags/dart
 
To file a bug report or feature request, go to http://www.dartbug.com/new
---

You received this message because you are subscribed to a topic in the Google Groups "Dart Misc" group.
To unsubscribe from this topic, visit https://groups.google.com/a/dartlang.org/d/topic/misc/GQok_RQ0Ico/unsubscribe.
To unsubscribe from this group and all its topics, send an email to misc+uns...@dartlang.org.

[Danny Tuppeny]

On Tue, 21 Feb 2017 at 08:59 'David "Morgan" Morgan' via Dart Misc <mi...@dartlang.org> wrote:

This is because I've worked with code that used general serialization (think Java serialization) and had some serious problems with inconsistent data types. Some mutable, some immutable, some with equals/hashCode, some without, ... small projects can afford to give developers freedom, in large projects though it's a disaster waiting to happen.

Yeah, I understand this, and we've started using immutable types everywhere at work; but sometimes it's just an unnecessary overhead (especially when they're really verbose because they don't have language support). It just seems weird to tie JSON to immutability - JavaScript already does JSON great and Dart is already poor at it even without this additional constraint :-(

In this case, I'm just trying to build some really trivial objects to serialise and pass over to DynamoDB (a tiny project of my own). The construction and serialisation are really close and these objects exist only to give typing over the API so there's a limit to how much effort/work I want to put into them (I was actually close to just changing to maps; every time I try to do JSON in Dart I get really frustrated!)..

 

BTW, built_value shouldn't be _too_ verbose -- did you discover that you don't have to write the Builder class if you don't want to? It can be fully generated. With that, and by adding extra factories by hand where you need them for convenience, it should be quite close. If there are any awkward cases I'd love to hear about them.

Yeah; I hadn't been building them but the classes just generally feel like I'm having to add a lot of noise. Eg:

abstract class CreateTableRequest

    implements Request, Built<CreateTableRequest, CreateTableRequestBuilder> {

  String get TableName;

  BuiltList<KeyAttribute> get KeySchema;

  BuiltList<Attribute> get AttributeDefinitions;

  Throughput get ProvisionedThroughput;

  BuiltList<GlobalSecondaryIndex> get GlobalSecondaryIndexes;

  BuiltList<LocalSecondaryIndex> get LocalSecondaryIndexes;

  static Serializer<CreateTableRequest> get serializer =>

      _$createTableRequestSerializer;

  CreateTableRequest._();

  factory CreateTableRequest([void updates(CreateTableRequestBuilder b)]) =

      _$CreateTableRequest;

}

The second half of this is all just to get the serialisation; I've had to change the lists to BuiltLists and the implements line is quite noisy. And then the enums have turned into this:

class AttributeTypes extends EnumClass {

  static const AttributeTypes S = _$s;

  static const AttributeTypes N = _$n;

  static const AttributeTypes B = _$b;

  static Serializer<AttributeTypes> get serializer =>

      _$attributeTypesSerializer;

  const AttributeTypes._(String name) : super(name);

  static BuiltSet<AttributeTypes> get values => _$attributetypes;

  static AttributeTypes valueOf(String name) => _$attributetypesv(name);

}

This, on top of having to run a script that watches for changes, a bunch of additional dependencies and the fact I didn't manage to get it working all added to the frustration! It feels like I'm wasting time doing basic plumbing rather than working on my app, which is really what I hoped to avoid using Dart.

I've been away for a while so not had chance to revisit and try to finish this off, but I'm hoping to get some time next weekend to get back to this.

[David "Morgan" Morgan]

Thanks for adding more info.

I agree that your use case makes sense --  I just don't want to mix the two.

I haven't tried it, but it sounds like this might do what you need:

http://stacktrace.ca/json-serialization-in-dart-with-dogma 

https://github.com/dogma-dart/dogma-codegen 

I guess that something nice for built_value would be to be able to help you bootstrap by generating classes from a specification of the data. There is an issue open for that

https://github.com/google/built_value.dart/issues/92 

but I haven't had time to look into it.

There's also a live template for IntelliJ/WebStorm:

https://github.com/google/built_value.dart/issues/96 

--

[Danny Tuppeny]

On Mon, 27 Feb 2017 at 08:34 'David "Morgan" Morgan' via Dart Misc <mi...@dartlang.org> wrote:

I haven't tried it, but it sounds like this might do what you need:

http://stacktrace.ca/json-serialization-in-dart-with-dogma 

https://github.com/dogma-dart/dogma-codegen 

The "status: alpha" and v0.0.2 doesn't fill me with confidence; though it did seem to what I want. I was really hoping for something built-in (or at least Google-maintained.. I'm aware OSS devs hate when people say this; but I seem to spend half my life dealing with abandoned/broken/version-incompatible dependencies!).

I guess that something nice for built_value would be to be able to help you bootstrap by generating classes from a specification of the data. There is an issue open for that

https://github.com/google/built_value.dart/issues/92 

I don't know what OpenAPI is, but this also sounds like a good idea - if the whole lot was generated code I didn't have to mess around in, it can be as noisy as it likes!

I was thinking it should be pretty trivial to generate fromJson/toJson methods (probably what that dogma library linked above does), so I might have a go at that... Tbh I do keep hoping that by the time I come to work on something real, Dart will have reasonable support for basic JSON.. it seems crazy that a language designed to replace JS doesn't have it (it's actually one of the reasons my company/colleagues rejected Dart for our new web modules.. I'm sure we weren't the only ones frustrated by this) :'(

[Joel Trottier-Hébert]

I think dogma is broken tbh

--

For other discussions, see https://groups.google.com/a/dartlang.org/
 
For HOWTO questions, visit http://stackoverflow.com/tags/dart
 
To file a bug report or feature request, go to http://www.dartbug.com/new
---

You received this message because you are subscribed to the Google Groups "Dart Misc" group.

To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

[Matan Lurey]

I'm really encouraged by all of the discussion and desires around this issue. I think there are a number of hurdles:

1. Dart relies on being tree-shaken/minified to produce better performance and application size

That means there is no easy way to say "copy all fields from this object to this object", modulo either mirrors (ick) or codegen (see below).

One quick note - as the JS ecosystem starts relying more on RollupJS and Closure, they'll run into similar issues.

2. Dartisans are opinionated on how code-generation should be done to create from/toMap/Json. There is no lack of JSON packages in pub:

https://pub.dartlang.org/search?q=json

This is pretty simple to map from JSON -> Dart:

class Animal {

  Animal({
    String name,
    int age,
    bool isMammal,
  });

}

This is not - or at least - it requires an opinion on how to map that will be incompatible with other opinions:

class Animal {
  Animal({
    Species species,
    List<Appendage> legs,
    Map<String, Animal> relationships,
  })
}

[Danny Tuppeny]

On Mon, 27 Feb 2017 at 18:40 Matan Lurey <ma...@lurey.org> wrote:

I'm really encouraged by all of the discussion and desires around this issue. I think there are a number of hurdles:

1. Dart relies on being tree-shaken/minified to produce better performance and application size

Sure; this seems to have been the main objection all along but it feels like a required feature is missing because of filesize optimisations. People are minifying JS today and still supporting JSON so it doesn't seem like an impossible problem (sure they don't have types, but you still can't change thing.name to thing.a so presumably JS minification libraries are selective; Dart could be too?).

That means there is no easy way to say "copy all fields from this object to this object", modulo either mirrors (ick) or codegen (see below).

Codegen is a reasonable solution IMO. It kinda sucked before (transformers) but I think the new build stuff is pretty good.

 

2. Dartisans are opinionated on how code-generation should be done to create from/toMap/Json.

Sure, but having limited JSON support and having to roll your own/use 3rd party for something complicated seems like a much better solution than nothing. Other frameworks have this - .NET's built-in JSON serialisers (all of them ;)) are pretty limited but they work in the majority of cases and people switch to something else (like json.net) when they need something more flexible.

 

There is no lack of JSON packages in pub:

https://pub.dartlang.org/search?q=json

That's true, though these aren't all json libraries - and something weird happens when you go to page 11, it highlights page 10?! But how many work, are still being maintained, don't require mirrors or transformers, are consistent in dev/production, authored by someone reputable, don't pull in boatloads of dependencies, etc.? And how much time am I supposed to spend figuring out which one is best for me? It seems like a lot of work for something that is basically free in other languages.

All these JSON packages (and all the SO questions about this) might mean there's a battery missing from Dart?

(I also think "it's hard because of minification" and "we have tons of JSON packages" somewhat contradict each other!)

 

This is not - or at least - it requires an opinion on how to map that will be incompatible with other opinions:

class Animal {
  Animal({
    Species species,
    List<Appendage> legs,
    Map<String, Animal> relationships,
  })
}

I'm not sure exactly what you're referring to here? It's perfectly valid to crash at runtime if the provided data doesn't fit or if it's not possible to deserialise into a type (for ex. in .NET the deserialiser may throw if the class doesn't have a parameter-less constructors or setters). You don't have to provide a huge library that supports every combination of everything in order to be useful; but today you* don't even support the most basic case. There are plenty of other languages/frameworks that do JSON and have all kinda of restrictions yet they're still incredibly useful and used in boatloads of projects!

I'm sure the Dart team are sick of my opinions on JSON so I don't want to drag this out; but it does seem to me like the value of a simple built-in (or Dart team pub package) JSON support is hugely underestimated by the Dart team. Everyone is using JSON for everything these days (I'm not saying this is good, but here we are) and this is a barrier that is frustrating to overcome (none of the current JSON solutions are trivial, not even for the most basic requirements). I think a Google-owned package that did basic JSON (via code-gen using build/watch scripts) would go a long way even if wasn't totally flexible.

* by "you", I mean Google; this isn't personal ;-)

[Matan Lurey]

On Mon, Feb 27, 2017 at 11:30 AM Danny Tuppeny <da...@tuppeny.com> wrote:

On Mon, 27 Feb 2017 at 18:40 Matan Lurey <ma...@lurey.org> wrote:

I'm really encouraged by all of the discussion and desires around this issue. I think there are a number of hurdles:

1. Dart relies on being tree-shaken/minified to produce better performance and application size

Sure; this seems to have been the main objection all along but it feels like a required feature is missing because of filesize optimisations. People are minifying JS today and still supporting JSON so it doesn't seem like an impossible problem (sure they don't have types, but you still can't change thing.name to thing.a so presumably JS minification libraries are selective; Dart could be too?).

I guess you'd be talking about something like explicitly not minifying constructors/methods that will be used for JSON serialization. I don't think we have any (reliable) way of doing this today, but maybe it's something we can help tackle for dart4web once the primary development model is DDC -> dart2js, as we can more closely align with (parts of) JavaScript.

I am not sure there is a great answer for say, VM or Flutter though without code generation, so my instinct is we should still focus on improving code generation and/or providing patterns and guidance to others to build tools around this.

 

That means there is no easy way to say "copy all fields from this object to this object", modulo either mirrors (ick) or codegen (see below).

Codegen is a reasonable solution IMO. It kinda sucked before (transformers) but I think the new build stuff is pretty good.

Thanks! We really <3 package:build, and the maintainers are super eager to add features/address issues if anything arise.

 

 

2. Dartisans are opinionated on how code-generation should be done to create from/toMap/Json.

Sure, but having limited JSON support and having to roll your own/use 3rd party for something complicated seems like a much better solution than nothing. Other frameworks have this - .NET's built-in JSON serialisers (all of them ;)) are pretty limited but they work in the majority of cases and people switch to something else (like json.net) when they need something more flexible.

So technically we have limited JSON support, right, it's the JSON.encode/decode "reviver"/"toEncodable" method. I realize this isn't quite as good as .NET's built in serializers.

re: JSON.net, I know they use runtime reflection - something we're trying to avoid, so I'm not sure we could ever be as "magical" - though definitely code generation can get pretty far.

 

 

There is no lack of JSON packages in pub:

https://pub.dartlang.org/search?q=json

That's true, though these aren't all json libraries - and something weird happens when you go to page 11, it highlights page 10?! But how many work, are still being maintained, don't require mirrors or transformers, are consistent in dev/production, authored by someone reputable, don't pull in boatloads of dependencies, etc.?

I think that's probably more of an issue of pub needing some TLC and is independent of JSON serializing - though I do understand how the difficulty of finding "good" packages makes re-using other's code that much harder.

 

And how much time am I supposed to spend figuring out which one is best for me? It seems like a lot of work for something that is basically free in other languages.

All these JSON packages (and all the SO questions about this) might mean there's a battery missing from Dart?

(I also think "it's hard because of minification" and "we have tons of JSON packages" somewhat contradict each other!)

You're not wrong - but clearly since we don't have universal adoption of say, build_value (which I think is the single most "full featured" implementation today) means that it perhaps isn't enough just to have a viable option.

A lot of the feedback has been around "I think it's [too heavyweight]" or "I just want it to do [X]" - both of those lead to rolling your own encoder/decoder, which sort of brings the problem back full circle. 

Maybe we should have a design session with some members of the community on "if we had a json.dart package, what would it look like/do"?

 

 

This is not - or at least - it requires an opinion on how to map that will be incompatible with other opinions:

class Animal {
  Animal({
    Species species,
    List<Appendage> legs,
    Map<String, Animal> relationships,
  })
}

I'm not sure exactly what you're referring to here? It's perfectly valid to crash at runtime if the provided data doesn't fit or if it's not possible to deserialise into a type (for ex. in .NET the deserialiser may throw if the class doesn't have a parameter-less constructors or setters). You don't have to provide a huge library that supports every combination of everything in order to be useful; but today you* don't even support the most basic case.

The issue is, see above, that for example built_value does handle the above case quite fine - but there is still resistance to using it due to it not being in the format people want (which is fine, but again, full circle).

 

There are plenty of other languages/frameworks that do JSON and have all kinda of restrictions yet they're still incredibly useful and used in boatloads of projects!

I'm sure the Dart team are sick of my opinions on JSON

I'm not, so keep it coming.

 

so I don't want to drag this out; but it does seem to me like the value of a simple built-in (or Dart team pub package) JSON support is hugely underestimated by the Dart team. Everyone is using JSON for everything these days (I'm not saying this is good, but here we are) and this is a barrier that is frustrating to overcome (none of the current JSON solutions are trivial, not even for the most basic requirements). I think a Google-owned package that did basic JSON (via code-gen using build/watch scripts) would go a long way even if wasn't totally flexible.

* by "you", I mean Google; this isn't personal ;-)

Just to be clear, even though Dart is a Google-owned product, we don't represent anything Google official to the same scale as say, Gmail is Google (the same goes for other products like Angular, Polymer, etc - they are individual Google teams - not representing "Google"). 

[Joel Trottier-Hébert]

I kind of agree with this though. Clearly this is a recurring problem. If you look at all the pub packages (maintained or not) that try to do JSON serialization, you can definitely see that people are satisfied with what comes built-in in the language.

Now Danny, about this issue, I was using Dogma a while back, but I had a few issues that made me have serious bugs in production. I think Dogma was one of the first (if not the first) to go down the code-gen way to do this. To me is was super close to be real good, but I don't think it's much active anymore.

I know some other people have developed things (namely the Jaguar guys who have got their own code-gen based json serializer too, but I haven't tried it), but I still haven't heard of anything that had become the "de-facto" (code-gen based) json serialization package.

--

[Istvan Soos]

I think such packages originate as "scratch your own itch", they are
very opinionated, they address different requirements, and as soon as
the authors solved their problem at hand, they are likely to move on
to the next issue they are facing. Having rolled out my own JSON
serializer in the code generated camp (package: owl), this describes
me pretty well.

While I would love to use a well-built serialization code, I feel it
is hard to satisfy everyone. As a starter: developers have strong
opinions whether the serialization should be done through mixins or as
a separate class. Should you define/annotate every field, or should
there be a default behavior if annotation is missing? If there is a
default, what should it be? How should it handle refactoring (aka
breaking changes) in the API? Is it designed to be thin wherever it is
included (web stack, mobile and server side)?

Serialization is easy to test, and it should be a thin library -> low
risk to adopt and easy to switch if needed. I hope that sooner or
later one or two of them will gain some traction and contributors, and
the choice will be clear which way to go. Until then, I suggest you
try one out, and engage with the authors.

Regards,
Istvan