Request for feedback: Discontinuing support for dart:mirrors

[Michael Thomsen]

We're considering discontinuing the dart:mirrors library, but would like to get some feedback before we make a decision. For further details, and to give us feedback, please see this feedback issue:

https://github.com/dart-lang/sdk/issues/44489

Regards, Michael

[Daniel Varga]

Hello Michael,

I might be reading too much into this but discontinuing `dart:mirrors` library has some serious consequences that were not mentioned in your mail. The most important is that we'd lose the ability to do runtime reflection at our server-side code. It is an HTTP server that uses mirrors for the most important bits of it. Transitioning away from it is a scary proposal for many reasons.

If anything I'd request to add mirrors to AOT snapshots too, instead of discontinuing, so that we'd get to deploy our code as a single binary with all the benefits and drawbacks it means. But... Let me address the concerns I have:

1. There is no proposal as to what would replace it, we need more info what you had in mind.
2. If there is no replacement, you'd effectively eliminate some useful features from the language/runtime we actively use. I don't think that's the case; I will assume for the rest of the mail that you are thinking in terms of migration to code generation.
   
3. We enjoy the ability of quick iterations on BE code without having to run builders all the time, which adds a lot of compile time overhead.
4. dart:mirrors is easy to learn and test, package:build is most definitely not
   
5. I keep struggling with different build_runner versions on FE code where package:analyzer constraints prevent me from using certain packages together (eg. Angular 5.3 and freezed). This issue does not exist with runtime reflection.
6. Builders can be really slow, in some cases I'd prefer the runtime overhead instead of the added minutes of wait time for the builders to complete.

I'm sure you have a list of pros and cons for mirrors vs. code generation too. Having said that, I'm not against the idea completely if there are good enough reasons.
I'd have to bite my lips hard if mirrors were removed today and would have to migrate to pacakge:build but luckily we still have some time. 

There have been several precedents where such a change was preceded by preparing users, crafting new tools or giving some extra attention to alternatives. Most recent one is the NNBD tooling to this. I hope it is the case here as well, in which case feedback on why I'm nervous about moving to pacakge:build is warranted here. In my opinion, build and associated packages are in a bad spot right now.

My recent incursions to writing builders were ... trying because:

1. after awhile, I gave up trying to find written documentation for the questions I had, I started digging around in the code - this shows that the there is too little information out there about how to write builders, how it works and how to configure them. Eg. I was surprised to see that `build.yaml` and `build.config.yaml` are merged together instead of being taken as an absolute configuration BUT with a huge caveat: builder declarations are ignored from `build.config.yaml`.
2. the configuration space is very complex and the tools do not aid me rooting out configurations that make no sense. ( ref ) Figuring out what is OK and what is not can only be done through trial and error or by reading the implementation. This is a problem for every user of package:build, not just builder authors.
   
3. Contributors of package:build stated it is intended for "superusers" ( ref ). Context to it is: I brought up that it is difficult to map how I see code as a developer to how analyzer sees the same code, then write a builder around that mapping. dart:mirrors make it fairly easy. I fear that I would be required to run around in language specifications to figure out what is what for a question which as simple as: 'What is the first method in this class with an annotation of `Route('/')?` Example of how complicated it can be just to resolve types here (used by Angular Compiler too).

I've had some steep learning curves over the years. package:build was one of them. The primary reason - which I consider the most important of all -  is a conceptual one: Code generation is becoming more and more important piece of the language ecosystem. As such, as a user of the language, my expectation is that such an important piece is well-documented, somewhat fool proof and fairly easily approachable. 

I hope that this is the kind of feedback you were looking for to help you make the right decisions.

Cheers, Daniel

--
For more ways to connect visit https://dart.dev/community
---
You received this message because you are subscribed to the Google Groups "Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.
To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/f66a16ff-0539-463a-9733-00296023058en%40dartlang.org.

[Michael Thomsen]

Thanks for those notes Daniel. Can you kindly post the feedback in the feedback issue on GitHub? It will be easier to keep track of things there.

[Daniel Varga]

Sure thing! 

[Istvan Soos]

On Wed, Dec 16, 2020 at 1:39 PM Andrew Mezoni <andrew...@gmail.com> wrote:
> What would you like to replace reflection in the "mustache" package used on the pub.dev website?
> If it is not a secret.

pub-dev uses package:mustache, but does not use reflection.
The data for the template is prepared as Map + List + simple values
like int or String.
A random example from the repo:
https://github.com/dart-lang/pub-dev/blob/master/app/lib/frontend/templates/listing.dart#L143-L164

Istvan

[Guyren Howe]

I’m not clear why Mirrors is not a good thing to keep, particularly since it’s already been implemented.

I get that code using Mirrors will be slower, but surely we can leave that up to the dev to decide, rather than just decreeing that they can’t use it.

If it must be removed, I can imagine implementing something that works the same using the build process, but again, why not just keep what we already have?

--

[Randal L. Schwartz]

>>>>> "Guyren" == Guyren Howe <guy...@gmail.com> writes:

Guyren> If it must be removed, I can imagine implementing something that
Guyren> works the same using the build process, but again, why not just
Guyren> keep what we already have?

That's not possible in general. A program using mirrors can get a class
name as a String (like from the CLI :), convert that to a Symbol, and
then work with that class.

That involves things that are not available at runtime to the build
process. When you use mirrors, there can be no tree-shaking!

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<mer...@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/Dart consulting, Technical writing, Comedy, etc. etc.
Still trying to think of something clever for the fourth line of this .sig

[Guyren Howe]

On Dec 16, 2020, at 15:30 , Randal L. Schwartz <mer...@stonehenge.com> wrote:

"Guyren" == Guyren Howe <guy...@gmail.com> writes:

Guyren> If it must be removed, I can imagine implementing something that
Guyren> works the same using the build process, but again, why not just
Guyren> keep what we already have?

That's not possible in general.  A program using mirrors can get a class
name as a String (like from the CLI :), convert that to a Symbol, and
then work with that class.

That involves things that are not available at runtime to the build
process.  When you use mirrors, there can be no tree-shaking!

I get the lack of tree shaking; as I say, I think this should be up to the developer to choose. It should certainly be that if an app doesn’t use mirrors, it should be the same as if there were no mirrors.

I’ve been digging into these Build Macros, or whatever that’s called, and I’m missing something if you’re right.

The build system can clearly crawl all the files in the app and access their syntax trees. Can we not use that to generate a file that declares a Mirrors object that provides a map between the strings and the symbols/classes?

[Jonathan Rezende]

> The build system can clearly crawl all the files in the app and access their syntax trees. Can we not use that to generate a file that declares a Mirrors object that provides a map between the strings and the symbols/classes?

That is pretty much what source generator packages do.
We used dart:mirrors then we replaced to Reflectable and finally we realized that we needed our own source generator for models where we could insert our business rules and stuff.
That was the best thing we could do because it made all of our business rules available to all platforms. We now share a lot, really lot of stuff between server and frontend applications while retaining full power of autocompletion, analysis, compile time to prevent errors and the benefits just goes on.
But that can just be our use case.

[Guyren Howe]

I would love to read about what you did and how you did it. The build system is not the best-documented or clear part of the Dart ecosystem.

--
For more ways to connect visit https://dart.dev/community
---
You received this message because you are subscribed to the Google Groups "Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/F3C77BF8-5B0A-454B-96F5-31BAD95CD030%40jode.com.br.

[Randal L. Schwartz]

>>>>> "Guyren" == Guyren Howe <guy...@gmail.com> writes:

Guyren> I would love to read about what you did and how you did it. The
Guyren> build system is not the best-documented or clear part of the
Guyren> Dart ecosystem.

Reflectable looks like it may get most people most of the way there.

[Ivan Vishnevskiy]

For me, one of the downsides of Dart is that I'm not able to convert objects to classes just like in C#(MyObject as MyClass) which is a lot of pain when dealing with lots of json data. I know there are packages that involves build_runner but I'm not huge fan of adding additional time when building apps. Can someone tell me how this feature I described will make tree-shaking difficult?

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/10dbf260-6905-4608-943a-ebcfbeb06160%40Spark.

[Michael Thomsen]

We're neither cutting development costs nor is funding the core issue.

But like any team, we have a finite size, and like most teams we find ourselves having a greater number of requests, options, and ideas than our capacity allows us to realize. So we try to apply a lot of thought into which things we select to build. In this case we felt like we needed to better understand the use cases to determine how to best support those.

On Thu, Dec 17, 2020 at 10:42 AM Andrew Mezoni <andrew...@gmail.com> wrote:

Can I ask a question here that I will most likely not get an amazing answer?

So the question.

What prevents you from continuing to develop and improve the mirrors system?

Maybe Google has decided to cut its development costs?

Or is the allocated funds not enough to work in this direction?

Everything that is now in Dart is in demand.

Is there a need to close other directions now, despite the obviousness that these directions are in demand?

четверг, 17 декабря 2020 г. в 13:33:27 UTC+5, iasta...@gmail.com:

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/05e5eeb5-fa26-40b0-bf54-a1c3b4e9e3bdn%40dartlang.org.

[Jonathan Rezende]

> For me, one of the downsides of Dart is that I'm not able to convert objects to classes just like in C#(MyObject as MyClass) which is a lot of pain when dealing with lots of json data. I know there are packages that involves build_runner but I'm not huge fan of adding additional time when building apps. Can someone tell me how this feature I described will make tree-shaking difficult?

Consider you are importing 10 packages that uses mirrors for several classes.
Also consider that your own project has mirrors.
Finally, lets totalize 100 mirrored classes.

Now, with mirrors, you could create any of those 100 classes in a way that compiler can not know which ones you are going to use or not.
For example: a column in a database that contains the name of the class.
With that in mind, the compiler must put all mirror stuff in your bundle.
in the end, your project really needed 2 classes to be created with the mirror system.

Nonetheless, let us imagine how would be today if the Dart team didn’t deprecate mirrors back then.

At least for me, I would have to dig the package’s internal before adding it to a project because it could literally impact my app just by importing it.

This subject remind me of a color package that I used few years ago.
Instead of having a class with functions, it used the noSuchMethod and used the function name to get the color, ie “red”, “cyan” etc.
I hated it because I couldn’t easily know which colors were available by command+click to navigate to the function or any autocompletion at all.

So I am in total favor for the Dart team to keep up an organized ecosystem where I can import a package with free of mind.

[Jonathan Rezende]

> We're neither cutting development costs nor is funding the core issue.
>
> But like any team, we have a finite size, and like most teams we find ourselves having a greater number of requests, options, and ideas than our capacity allows us to realize. So we try to apply a lot of thought into which things we select to build. In this case we felt like we needed to better understand the use cases to determine how to best support those.

What we could have is an interface system like in TS.
It is not a real class. It is just a virtual wrapper around a Map where it works only by giving us autocompletion and dev/compile time analysis.

and by virtual wrapper I mean something that would not make difference to the compiler if manipulating the Map with the interface or not. So it should not impact either performance or app size.

IMO that would solve most of our “mirror' problems, which is pretty much tied to JSON stuff.

[Guyren Howe]

Seems like a simple tool that checks a package for use of mirrors would mostly satisfy this complaint. It could be automatically applied in pub and displayed on packages and searchable.

[Ivan Vishnevskiy]

Thank you for the explanation. It's a bit unfortunate because I could've managed my own reflection myself with just the stuff I need, but I understand the point that not everyone will do that and if mirrors was a thing pretty much every random library would've used it rather mindlessly. And without that I'm able to make an android application which takes just 13mb in space, while react-native hello_world app is like 30+. Thank you!

[Randal L. Schwartz]

>>>>> "Andrew" == Andrew Mezoni <andrew...@gmail.com> writes:

Andrew> If there is a value of a certain type, it means that this type was not tree
Andrew> shaken.

Nor would any code method, or everything they call.... it's not as
simple as "including a class"... it's "include a class and everything it
might touch".

[Jonathan Rezende]

It can fit your use case, but Reflection is nowhere just that.

Take the Reflectable package. You even mentioned it can do more and that is exactly the point of Reflection: it can do a lot of stuff and the compiler can’t know what you will need or not.

Gee, you could even traverse the comment section, annotations and a lot of things of a class/functions/properties with Reflection. Imagine all that being bundled. That is why you must tell Reflectable what you want to be reflected., but even so, it can get insanely huge if widespread in the pub environment.

Also about the Reflectable package: it is pretty much a working out-of-the-box solution IMO.

Em 18 de dez. de 2020, à(s) 06:38, Andrew Mezoni <andrew...@gmail.com> escreveu:

I'll be honest about what I would need.

Extension of functionality (runtime) Type.

- Ability to query Type for a list of its member names

- Ability to read and write to a member by name through Type

final type = value.runtimeType;

final members = type.members; // List <String>

Type.getValue(value, 'foo');

Type.setValue (value, 'foo', 123);

final func = Type.getValue (value, 'func');

Function.apply (func, 1, 2, 3);

If there is a value of a certain type, it means that this type was not tree shaken.

It is unlikely that much space is needed to store additional type information.

At compilation, all this information can be placed in text tables (in json format) and compressed and can be decompressed at runtime.

The code size will also be small. It will only be necessary to number (identify) all the classes. This index can be used to reference the class.

You can also number all unique class member names.

Only classes that have not been deleted are used.

Reflectble does the same (without the table driven approach) and even a little more, but this is not an out-of-the-box solution.

--
For more ways to connect visit https://dart.dev/community
---
You received this message because you are subscribed to the Google Groups "Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/7a99abd7-02fc-431b-857b-90fd300c941dn%40dartlang.org.

[Erik Ernst]

On Tue, Dec 22, 2020 at 1:11 PM Andrew Mezoni <andrew...@gmail.com> wrote:

>> Also about the Reflectable package: it is pretty much a working out-of-the-box solution IMO.

Does it already know how to do reflection for classes that you don't have access to?

Those classes to which you cannot add metadata to mark them for reflection?

Reflectable code generation is intended to be applied to an entry point, that is, a library that exports a function named `main` and whose transitive import closure defines the contents of "a program". So let's call that a 'program'.

This is of course a limitation (you can't run reflectable code generation on a library L in isolation and have it produce, say, support for reflection on all subtypes of a given class declared in L, because some of those subtypes may be declared in libraries that aren't part of the transitive imports from L). You could also say that reflectable relies on a closed world assumption.

However, if you do run reflectable code generation on a library whose transitive imports will be just part of any given program, you will get support for the requested level of reflection for that part. This means that you could request reflection support for all subtypes of a given class C, and you will actually get support for only some of them (the ones that we can "see" via direct or indirect imports). If you're happy about that level of support and want to do this then go ahead and use that approach.

However, the main scenario is still that reflectable code generation is applied to a program as a whole. This means that if you encounter an instance of a class C at runtime, the program will include a library that contains the declaration of C, and reflectable code generation will provide the requested level of reflection support.

The addition of metadata is not crucial: For instance, if you request reflection for all subtypes of a given type then there is no need to edit each library that contains a declaration of any such subtype. You can also specify that a specific level of support for reflection (a specific 'reflector') should be used with a specific declaration. You have to do this in some cases, e.g., if you wish to reflect on instances of `int` or other built-in classes, because you don't get to edit 'dart:core'. You would then associate that class with that reflector using a global quantifier like this one:

@GlobalQuantifyCapability(r'^dart.core.int$', reflector)

In short: If your workflow allows for using a closed world assumption then reflectable code generation can provide reflection for any declaration in your program.

Let's say I'm generating classes using tool.

This tool is not able to add metadata to generated classes.

What is wrong with using this tool?

There is nothing wrong.

But would the "reflectable" package be useful in this case?

I don't think so.

Maybe I'm wrong?

You'd run that tool T first, and then run the reflectable code generator. You would use quantification (a subtype quantifier, or a global quantifier, or one of the several other kinds) to request specific kinds of reflection support for specific declarations in generated libraries. You may also be able to make T generate code where the appropriate declarations have some metadata, and then you can tell reflectable to associate those declarations with a specific reflector:

@GlobalQuantifyMetaCapability(MyMetadata, reflector)

which will ensure that any declaration that has metadata which is an instance of `MyMetadata` or a subtype thereof gets the level of reflection support that `reflector` specifies.

Reflectable code generation does have some additional limitations, though: It is not possible to generate code in Dart that looks up the value of a type argument and makes that value available for the creation of new entities. For instance, you may have a variable of type `List<Widget>` whose value has dynamic type `List<SpecialWidget>`, but you can't use code-generation-based reflection to learn that the type argument is `SpecialWidget`, and you can't create a new `Map<String, SpecialWidget>` based on that fact. 

Another limitation is that generated code has no way to determine the details of the type of a given function object (for instance, we can't obtain a list of names of named arguments accepted by a given function). We do have support for reporting the named arguments and such when it comes to a declaration (so you reflect on a class and iterate over its instance methods and investigate their formal parameters: no problem), but we do not have support for introspection on a function object at runtime.

The general issue is that we'd need a few more primitives in order to support these kinds of reflection, that is, we'd need a specialized, low-level version of a tiny portion of 'dart:mirrors'. It's not obvious how costly that would be, in terms of code size and performance.

In summary, the picture is somewhat complex, but you certainly can't just say that reflectable can't handle a class unless it's possible to edit its declaration and add some metadata to it.

--
For more ways to connect visit https://dart.dev/community
---
You received this message because you are subscribed to the Google Groups "Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/a2e729b6-827e-4ba6-a81a-7a5e30c9339dn%40dartlang.org.

--

Erik Ernst  -  Google Danmark ApS

Skt Petri Passage 5, 2 sal, 1165 København K, Denmark

CVR no. 28866984

[Guyren Howe]

The main concern appears to be that Mirrors prevent tree shaking.

Then let’s fix that: support a file where we can declare globs for the files we want to be able to reflect on. Folks will usually actually only want reflection for some subset of their app’s classes anyway.

On Dec 16, 2020, 02:52 -0800, 'Michael Thomsen' via Dart Misc <mi...@dartlang.org>, wrote:

We're considering discontinuing the dart:mirrors library, but would like to get some feedback before we make a decision. For further details, and to give us feedback, please see this feedback issue:

https://github.com/dart-lang/sdk/issues/44489

Regards, Michael

--

For more ways to connect visit https://dart.dev/community
---
You received this message because you are subscribed to the Google Groups "Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/f66a16ff-0539-463a-9733-00296023058en%40dartlang.org.

[Jonathan Rezende]

How would you do that for external packages?
Don’t think there is slightest chance that people would abuse mirrors?

After a lot of thought I am against dart:mirrors or reflection at all.

[Guyren Howe]

The external packages could have their own reflections file.

And I don’t in any way understand the sentiment “people would abuse mirrors”. It’s up to the developer whether they want to use reflection. I don’t even know what it would mean to abuse them (use them unnecessarily, thus slowing down their app?), but if the dev wants to do whatever this abuse is, that surely is up to them.

--
For more ways to connect visit https://dart.dev/community
---
You received this message because you are subscribed to the Google Groups "Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/1E5EFCBF-5EC9-490C-9919-ADF3FE53C370%40jode.com.br.

[re:fi.64]

GraalVM does this, it's honestly useless and a complete pain to work with. The only it's remotely tractable is if you limit reflection use to classes you control, but over here you might as well then use reflectable annotations anyway.

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/5f575fb6-3e45-4bfb-8c61-c480d6a93dc6%40Spark.

[Guyren Howe]

How is it useless and a pain to work with?

[re:fi.64]

Trying to figure out what classes in third-party libraries need to have reflection enabled is basically a terrible game of whack-a-mole, where you try building, let something break, add it to the list, etc. In the end, I ran a script to whitelist every single class in the libraries that caused trouble...essentially defeating the whole point of having an allowlist of sorts.

[Chris Norman]

I'm sorry if this is a stupid question (I've not been following the issues here), but if dart:mirrors is discontinued, what will the point of annotations be? I thought that dart:mirrors was the only way to work with them, as they are unlike Python decorators.

Thanks in advance.

Take care,

Chris Norman

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/CAHMoK3VKvpyG0LG_Z8hN-zUPp8UnwEEG9NE9iwfvRJ6ZRedksA%40mail.gmail.com.

[Jonathan Rezende]

Source generation

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/CANjhqb8MDfQRU97KVrCFw_3AxCe4CicSsjpc0gdo2siqaeALEA%40mail.gmail.com.

[Brian Wilkerson]

Or, more generally: annotations will still be useful for all forms of static analysis, which includes the kind of analysis done by both code generators and the analyzer.

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/misc/66EF53F0-F159-45DA-9A44-E67E0B8630C8%40jode.com.br.