Feature Request: Code Generation/ Macro System

[Isaac Shapira]

Continuing the discussion started here:

https://github.com/elm-lang/elm-compiler/issues/1413

[Isaac Shapira]

@mgold If you are suggesting that tedious to write elm code should be addressed by a distributed series of independent browser based online code generators, I'm going to say that's not a real solution. I don't want to go to a different set of browser bookmarks for different common derivable code scenarios. Nor do I want code generation outside of my build process.

If you are suggesting using things like json-to-elm inside of a build process, then we are potentially in some kind of module loader hell. json-to-elm looks like it uses python for generation, so now python is in the stack, and there is no consistent standard for how these distributed set of tools should work. If we wish to address 10+ boilerplate heavy scenarios, we now have to custom install and rig together disparate apis of these different generation tool into our build. And once all that is working, essentially we just invented fraken macros, where macro code is separate outside of .elm files.

A proper macro system seems like the only solution to me atm. That's not to say there is only one way of going about it.

One way that might address the "One Language" and maintainability concern, is to have macros that are outside of Elm. As in elm-make provides a facility to pipe code into an external process for expansion. Then type checks the expanded code. This could also mean that elm-package now needs the ability to let code generator authors package an executable with any surrounding elm files. I feel this could also discourage the abuse that can come from macros, as there are more barriers to set up the external code gen process and wire it in.

[Joey Eremondi]

Some things to consider:

Templating and Macros are Separate Issues

We can decide that we want to allow for macros for brief code generation, without making a separate templating language.

I'm with Max on the sentiment that having to learn a new language.

Macros *need* to be type safe, but this is possible

Rust macros are probably a good example of prior art, but they admit that they increase the complexity and reduce the readability of the code.

This is also a big problem with Template Haskell, when it fails, it's *really* hard to track down why it failed, because you have to mentally typecheck generated code.

We can't let arbitrary code be run at compile time

... because of this security issue where someone squatted on common typos of package names and gained access to machines by running malicious code during the install.

This means that we probably don't want to allow for any IO to happen in macros. Thankfully this is easy to enforce in Elm, but it's a design decision that is needed.

Even then, this opens up the possibility that compilation won't terminate, or will be really slow, in a way that the programmer can't control or understand.

--
You received this message because you are subscribed to the Google Groups "Elm Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Maxime Dantec]

Jumping in ! I said in the github issue that I was in favour of macros, but macros imply hacking the (haskell) core of the compiler. Wouldn't preprocessor be a better description ?

[Isaac Shapira]

@Maxime Call it what you will. Pre-processor, Macro, Template, they have different meanings but solve the same problem, generate code rather than write tedious code when possible. Regardless it going to mean some edits to the Haskell side, since it needs some kind of compile time support, as well as package management support.  

@Joey I understand the concern, but I think its worth the risk. Thank and typo-squatting seems like it could be resolved another way. elm-package could require package names to be at a certain "distance" from one another. That and since user names are required in the package name, the risk is much lower.

[Joey Eremondi]

elm-package could require package names to be at a certain "distance" from one another

That still doesn't resolve the problem of running arbitrary code. If our community gets big enough that I don't know and trust every author from this mailing list, we need a way to stop people from being malicious.

Elm is about reliability. We're trying to be better than JavaScript, so the security needs to be airtight.

Usernames increase the risk, I'd say. Someone could make evanzc/elm-graphics, and the typo is a lot harder to spot at first glance. And by your method, if evanzc was a legitimate elm developer, they'd need to make a whole new github account just to make a package.

The name collisions aren't the problem. Running arbitrary untrusted code at compile-time is. We can avoid this, but it needs to be central to the design, not an afterthought.

--

[Maxime Dantec]

On Saturday, June 11, 2016 at 10:29:05 PM UTC+2, Joey Eremondi wrote:

elm-package could require package names to be at a certain "distance" from one another

That still doesn't resolve the problem of running arbitrary code. If our community gets big enough that I don't know and trust every author from this mailing list, we need a way to stop people from being malicious.

Ok, this is a good enough reason not to have it in the core, but this is not a reason not to do it at all.

Plus, you assume that all elm code is only ment to be running in the browser. But it already run on node with tests or in the repl, and I'm expecting a lot of people starting to release node-elm-packages soon. It'll just fell into the whitelist processus, as it always should when it comes to native code.

[John Mayer]

Do you have a deadline? Ok. Then write a little external code generator, or fork the compiler and make your own technical decisions without any expectation that it will get merged into upstream.

Now, are you simply trying to improve the language? You really want some kind of macro system merged into upstream? Great. Realistically, how this is going to play out:

Build out a taxonomy of macro systems in, like, 10 major languages, maintain some kind of matrix of pros and cons of different kinds of macro systems, maybe come up with some stuff that no language does. Do the research. Crucially, make sure that "no macro system" is also on that list; try to open-mindedly come up with reasons why not to have a macro system at all. Get people to help you with this research. Collect and organize everything into one place for consumption.

Present that to the Elm maintainers (Evan, obviously) and share it on the mailing list.

And then wait, because (based on his track record) Evan is going to mull it over for a while. IMO, he gets that privilege because he's our BDFL. Throughout this process Evan's going to come down with a set of principles that are consistent with the rest of the Greater Goals of Elm. The choice of action will derive from those principles.

Obviously people want this. People want lots of things. Making it easy for Evan maximizes the chances of something happening.

--

[Maxime Dantec]

I also think that it should not be in the core. And I'd argue, that this thread is about polling the community about the idea :)

I have a tiny beginning of an Elm parser written using https://github.com/Bogdanp/elm-combine (awesome lib!!) that I could push once it compiles (^^).

[Aaron VonderHaar]

If someone created a macro system, it would be interesting to see what could be done with it.  But I think that would be extremely experimental.  I'm not convinced that having a macro system would lead to good solutions for the things it could be used to solve.

Specific to this conversation, there were three features mentioned in the original github issue:

 - make it easy to generate quickcheck producers

 - make `elmx` a core language feature

 - reduce boilerplate in `update` functions

 - other boilerplate scenearios

I don't think a macro system is going to be a great solution for any of those things.  (For quickcheck producers, I think having quickcheck automatically do that via native code, or having a general API for data structure reflection would probably be better.  For `elmx`, I personally don't think it should be in the core language.  For `update` boilerplate, I think a good solution will need to be part of the core framework and shouldn't depend on macros even if they existed.  And for the supposedly large number of other boilerplate scenarios, let's take a look at them and see what they have in common that might be able to be solved more simply than by having a macro language.)

In general, if someone wants to build an experimental macros system just to see what might come out of it, I think it would be great to see how that goes.  But if we are talking about specific problems, we should focus on trying to solve those problems rather than assuming that macros would solve those problems.

[Isaac Shapira]

Fair enough. However I still think "distance" could solve the typo problem. As in evanzc/elm-graphics would not be allowed as its too "close" to an existing package name.

[Isaac Shapira]

Let me make the scenarios I mentioned more clear. I'm not advocating for a macro language, I'm advocating for a means of doing code generation that is consistent and maintainable. Producers could be solved with code generation, elmx could be replaced with code generation (I don't think it should be core language), boilerplate in update functions could be replaced with code generation (and I don't see another path here, since it involves pattern matching, see original example).

[Noah Hall]

FWIW I've already turned json-to-elm into kind-of-macros for my own
personal usage along with a couple of other things.

I don't think that elmx is a compelling use case though. I think it's
best to leave that stuff to the react people. Things like decoders,
encoders, reducing boilerplate, generating ord and non-default
toString are good use cases of where Elm the language is missing
things _right now_, and therefore it makes sense to implement "macros"
or just editor plugins to generate that code for you. I don't think
they're compelling use cases for macros as part of Elm itself.

[Maxime Dantec]

or just editor plugins to generate that code for you. I don't think
they're compelling use cases for macros as part of Elm itself.

I agree. For example: as soon as you are generating union types, you are probably doing it wrong.

[Gage Peterson]

Macro systems add tons of power to a language. That power can be used to create really simple code but more often then not it will be used to create horrible code. I think this is perhaps my least favorite thing about Haskell. I feel like ML family languages are already pretty terse as they are and don't need much help to make them "prettier". One of the best things about Elm is that it routinely rejects power for simplicity.  I'm personally very happy with the current HTML syntax and see no benefit in that area. If some sort of syntax becomes widely useful (which is rare, as it should be) then it should be added to the language. I think a 3rd party thing akin to http://sweetjs.org/ would be a good playground to play with new syntax, however this shouldn't be widely used as a good idea. 

On Saturday, June 11, 2016 at 1:26:55 PM UTC-6, Isaac Shapira wrote:

[Zinggi]

I think the only thing that could really benefit from a macro system in Elm would be a deriving clause similar to Haskell.

This way we could get Json decoders for free:

type alias Model = { foo: String, bar: Int } deriving Json.Decoder

Which would generate modelDecoder : Json.Decoder Model

This way we could add more stuff after deriving for different scenarios.

[Gage Peterson]

Yes, that would be nice Zinggi

--
You received this message because you are subscribed to a topic in the Google Groups "Elm Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elm-discuss/im2up0f59qs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elm-discuss...@googlegroups.com.

[Tim Stewart]

At one stage there was suggestion of the compiler being able to derive JSON en/decoders automatically. This would be great - especially coming from JS-land where it's just JSON.parse(x) and JSON.stringify(y). I find the elm way of decoding JSON still a bit hard to grok. Defining a record "template" and having the encoder/decoders auto-generated at compile time would be ideal. I know someone made an external tool for this, I have yet to try, but baking it in to the core tooling and not having to even see deal with the generated code would be great.

[Ian Mackenzie]

Just to throw my two cents in, I'd want to be pretty conservative about any macro/code gen system if there is an official one at all. One of the things I love about Elm is that the restrictive nature of the language makes code very easy to follow and understand, and naturally causes libraries to follow similar patterns (things like JSON decoders, random generators and elm-check producers all follow very similar patterns largely because the constraints of the language mean that there's really only one good way to do it).

I'm worried that a macro system will let people get a bit "too creative" with their code and we'll end up with indecipherable stuff like injection-based HTTP servers with endpoints and parameter names magically deduced from function signatures. Like how Evan has said that the Elm Architecture "seems to emerge naturally in Elm", I think the current language structure does a great job of encouraging good design, and I'm worried that a macro system could hurt that.

[Max Goldstein]

I think Ian makes some great points and Iargely agree with him. I like that Elm is actually less to learn than React, Ember, etc, once you factor in ES2015 and JSX/handlebars and all the other stuff.

However, automatically derived JSON decoders are something I could potentially support. They seem to crop up a lot, and in application rather than library code (i.e. Random generators and Producers can hide the boilerplate behind a library but JSON can't be similarly carpeted over). There's also some precedent here with type alias of record types. I think a minimally obtrusive proposal would be no extra syntax, just some extra functions defined implicitly by a type alias. Example:

type alias User = { name : String, id : Int }

defines the type User and also the function User : String -> Int -> User. It's not a horrible idea to have it also define encodeUser : User -> Json.Value and decodeUser : Decoder User. If the type alias isn't JSON serializable, then nothing happens. If the programmer defines (en|de)codeUser herself, then the generated versions disappear.

[Peter Damoc]

It's not a horrible idea to have encodeUser/decodeUser but why not go one step further and have an internal typeclass like serializable that would only be used in the Json libraries. Also, pick a standard encoding for the UnionTypes so they can become serializable as well.  This will allow for compiler time errors if you try to serialize something that is not serializable and would simplify the handling of Json tremendously and would get rid of one of the silliest parts of Elm.  
 

--

There is NO FATE, we are the creators.
blog: http://damoc.ro/

[Tim Stewart]

Yeah just to be clear to Ian, I absolutely agree that a general macro system would be more trouble than it would be worth. It has a place in some languages but would take away from the simplicity that Elm promises - and increasingly delivers on. 0.17 was a big increment in that sense by making interfacing with JS and the external world in general more approachable.

I really was just hopping on the observation about JSON serialization and pointing out it has been talked about. If a future release could alleviate the currently somewhat cumbersome way of marshalling between JSON and Elm, that would be another great increment. Something along the lines that Max and Peter suggest would be awesome.

I am impressed by Elm's conservative approach in not granting developers (or DSL authors who might want macros) everything they might want, whilst still giving them what they need. In this case, not getting macros but getting compiler generated serialization would be a great outcome.

[Ian Mackenzie]

Tim, I'm glad we're in agreement, but I confess I wasn't specifically replying to your message - just wanted to generally chime in to this discussion =)

I'm personally indifferent as to whether or not there should be some sort of automatic JSON functionality. I'm sure it could be done in a clean and focused way that wouldn't detract from anything else, but so far I haven't been bothered by the need to write explicit encoders/decoders (and I kind of like how it forces you to be very explicit about exactly what kind of JSON you create and accept).

[杨博]

I thought a macro system should be a foundation of an extensible language. For example, Binding.scala on Scala.js implemented a reactive web framework based on some monads, then, Binding.scala provides a @dom macro that converts simple plain expressions into complex higher kinded monadic expressions.

Porting such a similar macro system for elm would give us the ability to separate underlying type classes and higher level syntaxes.

在 2016年6月12日星期日 UTC+8上午3:26:55，Isaac Shapira写道：

[Pablo Parga]

Hi!,

I've been playing with elm-combine to auto generate things and elm-format really make things easy, so I think that instead of having macros once we have AST working those things are going to be really easy to generate with an IDE. Anyone knows what is the current status of AST?