Elm Community Tooling 1 - Coordinating AST efforts

[Magnus Rundberget]

Hi !

I'm not 100% sure if this qualifies as "Work on Elm". If it isn't I'm sorry.


(Community ?) Elm AST

TLDR; It seems evident to me there is real and clear desire from the Elm community to have an accessible Elm AST for various use cases. Maybe efforts to make one could be coordinated, and if so it could communicated that such an effort is going on ? Maybe the the work done (and/or going on) in elm-format to create an AST for external consumption is just that and people interested and able should help contribute to that ?


Background:

Aaron through elm-format has done some really nice work with creating an AST. He's mentioned it shouldn't be too far off making that AST exposed as JSON and made available for tools (or whatever really that might find an Elm AST useful) to use. Relevant issue in elm-format : https://github.com/avh4/elm-format/issues/194

Pete has done some pretty exciting stuff with https://github.com/vilterp/elm-tracing-interpreter. It seems he's modified the elm-compiler and elm-make to do his bidding and provide a JSON AST :-)

I've gone ahead and created a very crude AST generating parser for use in my elm-light plugin for Light Table to gain some experience and ideas for what an AST might give me as an editor hacker. In Atoms elmjutsu (https://github.com/halohalospecial/atom-elmjutsu) I believe a regex based approach to enable cool and useful editor features   has been implemented to scratch a similar itch.

I've also seen various people on the Slack Channel talking about an Elm AST for various purposes.


Doing it right ?
----------------
While it might be useful with parallell efforts going on, it might also be some merit in an coordinated high-quality effort !?

Even "just" an AST like the one in elm-format made available for external consumption would probably enable a range of cool stuff to happen. An AST enriched with information from the elm-compiler would obviously be even more awesome.

Although I'm eager to get something yesterday (so I went ahead and did my own thing) I realize that a community effort would obviously require more care and thought.


What's next ?
-------------
I don't know! That's why I'm making this post. Maybe it's all taken care of and I should just keep quiet and wait for it!  I just wanted to get some feedback and if possible contribute beyond satisfying my own editor hacking desires.

cheers
-magnus

[Max Goldstein]

Just a quick note, there's a huge advantage to elm-format implementing its own parser and AST instead of relying on the compiler: it can format code that is not type correct.

Don't expect it to happen overnight, but some coordination to produce better tooling sounds like a great idea to me.

[Daniel Bachler]

I think a reusable AST would be great.

Maybe this could even run as a long running service that can
optionally watch files. AFAIK purescript's psc-ide-server does
something along these lines
(https://github.com/purescript/purescript/tree/master/psc-ide-server).

One thing that I could see being built on top of this is generating
json encoders/decoders from elm data definitions. This is something
that many beginners might be very happy about.
--
Daniel Bachler
http://www.danielbachler.de

On 21 September 2016 at 15:38, Max Goldstein <maxgol...@gmail.com> wrote:
> Just a quick note, there's a huge advantage to elm-format implementing its own parser and AST instead of relying on the compiler: it can format code that is not type correct.
>
> Don't expect it to happen overnight, but some coordination to produce better tooling sounds like a great idea to me.
>

> --
> You received this message because you are subscribed to the Google Groups "elm-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to elm-dev+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/elm-dev/01513d71-fcdc-4d73-8260-a86b61f2f519%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Evan Czaplicki]

Context

It definitely makes sense to have an AST that folks can mess with. The obvious usage is for editor plugins, but anyone who is creating an interpreter for Elm programs (like Pete) would want it to. Both Ravi (who works on sketch-n-sketch) and I would like to do some stuff with creative interpreters for Elm programs. In Ravi's case, he also wants to preserve comments (much like editor folks).

Current Plan

After talking with Ravi and Aaron last week after elm-conf, we came to this rough plan:

1. elm-format can go from .elm file to an AST with some JSON representation.
2. elm-format can take in an AST in that JSON format and turn it into an .elm file.

From there:

• Editor plugin authors can implement rename refactoring that preserves comments
• Ravi can be doing whatever crazy edits he wants

Now it seems plausible to allow folks to label where the cursor is on the AST, and elm-format could tell you where it should be after everything is formatted.

Okay, so that would be phase one. After that JSON format for an Elm program exists, we could make it so the compiler accepts this format as well. The idea would be that editor plugin authors could give the format, the compiler could do all the checks and possibly add some type information, and then hand back an augmented version of the AST.

So say the user has just typed something like this:

list

  |> List.filter .completed

  |> <CURSOR>

The editor could call elm-format to get the AST and replace the unknown part with some "fill this in" JSON node. Then the compiler can just figure out the type of what must go there. Then it's up to the editor plugin to make suggestions.

That's the general idea of things. A lot of this could change as we learn more by doing this work gradually and seeing what works well.

Advice

I'd say the best thing to do is to wait until elm-format does this well. More people is not necessarily better. If you want to help, the best thing you can do are things like:

1. Do a literature review to find what the AST looks like for other languages. Organize that information in a nice way.
2. Write about the specific things you would like to be able to do. What do you want? What do people ask for most?
3. Maybe draft an AST format that seems like it would work well for you? This may be helpful, but don't feel like we are just going to go with the first thing anyone suggests.

Doing any of these things poorly is probably not so helpful, but if we could get a high quality "literature review" that'd be awesome!

I think it's also important not to rush this. Both Aaron and I had a sense that if this got released in a alpha form, it would be in editor plugins really quickly. Please do not do this! It is way more important to get the AST format nice than to get it into editors as soon as possible. This format will likely be used in many different ways, so the fact that we are thinking and working on this should not imply that it is done and ready to be in everyone's editor.

[Aaron VonderHaar]

Yeah, writing the code to implement this is the easy part :)  The hard part is making sure the AST is easy to work with, not unnecessarily complicated, and flexible enough for expected uses.

If anyone wants to do literature review, you can join this thread: https://github.com/avh4/elm-format/issues/194

If anyone has a project that could make use of an AST, you can add yourself to the list here: https://github.com/avh4/elm-format/issues/236

If anyone wants to write up details of how their project could use such an AST and/or write up some samples of what those ASTs might look like in JSON form, it would be helpful either to share that here on elm-dev, and/or submit it as a github issue on the elm-format repo  https://github.com/avh4/elm-format/issues/new

In my opinion, the most valuable thing for others to do at the moment is to share detailed examples of how their project would like to use an AST.  (A good description would include a short summary of the scenario of what the user is trying to accomplish, along with the example Elm code in the scenario, and how your tool or project would be using the AST, and an example of what the relevant parts of the AST might look like.)

--

You received this message because you are subscribed to the Google Groups "elm-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elm-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elm-dev/CAF7GuPGGJbovH9-b9tv89QtjJtY2g0jtzLTmqbFh07QOc1A0jA%40mail.gmail.com.

[Dave Thomas (7sharp9)]

Is the AST mentioned here merely for parsing and understanding the Elm code in terms of editors or would there be scope for manipulating the AST too?

[Aaron VonderHaar]

elm-format would also read an AST and produce Elm source code, so transformations would be possible.

On Sep 24, 2016 11:06 AM, "Dave Thomas (7sharp9)" <kukulca...@gmail.com> wrote:

Is the AST mentioned here merely for parsing and understanding the Elm code in terms of editors or would there be scope for manipulating the AST too?

--

You received this message because you are subscribed to the Google Groups "elm-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elm-dev/25b0c747-e911-4572-94d9-936de5ff64f0%40googlegroups.com.

[Richard Feldman]

I'm not sure if this is the right place to put this, as it has to do with modules and not just ASTs, but here is a concrete use case I have for something along these lines in elm-test:

Status Quo

The only way to run a test is to have a port module containing a Port which sends the tests out. This means you have to combine all your tests manually and then send them to the port.

Problems with the Status Quo

1. Every time you write a new test, you have to add it to the "master list" of tests. That's slightly annoying.
2. If you edit some production code, e.g. SearchBar.elm, elm-test in watch mode (not yet implemented) can only do one thing: rerun the entire test suite, however long that may take. This is because elm-test has no knowledge of which tests depend on SearchBar.elm and which ones are unaffected by changes made to it.

What We'd Prefer

1. When you run elm-test it searches for .elm files in a certain directory which expose values of type Test. It runs all of those tests. No more "master list" to maintain, and you can easily run a subset of tests by pointing the elm-test CLI at a particular file—not just at the "master file" like today.
2. If you edit some production code, e.g. SearchBar.elm, elm-test in watch mode can:
1. Know which values SearchBar.elm exposes
2. Know which values each test module imports
   
3. Have stored all of these values from the previous run, including last modified timestamps
4. Detect that SearchBar.elm changed, and re-run only those test modules which depend on values it exposes
   

In an ideal world, it could know on a per-test basis (as in a value of type Test) how these dependencies worked, but that seems way harder (given closures and such) than doing it all at the module level. :)

[Zachary Kessin]

It seems to me that with an AST you could know what code each test runs and therefor could know this change only should affect this list of tests and run those (or at least run them first)

It is also possible that we could build something like a concolic testing tool like Cuter for erlang (See this talk by Kostis) http://www.erlang-factory.com/sfbay2016/kostis-sagonas as well as code coverage and even branch coverage in testing

Zach

ᐧ

--

You received this message because you are subscribed to the Google Groups "elm-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elm-dev/2cd8ec69-2edb-41cf-8891-75f9e8799a0a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Zach Kessin

SquareTarget

Twitter: @zkessin

Skype: zachkessin

[Francesco Orsenigo]

I would like to be able to automatically generate encoders/decoders for any defined type.

Right now, it is necessary to manually define JSON decoders for every type that needs to be serialized.
If the type changes, the decoder needs to be updated as well, usually getting in the way of compilation.

Using the AST for a type, I could automate any generator pattern on that specific type, removing repetitive manual work.
Besides JSON decoding, it could be used with Random generators; it would also make it very easy to automatically create generators for test data.

[Zachary Kessin]

Both of those would be useful

Zach

--
You received this message because you are subscribed to the Google Groups "elm-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elm-dev/71bfedbf-1b71-40cf-a6cf-a2322400ad32%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Magnus Rundberget]

Aaron,

I've submitted 5 new issues with examples of features it would be sweet that the AST supports. These are based on features I have working in elm-light already today.
- Completions in module declaration: https://github.com/avh4/elm-format/issues/239
- Import completions: https://github.com/avh4/elm-format/issues/240
- Completions in exposing node of import declarations: https://github.com/avh4/elm-format/issues/241
- Expose/unexpose: https://github.com/avh4/elm-format/issues/242
- Quick import: https://github.com/avh4/elm-format/issues/243


I can think of many more that'd be cool, but I'm reluctant to spam your repo if none of these are helpful (or if this is sufficient for giving input to the design of the AST format).

cheers
-magnus

To unsubscribe from this group and stop receiving emails from it, send an email to elm-dev+u...@googlegroups.com.

[Ulric Wilfred]

I think there are two different tasks and both are needed by IDEs.

• First is the AST tree and/or grammar, to know the type of the expression;
  
• Second is the tool to determine types of expressions very fast, like elm-repl or elm-oracle does, but without the requirement of internet connection and allowing to get types for several expressions with a single elm-make cycle;

Why these issues are different? In some cases, plugin only needs to know types (for type suggestion or for even cooler things like live-debugging/playgrounds for every expression) and currently it would also require programmer to parse user code with AST, which could be time-consuming. 

It would be very nice if programmer may ask some tool provided with Elm for a type of expression.

Currently, I was required to write binary parser for .elmi files and actually repeat what elm-repl does internally, but in node.js, since calling elm-repl from node.js is very slow and this approach works ~50-100ms for expression.

[Ulric Wilfred]

Sorry, this could look unrelated to the topic from a first sight, so I need to clarify some things:

We had a discussions in Slack regarding the topic "what people, developing Elm plugins for IDEs, would need from AST and along with AST?".

Surely, AST (and, currently, elm-format) is required for formatting Elm code, so I need to fix this:

> First is the AST tree and/or grammar, to know the type of the expression;

with this:

> First is the AST tree and/or grammar, to format code properly and to know the type of the expression;

During the discussion, we discovered that people who already try to develop IDE plugins, along with AST, have the needs I've mentioned previously and probably it's better to keep them here.

Also, for the sake of coordination, a link to the mentioned elmi-parser in node.js: https://github.com/shamansir/node-elm-repl