[early prototype] Elm-Data: a server loading code generator

[Max Goldstein]

Hi everyone,

I've been thinking for several months now about a useful abstraction for loading data from the server. It's painful to work with HTTP and JSON directly because you lack: caching, logic to not send a request already in flight again, error handling, and so on.

I came upon the idea that the client (that's you) will provide a type alias for each resource's attributes (and perhaps other crucial information), and I could generate code that would retrieve resources of those types. Generating code also allows the tool to create JSON encoders and decoders, a common pain point.

Each resource gets its own module with a standard set of functions. I've split out loading a resource (making a request if one is necessary) from getting a resource (obtaining it immediately from information available, if possible). Keeping track of what's been loaded requires an opaque Store model and StoreUpdate message.

I don't have this anywhere near usable yet; I haven't started working on the generating part of this at all. I first want to write the code I want to be generated by hand, and get feedback on that. Does this approach seem okay? Are there any problems you can envision? What you be good to work on next?

Have a look here:

https://github.com/mgold/elm-data/tree/master

[John Kelly]

I've done some work in this realm. See: https://github.com/john-kelly/elm-postgrest. The library is domain specific to a restful backend following the postgrest specification, however, there are a number of ideas that I think similar libraries could benefit from. 2 in particular that I think are worth looking at:

1. Handling of relationships (with user defined phantom types)
2. Building query and decoder simultaneously

In regards to code gen, as I see it, the only thing needing to be generated would be the Resource representation. Many people argue that decoders and encoders are the pain point, but I think with a good query builder API this pain is alleviated. I was unable to come up with a general API myself (hence the domain specific library I made), so I am interested in your efforts.

Would love to chat! I'm usually on slack + the sf meetup.

[Rupert Smith]

On Wednesday, January 18, 2017 at 7:03:37 AM UTC, Max Goldstein wrote:

I came upon the idea that the client (that's you) will provide a type alias for each resource's attributes (and perhaps other crucial information), and I could generate code that would retrieve resources of those types. Generating code also allows the tool to create JSON encoders and decoders, a common pain point.

I've been doing something very similar and it is proving very worthwhile, so I would encourage you to pursue this. In my case, I have a data model and a mapping of that onto a set of REST endpoints. The model that describes this is implemented Java and I use a (horrible) templating library called StringTemplate to do codegen from it.

For Elm, I output one (big) file called Model.elm, that gives me the data model mapped onto Elm, and encoders and decoders for it. Then for each grouping of endpoints (each service implemented in Java provides the grouping) I generate one Elm file that implements the Http logic for the service. I haven't open sourced this, but it might not be of much use to you anyway.

Each service requires a set of callback functions to be passed in - one for each endpoint, plus one for each endpoint when it results in an error, plus a default error handler for cases when the endpoint specific error handler doesn't process the error. The service also defines a set of convenience functions for triggering a call to the service. You nest the services update function inside yours and pass it the callbacks. Putting all the callback functions together in a record is a design choice I might re-think for something less monolithic. Anyway, here is an example of using it.

cseCallbacks : CSE.Callbacks Model Msg

cseCallbacks =

    let

        default =

            CSE.callbacks -- Default no-op callbacks so you only specify what you need.

    in

        { default

            | retrieveWithContainerBySlug = retrieveWithContainerBySlug

            , error = error

        }

update : Msg -> Model -> ( Model, Cmd Msg )

update action model =

    case (Debug.log "Client.State" action) of

        CSEApi action_ ->

            CSE.update cseCallbacks action_ model  -- Nesting the services update function.

        SelectLocation location ->

            ( model, CSE.invokeRetrieveWithContainerBySlug CSEApi location ) -- Invoking a service helper function.

Another possibility would be to use an API description meta-data such as Swagger. A Swagger codegen for Elm could easily output something useable.

Its definitely worth doing - the current project I am working on I have about 6K lines of Elm generated, all at practically zero effort (well, except the days and days I spent setting up the code generator, but that is effort I can re-use over and again). Project before that was about 4K lines. These were not huge APIs, maybe 4 or 5 entity types.

I am planning to move my codegen templates over from StringTemplate to Elm at some point. Now that I have server side rendering with Elm working. I defined a static Program type like this for it:

type alias StringProgram =

    Program Value String Never

https://github.com/rupertlssmith/elm-server-side-renderer/blob/master/src/ServerSide/Static.elm#L24

I think Elm will be well suited to writing code generators - in a manner similar to how it 'codegens' Html. You can design a set of functions for constructing the component parts of the output language you are working with. Elm is good for doing complex AST manipulation and it can be set up to guarantee correctly formed output. Code generation of Elm in Elm will be a mind bender.

[Max Goldstein]

John,

1. Handling of relationships (with user defined phantom types)

Elm does not support phantom types (to my understanding). What do you have in mind here? How can phantom types represent relationships?

In regards to code gen, as I see it, the only thing needing to be generated would be the Resource representation.

Interesting. I take it you are generating resource definitions from the Postgres schema. In my library, the resource's representation (as a type alias of a record of attributes) is the input to the code gen, the one thing that is not generated.

 

Many people argue that decoders and encoders are the pain point, but I think with a good query builder API this pain is alleviated. I was unable to come up with a general API myself (hence the domain specific library I made), so I am interested in your efforts.

I'm also limiting myself, specifically to JSON API, but that doesn't make assumptions about the database. And I think one might as well generate encoders and decoders, at least if not overridden, so that the code one must write by hand (those type aliases) is as minimal as possible.

 

Would love to chat! I'm usually on slack + the sf meetup.

I'm in San Jose, so hopefully I'll see you there in the next few months. Please introduce yourself and remind me of this email :) 

[Max Goldstein]

Rupert,

I've been doing something very similar and it is proving very worthwhile, so I would encourage you to pursue this.

Thanks (:

 

For Elm, I output one (big) file called Model.elm, that gives me the data model mapped onto Elm, and encoders and decoders for it. Then for each grouping of endpoints (each service implemented in Java provides the grouping) I generate one Elm file that implements the Http logic for the service.

I've come to the same file structure, because JSON API allows for included resources. For example, articles can include their author an vice versa, which means those decoders have to be defined in a common file. I'm trying to wrap them in one module per resource, though.

 

Each service requires a set of callback functions to be passed in - one for each endpoint, plus one for each endpoint when it results in an error, plus a default error handler for cases when the endpoint specific error handler doesn't process the error.

I'm taking a different approach. I'm using the RemoteData library to track the (possibly error) state of each ID'd resource, so if there's an error, it's just stored. You won't know until you try to get that ID and you wind up with the Failure case of the RemoteData type. I'm also planning on having an error log that's written to in opaque events and exposed read-only through an API. 

 

Another possibility would be to use an API description meta-data such as Swagger. A Swagger codegen for Elm could easily output something useable.

I've used both Swagger and JSON API before, and I'll say that JSON API has almost no downsides but Swagger is much more an engineering tradeoff. An Elm swagger codegen is something that should exist but I'm taking a different approach.

 

I think Elm will be well suited to writing code generators - in a manner similar to how it 'codegens' Html. You can design a set of functions for constructing the component parts of the output language you are working with. Elm is good for doing complex AST manipulation and it can be set up to guarantee correctly formed output. Code generation of Elm in Elm will be a mind bender.

I'm hesitant to rely on server-side rendering Elm when it's not officially supported (I say that as the co-author of Elm test and Richard's shell utility for it does exactly that, so maybe I'm not being consistent). I'm also a little less concerned with well-formed output, since elm-format can smooth over a lot of bumps in a hacked-together system. I'm more comfortable in Ruby but Noah Hall has written some JSON tools in Python that I'd like to build on if it makes sense to do so (he's left NRI so maybe not?).

Thanks again for your encouragement! 

[Andrew Radford]

FYI https://github.com/ahultgren/swagger-elm

still a work in progress though AFAIK.

[Rupert Smith]

On Thursday, January 19, 2017 at 5:18:33 AM UTC, Max Goldstein wrote:

For Elm, I output one (big) file called Model.elm, that gives me the data model mapped onto Elm, and encoders and decoders for it. Then for each grouping of endpoints (each service implemented in Java provides the grouping) I generate one Elm file that implements the Http logic for the service.

I've come to the same file structure, because JSON API allows for included resources. For example, articles can include their author an vice versa, which means those decoders have to be defined in a common file. I'm trying to wrap them in one module per resource, though.

In the general case one entity can reference another, and that other can reference the original (directly or via a chain of other references). In other words mutual recursion is possible. For that reason, I ended up wrapping all my records as single constructor union types.

A linked entity may be fetched or not. So I ended up using Maybes, except in situations where I know it will always be eagerly fetched.

These added complexity to the encoders/decoders but at least I got things into a form that would handle the most general cases that I needed to work with.

Another possibility is that for linked entities is that there are really 4 states; Nothing, meaning that the server has explicitly told me nothing is linked; Just a, meaning there is some linked entity and it has been fetched; Unknown, meaning that the server did not provide an explicit value so I don't know if something is linked or not; Ref id, meaning the server gave me the id of a linked entity, but did not fetch the actual entity. I couldn't face handling all of this, but might come back to it later. I use Nothing to mean really nothing or unknown and Just a to hold a linked entity or a linked entity in a degenerate form where just the id is present.

It would be nice if there was a simple mechanism to expand a Ref id into a Just entity, by which I mean a function I could call and pass in the reference and it will fetch it and then place the results back into a new data structure in the correct place. Then I could represent the data models as HAL or JSON-LD and then codegen a client in Elm for it that gives me a convenient tool to navigate around and fetch data.

[Rupert Smith]

On Thursday, January 19, 2017 at 4:10:06 PM UTC, Rupert Smith wrote:

Then I could represent the data models as HAL or JSON-LD

Or JSON-API, there are way too many emerging standards in this area... 

[Rupert Smith]

Actually, I want to try and keep the data representation in Elm and in the meta-model that codegen sources from abstract enough that it does not contain details of the specific wire format being used, or is particular to any single API description meta-language. Enough details that it can be mapped down to anything reasonable. If I manage that then I can always move to different formats as things evolve, or use Protocol Buffers to be more efficient, and so on.

[Noah Hall]

> I'm hesitant to rely on server-side rendering Elm when it's not officially supported (I say that as the co-author of Elm test and Richard's shell utility for it does exactly that, so maybe I'm not being consistent). I'm also a little less concerned with well-formed output, since elm-format can smooth over a lot of bumps in a hacked-together system. I'm more comfortable in Ruby but Noah Hall has written some JSON tools in Python that I'd like to build on if it makes sense to do so (he's left NRI so maybe not?).

I've left NRI, not vanished and therefore deleted every thing of note
I've ever done and banned everyone from using my things. Please use
it. It's not just in Python, there is a Elm version (with native) and
a node version (just js).

> --
> You received this message because you are subscribed to the Google Groups
> "Elm Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elm-discuss...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[John Kelly]

Max, 

Elm does not support phantom types (to my understanding). What do you have in mind here? How can phantom types represent relationships?

Based on my understanding/definition of Phantom Types, they are very much possible in Elm. An explanation + conversation over this medium is likely not going to be effective, so I'll just provide 2 links for those interested in how it is that I am making use of this idea. library relation type and sample usage

Interesting. I take it you are generating resource definitions from the Postgres schema. In my library, the resource's representation (as a type alias of a record of attributes) is the input to the code gen, the one thing that is not generated.

If I were to be using codegen, yes I would be using the schema of the db table or view. More importantly though, I think there is a slight confusion in what it is that I mean by Resource Representation. I don't claim that my naming is good, but what i mean by Resource Representation is a representation of the shape of the backend/api server on the frontend. Maybe it would be better to refer to this as a Schema? see here for what a "Resource Representation" or "Schema" looks like in the library. This representation serves as the single point in the application where the shape of backend/json api server lives. The representation contains all information related to fields, types of those fields, and relations. As you'll see if you dig into the library, this is how we can build up decoders while building up queries. 

In the library, the user must still provide how they wish the data received from the server to take form in elm. Whether that is a record or some union type. I think that taking a look at the hello world example of the library will make this point clear.

And I think one might as well generate encoders and decoders, at least if not overridden, so that the code one must write by hand (those type aliases) is as minimal as possible.

The api of the library allows one to build up the decoders and encoders while building up the query. In postgREST (and the more popular graphql), the user specifies what fields they would like to select from the backend. The idea of the library's api is: "Why not build up the decoder while the user is specifying what fields to select?" The user is not using decode pipeline or the standard json decoder api; the library abstracts this away and does it for you!

Anyways, I see now that explanation of this is a bit tricky via email! Especially when definitions for things are so fuzzy! If anyone is interested in chatting about this more -- feel free to pm me on slack @johnkelly