Metalinguistic abstractions over databases

[Peter Damoc]

The problem 

I know how to implement a REST API and interrogate that API from Elm but that seams very low level. 

I was curious about a language abstraction that would isolate my Elm program from the actual database strategy.  

How do you approach dealing with the databases? What is your strategy? 

What options are there?
Do you have any pointers to interesting perspective/approaches? 

--

There is NO FATE, we are the creators.
blog: http://damoc.ro/

[John Orford]

like graphql?

--
You received this message because you are subscribed to the Google Groups "Elm Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Damoc]

GraphQL could be part of the answer. 

I would love to see an answer that includes GraphQL being explored. 

Ideally, I would love to see it explored in a more complex context where the data set is very large. 

e.g. some kind of application over the data from MusicBrainz. 
Let's say that I want to have a bunch of views over a subset of that data (Artists, Albums, Genres).
How would one handle relations in Elm types?
How would one handle performance considerations in a nice way (issues of pagination, delayed loading for large sets, parallel loading of resources, cancelling of ongoing transactions)
 

like graphql?

To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Elm Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Rupert Smith]

On Wednesday, October 19, 2016 at 9:54:44 AM UTC+1, Peter Damoc wrote:

The problem 

I know how to implement a REST API and interrogate that API from Elm but that seams very low level. 

I was curious about a language abstraction that would isolate my Elm program from the actual database strategy.  

How do you approach dealing with the databases? What is your strategy? 

What options are there?
Do you have any pointers to interesting perspective/approaches?

Interesting that you jump straight from 'REST API' to 'database'. You can map database CRUD operations straight onto REST POST, GET, UPDATE, DELETE operations, it is true. Many APIs even if they do not expose all of CRUD through REST in this way, often have a trace of it in the API since the mapping is an obvious one. However, there is more to APIs than CRUD, and in many situations CRUD is not what you want for an API at all.

Here are some examples to illustrate.

Suppose I have an API that lets a user create a profile, modify their profile as they see fit, use their profile to access some applications, and if they choose to, delete their profile permanently. This fits naturally to a CRUD mapping of the profile.

Suppose I have an API for a bank account. The user cannot simply make changes to their account, they must always do so by requesting balancing transactions. So take some money from my account, and put it in another account. This does not map to CRUD on accounts at all, the API will describe a carefully chosen set of transactions that the user can fill in and request to have processed.

CRUD on single tables does not necessarily make a good API, as tables can be too small a fragment to deal with at the API level. Suppose in my profile example, that the profile has a list of email addresses in it, one of which is marked 'active'. There will be at least 1 profile table with a 1-to-many to the email addresses table. When I fetch the profile, I likely also want to fetch all the emails at the same time. The profile is what I call a 'slice' of data and it corresponds to a little document model. You could equally well store this document model in a non-relational database as json, say on MongoDB for example.

This is often how I think when designing APIs - what are the slices that the consumer needs to be able to work with to get the level of abstraction right. Having designed a data model and considered its slicing, I produce a CRUD API (+ finders) but at the level of abstraction of the slices. Later on, I then tidy this up by removing operations that should not be exposed in the API, say I don't want the delete operation for example. Or sometimes I remove the whole CRUD API for a certain class of entities, like the bank accoutn example, as a more transactional style API will be used instead.

'graphql' looks very interesting. I had a wee look after the recend post about an Elm client for it. Remember you will need a graphql implementation on the server though. And that for some situations, graphql might not be appropriate. As in the bank account example, you don't necessarily want the consumer to be able to freely navigate the whole graph of your data model, certain parts might require more restricted access.

Actually, I was thinking I'd like to look into graphql more deeply at some point. It essentially provides a way for the consumer of an API to dynamically specify the 'slice' that they want. It could also be used purely on the server side as a way of defining slices but exposing them as particular end points for a particular slice. Like:

GET /api/profile/<id>/summary - Gets just some of the profile data.

GET /api/profile/<id> - Get the full profile

Would be easy to do, if my server code let me set up these end points by writing them as graphql queries and automatically map that onto whatever underlying database technology I am using. I have no idea where we are at for good server side libraries for doing this...

[OvermindDL1]

I've been using absinthe on the server side and I am able to control access permissions quite well for GraphQL hosting, though I am using a custom permission library that ties to our local LDAP server so that code would not be generically useful to release.

But yes, I am a fan of GraphQL.  I just expose 'functionality', not raw tables.  Many of my GraphQL calls do not map to a database table but rather potentially multiple (or even none in some cases).

[Rupert Smith]

On Wednesday, October 19, 2016 at 4:39:00 PM UTC+1, OvermindDL1 wrote:

I've been using absinthe on the server side and I am able to control access permissions quite well for GraphQL hosting, though I am using a custom permission library that ties to our local LDAP server so that code would not be generically useful to release.

Sounds interesting. Is hooking custom permissions into graphql part of the graphql spec? Or is it some mechanism specific to absinthe that lets you do this?

Also, is graphql just for fetching data? Or it also allows you to create new data on the server, or make updates to existing data?

 

But yes, I am a fan of GraphQL.  I just expose 'functionality', not raw tables.  Many of my GraphQL calls do not map to a database table but rather potentially multiple (or even none in some cases).

Yes, I also do not expose raw tables in the API, each entity is typically a small document tree which maps to >1 table. Also depending on what you fetch and what condition you use to filter the query results I would expect hitting >1 table to be quite common.

[OvermindDL1]

On Wednesday, October 19, 2016 at 9:58:07 AM UTC-6, Rupert Smith wrote:

On Wednesday, October 19, 2016 at 4:39:00 PM UTC+1, OvermindDL1 wrote:

I've been using absinthe on the server side and I am able to control access permissions quite well for GraphQL hosting, though I am using a custom permission library that ties to our local LDAP server so that code would not be generically useful to release.

Sounds interesting. Is hooking custom permissions into graphql part of the graphql spec? Or is it some mechanism specific to absinthe that lets you do this?

The Elixir language (which Absinthe is built for and of which I use on the back-end) has a very powerful hygienic macro system.  With absinthe you define (taking these examples from the manual examples) a 'schema', which is an endpoint that can be queried, a specific graphql command in other words.  An example would be this:

```elixir

@desc "An item"
object :item do
  field :id, :id
  field :name, :string
end

 ```

This as it is does nothing as no action ('query') bound to it yet, this is just the data structure and the types (of which you can define your own types, use other objects, etc...), but we can bind such a query like:

```elixir

  query do
    field :item, :item do
      arg :id, non_null(:id)
      resolve fn %{id: item_id}, _ ->
        {:ok, @items[item_id]}
      end
    end
  end

```

This defines a query that lets a client query by 'id' and only an 'id', which must not be null.  The resolve function in this case is what will do the actual database look up (in this example it is just returning a static data element or 'nil' if it does not exist).  This resolve function is where I put my access control.  My access control works by adding this kind of thing into where-ever I want to enforce a permission (which I can make as fine or coarse-grained as I want, the permission system is very finely grained):

```elixir

@perm true = user |> can?(select(%Perms.WhateverPerm{id: id}))

```

Where user would be passed in the 'context' of the map passed in to the resolve function (detailed at http://absinthe-graphql.org/guides/context/ for note).

Absinthe handles all the nasty parts of GraphQL though, the combining of queries, the real-time type documentation generation, etc... etc...

On Wednesday, October 19, 2016 at 9:58:07 AM UTC-6, Rupert Smith wrote: 

Also, is graphql just for fetching data? Or it also allows you to create new data on the server, or make updates to existing data?

Nope, you can query for (and only get back what you request, like if an 'object' has 3 fields but the client only asks for 1 of them, you get that information in the resolver so you can optimize the DB queries accordingly), you can insert new things (if they fulfill your requirements/authorization/etc...), update things, delete things, etc...  Or anything else.  Technically a GraphQL 'command' is just like a function call where the arguments are well typed and defined and the return value is configurable based on the input, it is basically just RPC but a bit more optimized and self-documenting, and as such the 'functions'/queries can do whatever you want.

On Wednesday, October 19, 2016 at 9:58:07 AM UTC-6, Rupert Smith wrote:  

But yes, I am a fan of GraphQL.  I just expose 'functionality', not raw tables.  Many of my GraphQL calls do not map to a database table but rather potentially multiple (or even none in some cases).

Yes, I also do not expose raw tables in the API, each entity is typically a small document tree which maps to >1 table. Also depending on what you fetch and what condition you use to filter the query results I would expect hitting >1 table to be quite common.

Indeed, I have a caching system that caches immutable DB queries across servers (of which I try to design the DB to use as many immutable rows as possible, lots of SELECT and INSERT, few if any UPDATE's).

[Rupert Smith]

On Wednesday, October 19, 2016 at 8:23:46 PM UTC+1, OvermindDL1 wrote: 

Absinthe handles all the nasty parts of GraphQL though, the combining of queries, the real-time type documentation generation, etc... etc...

What database do you use? Is it always a SQL database or can Absinthe work with noSQL too?

Also, when it combines queries, does it translate that down into an efficient SQL join? Or does it process the joins outside of the database, in the server code? 

[OvermindDL1]

It is storage agnostic, and technically you do not even need a storage, remember that GraphQL calls are basically just RPC, you could have a `fib` GraphQL call that just calculates that.

The database I use is PostgreSQL via the Ecto library though.  Absinthe is database and all agnostic, however it does have a section at http://absinthe-graphql.org/guides/ecto-best-practices/ talking about the best ways to use it with ecto for optimization purposes, and they do have planned more detailed ecto integration in the future, but for now it is do-it-yourself (which I prefer, means I can use my permission system to only return specific things that they have access to).  Absinthe itself does not combine queries, it has no clue what a query is, it just gives the graphql 'function call' setup to you, what the user requested, what they passed in, etc...  With proper Ecto work all the joins are in-database.  With ecto it is trivial to build up database joins in piecemeal, so it works fantastically with graphql.

[John Kelly]

I'm sorry to link drop, but I've been doing a bit of work on a library to remove some of the boilerplate when writing client code for a REST API. The library is currently locked in / specific to what is called PostgREST, but I imagine that the patterns could be applied to any REST backend. Check it out: https://github.com/john-kelly/elm-postgrest/

The core idea is to remove the boilerplate of always having to define encoder, decoder and schema. Would love to chat.

[Kasey Speakman]

I like to put databases behind APIs. But I don't map URIs directly to database entities. I map URIs to use cases and queries. A use case could make multiple changes to the system, not just change one table. A query doesn't necessarily map to a single entity either. The result of one query might contain data from several entities.

So the key here is to use the API like an entry point into your system, NOT an entry point into your database.

[Peter Damoc]

Hi John, 

The project you linked to looks great. 
How do you deal with references? (entities referencing other entities)  

On Thu, Oct 20, 2016 at 9:19 PM, John Kelly <jdrke...@gmail.com> wrote:

I'm sorry to link drop, but I've been doing a bit of work on a library to remove some of the boilerplate when writing client code for a REST API. The library is currently locked in / specific to what is called PostgREST, but I imagine that the patterns could be applied to any REST backend. Check it out: https://github.com/john-kelly/elm-postgrest/

The core idea is to remove the boilerplate of always having to define encoder, decoder and schema. Would love to chat.

--
You received this message because you are subscribed to the Google Groups "Elm Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Kelly]

Great Question!

You can checkout an example here. It builds off of the example presented in the docs. 

Currently, the library does not support representing relationships in the Resource definition, however, the library does support representing the relationships in the queries (see example). I'm not yet sure the best way / whether it will be possible to represent the relationships in the Resource definition. Would love to chat if you have any ideas!

On Friday, October 21, 2016 at 1:25:10 AM UTC-7, Peter Damoc wrote:

Hi John, 

The project you linked to looks great. 
How do you deal with references? (entities referencing other entities)  

On Thu, Oct 20, 2016 at 9:19 PM, John Kelly <jdrke...@gmail.com> wrote:

I'm sorry to link drop, but I've been doing a bit of work on a library to remove some of the boilerplate when writing client code for a REST API. The library is currently locked in / specific to what is called PostgREST, but I imagine that the patterns could be applied to any REST backend. Check it out: https://github.com/john-kelly/elm-postgrest/

The core idea is to remove the boilerplate of always having to define encoder, decoder and schema. Would love to chat.

--
You received this message because you are subscribed to the Google Groups "Elm Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[John Kelly]

Just to follow up on the limitations in my library I spoke about -- namely, not being able to represent the relationships in the resource definition. I spent a bit of time drafting up some potential api changes that would make it possible: here. 

Handling the recursive nature of relationships was influenced by Json.Decode.Extra.lazy

[John Kelly]

I'm coming to the sad realization that an api like this is simply not possible:

```

session =

    resource "sessions"

        { id = int "id"

        , speaker_id = int "speaker_id"

        , start_time = string "start_time"

        , end_time = string "end_time"

        , location = string "location"

        , session_type = int "session_type"

        , speaker = hasOne (\_ -> speaker)

        }

speaker =

    resource "speakers"

        { id = int "id"

        , name = string "name"

        , sessions = hasMany (\_ -> session)

        }
```

Any ideas? I was under the impression that the lambda would fix the recursive type issue, but now i see that elm still has trouble with the type of the record. 

[Nick H]

If you are trying to make a recursive type definition, you need to use union types.

E.G. This is not OK:

type alias Session = { speaker : Speaker }

type alias Speaker = { sessions : List Session }

But this will compile.

type Session = Session { speaker : Speaker }
type Speaker = Speaker { sessions : List Session }

Think of a type alias as a kind of search-and-replace. A recursive type alias leads to a never-ending search-and-replace process.

To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.

[Nick H]

I guess you weren't explicitly defining type aliases for those records. But still, I think you might be able to salvage the API if you wrap the records in union type constructors.

[John Kelly]

To my knowledge, the recursive type that you specified will compile. See here. 

My issue is in fact related to mutually recursive types, it's just that my types are truly infinite. According to the linked doc about recursive types:

"Somewhere in that cycle, you need to define an actual type to end the infinite expansion." Which mine does not.

And to address your comment in regards to: "But still, I think you might be able to salvage the API if you wrap the records in union type constructors." This is the idea I will be exploring next, thank you for the recommendation.

[Nick H]

My issue is in fact related to mutually recursive types, it's just that my types are truly infinite. According to the linked doc about recursive types:

"Somewhere in that cycle, you need to define an actual type to end the infinite expansion." Which mine does not.

That's just what I was trying to say.  Self-recursive or mutually recursive, the solution is the same... replace a type alias with a type definition.

Good luck!

To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.

[Rupert Smith]

On Monday, October 24, 2016 at 1:17:30 AM UTC+1, John Kelly wrote:

I'm coming to the sad realization that an api like this is simply not possible:

```

session =

    resource "sessions"

        { id = int "id"

        , speaker_id = int "speaker_id"

        , start_time = string "start_time"

        , end_time = string "end_time"

        , location = string "location"

        , session_type = int "session_type"

        , speaker = hasOne (\_ -> speaker)

        }

speaker =

    resource "speakers"

        { id = int "id"

        , name = string "name"

        , sessions = hasMany (\_ -> session)

        }
```

Any ideas? I was under the impression that the lambda would fix the recursive type issue, but now i see that elm still has trouble with the type of the record. 

Yes, I also ran into this issue with mutual recursion. I simply followed the advice here and created wrapper types for all my records:

https://github.com/elm-lang/elm-compiler/blob/master/hints/recursive-alias.md 

type Session = Session { ... }

type Speaker = Speaker { ... }

Its a bit of a PITA, since you end up having to wrap and unwrap records all the time. Defining an unwrap function can help things along:

unwrapSpeaker (Speaker speaker) = speaker

[Rupert Smith]

On Thursday, October 20, 2016 at 7:19:05 PM UTC+1, John Kelly wrote:

I'm sorry to link drop, but I've been doing a bit of work on a library to remove some of the boilerplate when writing client code for a REST API. The library is currently locked in / specific to what is called PostgREST, but I imagine that the patterns could be applied to any REST backend. Check it out: https://github.com/john-kelly/elm-postgrest/

The core idea is to remove the boilerplate of always having to define encoder, decoder and schema. Would love to chat.

Seems like a fairly nice idea - you provide functions like 'string', 'int' to construct the 'Fields' which are self contained in the sense that they provide the Decoder and the name of the field.

Mostly field names will match record fields, but sometimes they might not, so having an ability to name them differently to the name of the field in the Elm record could be useful.

I also quite like the little DSL you provide for field projection, filtering and sorting records. In some cases an API might already do that for you, but convenient to have this facility on the client side. I can see that coming in handy for fancy data tables for example.

[Rupert Smith]

On Thursday, October 20, 2016 at 7:19:05 PM UTC+1, John Kelly wrote:

I'm sorry to link drop, but I've been doing a bit of work on a library to remove some of the boilerplate when writing client code for a REST API. The library is currently locked in / specific to what is called PostgREST, but I imagine that the patterns could be applied to any REST backend. Check it out: https://github.com/john-kelly/elm-postgrest/

The core idea is to remove the boilerplate of always having to define encoder, decoder and schema. Would love to chat.

What do you do with fields in the json that are missing or null? I can see for example that for an 'int' you just use Decode.int as the decoder, so I guess you will get a Result.Err and fail when the value is missing?

This is another awkward corner that needs to be dealt with. Until recently I ignored it, or used a default value like "". But of course "" just defeats the point, so I recently updated all my code to use Maybe fields, and Nothing for missing or null values. Of course, not all values can be missing or null, but for the ones that can be optional in input/output of the API it makes sense. After I went through all the code and updated it to deal gracefully with missing values it felt like the Elm compiler had forced me to much more thorough in accounting for all the possible interactions with the API.

[Rupert Smith]

On Wednesday, October 19, 2016 at 9:54:44 AM UTC+1, Peter Damoc wrote:

What options are there?

The option that probably has the most traction is Swagger (now the Open API Initiative) :

https://openapis.org/

In terms of what has been discussed here, graphql or PostgREST are more sophisticated than Swagger, but Swagger may have relevance if its following grows. That said, the news section in the open API initiaive is dissapointingly empty beyond the initial announcement of the initiative:

https://openapis.org/news

I would find a json-schema -> Elm decoders/encoders/records converter useful and the ability to generate the relevant Http boilerplate needed to call endpoints defined in Swagger. It would be nice if you could pick up a Swagger definition and create an Elm client module to work with that API with next to no effort.

[Rupert Smith]

There are some blog posts, so perhaps things are progressing after all:

https://openapis.org/news-faq/blog 

[Peter Damoc]

What I'm looking here is more like a strategy that takes into account the entire persistence layer from the perspective of an Elm Architecture app. 

If one doesn't need to store data remotely, one could simply either use just some transient model OR, use something like localstorage to persist some small bits of data with an update on every relevant change to the model. 

But when one needs remote data, suddenly it comes the problem of how do you approach solving the fact that there is state on the server and state on your computer and there are performance considerations in retrieving the entirety of the state. 

I'm more interested in how one would solve this in a multilayer system where the actual remote persistence is abstracted from the app. 

The actual remote persistence might be implemented with REST, or it might be some kind of websockets thing.
It might involve a SQL database, or maybe a NoSQL database. 
It might be something like Datomic. 

I'm interested in how would one implement this abstraction, this separation of concerns. 

--

You received this message because you are subscribed to the Google Groups "Elm Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Kelly]

On Monday, October 24, 2016 at 7:28:30 AM UTC-7, Rupert Smith wrote:

What do you do with fields in the json that are missing or null? I can see for example that for an 'int' you just use Decode.int as the decoder, so I guess you will get a Result.Err and fail when the value is missing?

I was thinking something like this would work for that case: here 

On Monday, October 24, 2016 at 7:23:43 AM UTC-7, Rupert Smith wrote:

Mostly field names will match record fields, but sometimes they might not, so having an ability to name them differently to the name of the field in the Elm record could be useful.

This is a great suggestion! 

 

[John Kelly]

On Monday, October 24, 2016 at 8:33:55 AM UTC-7, Peter Damoc wrote:

I'm more interested in how one would solve this in a multilayer system where the actual remote persistence is abstracted from the app. 

The actual remote persistence might be implemented with REST, or it might be some kind of websockets thing.
It might involve a SQL database, or maybe a NoSQL database. 
It might be something like Datomic. 

I'm interested in how would one implement this abstraction, this separation of concerns. 

Could you provide more details of what you are looking for? Or some sample code?

Are you interested in how one would abstract away the data consumption? What I mean by that is: The library user simply requests data and the library determines if it needs to make an API call or if it can just get the data from localStorage.

[Peter Damoc]

I cannot provide sample code because I don't have a clear idea how the API could look. 

Think about the role that a ORM is playing. What I want to understand is what would a functional equivalent would look like in Elm.  

What would sit above graphQL or REST or some other lower level tech?

Another way to look at this would be to take the Json.Decode/Encode as example and imagine that one has the equivalent of a Decoder only that it would be some kind of a descriptor used by some lower level library that does the queries based on that descriptor data. 

Maybe this descriptor uses something like Datalog.

Another way to think about this is to try to imagine what would it take to reduce the cognitive overload of a beginner that does not really care that much about how the data is retrieved/saved remotely but cares to have the functionality present.
 

--

You received this message because you are subscribed to the Google Groups "Elm Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Kelly]

On Monday, October 24, 2016 at 1:15:14 PM UTC-7, Peter Damoc wrote:

Think about the role that a ORM is playing. What I want to understand is what would a functional equivalent would look like in Elm.

What would sit above graphQL or REST or some other lower level tech?

Another way to look at this would be to take the Json.Decode/Encode as example and imagine that one has the equivalent of a Decoder only that it would be some kind of a descriptor used by some lower level library that does the queries based on that descriptor data. 

Maybe this descriptor uses something like Datalog.

Unless I'm mistaken, that is the goal I have set for elm-postgrest. The equivalent of a `Decoder` in my library is a `Query`. I build the Query up in the same fashion that Decoders are built (with some extra functionality for filtering and ordering and such). The high level `Query` (descriptor as you called it) is then converted into an PostgREST compliant HTTP Request. I have yet to work out the write api -- which is undoubtably a large portion -- however I am convinced (egotistical, i know) that the library is moving in the right direction. 

I think the tricky part is that the API/functionality of the client is tightly coupled to to functionality of the server. This is why I have scoped my library to only support PostgREST. It is not always the case that the server supports all of the things (nesting, filtering, ordering, pagination, limit, offset, etc). I am unsure if you are suggesting that a general API could exist which encompasses all backends. I originally tried to create a general API, but quickly came to the conclusion that such a task was quite tricky.

Overall, (once again sorry for the plug) I think that elm-postgrest has made some decent steps in the right direction, and I think a more thorough audit / usage of the code could bring this conversation to another level. (also, for those too lazy to look up PostgREST -- it's basically the same as graphql -- but 0 coding required. some fancy people might say "isomorphic")

[Rupert Smith]

On Monday, October 24, 2016 at 9:15:14 PM UTC+1, Peter Damoc wrote:

I cannot provide sample code because I don't have a clear idea how the API could look. 

Think about the role that a ORM is playing. What I want to understand is what would a functional equivalent would look like in Elm.  

What would sit above graphQL or REST or some other lower level tech?

Another way to look at this would be to take the Json.Decode/Encode as example and imagine that one has the equivalent of a Decoder only that it would be some kind of a descriptor used by some lower level library that does the queries based on that descriptor data. 

Maybe this descriptor uses something like Datalog.

Another way to think about this is to try to imagine what would it take to reduce the cognitive overload of a beginner that does not really care that much about how the data is retrieved/saved remotely but cares to have the functionality present. 

I think the model that has the lowest 'cognitive overload' for a beginner is to simply work with the request/response model of HTTP, and to request data when needed, and POST data when you want to update something. Its a pretty obvious abstraction and fits nicely with Elms event handling.

Again, it comes down to Parnas principles of modular design. An API provides a minimal interface to the server code that sits behind it. To make use of that servers functionality, you only need to comprehend the API and what it does - the details are irrelevant. The API provides an abstraction that simplifies away the details.

I would say that persistence is not really the concern of the UI. It is nice to experiment with clever technologies that let the UI code build whatever query it needs, or update whatever data it needs but you have to remember that most of the time your UI is not runnning in a trusted environment - in the case of Elm/javascript its running in someones browsers. Therefore it is almost a necessity that you create an API and think carefully about what data can be seen through it, and what data updated by whom. That said, Overmind has provided some clear details of how access rights to data are protected when using graphql. Not sure if PostgREST provides any assistance with access rights?

Coming back to the 'cognitive overload'... I have found that simply setting up and working with the HTTP requests in Elm is a lot of work - you need encoders, decoders, functions to help build the requests, functions to help decode the responses and deal with errors, msgs to pass to update functions to update your model based on responses and so on. There is a lot of boilerplate in simply working with an API, although this has nothing to do with databases and persistence as such.

I have used code generation techniques to handle all this boilerplate for me, which makes for a much more pleasant experience. So when it comes to working with an API, I am satisfied that it can be done much more simply than it first appears.

Another simple model for persistence that should work with Elm is to simply serialize the application state and store it. So long as you don't have any functions in your model, it should be possible to write an encoder/decoder for the entire model. Why not just freeze dry your entire application state to some key/value store (on every update), then you have a simple way to rehydrate the whole application after a period of inactivity? If you want to get fancy, just as Elms virtual DOM is only updated as required, some technique to only save the deltas to the model, but be able to faithfully restore it in its entirety might be feasible? Perhaps draw inspiration for this from the 'event sourcing' pattern. Is this more the sort of thing that you are looking for?

[Peter Damoc]

On Tue, Oct 25, 2016 at 12:50 AM, John Kelly <jdrke...@gmail.com> wrote:

Unless I'm mistaken, that is the goal I have set for elm-postgrest. The equivalent of a `Decoder` in my library is a `Query`.

Yes, your project does look like a step in the right direction.   

 

I am unsure if you are suggesting that a general API could exist which encompasses all backends. I originally tried to create a general API, but quickly came to the conclusion that such a task was quite tricky.

Take a look at Datomic. If one goes from the database to the language, things might look impossible. If one goes from the language to the database, things might be doable. 

 

Overall, (once again sorry for the plug) I think that elm-postgrest has made some decent steps in the right direction, and I think a more thorough audit / usage of the code could bring this conversation to another level. (also, for those too lazy to look up PostgREST -- it's basically the same as graphql -- but 0 coding required. some fancy people might say "isomorphic")

Again, I would love to see the ideas you put in your project explored and taken further. 

I think that what is needed is some kind of complex enough example, like the music database that I mentioned earlier. 

It needs to be complex enough to be an example on how you do references and pagination. 

[Peter Damoc]

On Tue, Oct 25, 2016 at 2:06 AM, 'Rupert Smith' via Elm Discuss <elm-d...@googlegroups.com> wrote:

I think the model that has the lowest 'cognitive overload' for a beginner is to simply work with the request/response model of HTTP, and to request data when needed, and POST data when you want to update something. Its a pretty obvious abstraction and fits nicely with Elms event handling.

You are right if by "beginner" you understand someone new to Elm. This might not be true for someone who is using Elm to start learning to program. 

 

I would say that persistence is not really the concern of the UI. It is nice to experiment with clever technologies that let the UI code build whatever query it needs, or update whatever data it needs but you have to remember that most of the time your UI is not runnning in a trusted environment - in the case of Elm/javascript its running in someones browsers. Therefore it is almost a necessity that you create an API and think carefully about what data can be seen through it, and what data updated by whom. That said, Overmind has provided some clear details of how access rights to data are protected when using graphql. Not sure if PostgREST provides any assistance with access rights?

Browsers can provide a trusted environment through the use of https. This is what Gmail and Facebook and all other webapps are doing.
Also, you are right, persistence is not the concern of UI but Elm is not exclusively about the UI. 

What I dream to have available is more like full stack in the browser. Something more like what  Abadi Kurniawaan demoed at elm-conf.
This is doable with the current technologies and would be a killer feature if Elm could do it right. 

 

 

Coming back to the 'cognitive overload'... I have found that simply setting up and working with the HTTP requests in Elm is a lot of work - you need encoders, decoders, functions to help build the requests, functions to help decode the responses and deal with errors, msgs to pass to update functions to update your model based on responses and so on. There is a lot of boilerplate in simply working with an API, although this has nothing to do with databases and persistence as such.

I have had a similar experience and this is what motivated this topic. 

It is not that I mind the boilerplate all that much, it's that I lose cognitive resources by not having an official/semi-official way to do it. 

I keep asking myself if I took the right approach when I should be focusing on implementing some other feature.  

 

Why not just freeze dry your entire application state to some key/value store (on every update), then you have a simple way to rehydrate the whole application after a period of inactivity?

This is not an option on large datasets. I gave the example with the Music database in an earlier message specifically to address this. 
Pagination is a very real concern that seldom appears in toy examples.
 

[Rupert Smith]

On Tuesday, October 25, 2016 at 8:08:11 AM UTC+1, Peter Damoc wrote:

Browsers can provide a trusted environment through the use of https. This is what Gmail and Facebook and all other webapps are doing. 

What I mean is, there is nothing to stop whoever is running your application from subverting it. In the browser, there are even a lot of things you can do with the javascript console. If your 'persistence API' requires the application to behave correctly in order to not store invalid or maliciously altered data, you cannot guarantee that. This is one very good reason why business logic is typically implemented on the server behind an API that only provides the specific operations that a user is allowed to perform, whether they perform them through your application or otherwise.

You can use secure cookies with HTTPS. There is nothing to stop someone using a hacked version of the browser that lets them get the secure cookie in order to make malicious calls to your API.

[Rupert Smith]

On Tuesday, October 25, 2016 at 8:08:11 AM UTC+1, Peter Damoc wrote:

Why not just freeze dry your entire application state to some key/value store (on every update), then you have a simple way to rehydrate the whole application after a period of inactivity?

This is not an option on large datasets. I gave the example with the Music database in an earlier message specifically to address this. 
Pagination is a very real concern that seldom appears in toy examples.

Its not an option on shared data. Going with your MusicBrainz example, there is a shared data set. You could write an application that lets users read and write that data. I would think that such an application would only ever hold a small amount of the data at any one time, so serializing the application state is not the same thing as saving the whole MusicBrainz database. Also, this technique only saves the state for one user, there is no mechanism for integrating the changes made by several users into a single database.

Here is an example where it would be relevant: a single player game. I might periodically dump the game state (perhaps not the entire Elm model, but enough to be able to continue playing from last position). Each users game state is their own, and I don't really care if they cheat, for example by using curl on the command line to poke at the persistence API, perhaps changing game level from 11 to 12 say.

This is another very good reason that persistence is typically the concern of a server - globally, over the whole application state over all users, updates are being made concurrently. It is usually the responsibility of the server side to coordinate this concurrent state machine, in order to ensure it does not enter illegal states. The server is ideally positioned to take on this responsibility, being the hub in a hub and spokes model of your application. That said, a server centric model is not the only one possible.

Client side storage, such as sessionStorage and localStorage and Web SQL database are relevant to Elm. I do think that these server side responsibilities are not really within the domain of Elm.

[Rupert Smith]

On Tuesday, October 25, 2016 at 9:27:58 AM UTC+1, Rupert Smith wrote:

On Tuesday, October 25, 2016 at 8:08:11 AM UTC+1, Peter Damoc wrote:

Browsers can provide a trusted environment through the use of https. This is what Gmail and Facebook and all other webapps are doing. 

What I mean is, there is nothing to stop whoever is running your application from subverting it.

There are ways that untrusted code can be made to work to update shared state accross many actors whilst ensuring the integrity of the data - blockchain databases.

[Peter Damoc]

On Tue, Oct 25, 2016 at 11:27 AM, 'Rupert Smith' via Elm Discuss <elm-d...@googlegroups.com> wrote:

If your 'persistence API' requires the application to behave correctly in order to not store invalid or maliciously altered data, you cannot guarantee that.

This actually sounds more like a challenge to be faced rather that a technical impossibility. 
Maybe some kind of declarative access control embedded in a shared schema could solve this. 

 I do think that these server side responsibilities are not really within the domain of Elm.

Look at what happened with Javascript. Once it got useful in the client people wanted it on the server and then we got Node. 
We are nowhere near the popularity of javascript and yet, I already see frequent enough questions about using Elm on the server-side. 

There are two ways to address this: 

1. allocating resources to making Elm viable on the server
2. making the server part as small and automatic as possible as to not require much coding. 

To me, option 2 is much more attractive. 

[Rupert Smith]

On Tuesday, October 25, 2016 at 10:01:48 AM UTC+1, Peter Damoc wrote:

On Tue, Oct 25, 2016 at 11:27 AM, 'Rupert Smith' via Elm Discuss <elm-d...@googlegroups.com> wrote:

If your 'persistence API' requires the application to behave correctly in order to not store invalid or maliciously altered data, you cannot guarantee that.

This actually sounds more like a challenge to be faced rather that a technical impossibility. 
Maybe some kind of declarative access control embedded in a shared schema could solve this. 

Declaring what the access rights are to the client UI is useful, yes - it is just that they still need to be enforced by the server because you cannot fully trust the client to take care of it. This is actually what I am doing (in some cases), because when a user logs in they get back a JWT token. This token contains some information about who the user is, and what permissions they have. The UI can use this to only render screens and functionality that the user is allowed to use. This is merely to be helpful and provide a nice user experience, the permissions are always checked whenever a restricted API endpoint is invoked.

I don't always use the JWT tokens as bearer tokens in an HTTP Authorization header field, or as a secure cookie because they can grow quite large. However, I generally do provide an endpoint in my API where the JWT token can be requested in order to inspect the users declared access rights.

 

 I do think that these server side responsibilities are not really within the domain of Elm.

Look at what happened with Javascript. Once it got useful in the client people wanted it on the server and then we got Node. 
We are nowhere near the popularity of javascript and yet, I already see frequent enough questions about using Elm on the server-side. 

There are two ways to address this: 

1. allocating resources to making Elm viable on the server
2. making the server part as small and automatic as possible as to not require much coding. 

To me, option 2 is much more attractive. 

Ok, I think I now get a better idea of what you are after.  As per John Kellys PostgREST code: https://github.com/john-kelly/elm-postgrest. Have a way of defining a data model in Elm, and use that description to automatically generate a suitable server to persist it.

In this case defining the access rights in the Elm code would be ok, as the server that you generate would also securely check them at runtime.

[Kasey Speakman]

So to phrase what I previously said a different way, a database is the wrong level of abstraction to be shooting for. A database is just one integration that most business systems have.

The system itself exposes use cases and queries. Whether and which databases are involved (and their structural details, I contend) should be of no concern to client apps. The client apps exist to help the user fulfill the system's use cases, not just to connect to a database.

[Rupert Smith]

On Tuesday, October 25, 2016 at 3:12:53 PM UTC+1, Kasey Speakman wrote:

So to phrase what I previously said a different way, a database is the wrong level of abstraction to be shooting for. 

 

Yes, I think you are right. Much better if we can just think about the data model we need, and have some completely automated way of persisting it.

I was just thinking, with reference to "Making Impossible States Impossible", typed functional languages have some appealing aspects as a foundation for describing data models.

[Rupert Smith]

On Monday, October 24, 2016 at 9:15:14 PM UTC+1, Peter Damoc wrote:

I cannot provide sample code because I don't have a clear idea how the API could look. 

Think about the role that a ORM is playing. What I want to understand is what would a functional equivalent would look like in Elm.  

So you got me thinking. There is actually a parallel between an ORM and the client side using an API, because both work with complete data models but only fetch and update parts of the complete model.

I OO programming, when something has not been fetched, the ORM proxies it. So if you traverse to a proxies relation, the ORM automatically fetches it for you. You can also size your queries to do a certain amount of pre-fetching as your business logic settles into place and you strive to optimize things.

I don't think there can be such a thing as transparent proxying in Elm though? Since all HTTP requests require using the event driven system - in order to trigger a request you must explicitly create a Cmd?

Take this as an example data model:

Type Passenger

  =  Passenger 

  { name : String,

    flights : List Flight

  }

Type Flight

 = Flight

 { flightNo: String,

   passengers : List Passenger

 }

Its not so convenient, since I cannot put off fetching the related passengers or flights. With this model if I get one passenger, I need to get all their flights and all their passengers and..

What I did was to use a Maybe

Type Passenger

  =  Passenger 

  { name : String,

    flights : Maybe (List Flight)

  }

Type Flight

 = Flight

 { flightNo: String,

   passengers : Maybe (List Passenger)

 }

but this feels a little dishonest to me, since Nothing should really mean no Flights perhaps, even though I can use the empty list to stand for that.

I could instead have a type with Resolved and Unresolved constructors. Now all I need is a function that takes an Unresolved, fetches its slice of data, then updates the model to Resolved. If the server is supplying data in a HAL type format, then resolving will have URLs to follow in order to resolve data. Or if it is not a HAL type interface, the function needs to be written (or generated) to fit the API it is using - it probably takes some <id> and adds it to a URL it already knows to be the root of the API that deals with that type.

As I say, I don't think we can make the remote fetching transparent unless someone can tell me how that could be done in Elm? but providing a function to resolve, especially if smart enough to encapsulate knowing where to fetch the data and how to update the model with the result would make for quite a slick user experience.

[Kasey Speakman]

As best I can, I try to look at the system as use cases and events.

With legacy systems, we are tied to a normalized data model (which is really designed for writing and not reading), so for queries we have to "project" from that normalized data model into another model. But the place I'd really like to get to is storing the system's events, and creating whatever models are necessary to answer queries and fulfill use cases from those events. AKA Event Sourcing. I am finally getting to do this on a new project. Our existing systems will stay with a normalized data model for the foreseeable future as the cost of change is too high.

But I still try to take the principles of using business-specific events (like StudentRegistered or PaymentDeclined) in my business logic, then translate those into database calls when they are sent for persistence. That also allows me to use those events to update secondary models or trigger other logic. The common alternative, just updating state and doing `repository.Save()` makes it harder to hook into specific business happenings.

[Rupert Smith]

On Friday, October 28, 2016 at 6:26:46 PM UTC+1, Kasey Speakman wrote:

As best I can, I try to look at the system as use cases and events.

With legacy systems, we are tied to a normalized data model (which is really designed for writing and not reading), so for queries we have to "project" from that normalized data model into another model.

My understanding of normalization is that its purpose is to avoid duplicating data, and by avoiding duplication reduce the chance of errors in order to help ensure the quality of stored data.

But the place I'd really like to get to is storing the system's events, and creating whatever models are necessary to answer queries and fulfill use cases from those events. AKA Event Sourcing. I am finally getting to do this on a new project. Our existing systems will stay with a normalized data model for the foreseeable future as the cost of change is too high.

But I still try to take the principles of using business-specific events (like StudentRegistered or PaymentDeclined) in my business logic, then translate those into database calls when they are sent for persistence. That also allows me to use those events to update secondary models or trigger other logic. The common alternative, just updating state and doing `repository.Save()` makes it harder to hook into specific business happenings.

I only advocate the approach of updating state and saving it when that makes sense, and I would say those situations are:

Where a user completely owns the data and is free to modify it as they see fit.

Where, due to the nature of the data it is hard to automate use cases over it - textual data requiring processing by a human expert for example.

When prototyping a system and the use cases are still being developed.

I find this approach to prototyping beneficial compared with going straight to use cases as it leads to a more declarative and extendable data model. I mean in the sense that the database isn't straight away hidden behind a rigid set of use cases. It tends to lead to systems that are less layered too, many times I have seen very layered services when something simpler would have sufficed.

Also, if a data model can be built in such a way that illegal states are not possible (which is the purpose of normalization), there is less need for only allowing it to be modified according to a set of use cases, since only legal changes can be made to it. For example, if a user account must have an email address associated with it, if there is validation on the format of the email address and it cannot be null, then there is not need to write a specific transactional end-point to allow a user to update their email address, you can just let them modify and save the account record and they can still only perform that operation in a way that produces correct data.

I take your point though about being able to hook into changes relating to specific events.

 

[Rupert Smith]

On Friday, October 28, 2016 at 10:58:34 PM UTC+1, Rupert Smith wrote:

For example, if a user account must have an email address associated with it, if there is validation on the format of the email address and it cannot be null, then there is not need to write a specific transactional end-point to allow a user to update their email address, you can just let them modify and save the account record and they can still only perform that operation in a way that produces correct data.

I take your point though about being able to hook into changes relating to specific events.

I inadvertently picked an example with the email address that shows why you want to hook into specific business events, because in this case you might want to send a confirmation email with a link in it to confirm the address when the email address is changed. In that case I would not allow the email address to be set as part of a more generic 'save' endpoint, and add a new end-point for the change email address as its own operation. Its either that or add some code to detect the email change somehow - but I think you are right, it better to have an explicit endpoint for it, then very easy to hook into it as an 'event'. I like to start quickly by starting with the open and generic data modelling with CRUD over entities, and then refine things from there.

[Kasey Speakman]

There is a danger in focusing on data over use cases. It's not a guarantee that you make this mistake (I have), but you know you've gone too far with it when most workflows are Load/Edit/Save (or Create/Save). And the user is left to know what field to change to have a specific effect in the system. I've seen this termed as data-centric. Seems okay at first but after a few years in production this leads to long user training times, brittle workflows, and high support loads.

Most businesses invest in custom software to accelerate businesses processes. So those should be the focus. For just data storage, low-code solutions could do that much cheaper. That said, CRUD is a part of almost all custom software because data must be collected to fulfill use cases (usually for the human factor, like seeing names instead of just IDs). Often the easiest place to start is with CRUD, because you need a way to populate data to work with anyway. But the database structure is the wrong thing to center a design on.

[Peter Damoc]

On Mon, Oct 31, 2016 at 5:24 PM, Kasey Speakman <kjspe...@gmail.com> wrote:

There is a danger in focusing on data over use cases. It's not a guarantee that you make this mistake (I have), but you know you've gone too far with it when most workflows are Load/Edit/Save (or Create/Save). And the user is left to know what field to change to have a specific effect in the system. I've seen this termed as data-centric. Seems okay at first but after a few years in production this leads to long user training times, brittle workflows, and high support loads.

What are the alternatives? How would an approach focused on use cases look like? 

 

[Kasey Speakman]

On Friday, October 28, 2016 at 4:58:34 PM UTC-5, Rupert Smith wrote:

On Friday, October 28, 2016 at 6:26:46 PM UTC+1, Kasey Speakman wrote:

As best I can, I try to look at the system as use cases and events.

With legacy systems, we are tied to a normalized data model (which is really designed for writing and not reading), so for queries we have to "project" from that normalized data model into another model.

My understanding of normalization is that its purpose is to avoid duplicating data, and by avoiding duplication reduce the chance of errors in order to help ensure the quality of stored data.

Yes, these are benefits of normalization. However, normalization optimizes for writing. Reads become more expensive due to joining in all the relevant details you need to service a particular query. In most business systems, multiple reads are performed for every write performed. It's a minor point for most business systems I've been a part of. However, I do currently have a couple of views where the reads became too expensive and it was better to maintain a denormalized read model alongside the normalized version. You might recognize this as a periodically-run report. But when I use events internally, I can have the report be updated cheaply as each event occurs rather than projected expensively from the normalized form.

[Kasey Speakman]

"Explicit is better than implicit." 

-- Tim Peters

So the main way "data-centric" happens is by modeling behaviors implicitly in the data. A common example is an item being active or not. I often model this as a boolean property "isActive". The usual way to tweak this value is to have the user edit the item and toggle that box. So, the problem lurking here is that deactivation may, in fact, be a process. If you have to check on save `if oldValues.isActive <> newValues.isActive then // do extra de-/re-activation steps`, that is an indication that you have implicit behavior hiding in data. That should be changed to be an explicit action the user can perform (e.g. a button in the UI) and have its own use case (uri or method call) on the system API. (Imagine this: to cancel your Amazon order, you have to edit the order and check the IsCanceled box.)

So here's a concrete example of how we did it wrong in our legacy system. To close a trainee's registration as "No Show", an employee has to create an exam against that registration and grade it as 1%. This is an implicit concept which our employees and our software understand as "No Show". Instead of making it explicit by programming in a No Show button/action/status, we have to program the employees (current and future) to recognize this situation.

And this is just one example of probably dozens in our system. Because of that, it takes 6 months or more to get someone fully trained to use it properly. And even then it's hard for people to keep dozens of cases in their head at once, so minor mistakes/inconsistencies can happen easily.

It's also hard to add features and fix bugs in such a system. Not only because there's a lot of potential dev time spent on support, but also because we programmers can't keep all these implicit behaviors straight either. (We often don't know them as well as our users do, because we don't work with them every day.) So deployments always carry at least a moderate level of risk even when we believe the changes are minor.

Going forward, we refactor these implicit behaviors to be explicit actions/API calls as we have feature/bug requests against them. It takes more time, but improves quality in the long run. So we are pushing it in the right direction, and users generally still manage to make good use of it in the mean time.

I should also note that for non-functional reasons, it may not be possible to make behaviors explicit. I worked on another project where I was not given leave to do this. I figure it was deemed prohibitively expensive, because I would need too much time with subject matter experts. Such is life.

[Peter Damoc]

On Tue, Nov 1, 2016 at 2:25 AM, Kasey Speakman <kjspe...@gmail.com> wrote:

So here's a concrete example of how we did it wrong in our legacy system. To close a trainee's registration as "No Show", an employee has to create an exam against that registration and grade it as 1%. This is an implicit concept which our employees and our software understand as "No Show". Instead of making it explicit by programming in a No Show button/action/status, we have to program the employees (current and future) to recognize this situation.

Wow... this is so silly that it almost looks like a joke. Unfortunately, I've seen enough to know that it happens. 

However, looking at a fresh system that one might want to design it seams to me like there are 3 possible layers 

Layer 3. Business Objects Layer - concerned with validity of state transactions 

Layer 2. Data Modeling Layer - concerned with what needs to be persistent 

Layer 1. Storage Layer - concerned with connections, locations, raw entity storage 

Layer 1 would be the implementation of the library I would like to have in Elm. Ideally, something similar to Datomic.  
Layer 2 would be implemented by the user using Layer 1 in a declarative way similar to Json.Decode 
Layer 3 would be implemented by the user using Layer 2 in a way that is similar to the Elm Architecture (layer 2 Model + some update) 

What do you think?
Am I misunderstanding what you described? 

[Rupert Smith]

On Tuesday, November 1, 2016 at 8:31:07 AM UTC, Peter Damoc wrote:

However, looking at a fresh system that one might want to design it seams to me like there are 3 possible layers 

Layer 3. Business Objects Layer - concerned with validity of state transactions 

Layer 2. Data Modeling Layer - concerned with what needs to be persistent 

Layer 1. Storage Layer - concerned with connections, locations, raw entity storage 

Layer 1 would be the implementation of the library I would like to have in Elm. Ideally, something similar to Datomic.  
Layer 2 would be implemented by the user using Layer 1 in a declarative way similar to Json.Decode 
Layer 3 would be implemented by the user using Layer 2 in a way that is similar to the Elm Architecture (layer 2 Model + some update) 

What do you think?
Am I misunderstanding what you described? 

Are you talking Elm running server side for this? 

[Peter Damoc]

On Tue, Nov 1, 2016 at 11:30 AM, 'Rupert Smith' via Elm Discuss <elm-d...@googlegroups.com> wrote:

Are you talking Elm running server side for this? 

I haven't thought about that too much.
In theory, there should be some kind of schema common to both the front-end and the back end that would capture authorization requirements but, ideally it would be something that sits mainly in the client. 
Ideally it would be something similar to using Horizon. 
The server side should be mainly automated. 

[Rupert Smith]

On Tuesday, November 1, 2016 at 10:00:08 AM UTC, Peter Damoc wrote:

On Tue, Nov 1, 2016 at 11:30 AM, 'Rupert Smith' via Elm Discuss <elm-d...@googlegroups.com> wrote:

Are you talking Elm running server side for this? 

I haven't thought about that too much.
In theory, there should be some kind of schema common to both the front-end and the back end that would capture authorization requirements but, ideally it would be something that sits mainly in the client. 
Ideally it would be something similar to using Horizon. 
The server side should be mainly automated.

Its just for reasons previously discusses, I don't think your layers 1, 2 & 3 belong on the client side. How can you trust an unknown person or organisation to run your client code and not deliberately alter or bypass your business logic? Instead of posting requests to make balanced transactions between bank accounts, just add a few million to my account and then save it.

If we follow Kasey's advice and make an API with endpoints for each use case, all that the client should see is the available endpoints, and the data models that they accept. Like this:

Client layer: API data model + functions to invoke endpoints + some way of receiving back the responses.

Server API layer: API data model + endpoints

Server Business logic layer: functions/methods implementing the business logic

Server Persistence layer: code to shove stuff in the database etc.

There may or may not be 2 data models on the server side, one for the API and one for the database. I don't tend to automatically choose this approach, in particular I don't like code that copies from one data model to another:

myDBObject.setFoo(myAPIObject.getFoo()); 

myDBObject.setBar(myAPIObject.getBar());

...

The reason being that more often than not, someone will add something to this data mapping that should actually be part of the business logic. Even setting a default value in the mapping should not be done, the default should instead be made very clear in the business logic. The other reason I don't like it, is that often the code that constructs the API object from the database objects is the source of N+1 selects.

So I make a judgement call on where it is appropriate to use the same data model for both.

[Kasey Speakman]

It is silly, and I don't know why it was done this way. But that's the world I live in now. It's easy to justify one case at a time, but all tolled it adds up.

As far as "layers", you should check out these two posts in order.

http://blog.ploeh.dk/2013/12/03/layers-onions-ports-adapters-its-all-the-same/

http://blog.ploeh.dk/2016/03/18/functional-architecture-is-ports-and-adapters/

[Kasey Speakman]

Oh, and yes, most of these layers are on the server side if we are talking about an Elm client application. Generally speaking, the client app serves to collect and prepare data to submit use cases. So it reads data from an API and submits data to an API.

[Peter Damoc]

It would be awesome to have a more complex but practical example that captures this. 

The reservation scenario from the second post looks like an interesting use-case. 

I will think more about it. 

--
You received this message because you are subscribed to the Google Groups "Elm Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rupert Smith]

On Tuesday, November 1, 2016 at 1:16:40 PM UTC, Kasey Speakman wrote:

It is silly, and I don't know why it was done this way. But that's the world I live in now. It's easy to justify one case at a time, but all tolled it adds up.

As far as "layers", you should check out these two posts in order.

http://blog.ploeh.dk/2013/12/03/layers-onions-ports-adapters-its-all-the-same/

http://blog.ploeh.dk/2016/03/18/functional-architecture-is-ports-and-adapters/

I think people who write code like this must get paid by the line. A couple of years ago I worked some overtime over Christmass/New Year to refactor a system like this that was getting out of control. In 5 days I turned 10,000 lines of code with layers into 1,000 lines of code with just 1 layer that did exactly the same thing - then booked myself a cheap holiday in early January to enjoy my TOIL.

I'm not saying its always the right answer, but when someone tells you that you must have lots of layers, I would always question them and their motives. 

[Rupert Smith]

On Tuesday, November 1, 2016 at 2:10:05 PM UTC, Peter Damoc wrote:

It would be awesome to have a more complex but practical example that captures this. 

The reservation scenario from the second post looks like an interesting use-case. 

I will think more about it. 

On Tue, Nov 1, 2016 at 3:33 PM, Kasey Speakman <kjspe...@gmail.com> wrote:

Oh, and yes, most of these layers are on the server side if we are talking about an Elm client application. Generally speaking, the client app serves to collect and prepare data to submit use cases. So it reads data from an API and submits data to an API.

Lets take the example I gave before, of a user changing their email address on a profile. Except maybe lets make it some record type, like payment details that they are changing, instead of an email address. Just so that we have two 'objects' with a relationship between them.

The difference between a data-centric and service-oriented approach, in Elm, would be as follows.

In the data centric approach

type alias Profile =

 { id : String

 , username : String

 , paymentDetails : PaymentDetails

 -- and so on.

 }

type alias PaymentDetails =

 { creditCard : String

 , expiryDate : String

 -- and so on.

 }

In the service oriented approach

type alias Profile =

 { id : String

 , username : String

 -- and so on.

 }

type alias PaymentDetails =

 { creditCard : String

 , expiryDate : String

 -- and so on.

 }

In the data-centric approach, we'd just have one endpoint for saving the profile. This is a bit long in Elm to write out here, so I'll just summarize it as

PUT /api/profile/<id>  (In the request, put a Profile with PaymentDetails set on it).

In the service-oriented approach, we'd have one endpoint for saving the profile (maybe we would accept some fields to simply be set on the profile through its generic save endpoint, like the users prefered color scheme - stuff that doesn't need to trigger some business process). We'd have another endpoint for updating the payment details:

PUT /api/profile/<id> (In the request, put a Profile)

PUT /api/profile/<id>/paymentdetails (In the request put a PaymentDetails, the id of the profile to attach it to is in the URL).

In the second approach, it is easier to hook into a change of payment details as a business event - perhaps we need to trigger some procedure to verify the new details.

There is clever stuff we can do with the data-centric approach, in terms of using generic CRUD APIs, or tailoring graphql expressions to fetch or update particular paths in the data model. It is a bit harder to see how clever tricks can be used to cut down the larger amount of work that needs to be done with the service-oriented approach. I think by shifting from doing the clever bit at runtime to doing it at compile time and using code generation techniques, it is doable.

[Kasey Speakman]

I used to think that too. But then I fell into all the traps doing things naively and over time found myself moving towards similar things to the blog. Of course, such an OO architecture is not worth doing in every case... only in critical cases.

Then I eventually found functional programming. And if you read the followup post about "functional architecture", much of the overhead of "ports and adapters" is no longer present when it's done functional. For instance, single method interfaces are common in "ports and adapters" in OO languages, but in FP those are just functions. You probably don't even need to define a type alias. You get the benefits of dependency injection with almost none of the traditional overhead as seen in OO "ports and adapters".

[Kasey Speakman]

On Tuesday, November 1, 2016 at 9:53:55 AM UTC-5, Rupert Smith wrote:

...

There is clever stuff we can do with the data-centric approach, in terms of using generic CRUD APIs, or tailoring graphql expressions to fetch or update particular paths in the data model. It is a bit harder to see how clever tricks can be used to cut down the larger amount of work that needs to be done with the service-oriented approach. I think by shifting from doing the clever bit at runtime to doing it at compile time and using code generation techniques, it is doable.

http://softwareengineering.stackexchange.com/questions/25276/why-is-cleverness-considered-harmful-in-programming-by-some-people/25281 

[Kasey Speakman]

In a new project, we are going for something like this.

https://www.youtube.com/watch?v=GneIxgl71NM&index=18&list=PL9Z-JgiTsOYTdi91N_DlcpWqkCYvMrhA4

[Eric G]

Just wanted to say, thanks for posting this Kasey. It's great to hear the nasty details, though I'm sure it's not so great to live with them day to day :)  It's a great point about implicit behavior pushing off programming from the system onto the users.  Thanks also for the links.

I have been moving away from "data-encoded behavior" styles in my work too, especially for applications that have quite structured state transitions. A key thing I have realized is that for many applications a 'form submit' does not represent creating or updating an entity directly -- it represents an intent to transition state in a particular domain-specific way, depending on current state. In this way it is very similar to the `Msg -> Model -> Model` update within the Elm app, but extended to the backend.

[Rupert Smith]

On Tuesday, November 1, 2016 at 5:54:10 PM UTC, Kasey Speakman wrote:

In a new project, we are going for something like this.

https://www.youtube.com/watch?v=GneIxgl71NM&index=18&list=PL9Z-JgiTsOYTdi91N_DlcpWqkCYvMrhA4

What sort of throughput in events/sec are you aiming to support? 

[Rupert Smith]

On Tuesday, November 1, 2016 at 1:16:40 PM UTC, Kasey Speakman wrote:

http://blog.ploeh.dk/2013/12/03/layers-onions-ports-adapters-its-all-the-same/

I agree with this. I was very happy to see in the first couple of paragraphs a description of what usually goes wrong in OO implementations - inverting the dependency between the data layer and the domain. This is the pattern I am currently using. 

http://blog.ploeh.dk/2016/03/18/functional-architecture-is-ports-and-adapters/

Was pretty interesting. I agree that somehow implementing in a functional way makes the layering seem less onerous. A case of serendipity that the awkwardness of IO in functional languages tends to make this pattern a natural one.

To some extent, this also applies to a UI in Elm. The 'domain' of a UI is related to but not identical to the domain of the services it consumes - but there is still a domain. In Elm we implement that with our model, use the type system to try and constrain it to dissallow illegal states, and generally provide a set of convenience functions for manipulating the model or extracting features of interest from it. The view is a port. The interface onto the service, with its HTTP endpoints and encoder/decoders is another port.

[Kasey Speakman]

On Tuesday, November 1, 2016 at 3:31:07 AM UTC-5, Peter Damoc wrote:

On Tue, Nov 1, 2016 at 2:25 AM, Kasey Speakman <kjspe...@gmail.com> wrote:

So here's a concrete example of how we did it wrong in our legacy system. To close a trainee's registration as "No Show", an employee has to create an exam against that registration and grade it as 1%. This is an implicit concept which our employees and our software understand as "No Show". Instead of making it explicit by programming in a No Show button/action/status, we have to program the employees (current and future) to recognize this situation.

Wow... this is so silly that it almost looks like a joke. Unfortunately, I've seen enough to know that it happens. 

After thinking on this, I can tell you exactly how it happens. The program was made on a contract basis, so either the contractor didn't discover the No Show use case, or it was not in the business processes at the time it was implemented. Later, a user in charge decides they themselves can implement a No Show feature without going through contracting by making policy around the data. "If you see a grade of 1, that means it was a No Show." The user probably congratulates themselves for saving money. This kind of thinking leads to more of this. And that's where we are.

Sometimes this kind of shortcut might be the only way to get things done under time/budge pressure, from the user's perspective.

[Kasey Speakman]

Initially, the load will be small and our cloud-provisioned resources will reflect that. It will necessarily be multi-tenant and we've had a lot of interest expressed among clients. So there is a great potential for growth.

[Gavin Walsh]

When you use Absinthe, do you use the relay https://github.com/absinthe-graphql/absinthe_relay module as well? Even though there's no elm equivalent to relay, the relay module helps with pagination it looks like.. or did you do something else for pagination?

And are you using https://github.com/jahewson/elm-graphql for the frontend out of curiosity? 

Thanks!

On Thursday, October 20, 2016 at 10:34:23 AM UTC-4, OvermindDL1 wrote:

On Thursday, October 20, 2016 at 3:55:45 AM UTC-6, Rupert Smith wrote:

On Wednesday, October 19, 2016 at 8:23:46 PM UTC+1, OvermindDL1 wrote: 

Absinthe handles all the nasty parts of GraphQL though, the combining of queries, the real-time type documentation generation, etc... etc...

What database do you use? Is it always a SQL database or can Absinthe work with noSQL too?

Also, when it combines queries, does it translate that down into an efficient SQL join? Or does it process the joins outside of the database, in the server code? 

It is storage agnostic, and technically you do not even need a storage, remember that GraphQL calls are basically just RPC, you could have a `fib` GraphQL call that just calculates that.

The database I use is PostgreSQL via the Ecto library though.  Absinthe is database and all agnostic, however it does have a section at http://absinthe-graphql.org/guides/ecto-best-practices/ talking about the best ways to use it with ecto for optimization purposes, and they do have planned more detailed ecto integration in the future, but for now it is do-it-yourself (which I prefer, means I can use my permission system to only return specific things that they have access to).  Absinthe itself does not combine queries, it has no clue what a query is, it just gives the graphql 'function call' setup to you, what the user requested, what they passed in, etc...  With proper Ecto work all the joins are in-database.  With ecto it is trivial to build up database joins in piecemeal, so it works fantastically with graphql.

[OvermindDL1]

Ah, no, in fact I do not recall seeing that, it looks like it has had a lot of development recently so it appears that it might be newer than when I built my api pipeline.  I've not had a pagination setup yet as the client requests the range of what they want for the one set of where a range is useful.  It is an internal API so if I exposed it to the public I'd set a range limit of 100 or something.

And nope, it's tied in through a specialized websocket (via phoenix) so I just build the queries manually and send it in via my phoenix library currently, pretty basic, but nice to use even manually.

[Gavin Walsh]

Okay awesome. Thanks so much! 

--
You received this message because you are subscribed to a topic in the Google Groups "Elm Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elm-discuss/igxYW0Q3Clw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elm-discuss+unsubscribe@googlegroups.com.