Task ports: A proposal to make it easier to integrate JS with Elm.

[James Wilson]

The problem

ports as they stand are fundamentally incompatible with Tasks. Being backed by Cmd's, they are harder to compose. A frustration of mine is that often we are directed to "just use ports" when a proper interface to some native API is not yet available, but this leads to our Msg types growing and more significant changes being required when eventually the proper interface is made available.

Also, many JS interop things I find myself wanting to do are fundamentally one-shot functions which I expect a result back into Elm from immediately, or otherwise just want to compose with other Task based things. Some examples that come to mind of one-shot tasks you may want to compose rather than use the streaming interface that ports provide:

• Getting items from local/sessionStorage
  
• .. really, most things involving working with the Web API that arent yet implemented in Elm.
  
• Embedding JS widgets into Elm elements
  
• Using a JS library for doing things like hashing passwords or obtaining some data back from some custom service
  
• Interacting with things like Electron for creating apps that can run in the desktop and interact with the filesystem etc.
  

The solution

Task ports. The idea is that these are defined the same way that Ports in elm currently are, but they return a Task type rather than a Cmd or Sub type. On the JS Side, we attach a function to the Elm app that returns a Promise, and on the Elm side we wait for the Promise returned to reject or resolve, and marhsall the error or result from the promise into the error or result type required by the Task type of the port.

Let's see how this might work:

Ports.elm:

port apiSession: Task String SessionId

Main.elm:

import Ports
import Json.Decode as Decode
import Task exposing (andThen)


-- get an API session from JS land and make an http request using it
-- given some path and a decoder to decipher the result:
apiRequest : String -> Decoder a -> Task ApiError a
apiRequest path decoder =
  let
    headers sessId =
        [ ("Content-Type", "application/json")
        , ("MyApp-SessionId", sessId)
        ]


    req sessId = Http.send Http.defaultSettings
        { verb = "POST"
        , headers = headers sessId
        , url = path
        }


    decodeResponse res = Decode.decodeString decoder -- ...handle error etc
  in
    Ports.apiSession `andThen` req `andThen` decodeResponse

App.js:

Elm.Main.ports.apiSession = function(){
    return new Promise(function(resolve,reject){


        var sess = localStorage.getItem("sessionId");
        if(!sess) reject("NO_SESSION");
        else resolve(sess);


    });
}

var app = Elm.Main.fullscreen();

Here, we use a tiny bit of JS to access localStorage and pull out a session ID. This function is used whenever the apiRequest Task is performed in Elm, and composes nicely into our apiRequest without the need for a complicated effect manager or threading a sessionId through everywhere just because we need to get it from a Cmd based port.

One of the nice things about this is that there is minimal refactoring to do for those things that do eventually receive coverage in the Elm Web API - you're just swapping out Tasks for other Tasks. As the Web API will always be changing, I think that having a nice way to make JS polyfills like this will always have some value, let alone for interacting with libraries written in JS that haven't or won't ever be ported to Elm.

Elm would continue to make the same guarantees as with other ports; if the task port can't marshall the response back into Elm an error would be thrown along the same lines as is currently done via ports.

Summary

- regular ports only let you send data off or receive data back, not both.

- Cmd's and Sub's are not composable

- Task based ports allow you to create a new Task that is backed by JS

- Task based ports allow for better composition and less friction when the backing JS is eventually implemented in Elm.

I'd love to hear what people think about this. Perhaps I'm missing some big issues with the idea for instance, or maybe it's an awesome idea :) What do you all think?

[José Lorenzo Rodríguez]

I would really love this, the lack of Task ports is the main reason I fallback to creating native modules.

[OvermindDL1]

A task-port to handle javascript promises would be optimal for sure, but for now I use a callback structure so the JS can call back into whatever port I want.  Definitely a hack but it also fulfills the situations where there can be multiple callbacks and not just one.

[RGBboy]

I have just been thinking the exact same thing over the past couple of days. 

If you wanted to keep compatibility with older browsers this could also just be a function that takes a value and a node style callback:

Elm.Main.ports.apiSession = function (value, cb) {
  var sess = localStorage.getItem(value);
  if (!sess) {
    cb("NO_SESSION"); // first argument error
  } else {
    cb(null, sess);
  }
}

You could add a guard in the callback so that it can not be called more than once. 

I think on the elm side you would need to restrict the errors to values that can be sent through ports, and your port definition should probably take this into account along with the output type:

port apiSession: String -> Task Decode.Value Decode.Value

Then you can use the task helpers to decode either.

What I really like about this is that you should now be able to write a lot of the existing browser API's using ports rather than native code.

[Tim Stewart]

Fantastic proposal for a good solution to a very real problem. I hope this gets accepted.

[Maxwell Gurewitz]

I've had the same idea.  Really hope this gets noticed!

[Erik Lott]

Although I'd love to be able to use and compose port commands like tasks, I'm not a big fan of this solution. 

regular ports only let you send data off or receive data back, not both.

This is a good design decision. Ports isolate Elm from the underlaying language (javascript) and avoids creating dependencies on the outside implementation: you fire commands out of ports, and don't need to be concerned about what happens to that data once it's exited the port. You receive subscription messages from js, and don't have to be concerned about how they were generated.

As soon as you send a command out of a port and expect something in return, you've created a dependency, and a new source of runtime errors that the compiler can't catch.

[Tim Stewart]

But isn't that covered by the error handling of tasks? The same way as when you use the "built-in" tasks like Http? Http gets around the general issue of "what happens if this never returns" using the Http.Error value Timeout. Maybe something similar could be used for Task Ports.

In the same case with standard Cmd / Sub, you would just never receive the Sub. It gives you LESS opportunity to handle the JS error through a timeout or similar mechanism.

Not to mention the added complexity when using Cmd / Sub of tracking separate, parallel calls to the same port. You have the overhead not only of managing two different entry points to your update function (as with any Cmd/Sub pair for an essentially single asynchronous operation), but of encapsulating some kind of state identifier that lets you match the incoming Subs with whatever Cmd they related to in the first place. The use of Tasks on the Elm side and Promises on the JS side makes this unnecessary.

I know, Elm design decisions are based on concrete use cases not on waffly hand-waving. I'll try and find time to write one up.

[Erik Lott]

But isn't that covered by the error handling of tasks? The same way as when you use the "built-in" tasks like Http? Http gets around the general issue of "what happens if this never returns" using the Http.Error value Timeout. Maybe something similar could be used for Task Ports.

It's hard to disagree with that rationale. A "Timeout" and "UnexpectedValue" error would be all that is needed to handle cases where functions return no promise, a non-resolving promise, or a promise with an incorrect return type. 

[Maxwell Gurewitz]

My only comment would be that the interface should not rely on promises, which are not supported by IE.  Instead it should use node style callbacks.

On Saturday, August 13, 2016 at 8:31:07 AM UTC-7, James Wilson wrote:

[James Wilson]

It wouldn't be hard to provide a promise shim, although I'm not sure how I feel about that. 

Callbacks would be the well supported option, although the interface to promises maps better. Promises can only resolve/reject once, always return something, and the function attached to the task port could just return a plain old value too, which would be equivalent to resolving a promise immediately with that value. This means that for synchronous calls the user can just provide a regular function to the JS side of the task port and not do anything Promisy at all. If we want to provide callbacks instead I would be tempted to provide 2 - one for resolve and 1 for rejection, and then add the promise logic (first one to be called wins) behind the scenes.

In the promise scenario, these are all valid:

App.ports.myTaskFunc = function(val){
    return (val + 2) //equivalent to returning a Promise that is resolved to (val+2) immediately
}

App.ports.myTaskFunc = function(val){
    return new Promise(function(resolve){
        resolve(val+2)
        reject("err") // this is ignored since we've already resolved.
    });
}

App.ports.myTaskFunc = function(val){
    return Promise.resolve(val + 2)
}

And the equivalent callback style might look like:

App.ports.myTaskFunc = function(val,resolve,reject){
    resolve(val+2);
    reject("err") // this is ignored since we've already resolved.
}

[Maxwell Gurewitz]

It'd be more in line with community standards if the callback followed the node convention

App.ports.myTaskFunc = function(val, cb) {
  cb(null, val + 2);
}

[James]

Good point, I had node in the back of my mind but somehow forgot what they did!

--
You received this message because you are subscribed to a topic in the Google Groups "Elm Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elm-discuss/TjWoacZobWw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elm-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Stewart]

Maybe it could accept either a promise or a (callback) function and behave accordingly? Anyone targeting IE could then decide whether to use callbacks or add a shim. 

To unsubscribe from this group and all its topics, send an email to elm-discuss...@googlegroups.com.

[Eirik Sletteberg]

I think this would be a very useful feature!

I forked the Elm compiler and made a working proof of concept: https://github.com/eirslett/elm-task-port-example

Is this still something people think would be a good idea?

[Witold Szczerba]

I think that you cannot say that promises are supported by IE or are not. It's just that IE 11 does not provide built-in implementation of ES6 "Promise" interface. It does not mean you cannot use any of the "Promises/A+" libraries. Promises, as a concept described by "Promises/A+ standard specification" were widely used before any browser provided a default ES6 implementation. This is really a great step forward compared to the callback equivalent.

Regards,

Witold Szczerba

[Oliver Searle-Barnes]

Having used Elm for 6 months now I'm not entirely sure why any API creates commands rather than tasks. Wouldn't it always be beneficial to have the option of composing rather than being forced into using an intermediate Msg? If you don't need to compose with another task then you can always add helpers functions to make it easy to create a command, but the opposite is impossible. Is there a scenario where you would want the API to force consumption as a command?

[Tim Stewart]

The lack of Task Ports basically lead to me putting Elm on the "watch-list" rather than actively using it on anything. Commands make it too difficult to track interdependencies between multiple separate async operations. I'm glad to see revived interest in this area, hopefully the project leadership takes interest.  

[Eirik Sletteberg]

It's easy to find concrete issues - a whole class of issues would be solved with task ports.

All problems that require interop between Elm and JS, where a message from Elm to JS must be correlated with a message from JS to Elm. The async request-response pattern. Like callbacks or Promises in JS land.

- Interacting with new web APIs that are not yet supported by Elm:

- IndexedDB, LocalStorage, SessionStorage

- Messages to/from Web Workers and Service Workers

- Sending two messages to the same outgoing port, receiving two responses on an incoming port, the responses may come out-of-order, correlating which response belongs to which outgoing message

With task ports, one can write less JS, more generalised, and then build business logic in Elm on top of that generic JS code. With the standard setup today, workarounds will generally involve more business logic on the JS side. Instead of using the elm-xxx library for example, for web API xxx (localstorage for example, where Elm development has stalled), users can make their own ad-hoc bridge, get Tasks on the Elm side, and compose these Tasks into more complicated Tasks. Where most of the complexity ends up in Elm, and as little as possible in JS.

[Jeff Schomay]

Just want to add my desires for this functionality and bump this topic.

My current need is to add a new `li` item to a `ul`, find its offsetTop and scroll to it (so that the top of the new item is at the top of the window).  I can't find a more "Elm-y" way of doing this besides hitting a port to get the offsetTop and then use Dom.Scroll.toY to get there (maybe with some animation).  It would be great to chain all of that in a series of tasks, but currently I have to make a new message for the port and respond to that with my scroll code :-/.

In general, after using Elm a lot over the many months, I'm learning that the monadic properties of Tasks (ie. chaining with `andThen`) are extremely powerful and expressive and composable, not to mention easier to test with libraries like arborist.  I'm just sad that there aren't more api's that return Tasks.

[Mark Hamburg]

Agreed.

Tasks actually were the core way to do things in 0.16 — though ports then were more focused on reactive programming models as I recall. Effects (what essentially became commands) were largely a shim on top of tasks. But then along came Elm 0.17 and it's emphasis on effects managers accessed via commands and subscriptions and we started to see less functionality delivered via tasks. But it feels like effects managers have stalled out — e.g., Evan's local storage effects manager, documentation, etc — leaving few clear answers here.

Ports are interesting but as noted don't really scale well for any cases where you need to send in messages and get back matched responses — particularly if those responses need to result in different messages to the app.

So, is there a reason for why Elm doesn't place more emphasis on tasks and provide something like task ports as a way to leverage JavaScript functionality?

Mark

--
You received this message because you are subscribed to the Google Groups "Elm Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.

[Martin Norbäck Olivers]

Yes, right now I want to make uuids by getting random numbers from window.crypto.getRandomValues. To generate good uuids without risk of collisions we need 128 bits of randomness, which the current random packages do not supply (they only supply 32, which is not enough).

It would be great to be able to do

port random : Int -> Task Never (List Int)

and in javascript do something like this:

app.ports.random.handle(function (n, callback) {   var array = new Uint32Array(n);   window.crypto.getRandomValues(array);   callback(null, array); });

Right now, I have to write a native function to make the task, or make two ports and subscribe and keep track of the random numbers somehow.

[John Kelly]

Here's​ some context on this request / discussion: https://www.reddit.com/r/elm/comments/61trtm/hard_things_about_ports_as_task_in_elm/?ref=search_posts

[Martin Norbäck Olivers]

An interesting read!

However it seems they sort of talk beside a big point of "task ports". My example is not so much communicating with the outside world in a general sense but more communicating with the browser, using in Elm unimplemented browser APIs.

The responsibility would be completely on the javascript side to return a result, but right now the only way to call these API:s is to write "Native" code, or to make "dual ports", one for sending a command, the result of which is then returned in a subscription and need to be paired with the call without any guarantee of the order of the results etc.

The question is, why is it worse to have a Task not return a result than to have this other mechanism not deliver a Msg? In both cases you'd get nothing.

Especially if Native becomes Kernel and even more unapproachable (which in itself is probably a good thing), then we really need a way to have the user make Tasks, or something similar to be able to use the browser APIs.

For databases, websocket-like stuff, http calls etc, that have internal state, then effect modules are probably better, but for calling a function to get some crypto random or to access LocalStorage? Seems like it's too much to create an effect module for that, just look at the code in my example above. That level of simplicity should really be available if people are going to be able to use browser apis productively.

Regards,

Martin

[Mark Hamburg]

Given the existence of Process.sleep, the argument that tasks have to return at some point and we couldn't guarantee that from JavaScript seems incredibly weak.

Given that effects manager development seems to have ground to a halt — Local storage? Phoenix? — it seems pretty vital for a language that talks up integration with JavaScript as one of its top features that there be a straightforward way to extend it with JavaScript. Command and subscription ports don't qualify because they don't provide a clean way to match requests to responses.

Mark

--

You received this message because you are subscribed to the Google Groups "Elm Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss...@googlegroups.com.

[Conner Ruhl]

What is the process for requesting these sort of features? Is there one?

[Nicholas Hollon]

The process:

1. Share your ideas, experience, & code on elm-discuss.

2. Accept that you have no direct influence over what Evan works on next.

3. Look at the release history for Elm. Notice that changes happen slowly. Notice that improvements to the JavaScript interface (ports, tasks, subscriptions) are spaced out by at least a year. Notice that 0.17 has only been out for about a year.

4. Remember that things are going to get better! Just because something isn't being worked on right this minute doesn't mean that it isn't going to be improved in the future.

6. Do things in life that make you happy. If it upsets you that Elm lacks something you think is super important, maybe take a break and come back later.

[Simon]

+1 for Task ports.

[Mark Hamburg]

Since the call is for concrete use cases, here is one: Reading from local storage.

My program wants to cache various things in local storage or for the purposes of this example it wants to read from various locations in local storage to get the values needed at various points in the model/UX. If we didn't use local storage and just used HTTP, we would generate HTTP requests at the points where we needed the information and using Elm message routing get the results delivered back. We would like to do the same thing with local storage as a cache — possibly followed by an HTTP fallback if we found nothing in local storage but that's beyond the scope here.

The analogous API to the HTTP case would be something like:

get : (Result Error Json.Decode.Value -> msg) -> String -> Cmd msg

Or to make it more composable — cough, tasks v commands, cough — we might have:

getTask : String -> Task Error Json.Decode.Value

Given that the Elm local storage effects manager seems to be on indefinite hold, we need to use ports. So, what does it take to do this using ports?

Command ports don't return results. Subscription ports don't convey information to the JavaScript side. So, we have to use a pair of them. A command to trigger work on the JavaScript side and a subscription to receive values back. But how do we match results coming back with requests coming in? How do we tag those results appropriately for delivery?

There would seem to be two options:

1. For every request we need to make, create a separate pair of ports. Now, we don't need to pass in the key string because the key is built into the port identity. In fact, it would be bad to pass in the key string since that would reintroduce the question of which response goes with which request. This approach will work if we can statically enumerate ahead of time all of the keys we are interested in AND we are prepared to write JavaScript handler code for each and every request. This might be viable for the big flat applications that some people like to advocate but it's a pretty messy maintenance situation and it breaks the moment we can't statically enumerate the keys of interest.

2. When we send a request in, we tag it with an identifying number and maintain a dictionary mapping those id's to tagger functions. The JavaScript side includes the id in its response — basically the same thing Phoenix push messages do — and the subscription port feeds into logic that looks up the id and tags and delivers the result. Those of you who are horrified at the thought of storing functions in the model should perhaps stop right here because that id to tagger function dictionary is doing exactly that. But even if one gets beyond that, note that this approach won't work just like HTTP because we will need at some point before this becomes a command to update the delivery map. That can either happen by replacing commands with requests or by allowing arbitrary update code to touch the shared global id generator and delivery map. Neither approach is as straightforward as the HTTP case.

This could be addressed by extending command ports to allow them to return responses but if we're doing that, why not just make them composable and generate tasks that take a Json.Encode.Value as input and return a Task PortError Json.Decode.Value (where PortError might just be another Json.Decode.Value or it might be some other response structure that could reflect other implementation errors TBD):

port get : Json.Encode.Value -> Task Task.PortError Json.Decode.Value

Now, I can wrap this up in code to do the encoding of strings to JSON (trivial) and map the resulting task result to do any appropriate decoding and delivery.

One could argue that this would all be handled in the local storage effect manager thereby rendering this example moot. But as noted above, that effects manager seems to be on indefinite hold. Furthermore, local storage was just a convenient and simple example. The exact same arguments could apply to almost any storage mechanism we wanted to talk to. If the Elm community were cranking out effects managers to handle these cases that would clearly mitigate this issue but that hasn't been happening. What's more such effects managers would almost certainly end up writing kernel (née native) code to provide task support for the interaction with JavaScript so instead of having a single mechanism for marshaling these interactions and keeping the relevant code out of the kernel, we would  instead have a growing number of pieces of code wanting kernel rights.

Mark

--
You received this message because you are subscribed to the Google Groups "Elm Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.

[Tim Stewart]

Very well put Mark.

Regarding Nicholas' advice on "The process"

> 6. Do things in life that make you happy. If it upsets you that Elm lacks something you think is super important, maybe take a break and come back later.

That's what I've done. I'm back in JS land and Getting Things Done. Elm's lack of a simple mechanism for interaction with external APIs (other than pub/sub style APIs, which it does well, but they are the minority) limits its practical application. The proposition that the tiny core Elm team will deliver built-in integration to all APIs in every JS runtime context (Electron? Cordova? Lambda? Chrome Extension? React Native? Arango? Espruino? ...) is clearly not realistic. 

To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss...@googlegroups.com.