Batching HTTP requests and then performing something after they all finish

[Rafał Cieślak]

I have a dict which acts as an album cover cache:

1. I download a list of albums from Last.fm.

2. I send a separate request for each album to get its cover image URL (right now i use Cmd.batch).

2.1 After I receive an image URL from Last.fm, I add it to the cache, so that I have a mapping from album ID to album image URL.

3. Once all the album image URL requests finish (no matter if any of them fails), I'd like to send the cache through a port so that JS can save it in localStorage.

I don't know how to approach this. The Cmd API allows for batching, but not sequencing and the Task API allows for sequencing, but not batching. I can turn a Task into Cmd, but not the other way around.

So if I have a bunch of batched HTTP requests, I'm not really able to sequence something after they happen, or at least I don't see how I can do this without adding some dirty workarounds to the update function.

Theoretically I could wrap the album image URLs into RemoteData and then after receiving any of them, see if the rest of the album image URLs is finished and if so, send the cache to JS.

[Brian Hicks]

It sounds like you might actually want to do this in update! You're going to have to write an Msg for getting new album covers anyway, right? And, conceptually, it sounds like you want to save the results you get but cache them for future use through a port.

Your logic branches there, so it makes sense to say, in response to your Msg, "save these to display to the user. Also, store them so I don't have to make this request again."

If my assumption there is wrong, please correct me. The mechanical way to do exactly what you're asking is putting your tasks in Task.sequence and sending the result through Task.andThen. But I'm not sure you want to do that, for the reasons above. Your task handling logic would get really hairy, when it has a natural fit in the Elm Architecture.

[Eric G]

If you model the dict values (image URLs) with something like

type AlbumImageState

    = NotLoaded
    | Cached Url
    | Loaded Url
    | Error Http.Error

Would that help?  When you get back the list in the first request, initialize the values to NotLoaded. Then you could then write a function like

allLoadedOrError : List AlbumImageState -> Bool

to check if all the album-image requests have returned, before writing the successes to localstorage.

[Rafał Cieślak]

Eric: This is what I meant when I suggested wrapping the URLs with RemoteData. ;) It does pretty much exactly what you described and I'm thinking about using it anyway, as it seems that it's going to be helpful in other parts of the app as well.

Brian: As far as I understand your approach, your way of solving that involves sending each new album cover through a port separately, which is okay. I was thinking about sending the whole dict (cache) to JS once I know that I finished downloading all the images.

 it sounds like you want to save the results you get but cache them for future use through a port 

More-or-less yes, with the difference that right now the cache lives in Elm and the only thing I want to do outside of Elm is to save it to localStorage so that I can rebuild the cache the next time I reload the app.

So your approach could certainly work, but I think I'm going to use the second solution and wrap the values in the cache with RemoteData. I think I'd rather have as few interactions with JS as possible – in the second solution I'd pass the cache from JS to Elm through flags and then pass it back from Elm to JS once I finish downloading all images.

---

Obviously we could come up with even more ideas. I think my main gripe was that you can either sequence then batch or just batch, but you can't batch and then sequence, because the API doesn't allow you to do that.

I haven't given much thought to it, but I feel it's because Task and Cmd have fundamentally different approach to error handling. Adding Task.batch would require either imposing certain semantics of errors (like Task.sequence does) or offering an API that would allow the user to implement error semantics themselves. And I'm not sure how easy of a task the latter thing is.

Even in JS with promises such sequencing would require some not-quite-elegant solutions.

[Brian Hicks]

I think there's been something lost here. I'm suggesting you do:

    Task.sequence [ allYourThings ] |> Task.perform Cmd

And then in the update,

    { model | covers = covers } ! [ storeInJS covers ]

--
You received this message because you are subscribed to a topic in the Google Groups "Elm Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elm-discuss/22Tt_p2OpMY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elm-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rafał Cieślak]

Oh okay, sorry. :D

But this won't batch the requests, right? By "batching" I mean "run all in parallel" as Cmd.batch seems to do. As far as I understand Task.sequence docs, it's going to make the requests one-by-one. I'd rather load them in parallel.

What's more important though is that Task.sequence is going to fail if any task fails – I'm okay with some of the requests failing and this behavior would make this API a bit harder to work with in my case.

[Eric G]

On Sunday, December 3, 2017 at 8:25:37 AM UTC-5, Rafał Cieślak wrote:

Eric: This is what I meant when I suggested wrapping the URLs with RemoteData. ;) It does pretty much exactly what you described and I'm thinking about using it anyway, as it seems that it's going to be helpful in other parts of the app as well.

I had assumed you were caching at the individual album-image level, which is why I had that additional `Cached Url` case. But maybe you don't need it.