Collecting use cases for File, ArrayBuffer and TypedArrays/DataViews

[Daniel Bachler]

I'd love to see support for the File and ArrayBuffer Apis, and maybe TypedArrays/DataViews as well. IMHO they are an important piece of the Web Platform that is still missing in Elm.

Evan suggested collecting concrete use cases to guide the design. I would like this thread to be the starting point of this effort. I would like to ask anyone who would also like this feature or who has substantial experience using either Api to add use cases or comment here so that we can try to define the user story for both apis. From there, we could decide what we would like to see supported and what, if anything, we don't need for now and suggest Elm Apis.

I have two stories from a side project of mine. It is a slideshow editor that allows the user to select photos and audio files from the local system, uploads them to a web service, let's the user arrange and manipulate photos and music and then share the result with others. For this, I have two immediate use cases plus some more ideas:

Upload local files as binary blob to AWS S3

In my current, (hacky) version, I use the FileReader api (via simonH1000's filereader library) to read the content of a file into an ArrayBuffer, (represented as Json.Value in Elm) then use a modified version of elm-http to upload the content of the ArrayBuffer to an S3 storage bucket.

Download mp3 files, decode them and play them back via the AudioApi

Currently I do this with my modified http library to download the mp3 file into an arraybuffer, then pass the resulting arraybuffer through a port to some native javascript that then uses the Audio Api to decode the mp3 file into a playable audiobuffer.

Parsing or otherwise processing local text files.

For another project I would be interested in reading and parsing Swagger/OpenAPI definition files and then providing a UI to compare rest apis. Since the processing will be done on simple Strings, this would only require FileReader support (specifically the readAsText method). This would already work with the FileReader library as is (though that one is not available on package.elm-lang.org because it contains native code and is not whitelisted).

TypedArrays and DataViews

I haven't worked with these yet, but I can anticipate some cases that would be interesting:

Parsing/manipulating of binary data via the ArrayBuffer api.

One case I personally would like to do with this, is to parse the Exif header of the jpeg files the user loaded from the local file system. My slideshow could then display metadata information without roundtripping to the server.

Create geometry for WebGL in the form of Vertex Buffers

Generating sound/music by writing raw audio samples

These could then be played back via the Web audio apis.


Please add your own ideas to this thread. Once we have compiled a list of use cases, we can look at the JS Apis available under the Web Platform for Files, ArrayBuffers, Typed Arrays etc. and think how these could be exposed to Elm.

[John Watson]

Hi, Daniel,  thanks for starting this off.  One thing I'd like to add:

Binary Base64 decoders.

These could obviously be for any binary resource that is smuggled into elm as text.  In my case I need to reconstitute soundfonts that are available as base64 encoded resources embedded in JSON.

I'm mainly interested in web-audio.  It seems to me if ever Evan (or somebody) gets round to doing elm-lang/web-audio, then binary types will be needed.  And I really would like access to web-audio (and web-midi) by means of a core elm library.   

[Michael B]

I'm making a browser based IDE, so support for the FileReader API is essential for opening local files.

[Michael B]

..and it's for an audio programming language, so the ability to read and write binary data would be fantastic.

[Peter Damoc]

I need to upload user avatar image and company logo. :) 

On Fri, Jul 29, 2016 at 11:54 AM, Michael B <mbyl...@gmail.com> wrote:

..and it's for an audio programming language, so the ability to read and write binary data would be fantastic.

--
You received this message because you are subscribed to the Google Groups "Elm Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

There is NO FATE, we are the creators.
blog: http://damoc.ro/

[Job van der Zwan]

On Thursday, 28 July 2016 23:17:51 UTC+2, Daniel Bachler wrote:

TypedArrays and DataViews

Please add your own ideas to this thread. Once we have compiled a list of use cases, we can look at the JS Apis available under the Web Platform for Files, ArrayBuffers, Typed Arrays etc. and think how these could be exposed to Elm.

Hi, I'm not using Elm yet but thinking about it because I'm bitten by undefined variables way to often currently and miss my statically typed languages. 

I'm developing for a research group to build a data browser for their growing RNA database. Basically, we'll be dealing thousands of cells with a arrays of thousands of genes each, and for each gene have an integer value representing how much RNA activation was measured, plus a bunch of metadata per cell. This data itself is immutable of course, but the plan is to on-the-fly render the datavisualisations (with as little rendering as possible - we'll probably use <canvas> for this).

So a compact way of storing and a fast way of accessing these gene values is important. TypedArrays would probably help here.

(also, suggestions on how to best use the canvas element in Elm are welcome, since elm-graphics appears more-or-less deprecated, but maybe I should open a separate topic for that?)

[Jan Weitz]

I would like Binary decoders/encoders as John Watson  mentioned, as well.

- Encoding/Decoding MQTT messages (which itself might contain Protobuf messages) [http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718013]

- Encoding/Decoding Protobuf3 binary messages [https://developers.google.com/protocol-buffers/docs/encoding]

Inspirations for native interfaces (for web or server-side) might be:

- https://github.com/dcodeIO/bytebuffer.js [Native/bytebuffer.js and ByteBuffer.elm]

- https://github.com/dcodeIO/long.js

- (https://github.com/dcodeIO/ProtoBuf.js) [Should be regular elm code]

- (https://github.com/google/protobuf/tree/master/js/binary) [Should be regular elm code]

[Simon]

I think this is the sort of code I would like to be able to write to do signed S3 uploads

FileReader API

readFile : FileRef -> Task Error Blob

and in the Http library

blobData : Blob -> Http.Data

such that

get ("signature" := string) "/get_signature"
`Task.andThen`
    \sig -> 
        readFile ref
        `Task.andThen`
            \content ->
                let 
                    body = 
                        multipart 
                            [ ("signature", stringData signature)
                            , ("file", blobData content)
                            ]
                in
                Http.send defaultSettings
                    { verb = "POST"
                    , headers = [("Origin", origin)]
                    , url = "http://bucket-name.s3.amazonaws.com/"
                    , body = body
                    }
                |> Http.fromJson ("ETag" := string)

[Daniel Bachler]

Thank you Simon for the example, this is great! I created a new thread to collect design ideas. Could you create a gist and post the link there?

[Erik Lott]

Our primary application allows photographers to upload hundreds/thousands of images for portfolio display. image delivery, etc. I guess you could say that our app is generally driven by image uploads. I would love to see this functionality provided by Elm, rather than having to use ports...

[Simon]

Hi all,

thought it might be worth updating this thread on my progress with S3, which I have now solved by adding one small hack to the Filereader library (github only because of the native code). I have documented this at https://simonh1000.github.io/2016/12/elm-s3-uploads/ 

Meanwhile, I have a new use to register. I was looking at MessagePack and noted that Elm is not included in the long list of supported languages. I suspect that it currently could not be because of the lack of ArrayBuffer support?

Simon

[Zachary Kessin]

I have an app I am working on that has to upload a file to a server for server side processing

Zach

ᐧ

--

You received this message because you are subscribed to the Google Groups "Elm Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Zach Kessin

SquareTarget

Twitter: @zkessin

Skype: zachkessin

[Simon]

My company uses websockets for mobile apps and thus wants to save on bandwidth. As a result all websocket traffic is encoded using msgpack, but the current library cannot handle that - presumably because of Elm's lack of underlying support for binary data.

On Thursday, 28 July 2016 23:17:51 UTC+2, Daniel Bachler wrote:

[Wojtek Piekutowski]

It would be awesome to be able to do directly in Elm:

- consume/emit msgpack

- use File* APIs

- direct upload to S3

On Thursday, 28 July 2016 23:17:51 UTC+2, Daniel Bachler wrote:

[Will Clardy]

I have a small hobby project that uses File and TypedArray to read and parse NES rom files, display their CHR data, and print interesting information from the iNES headers. I'd love to rewrite what I have in Elm and expand the project to include a pixel editor for the CHR data.

[art yerkes]

Since this thread has risen from the grave completely, handling JWT validation is kind of gross in elm.

https://gist.github.com/prozacchiwawa/d51b4e49e59a2aa0d3a28b328f62627d

Note that this requires a version of billstclair's sha256 that disables internal utf-8 handling (if there's demand, I'll submit a PR).

[Rupert Smith]

Just looking back at what I did for JWT tokens. Realised that all I had done was to decode the token (using truqu/elm-base64) and had not validated its MAC on the client. Not sure I need to validate on the client though - authentication/authorization is checked on all incoming server calls and all I need from the token is the user id, and their permission scopes.

+1 for ArrayBuffers for Sha256 and all other encrypted binary data formats that we might have to deal with.

[art yerkes]

Clients usually don't need to validate tokens.  I'm doing something a little special.  Still, it's a pain.

--
You received this message because you are subscribed to a topic in the Google Groups "Elm Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elm-discuss/u_ui3PlDwLc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elm-discuss+unsubscribe@googlegroups.com.

[Matthieu Pizenberg]

A bit late, but I can add my two cents on this matter.

What I needed to do was to display binary masks on images like the red Taj Mahal mask (01_mask.png).
Those binary masks beeing dynamically computed from an original mask and a user interaction
(like bounding box drawing in 01_bbox.png).

Doing this in elm + JS was a bit of a nightmare but I finally succeeded in a very hacky and computive intensive way. The process was the following if that is of interest to anyone. Pain points are in bold.

---- precomputed step
1. Transform the binary encoded RLE* mask to a JSON RLE that would be elm friendly (with Ints instead of compressed chars).

---- setup steps
1. Load the web page including the image (just providing the image url).
    1.1 send the url to a JS port
    1.2 generate a blob url for the image
    1.3 send back to elm this blob url to display it somewhere
2. Load the JSON version of the RLE mask.
3. Transform the RLE data to an actual Matrix data type (using a rewrite of eeue56/elm-flat-matrix to use an alternative Array implementation because elm core Arrays are broken -- see https://github.com/elm-lang/core/issues/649).

---- live steps
1. Display the image inside an SVG tag (see image 01.jpg)
2. Wait for user interaction (the bounding box drawing that is drawn live with SVG).
3. Convert the bounding box to a mask (Matrix Bool)
4. Compute logical operations beetween bounding box matrix and mask matrix to get a new mask matrix.
5. Send through port this computed mask (Matrix Bool) and the tag id of the svgImg tag prepared in elm to receive the image to be generated in JS.
6. Convert this Bool Matrix to a Uint8ClampedArray
7. Put this in a temporary canvas context2d (ctx.putImageData)
8. Generate an image url from this canvas (canvas.toDataURL)
9. Set the href attribute inside the prepared svg image tag (see 5.)

--> image 01_bbox.png

TADA! :S


* RLE (Run Length Encoding): https://en.wikipedia.org/wiki/Run-length_encoding

[Kasey Speakman]

Uploading files to AWS S3

That's my only use case so far for binary-type APIs of JS. I first call my own API to get a signed upload link, then PUT a file at that link. My simple way to upload files using Elm's Http module can be found here.

I'm also delivering streaming media to the browser, but I'm doing that through existing JS libraries in an iframe.

[Zachary Kessin]

I had a use in my old job where I had to upload a CSV file to a server for parsing etc. Pretty simple really, I used some package someone did with some native code.

Zach

ᐧ

--

You received this message because you are subscribed to the Google Groups "Elm Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Zach Kessin

Teaching Web Developers to test code to find more bugs in less time

Skype: zachkessin

+972 54 234 3956 / +44 203 734 9790 / +1 617 778 7213