Support for binary data

[John Watson]

Can anyone tell me what the plans are for supporting binary data in elm?  I'm thinking of a Byte (and some sort of Byte Array) type and also implementing Blob in HTTP responses. 

[Evan Czaplicki]

I have been drafting Blob and ArrayBuffer APIs, but I wasn't sure who needed them.

What is your particular use case?

On Mon, Jan 11, 2016 at 4:55 AM, John Watson <john....@gmx.co.uk> wrote:

Can anyone tell me what the plans are for supporting binary data in elm?  I'm thinking of a Byte (and some sort of Byte Array) type and also implementing Blob in HTTP responses. 

--
You received this message because you are subscribed to the Google Groups "Elm Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Watson]

Thanks for replying, Evan.  I need to download soundfonts (these are maintained as JSON and contain Base64-encoded data), decode them and pass them to Web-Audio. I can do this all in a native library but I have been attempting to follow your advice and do as much work as possible in pure elm - i.e. binary base64 conversions. 

[Daniel Bachler]

I need it too, for these use cases:

My app is a slideshow editor. You drag photos into the webapp (extracting the File metadata like the filesize via a Json decoder in the ondrop event), then the files are read with a FileReader as ArrayBuffers (via SimonH's FileReader module). The library treats the ArrayBuffer as a Json.Value ATM. Next they need to be uploaded to a server, so round-tripping the ArrayBuffer to Http.send would be required.

A more advanced use case for working with Files would be a kind of "Lens" library to work on ArrayBuffers/ArrayBufferViews - then I could write a Library that decodes the Jpeg header in Elm to check if it is a valid jpeg image and give the user feedback if the dimensions are too small or too big etc.

My app also accepts sound files and after loading they are currently handed to a port to be decoded in native JS (and stored there because I kept the audio playback library in native JS and control playback via ports).

[Jesse Levine]

I have a very similar use case to Jon. I would like to download raw mp3 data and pass it over to Web-Audio. I could do this in not-Elm but I would prefer not-not-Elm!

[Jesse Levine]

ArrayBuffer is the API I would need btw. 

[Dave Keen]

I'm doing audio recording with WebRTC and it would be useful to have an ArrayBuffer so I could mostly stay in Elm-land.  Also I'm doing some uploading to S3 buckets and it would be great to be able to use Blobs in Elm instead of ObjectURL strings.

[Daniel Bachler]

You can hack this ATM if you need to go via blobs. The filereader library can already handle binary blobs as opaque Json.Value values and you can hack elm-http to be able to pass those through when uploading files (like I did on a branched version of elm-http here: https://github.com/danyx23/elm-http/tree/encode-as-json ). You can't use the package manager for the filereader library and my fork because they contain native code and native module review has stopped. Altogether it's quite a hack, but if all you need is to ingest blobs and upload them, this is a relatively painless way that allows you to stay in elm land. Let me know if you need more details, I have used this exact path in an app I developed.

AFAICS the upcoming 0.17 version should make this workaround superfluous.

--
Daniel Bachler
http://www.danielbachler.de

--
You received this message because you are subscribed to a topic in the Google Groups "Elm Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elm-discuss/spr621OlUeo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elm-discuss...@googlegroups.com.

[Dave Keen]

Thanks!  That's a nice approach, but I've cobbled something together with ports for now and with 0.17 round the corner I'll wait and see what it brings before changing anything over.

[Alex Shroyer]

I could use ArrayBuffer for lots of things, for what it's worth.

[Ian Mackenzie]

I'd love to have ArrayBuffers supported for doing WebGL stuff. For rendering very large and complex scenes, I think it would be useful to be able to do a lot of the heavy lifting (level of detail and occlusion calculations etc.) on the server, and then just send compact ArrayBuffers to the client that could be passed directly to WebGLRenderingContext.bufferData() or similar (via an Elm wrapper, presumably, but without any parsing/conversion/copying).

[Gábor Varga]

This feature would come handy for me too.

Our use-case is that we get some encrypted data from an MQTT broker and it might be broken up to several messages. 

The binary data first has to be reassembled before it can be converted to UTF8 encoded strings.

[John Mayer]

I took a shot at starting a basic version of this in my fork of websocket and a new package simply called binary. My approach was largely thin wrappers on top of the spec.

Evan, is your draft public? No updates from you since January. IMHO, don't try to carry this one yourself. This is an opportunity to spread the load and develop yourself as a manager of contributors while building out the process for accepting large contributions.

[Yosuke Torii]

I need binary things too for two cases, uploading files and Web Audio. These are not special things.

I'd like to know the plans, when it is most likely to come (or how these features are prioritized).

2016年6月21日火曜日 19時36分59秒 UTC+9 John Mayer:

[Simon]

I need to be able to upload PDFs

Simon

[Gábor Varga]

Another use case:

I would like to add Elm support to the flatbuffers compiler. 

Flatbuffers let you describe your protocol in a lightweight schema description language and generate low-level DTOs and serialization logic in different languages. If Elm supported binary data we could generate flatbuffers messages in pure Elm, in a lot friendlier way than the JavaScript implementation.

We use flatbuffers in a realtime video streaming application.

[W. Brian Gourlie]

My use-case:  I'm creating a debugger front-end for a NES emulator I'm writing.  Having to serialize binary data (in particular, 16KB of address space) to be consumed by elm is inefficient in about every way possible.  I currently pack it into an array of 32-bit signed ints before serialization and use a port to convert that to a UInt8Array.

Also, I wrote the disassembler in typescript because I could operate on UInt8Array (again integrated via port).  It would be possible in elm, but would have felt a bit hacky.  ArrayBuffer would fix that!

Brian

[Simon]

My need is to upload PDFs directly to S3. That requires two additional things

- the ability to read a file from the hard disk - the sort of native code needed is at https://github.com/simonh1000/file-reader/blob/master/src/Native/FileReader.js

- an extra function in Http with signature `blobPart : String -> Blob -> Part`

Simon

[Martin DeMello]

Seconding this one - I need to be able to pass a binary file to my backend, which requires the ability to both load the file and attach binary data to a POST request.

martin

--

You received this message because you are subscribed to the Google Groups "Elm Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.

[Duane Johnson]

My use case:

I have a compiled microcontroller file (a ".hex" file) delivered to the client browser. I use the data to draw a fancy SVG barcode that is subsequently displayed on the screen and scanned by a camera (like a QR Code).

I need a way to retrieve binary data from the server, and iterate over 16-bit integer values.

[Simon]

My use case:

My company uses websockets for mobile apps and thus wants to save on bandwidth. As a result all websocket traffic is encoded using msgpack, but the current library cannot handle that.

On Tuesday, 12 January 2016 01:32:43 UTC+1, Evan wrote:

[Coury Ditch]

Our use case: Pure elm library for cryptographic functions. 

We are creating a pure Elm web3.js equivalent. We think Ethereum developers and the blockchain developer community at large would benefit from the strong guarantees of using Elm.

Thanks

[Berry Groenendijk]

Coury,

I am very interested in an Elm version of web3.js. For exactly the reason you mentioned. Where can I follow the progress on this project?

Berry

Op zaterdag 8 juli 2017 21:30:53 UTC+2 schreef Coury Ditch:

[Coury Ditch]

Here are a couple relevant repos so far - they are mostly barren at the moment, as we're trying to solve some hard API problems before we put too much code down.

https://github.com/cmditch/elm-web3

https://github.com/cmditch/elm-truffle-webpack

[Zachary Kessin]

I would really like to have a way to do file uploads in Elm, I needed that at a project a while back

Zach

ᐧ

To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Zach Kessin

Teaching Web Developers to test code to find more bugs in less time

Skype: zachkessin

+972 54 234 3956 / +44 203 734 9790 / +1 617 778 7213

[Martin Bailey]

What kind of interaction with binary data would you require for your projects?

-Blob : black box to store and pass opaque data around (anything that isn't a valid Unicode string)
-Byte array : equivalent to char/string for arbitrary bytes
-Bit stream : more flexible mechanism to implement binary protocols or binary coding

Byte array would remove the need for blobs and would also allow building a clean bitstream library in pure Elm. It needs to be designed properly for binary data to be reliable and fun to work with, but that would probably be the best abstraction. Haskell calls theirs Data.ByteString.

[pukkamustard]

Our use-case: Decoding and encoding binary hardware data.

We connect to various hardware via a TCP <-> WebSocket bridge, via WebBluetooth and via a Chrome App. Currently we do all low-level hardware connectivity (including binary decoding/encoding) outside of Elm in Javascript and exchange data with ports. The complexity required to manage all the hardware on the Javascript side is increasing and it would be very nice to handle all that neatly in Elm.

On Wednesday, 19 July 2017 04:34:06 UTC+2, Martin Bailey wrote:

What kind of interaction with binary data would you require for your projects?

A byte array (ArrayBuffer) seems to be what we need. We need enough low-level access to decode binary protocols (currently we use Javascript DataView). It should be usable with WebSocket and other WebAPI modules (e.g. WebBluetooth).

[Coury Ditch]

I think it would be really excellent to begin work on something like crypto-elm (a la crypto-js). It will take a lot of work, but I find it a sorely needed foundational library for many other elm libraries I'd like to build out.

A byte array seems like the only option viable in this particular case. (Please correct me if I'm mistaken).

[Christophe de Vienne]

Our use-case:

We connect our elm application to a NATS server through a websocket [1]

The protocol is text-based but the payload can be binary. We would like
to switch to protobuf to encode our payloads but cannot do that because
of the elm client. So we stick to json for now.

So the need is to be able to encode/decode data structures to a binary
websocket connection.


[1] https://github.com/orus-io/elm-nats

Le 19/07/2017 à 04:34, Martin Bailey a écrit :
> What kind of interaction with binary data would you require for your
> projects?
>
> -Blob : black box to store and pass opaque data around (anything that
> isn't a valid Unicode string)
> -Byte array : equivalent to char/string for arbitrary bytes
> -Bit stream : more flexible mechanism to implement binary protocols or
> binary coding
>
> Byte array would remove the need for blobs and would also allow building
> a clean bitstream library in pure Elm. It needs to be designed properly
> for binary data to be reliable and fun to work with, but that would
> probably be the best abstraction. Haskell calls theirs Data.ByteString.
>
> On Tuesday, July 18, 2017 at 3:10:33 AM UTC-4, Zachary Kessin wrote:
>
> I would really like to have a way to do file uploads in Elm, I
> needed that at a project a while back
>
> Zach

> ᐧ
>
> On Wed, Jul 12, 2017 at 8:18 PM, Coury Ditch <cmd...@gmail.com
> <javascript:>> wrote:
>
> Here are a couple relevant repos so far - they are mostly barren
> at the moment, as we're trying to solve some hard API problems
> before we put too much code down.
>
> https://github.com/cmditch/elm-web3

> <https://github.com/cmditch/elm-web3>
> https://github.com/cmditch/elm-truffle-webpack

> <https://groups.google.com/d/optout>.

>
>
> --
> You received this message because you are subscribed to the
> Google Groups "Elm Discuss" group.
> To unsubscribe from this group and stop receiving emails from

> it, send an email to elm-discuss...@googlegroups.com <javascript:>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
>
>
> --
> Zach Kessin
> Teaching Web Developers to test code to find more bugs in less time
> Skype: zachkessin
> +972 54 234 3956 / +44 203 734 9790 / +1 617 778 7213
>

> --
> You received this message because you are subscribed to the Google
> Groups "Elm Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to elm-discuss...@googlegroups.com

> <mailto:elm-discuss...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.

--

Christophe de Vienne