Foundation for planning "native" APIs in 0.17

[Evan Czaplicki]

There has been a lot of talk of this recently, so I wanted to outline how I am thinking about this. It is quite complicated and interconnected, but it seems that it'd be good to share it all in one place. My goal here is to help, not to start any conflict.

In the past few weeks, I have been talking with Richard more about how they use Elm at NoRedInk, so the following is the result of our experiences and discussions.

Context

Goals and values of elm-package

Fundamentally, elm-package is about sharing high-quality code. The fact that the code there is very reliable is what makes it great. This is what makes Elm great overall.

This differentiates us from competitors where there are always 10 alternatives, many of them broken. Or competitors where a stranger's changes can break your code from a distance.

There's more we can do here, but that's what we already offer. And this is part of why companies like NoRedInk are using Elm!

Native bindings in healthy package ecosystems

When you think of "healthy" package ecosystems, you might think of Ruby or npm or maybe Haskell.

One thing they all have in common is that 99% of the packages in those ecosystems are written in the source language! You can bind directly to C in Ruby, but people do not do it. You can bind directly to C with npm, but people do not do it. You can bind to C in Haskell, but people do not do it.

Why? In all these cases, you do not actually want C bindings. On top of that folks probably don't want to write C even for the library, it'd be easier to just do it from scratch. Portability is also an issue in practice.

Okay, but JS is different. In our case everybody knows JavaScript. It is kind of platform independent (though in practice it is not with node). On top of this, a lot of folks may want to use a library they are already familiar with. So why not bind to that JS library? Over the course of 10 years, this would probably mean a significant fraction of Elm libraries are varying quality JS libs that got wrapped up and no one ever decided to do it in Elm.

Point is: our % of wrapped JS will be waaaaaay higher than in Ruby or npm or Haskell. Most people learning Elm are comfortable with JS, whereas this is way less true about C.

Evaluating the easy road

What would having lots of JS mean?

Remember the values of elm-package: high-quality, well-documented, reliable, etc. If I am scrolling through package.elm-lang.org and even 20 or 30% are thin wrappers around JS, I think we'd have a pretty serious problem:

• Native JS gives no guarantees. When you grab an arbitrary Elm package, the promise of Elm is that you have a near 0% chance of running into a runtime error. Wrapping JS means this goes away. It no longer "just works". This degrades how good the Elm ecosystem could ever be in my opinion.
• APIs are going to be ugly and forced. The mailcheck rewrite had an example where rewriting in Elm led to rethinking how some parts work. Talking with Richard, who has done some rewrites of his own, he has found that rewriting to Elm finds a lot of silly things. Richard is a badass, so I bet this is true of many JS libraries. Furthermore, the entire premise of Elm is that your code is going to be more reliable and easier to use if you do it Elm. It's not less true because there is a JS equivalent.
• It ties us to JS way more. I guess not many folks know this, but Elm is designed to be 100% independent of JS. It happens to compile to JS now because that was the right thing to do, but in a decade, I expect we'll be compiling to something more efficient. There is a small set of "native" stuff for now, but that means supporting the entire ecosystem on a new platform is fairly easy.

Over the course of a decade, I think these are all very very serious problems. This is why I have been conservative here.

Part of the initial reasoning for the review process was so that we could hear about all the different things folks wanted to do, so we'd have a better idea of what makes sense and what does not.

Outlining a better future

Recently someone asked: once you have "the web platform" covered, do you really want anything else done with native bindings? Do you want more than one binding to the geo library?

And my gut answer was no. If I had to choose between a Dropbox library done in JS or one done in Elm, I'd choose the Elm one every time.

To change the context: if the Dropbox library is already written in C, no Ruby programmer goes and wraps that up. They just make a version that makes sense for Ruby. One of the major benefits or Ruby is that you are not writing shitty, unsafe difficult, crashy C anymore. Why would they bring that into Ruby willingly?

So once people have full access to the web platform (e.g. websockets, blobs, geo, etc.) they can build anything in Elm. This is certainly more work, but I think it's a better future.

For reference, this is actually how Haskell does things in practice. There is a nice core of platform bindings for files and consoles and time that all work across all OSes. From there, everyone builds their stuff in Haskell.

Changing our thinking

Regardless of the details of native code in Elm, I think we need to flip our mindset.

Instead of thinking "oh, there's a JS library, I wish it was wrapped up" think "there's a great opportunity to contribute to the Elm community by writing more Elm". I think the mailcheck rewrite could be a great model. There certainly could be exceptions (perhaps crypto) but I think this should be the rule in 99% of cases, just like any other language.

The goal is to build high-quality, reliable libraries with well-documented APIs that are fun and easy to use. Some are blocked for now. Some are not. Focus on those!

My current thoughts on native bindings

So finally we get to some more concrete thinking about how to get to this better world.

Main goal: focus on creating bindings to the web platform in something like elm-lang/platform. It is too early to tell exactly what this will look like, but I'll be sharing my progress as there are more details!

Side goal: Get folks to accept that ports can do what you want 95% of the time, but make it much clearer how to do a one-off hack if that is what you need to do at work as a stopgap. If you think the burden of this is too high, don't use Elm yet. We'll get there! No one wants you to have a bad experience, and it's possible that Elm is not ready for the thing you want yet! That's fine. Circle back after a few releases.

Goal for the community: Build more cool stuff in Elm. Once the "platform" is available, this will expand out even more.

I think things are relatively vague here, so I will be outlining more of the practical particulars in another email and as I make more progress.

Chill Out

I hope I have earned at least some amount of trust in this community when it comes to design and quality, so I ask that you always keep in mind that I stress about this every day. I am trying to go fast. I am trying to do a great job technically. I am trying to respond to people's concerns. I am trying to create delightful features and blog about them. etc. etc.

In the meantime, some folks are building amazing projects, like this website and everything on it. Some folks are converting mailcheck. Some folks are creating elm-format or great editor plugins. There is tons of stuff out there! Choose not to block. Choose to make a cool thing in Elm.

If you are blocked, document why in a clear way and move on. If you are frustrated with something, bring it up directly, not in a vaguely related thread. Maybe there's a good reason or simple fix or something! I see this stuff. I'm not a dummy. Sometimes things are blocked on purpose. Not everything is a good idea. I can't send a massive email like this every time this happens, especially if things are early and more exploration is still needed.

So overall, please be more chill! We can't accomplish everything right this second, and throwing more people at these problems isn't the answer. I am working really hard, and this is a hard problem that has implications for the next couple decades of my life, not just Elm.

So again, choose not to block. I am working on this.

P.S. This thread is intended to clarify what I am thinking, not to solicit suggestions. I really don't think line-by-line quotes are going to be helpful, especially for something as long as this. If you have specific ideas about the technical content here that have not been surfaced already, it probably makes sense to start a clear coherent thread about it rather than hijacking this (or other) threads. If you want to obliquely tell me to go faster, please understand that I already know people feel this way. In any case, please pause before responding to this thread and ask yourself if it will make the thread more or less productive. On track or off track? New thread?

[Ryan Rempel]

For what it is worth, I entirely agree that it would be wise to focus on providing a set of bindings to platform facilities, rather than wrapping existing JavaScript libraries. I think you are right to worry about the problems which wrapping existing Javascipt libraries causes.

[Max Goldstein]

I also agree with the idea of a "platform". The websockets library that I wrote fits into that nicely.

One approach to writing a library, which is a bit clunky but follows the "choose not to block" dictum, is to output a signal of JSON values which the client must wire to a port, and then attach a provided JS handler. For example, if you want to write a sound library, instead of wrapping howler.js you can come up with an interface that feels "native" to Elm, and then work with the web audio technologies directly rather than through a library. Although, this assumes that the underlying web technologies will be simple. (On the flip side, JS libraries usually have mutation baked into them, so it wouldn't be "fair" for an Elm type to correspond directly to their JS object, even in a native library.) This leaves the library in a state in which it can become a native library fairly easily, if desired.

Come to think of it, there doesn't seem to be any sort of audio library yet. I might have to do that (though someone else please feel free).

[Luke Westby]

Will the idea of a set of platform bindings for "the web platform" extend to similarly embedded bindings for other environments where Elm might thrive as a compile-to-js language? The specific example I mean to reference is Electron, in my opinion an incredible use-case for Elm, which ships with embedded APIs that augment the chromium wrapper and are not implementable in Elm. I've begun working on these bindings at https://github.com/elm-electron and I'm not sure from the above what the future status of this project might be since it doesn't quite fit into one side or the other (web platform vs. rewritable in Elm). 

[Evan Czaplicki]

Luke, I agree that that is an important thing to support. Other examples include elm-webgl and node. Basically other "platforms" in my mind. This gets us into a world where a certain package may only work on certain "platforms", so it is not enough to solve constraints, we need to label each package with the platforms it needs.

So I'd like to get one platform sorted out and see what that looks like before building in language support for many platforms. I think those are two separate and difficult problems, and if I try both at once, I'm probably going to have a bad time.

If there is more to say, let's discuss it in a separate thread.

[Irakli Gozalishvili]

Hi Evan,

I absolutely share your concerns about the code quality that wrapped JS libraries may cause. That being said I can't help to think that even without much governing things will play out just fine. It also maybe worth considering ecosystem of languages other than JS, Ruby & Haskell that have more common with Elm in terms of platform bindings. For example Clojure(Script) in many ways is in the same shoes and while many libraries do wrap existing Java / JS libraries that still tend to provide an idiomatic Clojure interface for working with them & reason is users of Clojure like those idioms and they do their best to stick with them. Of course there are some bad examples there as well, but typically such libraries are overthrown by  more idiomatic alternatives fairly soon. Unfortunately Clojure(Script) does have runtime exceptions so arguments can be made that bad wrappers may harm Elm much more than it could Clojure(Script).

I don't personally know much about ecosystem of Scala but I imagine they likely have similar concerns by having access to all the Java libraries. Might be worth learning how that played out on code quality in their ecosystem.

BTW people also prefer packages with pure JS over packages with native bindings in npm because later needs to be build and rebuild on node updates & they also can crash node. So inconveniences of platform does stare ppl away which is very likely to be true with Elm as well. In fact just making packages with platform bindings harder to find on http://package.elm-lang.org and warning signs on those is likely to help steer authors towards pure elm alternatives ;) 

Please don't take this comment a wrong way, I'm not trying to rush you or argue, just trying to bring up more info that may help you while thinking through this problem & if you'll come up with something more robust and better then everyone here wins ;)

[Naman Goel]

I had a completely different perspective to this, but reading your post has convinced me.

I have a question about a specific use-case. How should we approach problems that are not web-apps. I'm thinking things like plugins for Atom and VS Code, Scripts for OS X etc (Applescript and JS is supported).

Of course, you're building bindings to the native web platform. But if I want to write plugins for Atom in Elm, there needs to be a package with binding to it's API. It can essentially be thought of as another platform.

What are your thoughts on this? I can think of three options:

1. Elm is for web-apps for now, and we should not use Elm for these use-cases

2. Use ports with a JS app, and use a boilerplate instead of a package. (A personal thing, this is probably easier as I don't have to write bindings for everything, but just the stuff I use)

3. Start work on bindings to various projects. It is a valid use-case for 'native' code.

[Evan Czaplicki]

I always recommend ports. Start a new thread about your particular scenario though. Tell us more about what you want to do and what the constraints are.

--
You received this message because you are subscribed to the Google Groups "elm-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elm-dev/a13a9357-ea64-412d-9e34-18fadacef515%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Eugene Nikolaev]

Sorry, I am not 100% sure it is appropriate thread for my comment.

To be honest I've found the thread while googling how to use native code in Elm)
But you has convinced me too...
I am tired of JS and bunch of new fashionable frameworks. I like Elm.
We definitely should have our own libraries and not to wrap legacy js code.
But I wonder what to do if say, we need a really complex library?

Say, highcharts or ace editor etc.
I guess it is possible to use them through ports, but I see, it is like a palliative.
I am not sure my question requires a clear answer. I just would be very grateful if you wrote what you think on this.

воскресенье, 4 сентября 2016 г., 19:58:45 UTC+3 пользователь Evan Czaplicki написал:

To unsubscribe from this group and stop receiving emails from it, send an email to elm-dev+u...@googlegroups.com.

[Murphy Randle]

Hi, Eugene!

I've recently needed to use some more complex 3rd party libraries (a markdown editor, for example), and ports have proved a pleasant way to get that done.