Reframing "native" code as "kernel" code

[Evan Czaplicki]

Sometimes new folks wonder, "should I be writing native code?"

The answer is no and I have elaborated on why in this 2015 post.

A great example is the question "Should I wrap D3 for Elm?" This was not permitted, and now we have elm-plot which is entirely in Elm and does things in a way that makes sense for Elm. When charts are a view function, we can have a totally separate library for animation. Each can be simple and used together or separately or whatever. And everything can work nicely within the Elm Architecture!

So it takes longer, but I think the end result will be so much better!

So why does the question persist?

I think the question "should I be writing native code?" persists because of poor framing and communication on my part.

From my perspective, this means "Should I be writing x86?" For folks from other backgrounds, it means "Should I be writing JS?" or "Should I be wrapping JS libraries?" When I chose the term "native" however many years ago (probably before elm-get even!) I did not appreciate how it would sound from other perspectives.

Renaming "native" to "kernel"

One part of the plan is to rename Native.* to Elm.Kernel.*

Check out the description of a kernel. That is exactly the role JS needs to have in the Elm ecosystem. A minimal set of code that lets developers have access to things like device drivers (mouse, keyboard, geolocation, etc.) without giving full access to the machine.

Like I say in this 2015 post, the core benefits of this are reliability and portability. The main value of a language is in the ecosystem, and restricting kernel code means (1) there are no runtime errors in the entire ecosystem and (2) the entire ecosystem can come along to platforms besides JS without major changes.

I hope the name "kernel" will suggest all these things more clearly.

Note: Another example of "kernel" code comes from JavaScript itself. When you write JavaScript, you are not able to distribute C++ that extends and modifies Chrome. This is about security, reliability, and basic sanity. JavaScript doesn't have pointers. JavaScript doesn't have memory shared between different threads. "But the web could move forward faster if I had access to those things." Sometimes it's more complicated than that.

Ending the "Native Whitelist"

The native whitelist is kind of a mess. "Why this package and not that other one?" Basically, we were learning what the limits should be, so there is a lot of historical noise in there.

I think the idea of "kernel" code would be even clearer if only projects in @elm-lang and @elm-explorations can have kernel code. That means no more native whitelist.

The @elm-lang organization will be for high quality code that can be recommended as the default package to use for a particular problem.

The @elm-explorations organization will be for code that may one day be included in @elm-lang. Based on the current native whitelist, that would include these:

• elm-community/linear-algebra
• elm-community/webgl
  
• Skinney/elm-array-exploration
• BrianHicks/elm-benchmark
  

Affordances may need to be made for other packages though. For example, I know that elm-graphics is used in educational settings, so it would need to survive even if it should not ultimately have kernel code in it.

How does @elm-explorations work?

For things to get into @elm-lang, I go through the API and make sure it is up to a certain standard. This takes quite a lot of time and research, so it must be balanced with other priorities.

The four projects listed above (for WebGL, new arrays, and benchmarking) all have coordinated with me to plan out the long term life of these projects and to go through the API. When you are serious about adding kernel code, this coordination is required. In those four cases, the packages were quite far along by the time we did any coordination, and the authors were able to make a clear and compelling case for the coordination being worthwhile.

I tried to talk about how this works in this talk, but there is no simple recipe. It's not like "the secret" where if you visualize it, it will happen. Maybe the API just sucks. Maybe it's great, but a fundamental premise of the library is off. Maybe it's great, but it's not higher priority than other things going on. Etc.

Final Thoughts

Having Elm expose the entire Web Platform is the long-term goal. The process outlined here will be slower than other methods, but when the priority is quality, I think this is the best path.

My focus right now is getting 0.19 ready to go. It has many features that are very important to Elm's long-term health, and it'll make sense to cover more Web Platform after it is out.

Anyway, just wanted to outline some thoughts on this topic. Hopefully this message is helpful for folks who find themselves answering the question "should I be writing native code?"

[Mark Hamburg]

One further distinction that seems useful is between JavaScript code that can run as a synchronous function with access to Elm's data structures and JavaScript code that extends Elm but with a stronger abstraction barrier in place. If I read you correctly, what you are mostly focusing on is restricting/reducing the former. The 2015 post actually seems relatively down on JavaScript code in general for reasons of code quality and fit and yet at the same time we have people like Richard Feldman telling people over and over again to use ports to talk to native libraries rather than trying to re-implement various UI widgets in Elm. So, the middle way between these perspectives seems to be to aggressively avoid synchronous JavaScript code with "kernel access", wish that more things were written in pure Elm, but acknowledge that there are a lot of useful JavaScript libraries and a lot of functionality that Elm actually has trouble implementing in a clean manner (e.g., stateful UI widgets).

From an OS perspective, the key to having less code operating in kernel space is figuring out how to have more code operating in user space. This means putting attention on what sorts of code people are writing that operate in kernel space right now and looking for ways to get them to operate in user space.

The chief use I've seen for kernel access in JavaScript code has tended to be around creating tasks. As documented on another recent thread, ports simply don't scale all that well as a way to access JavaScript services. For example, if one needs to read and write different parts of local storage right now, it's a gnarly problem to do this in a general way with ports. On the other hand, the task ports proposal that has floated around would allow access to arbitrary JavaScript code from arbitrary points within ones code but would do so just the way any other tasks work and allows the Elm kernel to protect itself from JavaScript mischief.

So, I can strongly sympathize with an aggressive effort to push back on JavaScript in the kernel. But in support of that effort, there needs to be an aggressive drive to look at the cases where people are writing kernel access JavaScript code right now and find other ways to address those needs.

Mark

--
You received this message because you are subscribed to the Google Groups "elm-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elm-dev/CAF7GuPH7KSAV2%2BzFCAV517Y2%3Dsj_SdsF%2B5tiuq6UGCg2-z5Weg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Evan Czaplicki]

I'm pretty sure we agree. I emphasized improving Web Platform support, but making ports cover certain scenarios is also a relevant idea. I'm sure we can list other ideas that are related as well, but please let's not do that in this thread.

My point is mainly that the term "native" is misleading and clarifying that is part of a larger project to do interop right overall.

[Richard Feldman]

we have people like Richard Feldman telling people over and over again to use ports to talk to native libraries rather than trying to re-implement various UI widgets in Elm

I really don't want to derail this thread, but I also don't want someone to scan this and mistake my silence for tacit agreement; I actually laughed out loud when I read this. Mark, if you genuinely think this statement reflects reality, please ping me on Slack or email me and I'd be happy to discuss.

Apologies for the tangent. Back to comments on the OP. :)

[Max Goldstein]

I think the term "kernel" much better describes what this code actually is, and I support the renaming.

What happens to elm-tools/parser-primitives? I think its premise of giving everyone fair access to native/kernel optimized functions is a good one.

[Liam Curry]

I'm ambivalent about renaming "Native" to "Kernel". Either name is fine but Native seems clearer to me (coming from a JS background). Will custom Native modules (Elm.Kernel modules?) still be possible?

[Eirik Sletteberg]

> (2) the entire ecosystem can come along to platforms besides JS without major changes.

Is this an actual practical direction Elm is going into, or just a theoretical thought? I can see lots of reasons why Elm would be better off as an ecosystem designed for the js/css/html platform. (Use case driven development?)

Should I be writing kernel code? It just so happens that I need to access the localStorage API, and that cannot be done in Elm. So I copy-pasted the localStorage module from elm-persistentcache into my project. Works like a charm.

There are also web APIs X, Y, Z etc. that I need access to, with Tasks. Then, I can build business-specific web APIs on top of the native ones (wrap localStorage .get with my own .get which prepends APP_{userId}_ to all key names for example).

Same with the HTTP API, I can build a custom HTTP client which is composed on top of the native Elm client - it basically adds a configured root path to the API, adds authentication headers, wraps failures to more business-specific failures etc.

These constructs are enabled by Tasks, which are composable, while ports are not. (At least not without task ports) I would have to put all that business logic in JS instead.

So the simple answer is, yes, I have to write kernel code, because I want to have as much business logic as possible in Elm, not in JS.
When will it change? Whenever Web API X, Y and Z are supported by Elm. When will that happen? I guess maybe in 5-6 years. But I need to ship my app today.

I'm not saying Elm is not ready for prime time, it is - it's just that until most of the Web API is supported (or we have something like task ports), writing kernel code is the easiest solution.

[Max Goldstein]

A recent change to the JS code in elm-lang/linear-algebra -- one of the four "approved" packages -- broke everything unexpectedly. Native/kernel code is incredibly finicky and even "obviously no deficiencies" changes can introduce bugs. Writing your own is an incredibly bad idea. If that means you can't use Elm yet, that's how it is.

That said, I can see us making some changes to the ports API to encourage people to talk to JS that way. (Without endorsing any particular proposal.)

[Eirik Sletteberg]

I get your idea. But not being able to use LocalStorage shouldn't mean we cannot use Elm yet! I just think that's an unreasonable statement. Even if kernel code breaks between releases, if you have 200 lines of native/kernel code, then I guess you can live with that. (Until a better solution appears, at least)

[Liam Curry]

I agree with your sentiment, but to be fair LocalStorage is actually a pretty good candidate for ports. In my Elm apps I have a generic getter/setter that lets me store an object with LocalStorage with ports. Here is the JS code: https://github.com/liamcurry/frontend-skeleton/blob/master/src/js/runner.js#L9

Then any time you want to store something you can just call "setStored" https://github.com/liamcurry/frontend-skeleton/blob/master/src/elm/State.elm#L9

This is a pretty simple use case but it could easily be expanded.

[W. Brian Gourlie]

Having a blessed list of kernel modules does not have to be at odds with accommodating the writing and distributing of non-blessed kernel modules. It's needlessly restrictive at the moment, and an example of "perfect is the enemy of good."

Case in point: I've added features and fixed a major bug in the Websocket library, and the only way to use it that doesn't feel like a big hack is to include the entire native module in my project's repo. There's no nice way for other people to consume the updated library, so it's hard to get meaningful feedback. The consequence of all this is that additional work on this native module will exist in my project's repo, and there's no way for it to benefit the greater community, and less likely to influence the design of the blessed websocket library. Likewise with a ByteArray native module I wrote.

Developing native features in isolation, submitting a proposal and going through a lengthy feedback process simply does not scale. This model works fine for certain high-profile members of the community, but the vast majority of us are simply ignored. Allowing non-blessed kernel modules and letting people opt-in to using them (while making clear the consequences) will accommodate the community in developing and consuming things that the community needs. IMHO, this should be the first step to designing a blessed API: What is the community actually using? How are they using it? What can be learned from this initial implementation?

The consequence of the restrictive nature of kernel modules doesn't mean that people won't write them or consume them, they just won't have a nice way of distributing them. The actual consequence of this is that:

• It results in hacks like elm-package-install
• It discourages community development and feedback of much needed features
• There is a ton of untapped data regarding what the community actually needs, because the most natural source of that information disallows it.

I really hope the whole stance of kernel modules is re-examined at some point. 

Brian

[Maxwell Gurewitz]

One thing I don't understand at all is why web components are being pushed in the community when they suffer from all of the same problems and lack of guarantees as Native/Kernal code (lack of type safety, lack of determinacy, lack of versioning guarantees provided by elm-package) while requiring polyfills and a whole new build system (bower).

[Martin Bailey]

Should we be writing native/kernel code? It's a great question as Elm tries to keep JS at arm's length to simplify a switch to WebAssembly.

Web API's are an obvious use case for kernel code, but there are others where Elm just cannot be used today. Elm works well with text and JSON, but has no support for binary blobs. MsgPack/FlatBuffers/LZ-String come to mind. All my JS code could be removed if Elm had HTTP Blob and BitStream modules. Ports work despite the non-composable issues but it means I can't really contribute back to the community as it allows temporary hacks and runtime exceptions. How can we best enable Elm to cover such use cases without resorting to kernel code? Can we contribute there?

[OvermindDL1]

Can you elaborate on that?  The Web Components HTML spec should have nothing of the sort.  Things communicate 'to' something by setting attributes and properties (as you can do now in elm) and gets information back by receiving standard events (which you can 'kind of' do in Elm currently).  I think the main issue currently is that elm does not fit 'into' a web component well just yet as it has no method to send an event nor does it have a subscription ability on receiving property/attribute setters.  Those two things would make it pretty perfectly composable.

[Evan Czaplicki]

I want to remind folks of the What is elm-dev for? post.

The data and arguments people are making here are well-known. Many people want localStorage in Elm. Web sockets could be improved in various ways. Ports could handle certain cases different. Etc. I am aware of these things, and I suspect everyone on elm-dev is familiar with the various perspectives on each case. People will disagree about my prioritization choices, but that's not what elm-dev is for.

So let's move the discussion about Web Components to a different thread on elm-discuss or slack.

If you want to say more, and after reading What is elm-dev for? you think it belongs on elm-dev, please start a new thread about it.