Let's Auth v0.9 : The Rewriting

[Dan Callahan]

I'm confident in the broad shape of the experience we've prototyped with OpenID Connect, which means it's time to start talking about rewriting the daemon in a more production-ready language.

Reminder: The production daemon most be trivially self-hostable. For this reason, we're favoring languages that can produce statically compiled binaries.

The most promising languages in this domain are Rust and Go.

Rust:

- Can be very efficient and predictable, since it does not have a GC.
- Guarantees that compiled code is free from data races.
- Trivially supports FFI calls, thanks to its C-compatible ABI.

However, I don't find these attributes compelling in this case:

- We're a network service; round-trip latency will dwarf any GC pause.
- We're unlikely to need to share mutable structures between threads.
- We're a service, not a library. We care about HTTP more than libffi.

I specifically favor Go because:

- It's a simpler language, and thus has a lower barrier to contribution.
- It has a larger community, and thus more potential contributors.
- It has M:N threading built-in, trivially handling many concurrent clients.
- It's significantly easier to cross-compile projects.
- It has a more mature and broad ecosystem of third-party libraries.
- It was specifically designed for developing network services.

I believe those attributes are important to our success, and we would regret not having them.

Possible mitigating factors:

- Rust is still novel; contributors may be willing to overcome barriers.
- A small community is fine as long as we can find *some* contributors.
- The Iron web framework handles request concurrency as a library concern.
- Cross-compilation *is* possible with Rustup, it's just higher friction.
- We only need a few libraries; we'll probably be OK.
- It's not as pretty, but Rust *can* be made to do everything Go can do.

And the elephant in the room: djc has already begun hacking on a daemon written in Rust at https://github.com/djc/ladaemon.

Thoughts? Where should we go with this?

Possible options:

1. Commit to Rust.
2. Commit to Go; djc's daemon potentially becomes a competing implementation.
3. Build a temporary experiment in Go, compare the results to Rust.
4. Something else?

Please discuss,
-Dan

[Dirkjan Ochtman]

On Wed, Jun 8, 2016 at 10:24 PM, Dan Callahan <dan.ca...@gmail.com> wrote:
> Thoughts? Where should we go with this?

I think your bias somewhat shines through in the manner of your
reasoning, here. :) While I'm also biased, let me try to pivot your
thinking into a more objective line of reasoning, defining
(non-functional) requirements first and validating the options against
those later. The daemon:

1. Should run in a variety of environments
2. Should be easy to install in those environments
3. Should perform reasonably well
4. Needs some libraries (HTTP, crypto, email, key/value store)
5. Needs a low barrier of contribution

Going over these for Go and Rust, I would argue 1-3 are basically a
wash. Maybe cross-compiling is slightly more difficult for Rust, but
not dramatically so; if we just agree that it needs doing, it'll get
done. I totally agree that GC pauses won't have much of an impact for
this project. I think you're arguing that Go has more libraries and a
larger ecosystem. I don't disagree, but I also think that the Rust
ecosystem is large enough for our purposes, and quite vibrant itself.

As for low barrier of contribution, yes, more people know Go right
now. On the other hand, the Rust ecosystem is quite active. I think
you estimated 20-30 hours to build out a first version of the daemon
in Go, and you are at least somewhat familiar with Go. Without having
written more than hello, world in Rust before, I estimate that I spent
less than 30 hours on my version, suggesting it's not as inaccessible
as you might think. If you are put off by all the import lines,
remember that the import lines will make it very easy to find exactly
where some symbol is defined, which I find to be extremely helpful
when reading code.

I would argue there is one requirement you didn't mention explicitly
that I think is important, and that is robustness/reliability.
Especially because we're dealing with security-sensitive data, and
because we'd like to be internet-scale, I think robustness is pretty
important to this project. It is my feeling that Rust wins out over Go
here, because this is exactly what it was designed for. The strict
type system makes it so that successful compilation almost always
means code that does the right thing. The amount of unwrap calls in my
prototype may look unwieldy, they also provide very obvious markers
where we need some error handling, and if we replace these calls with
proper error handling, I think we can be very confident indeed in the
robustness/reliability of the Rust daemon. (Having no Go experience, I
don't know how it does error handling exactly; but my impression from
what I've read is that it's quite a bit easier to leave mistakes in
it.)

For me, the daemon prototype was a good project for learning Rust.
Having learned it, I've found myself immediately wanting to use it
more. I don't mind if the community prefers a Go implementation; in
that case, my implementation can either just linger or be an
alternative implementation. Personally I don't think I will contribute
much to a Go implementation, just because I'm not interested in Go as
a language.

Finally, I would argue that, at a meta level, this community doesn't
seem to have a large amount of resources at its disposal. Because I
was intrinsically motivated to build a Rust version, and not a Go
version. Even if, based on the requirements we've described so far, I
argue that Rust is better, and you argue Go is better, I think we can
agree that the difference is not very large at all. If you're
motivated now by having a production version of the daemon, it would
be more efficient to spend a few hours with this Rust code rather than
spending (probably) more hours on building it again in Go.

Still, I'm clearly biased, and I don't want to force the community to
go in any particular direction. If the rest of the you think Go is the
better option, you should definitely go for it.

Cheers,

Dirkjan

[David Kitchen]

I'm pro-Go, but I do not think it is productive to debate what language the server is written in.

What matters more is that:

* We have a reference version of the daemon

* The reference version is incredibly well documented

* The reference version is actively maintained

* The reference version is easy to deploy

* The reference version is easy to support

* The reference version is what we install/use

That other versions of the daemon will exist, ports be made, and variants emerge in other languages is a good thing and the competition is healthy and it acknowledges that different people may have different priorities (i.e. the deploy, support, to customise, etc).

I know resources are limited but it is not a drain on resources if a developer wants to contribute a version in Ruby and the reference is in Rust (or Go, etc). The Ruby dev is unlikely to have contributed to the Rust version and it is not a loss of resources for multiple versions to exist. Besides, when other instances emerge it does feed back into and improve the reference version by pointing out which interfaces don't work well across languages, or which flows and concepts have small flaws or are misunderstood and should be documented better or changed.

If we're happy acknowledging that servers in other languages may be created, then the question is really which language should be chosen for the reference daemon? That's less binary a choice then just thinking it's between Rust vs Go.

I'd say stick with the Rust, document it, and if others are motivated to produce a Go version then that's cool too and we should embrace that. It's not a split or schism to have more than one version, more a fun race to deliver the best solution.

If when we come to launch/announce we have just a Rust version, it's obviously the reference daemon and we're all good.

If at that time we have 2 daemons to choose from because that's what people wanted to build, then whatever best meets the criteria for being a reference version should be declared that but both can live and be used and now those deploying their own have a choice.

A version in Rust (or Go) doesn't preclude all others, after launch we should collectively embrace/support whichever is best suited to being the reference implementation.

Basically to go back to Dan's options:

1. Commit to Rust.
2. Commit to Go; djc's daemon potentially becomes a competing implementation.
3. Build a temporary experiment in Go, compare the results to Rust.
4. Something else?

I offer a something else:

5. Whomever is motivated to build and document a daemon should do so, if we end up with both a Go and Rust one we pick the one that best meets the criteria of a reference implementation to champion whilst acknowledging other versions.

--
You received this message because you are subscribed to the Google Groups "Let's Auth" group.
To unsubscribe from this group and stop receiving emails from it, send an email to letsauth+u...@googlegroups.com.
To post to this group, send email to lets...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/letsauth/CAKmKYaBBsmNboE%3D7paaREy1rbSPfcAQbYbpb5zsr_mqbPWzK%2Bw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[onli onli]

>

5. Whomever is motivated to build and document a daemon should do so, if we end up with both a Go and Rust one we pick the one that best meets the criteria of a reference implementation to champion whilst acknowledging other versions.

We still need to choose one as our reference version, and we need the one daemon where we focus our effort one. If I want to help with the implementation, I don't know how, maybe by implementing another OpenID IdP, then it has to be clear where that effort of mine should go to. Of course we'll forbid no-one to work on another implementation, and if some other implementation gets better we should switch. But it needs to eb clear what is the version to install for users, and what to work on for contributors.

Like said in IRC, I'm not familiar enough with those two languages to properly help decide. But I can say that I find Dirkjans post convincing. I still have problem with the actual rust code though. In which state is that version on https://github.com/djc/ladaemon/ from your perspective, codewise? Is that as clean as it gets from your point of view? I think it is really problematic that it is one single file. The routing is mangled in the main file, there is no inner abstraction via objects, also the HTML is in there. I find the code hard to read, there is not one single comment describing what functions/structs are for. That makes the advantage of having a rust client now a lot less convincing to me.

So from my perspective, we can also set on rust. But if we take the rust ladeamon as a starting point, I'd appreciate a lot if that in a first step would code-wise get improved for readability, by comments (functions declaration at least), structure and an external html template. Only then can we expect others to help implementing.

[Dirkjan Ochtman]

On Mon, Jun 13, 2016 at 12:33 PM, onli onli <onl...@gmail.com> wrote:
> I still have problem with the actual rust code though. In which state is that
> version on https://github.com/djc/ladaemon/ from your perspective, codewise?
> Is that as clean as it gets from your point of view? I think it is really
> problematic that it is one single file. The routing is mangled in the main
> file, there is no inner abstraction via objects, also the HTML is in there.
> I find the code hard to read, there is not one single comment describing
> what functions/structs are for. That makes the advantage of having a rust
> client now a lot less convincing to me.

I guess clean is a relative thing. To me it is very clean, in the
sense that it's the simplest thing that is needed to handle the two
happy paths for email loop and famous IdP login.

What's your problem with it being in one file? It's less than 500
lines, with some clearly delineated blocks of code. To me,
distributing that over multiple files only makes it harder to work
with, as you have to skip around in different files. Similarly, I
don't see how "abstraction" or "objects" are needed for this stuff,
where we have a few fairly simple HTTP request handlers and some
helper functions. Having straight HTML in the source is the simplest
thing that could possibly work, and seems more than enough for this
particular use case at this stage. Yes, having templates in separate
files is nicer, but it also requires adding yet another dependency,
with its own added complexity.

Obviously, this is not a mature project. It's more proof-of-concept or
prototype stage. Still, there is no unnecessary code in there at all,
and I think the code is organized well.

> So from my perspective, we can also set on rust. But if we take the rust
> ladeamon as a starting point, I'd appreciate a lot if that in a first step
> would code-wise get improved for readability, by comments (functions
> declaration at least), structure and an external html template. Only then
> can we expect others to help implementing.

I've added some ~150 lines of comments now. Let me know if this helps
your understanding, and if you still think it's disorganized/hard to
grok with these comments.

Cheers,

Dirkjan

[Stavros Korokithakis]

As a datapoint, I find it much more readable with the comments, thank
you. At a quick glance, Rust doesn't look too unreadable, although I'm
sure that if I tried to change anything I'd have to spend an hour
fighting the compiler. Even so, it looks easy enough to understand to
me, now that it's better documented.

Stavros

[Dirkjan Ochtman]

On Thu, Jun 16, 2016 at 11:50 AM, Stavros Korokithakis <h...@stavros.io> wrote:
> As a datapoint, I find it much more readable with the comments, thank
> you. At a quick glance, Rust doesn't look too unreadable, although I'm
> sure that if I tried to change anything I'd have to spend an hour
> fighting the compiler. Even so, it looks easy enough to understand to
> me, now that it's better documented.

Thanks, that's good to hear!

Don't overestimate how hard it is to make changes. Many of the
compiler messages are quite readable. As I mentioned earlier in this
thread, I was pleasantly surprised in the end how quickly I was able
to get up to speed.

Cheers,

Dirkjan

[Stavros Korokithakis]

I've been meaning to get into Rust, but my problem is that I find it
very hard unless I have a project that's a good fit for the language.
Hopefully, LA will be that project.

Stavros