Proposal: provide a basic set of crypto modules for use with Elixir

[Mark Madsen]

Crypto is super hard. And super important. 

Currently Elixir pushes users to call Erlang when working with crypto. This doesn't align with Elixirs goal of developer productivity.

My experience on working with Erlang's crypto application is:

• It is intimidating for newcomer who are just learning Elixir.
• While Erlang provides documentation, none of it refers back to Public Key Cryptography Standards directly making it difficult to work with or understand in the context of other libraries and code.
• Erlang provides building blocks, but they require writing a bunch of work to be useful (were starting to see this with pbcs extracted out of hex and now the work to extract plug_crypto out of plug).
• the data structures, for example the data structure for a certificate, are hard to work with. 

This has resulted in a couple hex packages up to this date to handle PKCS#5 and PKCS#7 padding. 

And more recently PBCS was extracted out of Hex https://github.com/hexpm/pbcs and there's work ongoing to extract the crypto out of Plug https://github.com/elixir-plug/plug_crypto

Another candidate is the certificate generation in Phoenix from https://github.com/phoenixframework/phoenix/blob/master/lib/mix/tasks/phx.gen.cert.ex - I think much of the code in there should be extracted. It shouldn't live in a mix task, but instead should come along with language. (not trying to pick on @voltone, I am very grateful for his work there, I wouldn't have done any better.)

But I'm personally guilty of worse. I have a bunch of code needed to parse certificates to extract details and provisioning profiles to extract PKCS#7 signature details. 

Crypto is so important that I think we need a central place to interact with any of the Public Key Cryptography Standards. Ideally we could have support for: 

• PublicKey/PrivateKey
• X509
• PKCS#5
• PKCS#7/CMS
• PKCS#10
• PKCS#12 

It would give us an opportunity to write documentation around the usage of the functions to help out newcomers and provide data structures that are easier to work with in Elixir. It also puts as much of this code as possible in a central place where we can get a lot of eyes 

I know this is going to be a lot of work. If I had to do it myself it'd probably take a couple years given the time I have to work on this kinda stuff and that assumes its my only priority (which it isn't). But I also see some of the pieces of this happening in separate places right now and if everyone contributed the work they are extracting from their codebases to one spot we'd probably be 30% or more done.

Eventually I think this is important enough that it should be part of Elixir, but to iterate on it initially I think it should be done separately. 

So I propose that someone makes https://github.com/elixir-lang/crypto and we start planning this out there. :)

Thanks

Mark (@idyll)

[Allen Madsen]

A proposal like this has popped up before. Perhaps you could address some of the arguments made there. The primary one being that improvements to crypto should happen in OTP.

https://groups.google.com/forum/#!searchin/elixir-lang-core/crypto%7Csort:date/elixir-lang-core/J-Idvs6ije8/eC-pmklSBQAJ

Allen Madsen
http://www.allenmadsen.com

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/dcf99a35-8a2b-4022-830e-983075b486d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Madsen]

Why not just do this in Erlang? Because it doesn't solve the problem of making Elixir easier for developers. If you need to use crypto in Elixir right now it's not easy.

Adding full PKCS7 to OTP still isn't going to make working with it in Elixir easy.

The plan here isn't to just wrap things. It's to make them easier.

But instead of arguments lets look at the facts:

 - some of code I am proposing is being written, already, in lots of places and lots of projects. examples were given. I'm sure there's more.

 - that same code is then being extracted into packages in an attempt to share it.

 - by the nature of software development if a bunch of people are writing the same code separately it's going to be less secure than if a bunch of people write it together and share/review it.

Now, there probably are some enhancements that do belong in OTP. And we should make them there. Off the top of my head I think that it's difficult to build a PKCS7 message in Erlang. Possibly dealing with detached signatures. We probably want those changes in OTP. But the way to discover that is to build this on top of OTP and find out what it's missing and then work on the pieces we need.

In my opinion this package is already being written. It's just not being planned out so we're only getting the pieces that people are extracting out of their code.

Instead of that approach lets plan this out, with a focus on developer productivity.

[José Valim]

Over my keynote at ElixirConf we discussed when to add things to the language, which are also in our development page, and although crypto is super important, having crypto functionality in Elixir would not provide anything different than if it belongs to a package. Therefore, I don't believe it is a good candidate for inclusion.

I would not treat the creation of plug_crypto and pbcs as a yellow flag but I would say quite the opposite, the community is finally starting to extract and share their crypto code into packages, which is exactly the direction we want to go.

In fact, if we want to move to a shared tool, breaking the existing code apart is a necessity, so we can preserve backwards compatibility for users of those APIs. However, both packages are still lacking in some areas, especially in the documentation front. pbcs focuses on algorithm names, so developers, including myself, have no idea how to use them. That's why the plug_crypto API focuses on use cases. At the same time, plug_crypto focusing on use cases means someone looking for an existing algorithm will have a hard time finding it.

I would also like to say that there are some disadvantages to creating a package that attempts to tackle everything related to crypto:

  * The package will provide too many functionality and become too large, which increases its footprint and makes it harder to maintain

  * The APIs will likely grow in size in terms of options, which also makes it hard to maintain and may become itself a security issue

Maybe the best approach is to create an elixir-crypto organization and have multiple packages in there. They can depend on a master package with basic functionality (like secure_compare) and have everything else built on top using shared conventions. I am also certain that this should not belong to the elixir-lang organization. If our answer to every problem is "let's put it to the Elixir team to maintain", then we are in deep problem. I do understand you were not necessarily asking the Elixir team to maintain it but, if it is in the elixir-lang organization, then it is OUR responsibility.

Thanks for the proposal, Mark!

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/57b38fc5-1eef-4d90-a7bf-87629759138b%40googlegroups.com.

[Mark Madsen]

I agree with a separate org. I picked here to start simply because most of the code out there that has relevant pieces probably belongs to someone on this mailing list. I think that once it is stable it may be a candidate for inclusion. But until it is stable I wouldn't consider it. 

I also agree that some of the packages can possibly kept separate from each other. And there's probably some base package where we'd want to extract some code common to all crypto as you mentioned. 

Also, I don't want to volunteer anyone, but I think there needs to be a single spot where we try to maintain important code related to crypto. If for no other reason than the trust aspect of it. I understand the concerns about security once it gets large but that's why a group of packages under a single org makes sense. And it provides us to be a little more forward thinking in what we want the packages to look like. The best example I can think of to illustrate my thoughts here is this:

Imagine we wanted to add a check when Phoenix starts up to ensure that the certificate being passed to Cowboy hasn't expired. Or log out a message with the expiry date so that it can be tracked easily.

That is reusable code. I can think of several other cases where people might want to check the expiry on a certificate. So then we extract it.

The problem is that the code needed to read out that date is very similar to code needed to read out any details of the certificate. And it probably belongs with code needed to manipulate certificates instead of being in a separate package, because it would either duplicate some of that code, or depend heavily on it and would be useful inside that package.

I am willing to help maintain it. But I definitely can't do this all by myself so I'm looking for support.

I also don't want hijack the code that is out there already. So I'm trying to gauge interest in this.

If I was to make elixir-crypto who should be a part of it?

Thanks for your feedback José. :)

[Christopher Keele]

Small nit:

Currently Elixir pushes users to call Erlang when working with crypto. This doesn't align with Elixirs goal of developer productivity.

I've always found the ability to dip into erlang a major factor of Elixir's promise of developer productivity. :D

I agree with the sentiment that the experience could be much improved, both by contributing to OTP, as well as perhaps creating a higher-level Elixir swiss army knife geared towards common applications of erlang's modules.

I wanted to contribute a point alluded to here but not directly stated: a major virtue of a vogue or standard crypto library living outside core is that vulnerabilities can be patched and deployed infinitely faster that the language itself. This agility is a commonly stated motive to have feature proposals be implemented outside of the language instead, but crypto seems a particularly compelling domain in which to prioritize that responsiveness.

I am willing to help maintain it. But I definitely can't do this all by myself so I'm looking for support.

I wish I could be of more help but I'd be useful as a maintenance triager only, it's beyond my realm of expertise. But I recommend making a post on the forums, perhaps to funnel interested parties into a dedicated #crypto channel on slack? I can see a lot of people there being interested in contributing.