Proposal: Stdlib Core Distribution

[Ed W]

Elixir has a goal to keep Stdlib light. This seems very sensible.

Perl solves a similar problem by keeping core lean, but bundling a
preferred group of modules as a "core" distribution (I don't think
anything would stop you distributing an alternative core bunch of
modules with a private build).

In practice this means modules have been added and dropped through time.

This seems like a very practical solution for Elixir. For example it
makes it feasible to ship Elixir + some modules for (say)
profiling/benchmarking. In the future if some other module takes the
lead on offering better profiling support, there is the option to
exchange profiling modules. Likewise you could solicit other data
structures such as Heaps, Trees, etc. If in the future it's felt they
no longer offer value, simply drop them (or change them for new best of
breed options)

The benefits:
- If you drop/replace a module it doesn't break anything, users simply
need to install it via the normal packaging methods

- It encourages a certain amount of centralisation of development
effort. ie some modules become "recognised" and slightly special and
there tends to be an increased probability to use those first. For me,
this means I don't install three modules, each of which uses completely
different date/time libraries and hence my project is less likely to be
pulling in tons of overlapping functionality...

- It encourages building small, solid, modular functionality (perl is
unafraid to release modules which are 2-3 code lines long, usually with
50-100 lines of documentation explaining the subtlety which is being
solved and wrapped)

- "standing on the shoulders of giants". Much innovation develops from
extending and expanding on existing projects. Base modules looking more
"official" encourages re-use, debugging and extending, rather than
re-inventing the wheel every time. With perl I regularly might see a
module dependency graph 2-3 deep (or longer) of small compartmentalised
functionality AND I will not be fearful of this. The dependencies
frequently look solid and I don't have fear of breakage. Contrast this
with the current state of Elixir where for example I can't point to
Ecto, and in turn see it build on a solid external validation or
date/time library... Were this Ruby/Perl I would expect to see a
dependency on a solid Date/Time library, which itself had an active
community building extensions to the date/time lib to support all kinds
of interesting corner cases.


The key point here is that there is a tendency to rebuild functionality
time and time again unless
a) the dependency feels long term solid and dependable (best of breed
and well supported)
b) Ideally that the dependency itself is extendible to handle future
corner cases, and/or, it's simple to drop in a replacement

Simple examples today might be:
- Ecto and Phoenix are fairly clearly established as "go to" libraries.
I would feel comfortable to build a project on one
- Point me to a date/time library? There are several, but I would feel
less comfortable picking a winner today? If I needed just to quickly
print or parse an ISO date format I would find myself very inclined to
re-invent the functionality.

A new approach from Elixir core could offer a new competitive
environment for "best of breed" to flourish and hence encourage re-use
and reduce splitting of effort

Thanks for listening

Ed W

[Peter Hamilton]

A better comparison is erlang. When installing erlang on ubuntu from the erlangsolutions ppa, here are all the packages available:

erlang-asn1 - Erlang/OTP modules for ASN.1 support

erlang-base - Erlang/OTP virtual machine and base applications

erlang-corba - Erlang/OTP applications for CORBA support

erlang-crypto - Erlang/OTP cryptographic modules

erlang-dbg - Erlang/OTP symbol files

erlang-dev - Erlang/OTP development libraries and headers

erlang-diameter - Erlang/OTP implementation of RFC 6733 protocol

erlang-doc - Erlang/OTP HTML/PDF documentation

erlang-edoc - Erlang/OTP module for generating documentation

erlang-eldap - Erlang/OTP LDAP library

erlang-erl-docgen - Erlang/OTP documentation stylesheets

erlang-eunit - Erlang/OTP module for unit testing

erlang-examples - Erlang/OTP application examples

erlang-gs - Erlang/OTP graphics system

erlang-ic - Erlang/OTP IDL compiler

erlang-inets - Erlang/OTP Internet clients and servers

erlang-megaco - Erlang/OTP implementation of Megaco/H.248 protocol

erlang-mnesia - Erlang/OTP distributed relational/object hybrid database

erlang-nox - Erlang/OTP applications that don't require X Window System

erlang-odbc - Erlang/OTP interface to SQL databases

erlang-os-mon - Erlang/OTP operating system monitor

erlang-parsetools - Erlang/OTP parsing tools

erlang-percept - Erlang/OTP concurrency profiling tool

erlang-public-key - Erlang/OTP public key infrastructure

erlang-runtime-tools - Erlang/OTP runtime tracing/debugging tools

erlang-snmp - Erlang/OTP SNMP applications

erlang-src - Erlang/OTP applications sources

erlang-ssh - Erlang/OTP implementation of SSH protocol

erlang-ssl - Erlang/OTP implementation of SSL

erlang-syntax-tools - Erlang/OTP modules for handling abstract Erlang syntax trees

erlang-tools - Erlang/OTP various tools

erlang-webtool - Erlang/OTP helper for web-based tools

erlang-xmerl - Erlang/OTP XML tools

erlang - Concurrent, real-time, distributed functional language

erlang-appmon - Erlang/OTP application monitor

erlang-base-hipe - Erlang/OTP HiPE enabled virtual machine and base applications

erlang-bear - Set of statistics functions for erlang

erlang-bitcask - Log-Structured Hash Table for Fast Key/Value Data

erlang-bitcask-dev - Log-Structured Hash Table for Fast Key/Value Data (development files)

erlang-cherly - Cherly (sher-lee) is an in-VM caching library for Erlang

erlang-cl - Erlang OpenCL bindings

erlang-common-test - Erlang/OTP application for automated testing

erlang-cowboy - Cowboy is a small, fast and modular HTTP server written in Erlang

erlang-cowboy-doc - Documentation files for erlang-cowboy

erlang-cowboy-examples - Examples for erlang-cowboy

erlang-debugger - Erlang/OTP application for debugging and testing

erlang-dialyzer - Erlang/OTP discrepancy analyzer application

erlang-esdl - Erlang bindings to the Simple Direct Media Library

erlang-esdl-dev - Erlang bindings to the SDL (development files)

erlang-esdl-doc - Erlang bindings to the SDL (documentation)

erlang-et - Erlang/OTP event tracer application

erlang-folsom - Erlang based metrics system inspired by Coda Hale's metrics

erlang-folsom-dev - Erlang based metrics system inspired by Coda Hale's metrics

erlang-guestfs - guest disk image management system - Erlang bindings

erlang-ic-java - Erlang/OTP IDL compiler (Java classes)

erlang-jiffy - JSON NIFs (Native Implemented Functions) for Erlang

erlang-jinterface - Java communication tool to Erlang

erlang-manpages - Erlang/OTP manual pages

erlang-meck - mocking library for Erlang

erlang-mode - Erlang major editing mode for Emacs

erlang-observer - Erlang/OTP application for investigating distributed systems

erlang-pman - Erlang/OTP process manager

erlang-proper - QuickCheck-inspired property-based testing tool for Erlang

erlang-proper-dev - QuickCheck-inspired property-based testing tool for Erlang - development files

erlang-proper-doc - QuickCheck-inspired property-based testing tool for Erlang - document files

erlang-ranch - socket acceptor pool for TCP protocols in Erlang

erlang-ranch-doc - Documentation of erlang-ranch

erlang-reltool - Erlang/OTP release management tool

erlang-test-server - Erlang/OTP server for automated application testing

erlang-toolbar - Erlang/OTP graphical toolbar

erlang-tv - Erlang/OTP table viewer

erlang-typer - Erlang/OTP code type annotator

erlang-x11 - Erlang/OTP applications that require X Window System

erlang-yaws - Erlang application which implements HTTP webserver

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/561985CC.1040309%40wildgooses.com.
For more options, visit https://groups.google.com/d/optout.

[Eric Meadows-Jönsson]

Erlang is actually an example of where this is bad, there are lots of applications bundled with OTP that should not be part of OTP, it has way too many applications to be manageable so we have outdated applications in OTP that should not be used or are not maintained. It also gives the OTP team less time to focus on their core applications.

I also don't like that ESL started shipping common libraries, that are not part of OTP, with their Erlang distribution. It becomes a problem because some developers depend on a globally installed library in their project but does not add it to their dependencies, when other people use their project it will break because they don't have that library on their machine or they have another version of it. Another issue can happen if they do add the library to their project but use another version than what is globally installed, now you have two versions of the application in your load path.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAOMhEnyapjO69rjAvoSyAjYop9hDzNvcf48PPzzK3K0qFYx-wA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Eric Meadows-Jönsson

[Tallak Tveide]

I suggested this earlier, but have since decided against it.

Better to have a single hex package, for instance called elixir-plus-pack or something. Then handle that outside Elixir core. The release cycles could follow elixir core, but some time in the future, elixir releases will probably become less frequent, while these kinds of libraries tend to evolve around a stable core forever.

I dont think it is trivial to remove a module that was earlier included in core.

[Ed W]

On 11/10/2015 00:10, Eric Meadows-Jönsson wrote:
> Erlang is actually an example of where this is bad, there are lots of
> applications bundled with OTP that should not be part of OTP, it has
> way too many applications to be manageable so we have outdated
> applications in OTP that should not be used or are not maintained. It
> also gives the OTP team less time to focus on their core applications.
>
> I also don't like that ESL started shipping common libraries, that are
> not part of OTP, with their Erlang distribution. It becomes a problem
> because some developers depend on a globally installed library in
> their project but does not add it to their dependencies, when other
> people use their project it will break because they don't have that
> library on their machine or they have another version of it. Another
> issue can happen if they do add the library to their project but use
> another version than what is globally installed, now you have two
> versions of the application in your load path.

These all seem fair criticisms. However, the counter point is that
without some kind of channelling and encouragement of some module to be
"the default", you end up with 3 date-time libraries in your Elixir
build because each of your dependencies picked a different one...

The problems above seem very important to solve with such a proposal,
but don't seem to be "unsolvable" either. Please keep in mind that one
of the important goals of such a project would be to have important
modules which aren't quite "core", but equally are very nearly core
quality/usefulness (eg datastructures, common parsing requirements, date
handling, etc). Done correctly this might reduce the load on core
developers...

[Eric Meadows-Jönsson]

> The problems above seem very important to solve with such a proposal, but don't seem to be "unsolvable" either.  Please keep in mind that one of the important goals of such a project would be to have important modules which aren't quite "core", but equally are very nearly core quality/usefulness (eg datastructures, common parsing requirements, date handling, etc).  Done correctly this might reduce the load on core developers...

Who should maintain this distribution? If it's core developers I don't see how it will reduce load, if it's not core developers there is not anything stopping you from creating a distribution and try having it adopted by the community.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/561D2799.3050008%40wildgooses.com.

For more options, visit https://groups.google.com/d/optout.

--

Eric Meadows-Jönsson

[Ed W]

On 14/10/2015 03:41, Eric Meadows-Jönsson wrote:

> The problems above seem very important to solve with such a proposal, but don't seem to be "unsolvable" either.  Please keep in mind that one of the important goals of such a project would be to have important modules which aren't quite "core", but equally are very nearly core quality/usefulness (eg datastructures, common parsing requirements, date handling, etc).  Done correctly this might reduce the load on core developers...

Who should maintain this distribution? If it's core developers I don't see how it will reduce load, if it's not core developers there is not anything stopping you from creating a distribution and try having it adopted by the community.

Is there not a middle ground?

a) The point is to have it "steered" by core.  I think the project doesn't fly without a benevolent dictator to guide
b) Its supposed to be a collection of best-of-breed projects developed by others. However, see a) above, you don't get into the club if you don't party with the benevolent dictator...

This is how it works for other projects.  eg going back to perl stdlib, its a bunch of projects, I guess some or all not developed by the perl core developers. Projects drop in or out of the stdlib. The definition of stdlib (versions, libs, etc) is frozen with each release of perl.  I am not involved with the politics, but I suspect that core most likely makes feature requests of the stdlib developers and influences direction of the individual modules.


OK, lets drop the theoretical and state the problem:

- Jose decides we should have a Map data type in Elixir
- He (and others) build a Map data type, maintain the data type, push for upstream changes
- Bingo, we have a high quality module.  Success! (for map type at least)

- Since then, several people suggested we need a DateTime and a Queue library (and much more)
- As a result of that we got a half dozen implementations of each, varying in quality and features and at least some of them have failed to achieve critical mass and start that nice spiral of success where they become popular, which in itself begats further quality improvements and contributions.
- So the status quo today is every time someone needs a tiny bit of datetime formatting, or a simple data structure, we generally just re-invent that bit or cut and paste it from another project to avoid adding a dependency to a bit of code of unknown long term availability.


So the problem is how to break out of this situation?  How do we get to an end position where people start to trust the dependencies and start to build on them? 

Today it's easy to point to examples of great "failings" (my definition). My biggest would be "datetime".  This is such a non trivial subject and hence errors are made repeatedly in implementations, which in turn means more sophisticated libraries should be used to avoid hitting these corner cases (eg witness how many busted implementations of ISO date formats are in the wild...).  So why isn't Ecto depending on a solid datetime lib?  I have no idea what Phoenix depends on, but if it's not a dependency on a solid external library then I'm sad...

I'm out of my depth, but I keep wondering why there are so many Elixir re-inventions of the equivalent of Ranch or Poolboy?  This seems like something with a lot of corner cases and yet everyone and his dog seem to keep re-inventing the core easy bit?


"Standing on the shoulders of giants" is where we always want to be.  My question is how can we redirect community efforts to drive forward, rather than losing effort repeatedly building the same "building blocks" again and again? 

Any thoughts? (or are we doing ok for the moment?)

Cheers

Ed W

[José Valim]

Folks,

I will say once more: depending on the core team *does not scale*. The language was designed to be extensible exactly so the core team does not become a bottleneck.

 

Today it's easy to point to examples of great "failings" (my definition). My biggest would be "datetime".  This is such a non trivial subject and hence errors are made repeatedly in implementations, which in turn means more sophisticated libraries should be used to avoid hitting these corner cases (eg witness how many busted implementations of ISO date formats are in the wild...).  So why isn't Ecto depending on a solid datetime lib?  I have no idea what Phoenix depends on, but if it's not a dependency on a solid external library then I'm sad...

I would actually say "datetime" is an example of great success. We have two main libraries: calendar and timex where the maintainers are syncing and helping each other. Not long ago I remember Laut providing some suggestions where timex could be improved. If we had merged timex when it first came out, as many proposed, it would be very likely Elixir v1.0 would ship with bugs in its datetime and we would be stuck with those for a good time. To prove the point further: none of those libraries have hit 1.0 yet and depending on the core team would make reaching 1.0 even slower.

Why Ecto isn't depending on it? Because Ecto is extensible and allows you to pick your favorite, specially now where there isn't a preferred option yet.

 

I'm out of my depth, but I keep wondering why there are so many Elixir re-inventions of the equivalent of Ranch or Poolboy?

I have no idea what you are asking. ranch is literally the 5th most download package on Hex. poolboy is right there on 8th. We have said over and over again that the Erlang ecosystem is a strength and to not wrap existing Erlang libraries and this is clearly being adopted by the community.

Sure, someone may reimplement a pool but that is a *good thing* if they are aware of the trade-offs. Maybe someone would provide a better solution than poolboy, like sbroker.

Finally, I don't think I can add anything more to this discussion. My instance on the topic has been pretty clear since the beginning, the core team won't be involved in a "plus pack" effort. Instead, we trust the community to build, grow and choose solutions that fit Elixir's goals. It won't come over the night, so patience is required.

[Booker Bense]

On Wednesday, October 14, 2015 at 1:25:10 PM UTC-7, Ed Wildgoose wrote:

On 14/10/2015 03:41, Eric Meadows-Jönsson wrote:

> The problems above seem very important to solve with such a proposal, but don't seem to be "unsolvable" either.  Please keep in mind that one of the important goals of such a project would be to have important modules which aren't quite "core", but equally are very nearly core quality/usefulness (eg datastructures, common parsing requirements, date handling, etc).  Done correctly this might reduce the load on core developers...

Who should maintain this distribution? If it's core developers I don't see how it will reduce load, if it's not core developers there is not anything stopping you from creating a distribution and try having it adopted by the community.

Is there not a middle ground?

a) The point is to have it "steered" by core.  I think the project doesn't fly without a benevolent dictator to guide
b) Its supposed to be a collection of best-of-breed projects developed by others. However, see a) above, you don't get into the club if you don't party with the benevolent dictator...

This is how it works for other projects.  eg going back to perl stdlib, its a bunch of projects, I guess some or all not developed by the perl core developers. Projects drop in or out of the stdlib. The definition of stdlib (versions, libs, etc) is frozen with each release of perl.  I am not involved with the politics, but I suspect that core most likely makes feature requests of the stdlib developers and influences direction of the individual modules.

The constraints under which the core libs in Perl were created are much different than the internet that exists today.

Back then a 9600 baud modem was "fast" and the idea of "loading on demand" was simply not possible.
Compiling an open source program was generally an all day process dealing with figuring whether you had
the right path to the *.h files. There was much less standardization among unix OS's in play than there
are today. To do all that work and still have to download files just to talk to an SMTP server would have
put a barrier to actually using the tool. 

It's a mistake to apply a solution that worked then to now. Personally, I think Ruby and Perl have shown
attempting to put the kitchen sink in the core distribution is a mistake in the long run.

On the other hand, I think you do have a point. The single developer supporting a library that hundreds of
projects depends on doesn't scale either. What Jose is saying is that it's not up to him and the core group
to solve this problem, it's a problem we need to solve as a larger community.

IMHO, the quality of a particular library is not the problem, the real problems are :

1. Is this API stable?

2. Is this API supported?

The real problem we need to solve in the short term is how do we make the lives of single developers
publishing API's easier. If you can do that, then their limited time can be spent on coding.

Once Elixir gains enough traction, (i.e. some company's bottom line depends on Library X working),
then you're over the single developer barrier, and market forces will largely take care of the
library maintenance problem.

What is required is an external to the Elixir core project to backfill around all the tedious chores
that keeping an API to production quality requires. We don't need a benevolent dictator, we need
an army of minions.

I have some thoughts on this problem and hope to have something to propose in the near future.

- Booker C. Bense