Haven't had a need for an HTTP server in Erlang for a while since I last
used inets-based http server five years back, and now examining what
exists in public domain.
The results are quite surprising - everyone who seems comfortable coding
in Erlang, feels compelled to implement an HTTP server. Previously we
had only one contender to inets - yaws. Now the landscape is quite a
mouthful - inets, yaws, cowboy, musultin, mochiweb, etc. As Steve
Vinoski rightfully points out (*) this doesn't help to pull the blanket
in all directions. From personal experience with inets, I recall that
once I figured out how to use it, it happened to be a very helpful and
powerful HTTP library for building an application server. So why don't
all the authors of those wonderful alternative libraries just patch the
inets to extend it with something they feel missing so that we only have
one good and documented library to use as the common denominator?
Biased Cowboy author here. If I got the wrong idea about another project please correct me. :)
On 09/30/2012 11:29 PM, Serge Aleynikov wrote:
> So why don't
> all the authors of those wonderful alternative libraries just patch the
> inets to extend it with something they feel missing so that we only have
> one good and documented library to use as the common denominator?
Because it's not about what's missing, but about how you use it, how it works and what you can do with it.
You could probably classify the servers into two groups, inets+yaws being monolithic servers with many features (that can be ignored/stripped down, at least for yaws) and with a more traditional configuration file based approach (unless you run yaws embedded), and mochiweb+misultin+cowboy being in the lightweight family, being pretty much bare bone HTTP (except mochiweb has a few extra unrelated modules like JSON).
Note that mochiweb is in maintenance mode and misultin has been dropped in favor of cowboy.
Cowboy itself has a focus on reduced memory use (uses binary), reduced latency, has long-polling and Websocket support (SPDY soon), while trying to stay close to OTP principles (long-polling/websocket connections will be special processes in the next version allowing you to use sys: and friends to debug your code, the handler modules are inspired by gen_server, everything is supervised, and so on). It also has Webmachine-inspired REST support.
> Haven't had a need for an HTTP server in Erlang for a while since I last
> used inets-based http server five years back, and now examining what
> exists in public domain.
> The results are quite surprising - everyone who seems comfortable coding
> in Erlang, feels compelled to implement an HTTP server. Previously we
> had only one contender to inets - yaws.
> On 30/09/12 5:29 PM, Serge Aleynikov wrote:
>> Haven't had a need for an HTTP server in Erlang for a while since I last
>> used inets-based http server five years back, and now examining what
>> exists in public domain.
>> The results are quite surprising - everyone who seems comfortable coding
>> in Erlang, feels compelled to implement an HTTP server. Previously we
>> had only one contender to inets - yaws.
On Sun, Sep 30, 2012 at 11:29 PM, Serge Aleynikov <se...@aleynikov.org> wrote:
> Haven't had a need for an HTTP server in Erlang for a while since I last
> used inets-based http server five years back, and now examining what
> exists in public domain.
> The results are quite surprising - everyone who seems comfortable coding
> in Erlang, feels compelled to implement an HTTP server. Previously we
> had only one contender to inets - yaws. Now the landscape is quite a
> mouthful - inets, yaws, cowboy, musultin, mochiweb, etc. As Steve
> Vinoski rightfully points out (*) this doesn't help to pull the blanket
> in all directions. From personal experience with inets, I recall that
> once I figured out how to use it, it happened to be a very helpful and
> powerful HTTP library for building an application server. So why don't
> all the authors of those wonderful alternative libraries just patch the
> inets to extend it with something they feel missing so that we only have
> one good and documented library to use as the common denominator?
I can think of several reasons why this does not happen:
a) You have to distinguish between functional and non-functional behavior.
All web-servers implement the same protocol (HTTP) - so they are
functionally
equivalent - but their non-functional behavior is very different
- one server
might be optimized for large numbers of short-lived sessions,
another for a few long-lived
sessions. One might cache content, another not. So even though they are
all http servers they have very different behaviour - this is good.
b) Servers have two interfaces - the protocol interface (HTTP) and
the interface towards
Erlang (the API if you like). As far as I can see most severs
have different Erlang APIs.
This is bad.
c) Inets is shipped with OTP ie built and maintained by the OTP group.
This is not really
"core business" as far as the OTP group is concerned, better for
everybody if non-core
applications get taken over by third-part vendors.
d) As things mature - people want more in the way of "service" - ie
good documentation and the ability
to buy support - the problem with software is not so much that of
writing it - which can be
easy or difficult depending upon the problem but with the "burden of support"
Personally I'm quite happy with there being several different http
servers, It gives me several to choose
from. No server in the world will do exactly what I want so they all
need tweaking in one way or another.
What I don't like is that all have different APIs - this make swapping
between severs a pain.
Some degree of standardization of the APIs necessary to setup and
manage the servers would be
highly desirable.
It would also be highly desirable to standardize the relationship
between a http uri and
and erlang function call. For example
means call foo:bar(...) on the server and return the value as html (or
something)
Currently the http-server side of things seems ok - what I'm missing is the
entire user management cycle - password management, authentication, cookies,
forgotten passwords etc. - this needs far more than an http server, it needs
a persistent database, security and so on ...
I did not have much to say about the rest (except that different interfaces may emerge based on the use cases that the server optimizes for), and I mostly nodded in agreement to most of your points, but the last one I believe is a very bad idea.
This is a problem in terms of:
1. *Maintenance*. You don't want to usually give such an obvious window
into your server as it somewhat crystallizes your internal
application as a public API, what the URI is usually used for. We
should strive to have URIs never change (see W3C - Cool URIs don't
change <http://www.w3.org/Provider/Style/URI.html> ). Unless you're
ready not to change the function calls under the hood, I would not
use this.
2. *Security*. How do you keep someone from calling
http://some.host/erlcall?mod=init&func=stop or
http://some.host/erlcall?mod=os&func=cmd&arg1="rm -rf /" ? If you go
look at tryerlang.org's repository, you'll see how restrictive
Roberto Aloi had to make it to avoid people doing tricky things like
passing binaries of funs calling things they shouldn't indirectly
just to crash the system. This is absolutely non-trivial as soon as
you move out from using a very restrictive white list of valid
functions -- at which point you may as well hide them behind URIs.
3. *Clarity*. What are the argument types? Should we treat the '111' as
a string? a binary? an integer? An IO List? Should the server force
all arguments to be of a given type that needs to be converted later
so you need to have some kind of intermediary function doing
conversion for you? It's not different from what we get right now,
but there's a difference in expectations from the developer. A minor
issue, I guess.
4. *RPC and HTTP*. Whether RPC is still a good way to do things is up
to debate (look for Steve Vinoski's essays and posts on the topic).
Shouldn't the erlcall URI be extended to also handle timeouts? Or
are these server options unrelated to the function call? More than
that is the idea of how you write things that may depend on
content-type (such as RESTful web services). For example, a browser
that sends a request to a page in HTML may be served HTML. If
Javascript asks for info in JSON, then the same URI may forward
JSON. There is an out-of-URL mechanism to deal with different
requests and ideas.
5. *HTTP Spec problems*. GET and HEAD requests should be idempotent.
PUT, POST, and DELETE can be used in ways that change information on
the server. If I use POST to
http://some.host/erlcall?mod=foo&func=bar&arg1=111 and it updates
data, what should it do when I use GET or HEAD on it? Do we prefer
to use the POST body for the requests that are done under the POST
method? What if I include both POST arguments and querystring
arguments and they are different?
More than that, HTTP doesn't specify what happens when you have more
than one querystring argument being the same (from memory). It
should thus be possible to have arg1 four times in the URL, with
arg3 being there only once, and arg2 being entirely missing. Same
with the module or function name. How do we make this interface
behave at that point? Things are way more complex than what they
look like.
There are likely more issues with this approach. Ideally, it should be impossible for the user of your server to know what you used to implement it -- this is what should give you the most freedom in terms of implementation when lots of people start using it. At this point I'm thinking we should start using different protocols from HTTP if we're not really willing to respect it, but that's for another discussion.
And yes, more user management would definitely be nice. I've written a tiny library to handle passwords themselves, but it's not close enough to be something a framework would use without surrounding support -- it just handles the core of password hashing and verification. It's currently hosted at https://github.com/ferd/erlpass . Then each platform such as Zotonic likely reimplemented their own, although I don't know how moveable they are outside of the project.
Alternatively https://npmjs.org/ has 15k+ packages, we dont need to add
user management and the kitchen sink to inets, we need to remove it from
the standard library and promote userland packages as first class citizens
(ie pick a tool to install packages and bundle it)
On 1 October 2012 13:57, Fred Hebert <monon...@ferd.ca> wrote:
> I did not have much to say about the rest (except that different
> interfaces may emerge based on the use cases that the server optimizes
> for), and I mostly nodded in agreement to most of your points, but the last
> one I believe is a very bad idea.
> This is a problem in terms of:
> 1. *Maintenance*. You don't want to usually give such an obvious
> window into your server as it somewhat crystallizes your internal
> application as a public API, what the URI is usually used for. We should
> strive to have URIs never change (see W3C - Cool URIs don't change<http://www.w3.org/Provider/Style/URI.html>). Unless you're ready not to change the function calls under the hood, I
> would not use this.
> 2. *Security*. How do you keep someone from calling
> http://some.host/erlcall?mod=init&func=stop or
> http://some.host/erlcall?mod=os&func=cmd&arg1="rm -rf /" ? If you go
> look at tryerlang.org's repository, you'll see how restrictive Roberto
> Aloi had to make it to avoid people doing tricky things like passing
> binaries of funs calling things they shouldn't indirectly just to crash the
> system. This is absolutely non-trivial as soon as you move out from using a
> very restrictive white list of valid functions -- at which point you may as
> well hide them behind URIs.
> 3. *Clarity*. What are the argument types? Should we treat the '111'
> as a string? a binary? an integer? An IO List? Should the server force all
> arguments to be of a given type that needs to be converted later so you
> need to have some kind of intermediary function doing conversion for you?
> It's not different from what we get right now, but there's a difference in
> expectations from the developer. A minor issue, I guess.
> 4. *RPC and HTTP*. Whether RPC is still a good way to do things is up
> to debate (look for Steve Vinoski's essays and posts on the topic).
> Shouldn't the erlcall URI be extended to also handle timeouts? Or are these
> server options unrelated to the function call? More than that is the idea
> of how you write things that may depend on content-type (such as RESTful
> web services). For example, a browser that sends a request to a page in
> HTML may be served HTML. If Javascript asks for info in JSON, then the same
> URI may forward JSON. There is an out-of-URL mechanism to deal with
> different requests and ideas.
> 5. *HTTP Spec problems*. GET and HEAD requests should be idempotent.
> PUT, POST, and DELETE can be used in ways that change information on the
> server. If I use POST to
> http://some.host/erlcall?mod=foo&func=bar&arg1=111 and it updates
> data, what should it do when I use GET or HEAD on it? Do we prefer to use
> the POST body for the requests that are done under the POST method? What if
> I include both POST arguments and querystring arguments and they are
> different?
> More than that, HTTP doesn't specify what happens when you have more
> than one querystring argument being the same (from memory). It should thus
> be possible to have arg1 four times in the URL, with arg3 being there
> only once, and arg2 being entirely missing. Same with the module or
> function name. How do we make this interface behave at that point? Things
> are way more complex than what they look like.
> There are likely more issues with this approach. Ideally, it should be
> impossible for the user of your server to know what you used to implement
> it -- this is what should give you the most freedom in terms of
> implementation when lots of people start using it. At this point I'm
> thinking we should start using different protocols from HTTP if we're not
> really willing to respect it, but that's for another discussion.
> And yes, more user management would definitely be nice. I've written a
> tiny library to handle passwords themselves, but it's not close enough to
> be something a framework would use without surrounding support -- it just
> handles the core of password hashing and verification. It's currently
> hosted at https://github.com/ferd/erlpass . Then each platform such as
> Zotonic likely reimplemented their own, although I don't know how moveable
> they are outside of the project.
> Regards,
> Fred.
> On 12-10-01 4:01 AM, Joe Armstrong wrote:
> On Sun, Sep 30, 2012 at 11:29 PM, Serge Aleynikov <se...@aleynikov.org> <se...@aleynikov.org>
> wrote:
> It would also be highly desirable to standardize the relationship
> between a http uri and
> and erlang function call. For example
> On Sun, Sep 30, 2012 at 11:29 PM, Serge Aleynikov <se...@aleynikov.org>
> wrote:
>> Haven't had a need for an HTTP server in Erlang for a while since I last
>> used inets-based http server five years back, and now examining what
>> exists in public domain.
>> The results are quite surprising - everyone who seems comfortable coding
>> in Erlang, feels compelled to implement an HTTP server. Previously we
>> had only one contender to inets - yaws. Now the landscape is quite a
>> mouthful - inets, yaws, cowboy, musultin, mochiweb, etc. As Steve
>> Vinoski rightfully points out (*) this doesn't help to pull the blanket
>> in all directions. From personal experience with inets, I recall that
>> once I figured out how to use it, it happened to be a very helpful and
>> powerful HTTP library for building an application server. So why don't
>> all the authors of those wonderful alternative libraries just patch the
>> inets to extend it with something they feel missing so that we only have
>> one good and documented library to use as the common denominator?
I agree with Serge that it is somewhat unfortunate that there are so
many initiatives
where people write their own implementation of something we already
have included in
the Erlang/OTP distribution and doing this without even suggesting or
asking us if we could
incorporate some missing feature or improve the current implementation
in various ways.
I think it is better to try to improve the existing functionality
first and only if that is not feasible make another implementation
with a new API.
In Erlang/OTP we have keep strict compatibility with many of our APIs
which is both a strength and sometimes a burden (when we are not happy
with the old API) but there are mostly ways around that if there is
strong case for a change.
> I can think of several reasons why this does not happen:
> a) You have to distinguish between functional and non-functional behavior.
> All web-servers implement the same protocol (HTTP) - so they are
> functionally
> equivalent - but their non-functional behavior is very different
> - one server
> might be optimized for large numbers of short-lived sessions,
> another for a few long-lived
> sessions. One might cache content, another not. So even though they
> are
> all http servers they have very different behaviour - this is good.
> b) Servers have two interfaces - the protocol interface (HTTP) and
> the interface towards
> Erlang (the API if you like). As far as I can see most severs
> have different Erlang APIs.
> This is bad.
Would be good if there was a kind of standardized API for HTTP-servers
and clients.
> c) Inets is shipped with OTP ie built and maintained by the OTP group.
> This is not really
> "core business" as far as the OTP group is concerned, better for
> everybody if non-core
> applications get taken over by third-part vendors.
Joe is right reagarding core business in that there are not much use
of HTTP-servers in
products developed with Erlang inside Ericsson (who is financing ERlang/OTP) and
if used the requirements are not very demanding.
> d) As things mature - people want more in the way of "service" - ie
> good documentation and the ability
> to buy support - the problem with software is not so much that of
> writing it - which can be
> easy or difficult depending upon the problem but with the "burden of
> support"
We still actively develop inets with its http client and server and I
am not sure that
all the reasons to develop alternatives still exist.
We are more than happy to receive contributions with bugfixes and new
functionality.
I think it is of great value to have a relative complete functionality
in Erlang/OTP so that
the user can implement basic functionality without having to search
for separate implementations of, with unknown quality and completeness
at least for a beginner.
> Personally I'm quite happy with there being several different http
> servers, It gives me several to choose
> from. No server in the world will do exactly what I want so they all
> need tweaking in one way or another.
> What I don't like is that all have different APIs - this make swapping
> between severs a pain.
> Some degree of standardization of the APIs necessary to setup and
> manage the servers would be
> highly desirable.
> It would also be highly desirable to standardize the relationship
> between a http uri and
> and erlang function call. For example
> means call foo:bar(...) on the server and return the value as html (or
> something)
> Currently the http-server side of things seems ok - what I'm missing is
> the
> entire user management cycle - password management, authentication,
> cookies,
> forgotten passwords etc. - this needs far more than an http server, it
> needs
> a persistent database, security and so on ...
> I agree with Serge that it is somewhat unfortunate that there are so
> many initiatives
> where people write their own implementation of something we already
> have included in
> the Erlang/OTP distribution and doing this without even suggesting or
> asking us if we could
> incorporate some missing feature or improve the current implementation
> in various ways.
It happens a lot. People tend to review available options, and if none fits their needs, write something themselves without consulting any of the authors. Sometimes they're right to do so, because the answer they'll get is obvious, and sometimes not.
There's a few reasons this happens for this specific case:
* HTTP is simple, it's easier to go ahead and experiment on a new project than on an existing one.
* If the experiment succeeds and the implementation has to differ significantly from other projects, it's not worth backporting since you'll just run into a wall of backward compatibility requirements.
* OTP has a long release process. Many companies need their fixes to be included quickly to avoid maintaining huge branches for their changes. They generally agree in return to keep track of changes that might happen during the product's development.
* Inets is seriously well hidden. I only heard about it after releasing Cowboy. I knew about Yaws/Mochiweb/Misultin but not the inets HTTP server. Many people I talked to didn't seem to know it existed.
* Sometimes removing features is an improvement. Inets is big, it's only natural that lightweight alternatives appear. Same happened to Apache, and it's no surprise that Inets is quite similar to Apache from a user point of view.
Note that while not everyone uses inets, most servers do use erlang:decode_packet/3 which does most of the job. Cowboy doesn't anymore, because it needed backward incompatible changes to erlang:decode_packet/3. I didn't ask your thoughts, simply because the experiment I did hinted that doing the parsing in full Erlang was a better idea: this change results in improved HiPE optimizations and better reduction management which results in better latency under load.
> In Erlang/OTP we have keep strict compatibility with many of our APIs
> which is both a strength and sometimes a burden (when we are not happy
> with the old API) but there are mostly ways around that if there is
> strong case for a change.
It's a big strength for people looking for stability and a big burden for people looking for reactivity. Can't really blame the second group for finding solutions to their problems.
> We still actively develop inets with its http client and server and I
> am not sure that
> all the reasons to develop alternatives still exist.
But like you said the level of support for applications included in OTP vary widely. It is also not clearly indicated which parts are the focus of recent works (there is a small roadmap for the most important stuff but that's about it), nor can it be deduced from reading tickets in a bug tracker or similar collaboration tool.
Which brings me to a very important point: please start using a public bug tracker! It doesn't have to be 100% public (as I'm sure you have some issues that are internal only) but it would help a lot if people could see the existing issues instead of stumbling upon the same bugs again and again. It probably also would help with contributions.
> I think it is of great value to have a relative complete functionality
> in Erlang/OTP so that
> the user can implement basic functionality without having to search
> for separate implementations of, with unknown quality and completeness
> at least for a beginner.
I would prefer Erlang/OTP to focus on the core features and leave the satellite features to third parties. OTP has an HTTP server but doesn't have proper Unicode support. OTP has OpenGL but doesn't have proper timezones support. These 2 examples missing from OTP are some of many things that are too complex for most people to implement, and that would not benefit from having alternatives.
Let's take another example. While taking a quick glance at the modules included in OTP I discovered there is a GD implementation in the percept application. I'm pretty sure most people didn't know it was there, as they usually end up using a NIF. It's quite unexpected that it's in OTP.
Finding things in OTP unexpectedly is a quite common occurence, though.
If the user had a package management application (perhaps working similarly to rebar's deps, = per project/release, no site-wide install by default) along with a package index website, it could quite easily find what he needs. It wouldn't be of unknown quality or completeness, simply because adding a comment and rating system to a package index website is fairly simple to do.
Plus, he can make an opinion for himself, instead of just using whatever someone decided for him. You might include a project in OTP that fits your criteria for quality, but it might be crap for someone else. A more democratic system where people choose what they use based on other users' feedback and ratings is not only a lot more powerful but also allows to bring some light on some more obscure projects that are sometimes quite hard to find armed only with Google.
It makes sense to have a single choice for big corporate environments, not so much for the rest of the world.
Den 1 okt 2012 17:39 skrev "Loďc Hoguin" <es...@ninenines.eu>:
> On 10/01/2012 04:30 PM, Kenneth Lundin wrote:
>> I agree with Serge that it is somewhat unfortunate that there are so
>> many initiatives
>> where people write their own implementation of something we already
>> have included in
>> the Erlang/OTP distribution and doing this without even suggesting or
>> asking us if we could
>> incorporate some missing feature or improve the current implementation
>> in various ways.
> It happens a lot. People tend to review available options, and if none
fits their needs, write something themselves without consulting any of the
authors. Sometimes they're right to do so, because the answer they'll get
is obvious, and sometimes not.
> There's a few reasons this happens for this specific case:
> * HTTP is simple, it's easier to go ahead and experiment on a new project
than on an existing one.
> * If the experiment succeeds and the implementation has to differ
significantly from other projects, it's not worth backporting since you'll
just run into a wall of backward compatibility requirements.
> * OTP has a long release process. Many companies need their fixes to be
included quickly to avoid maintaining huge branches for their changes. They
generally agree in return to keep track of changes that might happen during
the product's development.
I think we are pretty fast with service releases about every 3rd month and
updates on the maint branch in git between that.
> * Inets is seriously well hidden. I only heard about it after releasing
Cowboy. I knew about Yaws/Mochiweb/Misultin but not the inets HTTP server.
Many people I talked to didn't seem to know it existed.
I agree on that. I have always thought that the name inets is misleading.
> * Sometimes removing features is an improvement. Inets is big, it's only
natural that lightweight alternatives appear. Same happened to Apache, and
it's no surprise that Inets is quite similar to Apache from a user point of
view.
I agree here too. The http server included in OTP should have been a
lightweight one leaving for others to implement a more complete or
specialized one.
> Note that while not everyone uses inets, most servers do use
erlang:decode_packet/3 which does most of the job. Cowboy doesn't anymore,
because it needed backward incompatible changes to erlang:decode_packet/3.
I didn't ask your thoughts, simply because the experiment I did hinted that
doing the parsing in full Erlang was a better idea: this change results in
improved HiPE optimizations and better reduction management which results
in better latency under load.
Interesting, would be good if you could give us more info on this. The http
parsing in
decode_packet is there to improve performance. If it is not ( without using
HiPE) we should consider removing it!
>> In Erlang/OTP we have keep strict compatibility with many of our APIs
>> which is both a strength and sometimes a burden (when we are not happy
>> with the old API) but there are mostly ways around that if there is
>> strong case for a change.
> It's a big strength for people looking for stability and a big burden for
people looking for reactivity. Can't really blame the second group for
finding solutions to their problems.
I am not blaming people for implementing their own solution, it is good to
have a lot to choose from. But I still think the stability is very
important for the majority of users making systems with a longer life cycle
than a year.
Our experience is that we have to be very compatible in order to convince
our
users (often big projects) to step up to our latest release.
By this we can also stop supporting older versions.
>> We still actively develop inets with its http client and server and I
>> am not sure that
>> all the reasons to develop alternatives still exist.
> But like you said the level of support for applications included in OTP
vary widely. It is also not clearly indicated which parts are the focus of
recent works (there is a small roadmap for the most important stuff but
that's about it), nor can it be deduced from reading tickets in a bug
tracker or similar collaboration tool.
Yes we plan to improve in this area.
I would say that the level of support is equally high for all applications
we regard as important and heavily used by Ericsson and or the external
Erlang community.
> Which brings me to a very important point: please start using a public
bug tracker! It doesn't have to be 100% public (as I'm sure you have some
issues that are internal only) but it would help a lot if people could see
the existing issues instead of stumbling upon the same bugs again and
again. It probably also would help with contributions.
We have discussed this as well.
>> I think it is of great value to have a relative complete functionality
>> in Erlang/OTP so that
>> the user can implement basic functionality without having to search
>> for separate implementations of, with unknown quality and completeness
>> at least for a beginner.
> I would prefer Erlang/OTP to focus on the core features and leave the
satellite features to third parties. OTP has an HTTP server but doesn't
have proper Unicode support. OTP has OpenGL but doesn't have proper
timezones support. These 2 examples missing from OTP are some of many
things that are too complex for most people to implement, and that would
not benefit from having alternatives.
What is missing when it comes to timezones? Has it been mentioned in
another thread?
I think we rely on the OS configuration here.
> Let's take another example. While taking a quick glance at the modules
included in OTP I discovered there is a GD implementation in the percept
application. I'm pretty sure most people didn't know it was there, as they
usually end up using a NIF. It's quite unexpected that it's in OTP.
It was developed as part of Percept just to have something to use there. We
don't regard it complete or general enough to be supported as an app of its
own.
> Finding things in OTP unexpectedly is a quite common occurence, though.
I think most things are documented but the docs could have better search
facilities.
> If the user had a package management application (perhaps working
similarly to rebar's deps, = per project/release, no site-wide install by
default) along with a package index website, it could quite easily find
what he needs. It wouldn't be of unknown quality or completeness, simply
because adding a comment and rating system to a package index website is
fairly simple to do.
I agree that a package management system would be good. There are
initiatives in that direction.
> Plus, he can make an opinion for himself, instead of just using whatever
someone decided for him. You might include a project in OTP that fits your
criteria for quality, but it might be crap for someone else. A more
democratic system where people choose what they use based on other users'
feedback and ratings is not only a lot more powerful but also allows to
bring some light on some more obscure projects that are sometimes quite
hard to find armed only with Google.
> It makes sense to have a single choice for big corporate environments,
> > * OTP has a long release process. Many companies need their fixes to > be included quickly to avoid maintaining huge branches for their > changes. They generally agree in return to keep track of changes that > might happen during the product's development. > I think we are pretty fast with service releases about every 3rd month > and updates on the maint branch in git between that.
But doesn't that imply upgrading the whole of OTP? If it does, that means more testing and more chances for new issues. If you can build a release upgrade with one app from one OTP version and the rest from a previous version, I'd like to know how.
> > Note that while not everyone uses inets, most servers do use > erlang:decode_packet/3 which does most of the job. Cowboy doesn't > anymore, because it needed backward incompatible changes to > erlang:decode_packet/3. I didn't ask your thoughts, simply because the > experiment I did hinted that doing the parsing in full Erlang was a > better idea: this change results in improved HiPE optimizations and > better reduction management which results in better latency under load.
> Interesting, would be good if you could give us more info on this. The > http parsing in > decode_packet is there to improve performance. If it is not ( without > using HiPE) we should consider removing it!
Between now and then binaries appeared, got performance improvements, and considering Cowboy is full binary (no conversion to atom for methods or header names or anything) and only receives requests, not responses, that means it can skip many steps that decode_packet/3 can't.
Binaries also mean you can parse the request line and all the headers using a single match context which seems to be very fast (good job!) and I'm guessing is the main reason for improved performance under HiPE (I'm about to start reading the HiPE papers to understand that better).
> >> In Erlang/OTP we have keep strict compatibility with many of our APIs > >> which is both a strength and sometimes a burden (when we are not happy > >> with the old API) but there are mostly ways around that if there is > >> strong case for a change.
> > It's a big strength for people looking for stability and a big burden > for people looking for reactivity. Can't really blame the second group > for finding solutions to their problems.
> I am not blaming people
I used a word too strong there. :)
> for implementing their own solution, it is good > to have a lot to choose from. But I still think the stability is very > important for the majority of users making systems with a longer life > cycle than a year. > Our experience is that we have to be very compatible in order to > convince our > users (often big projects) to step up to our latest release. > By this we can also stop supporting older versions.
Yes and I wouldn't want that to stop, that's a strong point. In fact having both ways is great, it means you can make sure the important parts of your system are stable while the more bleeding-edge parts are kept, well, bleeding-edge.
> >> I think it is of great value to have a relative complete functionality > >> in Erlang/OTP so that > >> the user can implement basic functionality without having to search > >> for separate implementations of, with unknown quality and completeness > >> at least for a beginner.
> > I would prefer Erlang/OTP to focus on the core features and leave the > satellite features to third parties. OTP has an HTTP server but doesn't > have proper Unicode support. OTP has OpenGL but doesn't have proper > timezones support. These 2 examples missing from OTP are some of many > things that are too complex for most people to implement, and that would > not benefit from having alternatives.
> What is missing when it comes to timezones? Has it been mentioned in > another thread? > I think we rely on the OS configuration here.
Date/time conversion from one timezone to another. For example if your data is saved with UTC time but the user is in Paris you want to convert it to Europe/Paris.
Perhaps it exists, but I haven't found it.
I think there's a lot of work to be done with regards to localization and internationalization support, but I haven't had enough time to think about it in details yet. Unicode (not just Unicode though, but a native string type abstracting encodings, EEP coming in a few weeks), timezones, graphical drawing libraries are pretty much what I find lacks the most for the web side of things. Graphical drawing libraries are fine outside of OTP for a while (until there's one that can do near everything and do it well) though.
Thanks for the info on package management and bug tracker. Hope we'll be involved in the process of creating the former. And thanks for your nice reply. :)
On Mon, Oct 01, 2012 at 05:38:52PM +0200, Lo?c Hoguin wrote:
> Finding things in OTP unexpectedly is a quite common occurence, though.
Not OTP's fault, anybody can do (for example):
$ mandb ~/R15B01/lib/erlang/man
[... noise ...]
$ apropos -M ~/R15B01/lib/erlang/man http
http_uri (3) - URI utility module
httpc (3) - An HTTP/1.1 client
httpd (3) - An implementation of an HTTP 1.1 compliant Web server, as defined in RFC 2616.
httpd_conf (3) - Configuration utility functions to be used by the Erlang Web server API programmer.
httpd_socket (3) - Communication utility functions to be used by the Erlang Web server API programmer.
httpd_util (3) - Miscellaneous utility functions to be used when implementing Erlang Web server API modules.
Standard functionality, really. And there's always find(1) and grep(1)
to assist with detective work -- not perfect, but quite efficient
anyways.
Could you cite some examples of third-party libraries being absorbed into
the Erlang/OTP distribution and "dying"? Perhaps this is not as obvious to
everybody as you think.
> Alternatively https://npmjs.org/ has 15k+ packages, we dont need to add
> user management and the kitchen sink to inets, we need to remove it from
> the standard library and promote userland packages as first class citizens
> (ie pick a tool to install packages and bundle it)
I couldn't disagree with that much more ... npm/Node.js is cluttered with
packages of dubious quality that are not maintained in the slightest, and
while this "Wild West" they prefer may seem exciting, I do not think it
bodes well for the long-term stability of projects that depend on these
packages.
If some particular functionality becomes used in a large body of Erlang
projects, it should certainly be brought into the stdlib. Per your Python
reference, one need only look at the "json" module for an example where
bringing something into the stdlib has made deployment and maintenance of
Python code-bases simpler -- no more installing the "simplejson" package,
docs right there on docs.python.org, and JSON earning first-class status
as a serialization format for Python objects.
To advocate actually removing things from the Erlang stdlib, potentially
breaking working code on a large scale, is madness to me.
I agree with Joe that having a design of the HTTP server architecture included in OTP that would offer standardized internal API and allow developers to either replace selected functionality via providing callback modules or swap the HTTP servers easily would be a great plus.
Concerning inets, what's really strange is that whenever we see some benchmarks posted comparing performance of different HTTP implementations, inets is never included there, though I would think, it would make more sense in terms of showing how a custom implementation compares to what's included in OTP. Does it have to do something with what Loďc referred to as inets being well hidden in OTP? I doubt, but if not that, then what? There are not even benchmarks comparing yaws to inets, and despite the fact that both offer a rich set of functionality make one wonder was was the idea of writing yaws in the first place given existence of inets' httpd.
Diversity is good, but at some point it creates a burden for the user who needs to make a selection of an HTTP server for his project. Instead of solving the application-related problem, now the task becomes studying several implementations to see which one offers a feature-set best suited for the problem being solved, and the answer to that question is not always straightforward.
Perhaps a good initiative from the OTP team would be to spec out the open architecture of an HTTP server and welcome feedback from authors of current http libraries, so that the successor of inets would be flexible enough to accommodate building light-weight and heavy-weight HTTP servers based on the supported code base included in OTP? This would also open a possibility of diversity for library writers replacing/enhancing parts to suit their needs.
> On 10/01/2012 08:11 PM, Kenneth Lundin wrote: >> > * OTP has a long release process. Many companies need their fixes to >> be included quickly to avoid maintaining huge branches for their >> changes. They generally agree in return to keep track of changes that >> might happen during the product's development. >> I think we are pretty fast with service releases about every 3rd month >> and updates on the maint branch in git between that.
> But doesn't that imply upgrading the whole of OTP? If it does, that > means more testing and more chances for new issues. If you can build a > release upgrade with one app from one OTP version and the rest from a > previous version, I'd like to know how.
>> > Note that while not everyone uses inets, most servers do use >> erlang:decode_packet/3 which does most of the job. Cowboy doesn't >> anymore, because it needed backward incompatible changes to >> erlang:decode_packet/3. I didn't ask your thoughts, simply because the >> experiment I did hinted that doing the parsing in full Erlang was a >> better idea: this change results in improved HiPE optimizations and >> better reduction management which results in better latency under load.
>> Interesting, would be good if you could give us more info on this. The >> http parsing in >> decode_packet is there to improve performance. If it is not ( without >> using HiPE) we should consider removing it!
> Between now and then binaries appeared, got performance improvements, > and considering Cowboy is full binary (no conversion to atom for methods > or header names or anything) and only receives requests, not responses, > that means it can skip many steps that decode_packet/3 can't.
> Binaries also mean you can parse the request line and all the headers > using a single match context which seems to be very fast (good job!) and > I'm guessing is the main reason for improved performance under HiPE (I'm > about to start reading the HiPE papers to understand that better).
>> >> In Erlang/OTP we have keep strict compatibility with many of our APIs >> >> which is both a strength and sometimes a burden (when we are not >> happy >> >> with the old API) but there are mostly ways around that if there is >> >> strong case for a change.
>> > It's a big strength for people looking for stability and a big burden >> for people looking for reactivity. Can't really blame the second group >> for finding solutions to their problems.
>> I am not blaming people
> I used a word too strong there. :)
>> for implementing their own solution, it is good >> to have a lot to choose from. But I still think the stability is very >> important for the majority of users making systems with a longer life >> cycle than a year. >> Our experience is that we have to be very compatible in order to >> convince our >> users (often big projects) to step up to our latest release. >> By this we can also stop supporting older versions.
> Yes and I wouldn't want that to stop, that's a strong point. In fact > having both ways is great, it means you can make sure the important > parts of your system are stable while the more bleeding-edge parts are > kept, well, bleeding-edge.
>> >> I think it is of great value to have a relative complete >> functionality >> >> in Erlang/OTP so that >> >> the user can implement basic functionality without having to search >> >> for separate implementations of, with unknown quality and >> completeness >> >> at least for a beginner.
>> > I would prefer Erlang/OTP to focus on the core features and leave the >> satellite features to third parties. OTP has an HTTP server but doesn't >> have proper Unicode support. OTP has OpenGL but doesn't have proper >> timezones support. These 2 examples missing from OTP are some of many >> things that are too complex for most people to implement, and that would >> not benefit from having alternatives.
>> What is missing when it comes to timezones? Has it been mentioned in >> another thread? >> I think we rely on the OS configuration here.
> Date/time conversion from one timezone to another. For example if your > data is saved with UTC time but the user is in Paris you want to convert > it to Europe/Paris.
> Perhaps it exists, but I haven't found it.
> I think there's a lot of work to be done with regards to localization > and internationalization support, but I haven't had enough time to think > about it in details yet. Unicode (not just Unicode though, but a native > string type abstracting encodings, EEP coming in a few weeks), > timezones, graphical drawing libraries are pretty much what I find lacks > the most for the web side of things. Graphical drawing libraries are > fine outside of OTP for a while (until there's one that can do near > everything and do it well) though.
> Thanks for the info on package management and bug tracker. Hope we'll be > involved in the process of creating the former. And thanks for your nice > reply. :)
On Sun, Sep 30, 2012 at 2:29 PM, Serge Aleynikov <se...@aleynikov.org>wrote:
> The results are quite surprising - everyone who seems comfortable coding
> in Erlang, feels compelled to implement an HTTP server.
I find this quite offensive to all the people that are putting efforts and
releasing their own work open source (and especially in BSD/MIT terms). But
I'm sure that you only meant to be provocative. :)
I too believed that too many duplication of efforts were going on, and
that's the reason why I've stopped misultin development [1] something like
8 months ago. The main result that I've obtained in closing misultin was to
push people to concentrate efforts around cowboy and yaws, and at least as
far as cowboy goes, this has been a success (judging by the number of
products that I've helped switching over).
However, I believe everyone should do whatever they feel like. Open source
and many projects are the result of a bunch of intelligent people desires
to bring their own creatures to life., andI see nothing bad in that. I'm
actually sorry to read statements such as "diversity is bad" or "too many
options".
> On Sun, Sep 30, 2012 at 2:29 PM, Serge Aleynikov <se...@aleynikov.org
> <mailto:se...@aleynikov.org>> wrote:
> The results are quite surprising - everyone who seems comfortable coding
> in Erlang, feels compelled to implement an HTTP server.
> I find this quite offensive to all the people that are putting efforts
> and releasing their own work open source (and especially in BSD/MIT
> terms). But I'm sure that you only meant to be provocative. :)
I am guilty too of having implemented my own small Erlang HTTP server at
some point in the past before deciding to switch to using inets! :-)
> I too believed that too many duplication of efforts were going on, and
> that's the reason why I've stopped misultin development [1] something
> like 8 months ago. The main result that I've obtained in closing
> misultin was to push people to concentrate efforts around cowboy and
> yaws, and at least as far as cowboy goes, this has been a success
> (judging by the number of products that I've helped switching over).
Concerning misultin, I did previously look at it and thought it was
great. Despite the success, your decision to close it in favor of
promoting other http server initiatives was very much admirable.
> However, I believe everyone should do whatever they feel like. Open
> source and many projects are the result of a bunch of intelligent people
> desires to bring their own creatures to life., andI see nothing bad in
> that. I'm actually sorry to read statements such as "diversity is bad"
> or "too many options".
> This. is. open sooooooooooource. [2]
See my previous post. I don't think we disagree in terms of what you
said above.
> I agree with Joe that having a design of the HTTP server architecture > included in OTP that would offer standardized internal API and allow > developers to either replace selected functionality via providing > callback modules or swap the HTTP servers easily would be a great plus.
This wouldn't have helped Cowboy, as one of the biggest difference is the use of binary instead of strings.
There's another solution, which is to write wrappers for other APIs, for example mochicow allows Mochiweb applications to use Cowboy under the hood without having to change their code. This means that the switch can be progressive instead of being brutal like a direct code switch.
> Diversity is good, but at some point it creates a burden for the user > who needs to make a selection of an HTTP server for his project.
It doesn't!
> Instead of solving the application-related problem, now the task becomes > studying several implementations to see which one offers a feature-set > best suited for the problem being solved, and the answer to that > question is not always straightforward.
Erlang is wonderful in that it allows you to write your application 100% HTTP-free, and then write another application for the HTTP layer that will be a thin layer running alongside your application. The communication layer can be separate from the business logic.
If you go that way then your application's code isn't dependent on the transports used at all, and you can easily switch servers, change protocols, serialization formats or anything in your HTTP layer without this impacting your application in any way.
For example, when SPDY support will be added to Cowboy, only the HTTP layer will require a change to make use of it (start_http, which only does HTTP, can be replaced by start_spdy, which does SPDY or HTTP depending on client capabilities). The application behind it shouldn't have to care about that.
> Perhaps a good initiative from the OTP team would be to spec out the > open architecture of an HTTP server and welcome feedback from authors of > current http libraries, so that the successor of inets would be flexible > enough to accommodate building light-weight and heavy-weight HTTP > servers based on the supported code base included in OTP? This would > also open a possibility of diversity for library writers > replacing/enhancing parts to suit their needs.
I would even go so far as to propose that almost every single Erlang
project, for the foreseeable future will want an HTTP service. I would
even extend this to "most project in any realm in any language."
Reason: Even if you're not providing an external service, having a
web-status console, admin or REST-based health/monitoring/whatever port has
a whole lot of value.
Right now, if you want external services you have to meld
server+protocol+client. The HTTP's strongest suit is that, just like
javascript, it's ubiquitous. It may not be optimal, but it's everywhere
and it's easy.
Would I prefer a world built on client-side(browser!) Erlang, a
length-prefixed binary protocol and real sockets (instead of websocket
silliness)? Hell yes. But we don't have that, and likely won't anytime
soon, so HTTP+javascript continues to be the *lingua franca.*
Having too many options is a Bad Thing<tm>, IMO. Erlang is already enough
of a 'big scary monster in the dark' for a lot of people. If we had a
unified, documented, packaged and turnkey solution withing the stdlib so
that users could 'drop in and go' it would increase uptake as people at
least explored it. The are more web-coders than ever, many with just
enough knowledge to be dangerous. Apple vs PC back in the day --- snag all
of the low-hanging cheap fruit of PC clones instead of insisting on the
white-ivory-tower and then achieve market dominance (which is then yours to
lose/squander at your leisure... :)
On Tue, Oct 2, 2012 at 10:37 AM, Loďc Hoguin <es...@ninenines.eu> wrote:
> On 10/02/2012 05:52 PM, Serge Aleynikov wrote:
>> I agree with Joe that having a design of the HTTP server architecture
>> included in OTP that would offer standardized internal API and allow
>> developers to either replace selected functionality via providing
>> callback modules or swap the HTTP servers easily would be a great plus.
> This wouldn't have helped Cowboy, as one of the biggest difference is the
> use of binary instead of strings.
> There's another solution, which is to write wrappers for other APIs, for
> example mochicow allows Mochiweb applications to use Cowboy under the hood
> without having to change their code. This means that the switch can be
> progressive instead of being brutal like a direct code switch.
> Diversity is good, but at some point it creates a burden for the user
>> who needs to make a selection of an HTTP server for his project.
> It doesn't!
> > Instead of solving the application-related problem, now the task becomes
> > studying several implementations to see which one offers a feature-set
> > best suited for the problem being solved, and the answer to that
> > question is not always straightforward.
> Erlang is wonderful in that it allows you to write your application 100%
> HTTP-free, and then write another application for the HTTP layer that will
> be a thin layer running alongside your application. The communication layer
> can be separate from the business logic.
> If you go that way then your application's code isn't dependent on the
> transports used at all, and you can easily switch servers, change
> protocols, serialization formats or anything in your HTTP layer without
> this impacting your application in any way.
> For example, when SPDY support will be added to Cowboy, only the HTTP
> layer will require a change to make use of it (start_http, which only does
> HTTP, can be replaced by start_spdy, which does SPDY or HTTP depending on
> client capabilities). The application behind it shouldn't have to care
> about that.
> Perhaps a good initiative from the OTP team would be to spec out the
>> open architecture of an HTTP server and welcome feedback from authors of
>> current http libraries, so that the successor of inets would be flexible
>> enough to accommodate building light-weight and heavy-weight HTTP
>> servers based on the supported code base included in OTP? This would
>> also open a possibility of diversity for library writers
>> replacing/enhancing parts to suit their needs.
On 10/1/12 4:30 PM, Kenneth Lundin wrote:
>>
> I agree with Serge that it is somewhat unfortunate that there are so
> many initiatives
> where people write their own implementation of something we already
> have included in
> the Erlang/OTP distribution and doing this without even suggesting or
> asking us if we could
> incorporate some missing feature or improve the current implementation
> in various ways.
Why, the more the merrier, let the best library win.
>> b) Servers have two interfaces - the protocol interface (HTTP) and
>> the interface towards
>> Erlang (the API if you like). As far as I can see most severs
>> have different Erlang APIs.
>> This is bad.
> Would be good if there was a kind of standardized API for HTTP-servers
> and clients.
Disagree, in todays webdevelopment the DOM is the really important
interface, and that is standardized.
Switching servers is work, yes, but doable.
Actually, the API between the protocol HTTP and the erlang code
is pretty much what distinguishes the different servers, this api can
be more or less ... well good. Yaws has appmods to do this, other servers have different mechanisms. This is a matter of preference and standardizing on this pretty much removes the need for different servers and we're back on square one - we should all have contributed
to inets instead. This would have lead to a worse world.
I recently received a copy of Building Web Applications with Erlang,
which is almost entirely about yaws. The discussion of inets on this
thread spurred me to look in the index under "i" -- except, oops, the
book has no index. It does have an appendix about other webservers and
frameworks in Erlang -- but inets is not listed there.
On Wed, Oct 3, 2012 at 12:52 AM, Serge Aleynikov <se...@aleynikov.org> wrote:
> I agree with Joe that having a design of the HTTP server architecture
> included in OTP that would offer standardized internal API and allow
> developers to either replace selected functionality via providing
> callback modules or swap the HTTP servers easily would be a great plus.
> Concerning inets, what's really strange is that whenever we see some
> benchmarks posted comparing performance of different HTTP
> implementations, inets is never included there, though I would think, it
> would make more sense in terms of showing how a custom implementation
> compares to what's included in OTP. Does it have to do something with
> what Loďc referred to as inets being well hidden in OTP? I doubt, but
> if not that, then what? There are not even benchmarks comparing yaws to
> inets, and despite the fact that both offer a rich set of functionality
> make one wonder was was the idea of writing yaws in the first place
> given existence of inets' httpd.
> Diversity is good, but at some point it creates a burden for the user
> who needs to make a selection of an HTTP server for his project.
> Instead of solving the application-related problem, now the task becomes
> studying several implementations to see which one offers a feature-set
> best suited for the problem being solved, and the answer to that
> question is not always straightforward.
> Perhaps a good initiative from the OTP team would be to spec out the
> open architecture of an HTTP server and welcome feedback from authors of
> current http libraries, so that the successor of inets would be flexible
> enough to accommodate building light-weight and heavy-weight HTTP
> servers based on the supported code base included in OTP? This would
> also open a possibility of diversity for library writers
> replacing/enhancing parts to suit their needs.
> Serge
> On 10/1/2012 3:52 PM, Loďc Hoguin wrote:
>> On 10/01/2012 08:11 PM, Kenneth Lundin wrote:
>>> > * OTP has a long release process. Many companies need their fixes to
>>> be included quickly to avoid maintaining huge branches for their
>>> changes. They generally agree in return to keep track of changes that
>>> might happen during the product's development.
>>> I think we are pretty fast with service releases about every 3rd month
>>> and updates on the maint branch in git between that.
>> But doesn't that imply upgrading the whole of OTP? If it does, that
>> means more testing and more chances for new issues. If you can build a
>> release upgrade with one app from one OTP version and the rest from a
>> previous version, I'd like to know how.
>>> > Note that while not everyone uses inets, most servers do use
>>> erlang:decode_packet/3 which does most of the job. Cowboy doesn't
>>> anymore, because it needed backward incompatible changes to
>>> erlang:decode_packet/3. I didn't ask your thoughts, simply because the
>>> experiment I did hinted that doing the parsing in full Erlang was a
>>> better idea: this change results in improved HiPE optimizations and
>>> better reduction management which results in better latency under load.
>>> Interesting, would be good if you could give us more info on this. The
>>> http parsing in
>>> decode_packet is there to improve performance. If it is not ( without
>>> using HiPE) we should consider removing it!
>> Between now and then binaries appeared, got performance improvements,
>> and considering Cowboy is full binary (no conversion to atom for methods
>> or header names or anything) and only receives requests, not responses,
>> that means it can skip many steps that decode_packet/3 can't.
>> Binaries also mean you can parse the request line and all the headers
>> using a single match context which seems to be very fast (good job!) and
>> I'm guessing is the main reason for improved performance under HiPE (I'm
>> about to start reading the HiPE papers to understand that better).
>>> >> In Erlang/OTP we have keep strict compatibility with many of our APIs
>>> >> which is both a strength and sometimes a burden (when we are not
>>> happy
>>> >> with the old API) but there are mostly ways around that if there is
>>> >> strong case for a change.
>>> > It's a big strength for people looking for stability and a big burden
>>> for people looking for reactivity. Can't really blame the second group
>>> for finding solutions to their problems.
>>> I am not blaming people
>> I used a word too strong there. :)
>>> for implementing their own solution, it is good
>>> to have a lot to choose from. But I still think the stability is very
>>> important for the majority of users making systems with a longer life
>>> cycle than a year.
>>> Our experience is that we have to be very compatible in order to
>>> convince our
>>> users (often big projects) to step up to our latest release.
>>> By this we can also stop supporting older versions.
>> Yes and I wouldn't want that to stop, that's a strong point. In fact
>> having both ways is great, it means you can make sure the important
>> parts of your system are stable while the more bleeding-edge parts are
>> kept, well, bleeding-edge.
>>> >> I think it is of great value to have a relative complete
>>> functionality
>>> >> in Erlang/OTP so that
>>> >> the user can implement basic functionality without having to search
>>> >> for separate implementations of, with unknown quality and
>>> completeness
>>> >> at least for a beginner.
>>> > I would prefer Erlang/OTP to focus on the core features and leave the
>>> satellite features to third parties. OTP has an HTTP server but doesn't
>>> have proper Unicode support. OTP has OpenGL but doesn't have proper
>>> timezones support. These 2 examples missing from OTP are some of many
>>> things that are too complex for most people to implement, and that would
>>> not benefit from having alternatives.
>>> What is missing when it comes to timezones? Has it been mentioned in
>>> another thread?
>>> I think we rely on the OS configuration here.
>> Date/time conversion from one timezone to another. For example if your
>> data is saved with UTC time but the user is in Paris you want to convert
>> it to Europe/Paris.
>> Perhaps it exists, but I haven't found it.
>> I think there's a lot of work to be done with regards to localization
>> and internationalization support, but I haven't had enough time to think
>> about it in details yet. Unicode (not just Unicode though, but a native
>> string type abstracting encodings, EEP coming in a few weeks),
>> timezones, graphical drawing libraries are pretty much what I find lacks
>> the most for the web side of things. Graphical drawing libraries are
>> fine outside of OTP for a while (until there's one that can do near
>> everything and do it well) though.
>> Thanks for the info on package management and bug tracker. Hope we'll be
>> involved in the process of creating the former. And thanks for your nice
>> reply. :)
On Oct 2, 2012, at 4:18 PM, "Thomas Allen" <tho...@oinksoft.com> wrote:
> I couldn't disagree with that much more ... npm/Node.js is cluttered with
> packages of dubious quality that are not maintained in the slightest, and
> while this "Wild West" they prefer may seem exciting, I do not think it
> bodes well for the long-term stability of projects that depend on these
> packages.
Two ideas spring to mind immediately.
First, one should definitely not make the package repository a dump. You want to make package rules. Does it have documentation? Does it have a test suite? Can the test suite be run automatically? Semantic versioning? And so on. Is the package experimental, does it have real world uses?
Furthermore, we could definitely adopt the ideas from the haskell-platform: A set of well-defined and stable packages are made into a basis library. Packages from the dump "graduate".
Jesper Louis Andersen
Erlang Solutions Ltd., Copenhagen
On Wednesday, October 3, 2012 1:04:45 AM UTC-7, Jesper Louis Andersen wrote:
> > I couldn't disagree with that much more ... npm/Node.js is cluttered > with > > packages of dubious quality that are not maintained in the slightest, > and > > while this "Wild West" they prefer may seem exciting, I do not think it > > bodes well for the long-term stability of projects that depend on these > > packages.
> Two ideas spring to mind immediately.
> First, one should definitely not make the package repository a dump. You > want to make package rules. Does it have documentation? Does it have a test > suite? Can the test suite be run automatically? Semantic versioning? And so > on. Is the package experimental, does it have real world uses?
> Furthermore, we could definitely adopt the ideas from the > haskell-platform: A set of well-defined and stable packages are made into a > basis library. Packages from the dump "graduate".
I believe in giving people simple and flexible tools and let them build the rules later on. This is basically why, after doing my work on Agner, I recently launched EXPM (http://expm.co, http://rashkovskii.com/2012/10/01/expm-or-meet-agner-2/) that is much simpler and much more flexible. EXPM allows to host your own repos very trivially, therefore allowing to build any combinations, including the one suggested by you.
So far I am just running a "dump" index at expm.co to bootstrap this system (after all, publishing got magnitudes easier with expm comparing to Agner), but I can very much see how we can eventually add another repository (say, stable.expm.co) that will only "host" graduated packages.
On Wed, Oct 03, 2012 at 01:11:08AM -0700, Yurii Rashkovskii wrote:
> I believe in giving people simple and flexible tools and let them build the > rules later on. This is basically why, after doing my work on Agner, I > recently launched EXPM (http://expm.co, > http://rashkovskii.com/2012/10/01/expm-or-meet-agner-2/) that is much > simpler and much more flexible. EXPM allows to host your own repos very > trivially, therefore allowing to build any combinations, including the one > suggested by you.
... And why are you announcing this in a thread about "Erlang http
servers"? Please, make a proper announcement.
> So far I am just running a "dump" index at expm.co to bootstrap this system > (after all, publishing got magnitudes easier with expm comparing to Agner), > but I can very much see how we can eventually add another repository (say, > stable.expm.co) that will only "host" graduated packages.
From first sight it looks like a clone of PyPi (which, IMHO, is
great). Let's see how it goes.
>>> I believe in giving people simple and flexible tools and let them build the
>>> rules later on. This is basically why, after doing my work on Agner, I
>>> recently launched EXPM (http://expm.co,
>>> http://rashkovskii.com/2012/10/01/expm-or-meet-agner-2/) that is much
>>> simpler and much more flexible. EXPM allows to host your own repos very
>>> trivially, therefore allowing to build any combinations, including the one
>>> suggested by you.
>> ... And why are you announcing this in a thread about "Erlang http
>> servers"? Please, make a proper announcement.
I will, I am just fixing things here and there to make sure it doesn't
have too many embarrassingly stupid bugs when I make a "proper"
announcement.
But I do want to involve early adopters who like playing with new
stuff (that's why I started talking about it, after a long period of
silence on "Agner 2" project)
I feel compelled to contribute on the topic of the hidden inets
application. Once a upon a time someone in Ericsson wrote an http
server that was very much inspired by Apache, which probably seemed
sensible at the time but it does not feel like the erlang way of doing
things. It was almost not document but used extensively by
Ericsson projects, hence very hard to get rid of. Years later several
improvements have been made and inets httpd server can be run in a
much more erlangish way without Apache config files, even though we
still have to support them. The documentation is much improved but
still quite sparse. And now I think we come to the negative circle. A
lot of pople seem to think HTTP is easy and roll their own and the
Ericsson projects are mostly content and do not have so many new
requirements. As you know HTTP is not Ericsson core business and most
open source users used something else and hence inets have not been
given the highest priority. And there
has been hardly no user contributions. Most efforts to improve inets
has been to the HTTP client as it is more used. Lesson to be learnt it
is worth doing things properly from the start because other pople may
have to live with your code and bad naming for years and years to
come.
Regards Ingela Erlang/OTP team - Ericsson AB
2012/10/3 Michael Turner <michael.eugene.tur...@gmail.com>:
> I recently received a copy of Building Web Applications with Erlang,
> which is almost entirely about yaws. The discussion of inets on this
> thread spurred me to look in the index under "i" -- except, oops, the
> book has no index. It does have an appendix about other webservers and
> frameworks in Erlang -- but inets is not listed there.
> "Well-hidden" indeed.
> -michael turner
> On Wed, Oct 3, 2012 at 12:52 AM, Serge Aleynikov <se...@aleynikov.org> wrote:
>> I agree with Joe that having a design of the HTTP server architecture
>> included in OTP that would offer standardized internal API and allow
>> developers to either replace selected functionality via providing
>> callback modules or swap the HTTP servers easily would be a great plus.
>> Concerning inets, what's really strange is that whenever we see some
>> benchmarks posted comparing performance of different HTTP
>> implementations, inets is never included there, though I would think, it
>> would make more sense in terms of showing how a custom implementation
>> compares to what's included in OTP. Does it have to do something with
>> what Loďc referred to as inets being well hidden in OTP? I doubt, but
>> if not that, then what? There are not even benchmarks comparing yaws to
>> inets, and despite the fact that both offer a rich set of functionality
>> make one wonder was was the idea of writing yaws in the first place
>> given existence of inets' httpd.
>> Diversity is good, but at some point it creates a burden for the user
>> who needs to make a selection of an HTTP server for his project.
>> Instead of solving the application-related problem, now the task becomes
>> studying several implementations to see which one offers a feature-set
>> best suited for the problem being solved, and the answer to that
>> question is not always straightforward.
>> Perhaps a good initiative from the OTP team would be to spec out the
>> open architecture of an HTTP server and welcome feedback from authors of
>> current http libraries, so that the successor of inets would be flexible
>> enough to accommodate building light-weight and heavy-weight HTTP
>> servers based on the supported code base included in OTP? This would
>> also open a possibility of diversity for library writers
>> replacing/enhancing parts to suit their needs.
>> Serge
>> On 10/1/2012 3:52 PM, Loďc Hoguin wrote:
>>> On 10/01/2012 08:11 PM, Kenneth Lundin wrote:
>>>> > * OTP has a long release process. Many companies need their fixes to
>>>> be included quickly to avoid maintaining huge branches for their
>>>> changes. They generally agree in return to keep track of changes that
>>>> might happen during the product's development.
>>>> I think we are pretty fast with service releases about every 3rd month
>>>> and updates on the maint branch in git between that.
>>> But doesn't that imply upgrading the whole of OTP? If it does, that
>>> means more testing and more chances for new issues. If you can build a
>>> release upgrade with one app from one OTP version and the rest from a
>>> previous version, I'd like to know how.
>>>> > Note that while not everyone uses inets, most servers do use
>>>> erlang:decode_packet/3 which does most of the job. Cowboy doesn't
>>>> anymore, because it needed backward incompatible changes to
>>>> erlang:decode_packet/3. I didn't ask your thoughts, simply because the
>>>> experiment I did hinted that doing the parsing in full Erlang was a
>>>> better idea: this change results in improved HiPE optimizations and
>>>> better reduction management which results in better latency under load.
>>>> Interesting, would be good if you could give us more info on this. The
>>>> http parsing in
>>>> decode_packet is there to improve performance. If it is not ( without
>>>> using HiPE) we should consider removing it!
>>> Between now and then binaries appeared, got performance improvements,
>>> and considering Cowboy is full binary (no conversion to atom for methods
>>> or header names or anything) and only receives requests, not responses,
>>> that means it can skip many steps that decode_packet/3 can't.
>>> Binaries also mean you can parse the request line and all the headers
>>> using a single match context which seems to be very fast (good job!) and
>>> I'm guessing is the main reason for improved performance under HiPE (I'm
>>> about to start reading the HiPE papers to understand that better).
>>>> >> In Erlang/OTP we have keep strict compatibility with many of our APIs
>>>> >> which is both a strength and sometimes a burden (when we are not
>>>> happy
>>>> >> with the old API) but there are mostly ways around that if there is
>>>> >> strong case for a change.
>>>> > It's a big strength for people looking for stability and a big burden
>>>> for people looking for reactivity. Can't really blame the second group
>>>> for finding solutions to their problems.
>>>> I am not blaming people
>>> I used a word too strong there. :)
>>>> for implementing their own solution, it is good
>>>> to have a lot to choose from. But I still think the stability is very
>>>> important for the majority of users making systems with a longer life
>>>> cycle than a year.
>>>> Our experience is that we have to be very compatible in order to
>>>> convince our
>>>> users (often big projects) to step up to our latest release.
>>>> By this we can also stop supporting older versions.
>>> Yes and I wouldn't want that to stop, that's a strong point. In fact
>>> having both ways is great, it means you can make sure the important
>>> parts of your system are stable while the more bleeding-edge parts are
>>> kept, well, bleeding-edge.
>>>> >> I think it is of great value to have a relative complete
>>>> functionality
>>>> >> in Erlang/OTP so that
>>>> >> the user can implement basic functionality without having to search
>>>> >> for separate implementations of, with unknown quality and
>>>> completeness
>>>> >> at least for a beginner.
>>>> > I would prefer Erlang/OTP to focus on the core features and leave the
>>>> satellite features to third parties. OTP has an HTTP server but doesn't
>>>> have proper Unicode support. OTP has OpenGL but doesn't have proper
>>>> timezones support. These 2 examples missing from OTP are some of many
>>>> things that are too complex for most people to implement, and that would
>>>> not benefit from having alternatives.
>>>> What is missing when it comes to timezones? Has it been mentioned in
>>>> another thread?
>>>> I think we rely on the OS configuration here.
>>> Date/time conversion from one timezone to another. For example if your
>>> data is saved with UTC time but the user is in Paris you want to convert
>>> it to Europe/Paris.
>>> Perhaps it exists, but I haven't found it.
>>> I think there's a lot of work to be done with regards to localization
>>> and internationalization support, but I haven't had enough time to think
>>> about it in details yet. Unicode (not just Unicode though, but a native
>>> string type abstracting encodings, EEP coming in a few weeks),
>>> timezones, graphical drawing libraries are pretty much what I find lacks
>>> the most for the web side of things. Graphical drawing libraries are
>>> fine outside of OTP for a while (until there's one that can do near
>>> everything and do it well) though.
>>> Thanks for the info on package management and bug tracker. Hope we'll be
>>> involved in the process of creating the former. And thanks for your nice
>>> reply. :)
