Nonblocking I/O

[Trung Pham]

Do you know if Elixir implements non blocking i/o like NodeJS or Ruby EventMachine?

Is there wasted CPU cycle waiting for http or database calls to response? Or does it put the process to sleep and wake it up when the data is ready?

[José Valim]

The majority of languages have non-blocking IO. Ruby has non-blocking IO too and you don't even need Event Machine, just threads.

In Elixir, when you do an IO call, that particular Elixir process blocks, but of course, all other processes continue running, using IO and CPU resources as required. You will find that Elixir has better performance than both those technologies because although Node.js and Event Machine have non-blocking IO, any CPU related task blocks all other events from running (it is a single queue) while Elixir is able to multiplex on both IO and CPU efficiently using a single operating system process.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

On Fri, Mar 6, 2015 at 7:25 PM, Trung Pham <tr...@phamcom.com> wrote:

Do you know if Elixir implements non blocking i/o like NodeJS or Ruby EventMachine?

Is there wasted CPU cycle waiting for http or database calls to response? Or does it put the process to sleep and wake it up when the data is ready?

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/5e556f48-53e9-48a0-a31d-28bc408370ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Trung Pham]

Let's say if my machine has 4 cores, and I'm making 4 long i/o requests in 4 separate processes. Will the fifth process be blocked until one of the i/o requests finished?

I know in NodeJS, those 4 i/o requests will be suspended, while the main program continues on.

Thanks.

[José Valim]

No. It won't. The cores are being used by processes that are not waiting on I/O. The 5th process won't block. This is true for Ruby, Java, Haskell, Elixir, Erlang, etc.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/e2fecc02-2b9a-44cc-9160-3fe97bfd3567%40googlegroups.com.

[Wojciech Kaczmarek]

Hi!

I guess this may be helpful: If you have in mind a webservice implemented in Elixir, the instructions in one request will be interspersed with instructions executed in another request. This would be an equivalent of what Eventmachine does for Ruby, green threads or "stackless" stuff does for Python, NodeJS does by callback nesting ..etc. This comes for free in the Erlang VM and without any tricks done to the VM (like in Ruby or Python) or without special syntax (like in NodeJS). It's just different processes which are executed in parallel, even on one CPU core. Same goes for IO - if your web handler talks to the database and waits for its response, the requests will not block each other - you don't need inventions like putting many separate backend servers for your stuff.

I believe this is a huge advantage of our platform.

Best,

W.

On Fri, Mar 6, 2015 at 7:25 PM, Trung Pham <tr...@phamcom.com> wrote:

Do you know if Elixir implements non blocking i/o like NodeJS or Ruby EventMachine?

Is there wasted CPU cycle waiting for http or database calls to response? Or does it put the process to sleep and wake it up when the data is ready?

--

[Ed W]

Hi Jose

I'm not sure I agree with your description on the difference between blocking/non-blocking to be honest?

My definition would be:

1) You have a large quantities of threads running, to perform blocking IO. The thread scheduler is responsible for suspending threads which can't continue and giving the CPU to another thread
2) You have a single thread which "non-blocks" on a "select". The single thread is essentially woken up the instant an IO completes and can efficiently give the CPU to whatever procedure wants to continue

I would suggest that Erlang and threaded ruby would be 1) above ie blocking
Node or AnyEvent or Nginx or Event Machine are all examples of 2), ie non-blocking

The thread solution is the naive "obvious" way to do things. However, it turns out that all common operating system threads have significant overhead spinning trying to find a thread which can progress (ie you give a thread some CPU time, discover it's blocked, move to the next thread, etc - all that consumes CPU)

Non-Blocking is effectively just implementing your own very efficient thread scheduler!  You are literally woken up and told which "thread" can progress, you give it CPU (ie call your continuation for whatever IO handle you receive).  So it's really a massive optimisation at the cost of having to write all the details of the scheduler yourself

There are several examples of extremely high performing servers using non blocking techniques (nginx comes to mind as a poster child).

The main challenge I see with non-blocking kernels is efficiently using multi-cores.  You need to be reasonably clever as essentially you are using a small number of operating system threads and implementing your own green thread-alikes on that.  So additionally you (generally) lose your pre-emptability (or at least need to re-implement it yourself)

I think why Erlang/Elixir is interesting is that it has a very efficient green threads implementation which is also multi-core capable.  So whilst I don't know the benchmarks, it seems to be reasonably performant to implement a blocking-style io and leave it up to the BEAM scheduler to handle the details of unblocking the correct thread and assigning it some CPU time.  You get to write things the obvious way, not need to be building continuations all the time (witness the "callback hell" of most non block implementations), keep pre-emptability and multi-core, etc


So I think the answer is:
- Yes, Erlang can implement non blocking IO
- No you probably don't want to do it in general
- According to google, under the bonnet (hood) Erlang uses non blocking IO (sounds about right, I didn't check the code though).  So it can likely be quite optimised in suspending and waking up "threads".
- You would need to do some benchmarks to show you could actually be more efficient to implement your own thread-alike using non-block... You might not lower the overhead, or perhaps only for specific tight loops (socket accept loop perhaps?)

However, I would argue that generally Erlang/Elixir is coded in a "blocking" style. I do accept that the virtual machine may have an efficient non blocking implementation of that blocking style though.

Regards

Ed W

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAGnRm4L-wC%2Brb4poXyf5M3DaEyygPvOpSr8_XtYCQ54A0dNGJg%40mail.gmail.com.

[José Valim]

Thanks Ed!

Both your examples are non-blocking. They are just different implementations of non-blocking systems:

1. http://en.wikipedia.org/wiki/Asynchronous_I/O#Light-weight_processes_or_threads

2. http://en.wikipedia.org/wiki/Asynchronous_I/O#Select.28.2Fpoll.29_loops

Non-blocking is if your program can do anything while waiting on I/O, be it in another thread, using callbacks, goroutines, etc.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/54FA32F9.4000202%40wildgooses.com.

[Trung Pham]

Most Erlang io libraries are blocking by default.

Additional effort is required to make io request in a nonblocking manner. See example: https://github.com/cmullaparthi/ibrowse#asynchronous-requests

Even though, the other processes are not blocked, but the scheduler still allocates CPU time to the supposedly suspended io process. I think this is where NodeJs has an advantage over Erlang.

[José Valim]

Most Erlang io libraries are blocking by default.

Additional effort is required to make io request in a nonblocking manner. See example: https://github.com/cmullaparthi/ibrowse#asynchronous-requests

You are mixing two things. Because proceses are async, APIs like ibrowse can provide both sync and async operations. That is just an example of that. It is not about the VM at all just how someone chose to design an API.

Node also can read files in a sync ways. Doing it the async way requires extra work by passing a callback.

 

Even though, the other processes are not blocked, but the scheduler still allocates CPU time to the supposedly suspended io process. I think this is where NodeJs has an advantage over Erlang.

This is not true. The Erlang VM has schedulers with run queues per core. Those queues are composed mostly of work to be performed either by running processes and I/O jobs. Once an Erlang process starts waiting for I/O, it no longer does any work, so no work will be put on the queue and therefore no scheduler will allocate work for that process.

The VM is very efficient at keeping processes waiting, that's why folks like WhatsApp can have 1 millions processes on the same machine, because they are mostly waiting (or on messages or on IO). Once a process enters in waiting mode, it is not awaken unless a specific event happens, like receiving a message. The schedulers do not frequently check the processes to see what is happening.

There are talks about scheduling the Erlang VM online for those more interested.

[Ed W]

On 06/03/2015 23:16, José Valim wrote:

> Non-blocking is if your program can do anything while waiting on I/O,
> be it in another thread, using callbacks, goroutines, etc.
>

Not to over-argue the semantics here, but I have googled this for a few
mins now and I think I see two common definitions for "non block"

1) Where your runtime has only one thread and no operating system
threads or green-threads (Javascript say). So you are manufacturing your
own concurrency primitives. This I believe is the definition Jose is
promoting. Also I think it clarifies your original claim that Ruby is
"non blocking using threads", here I think I agree that ruby offers
fibres and other lightweight concurrency primitives, so you can
effectively "block" and leave your concurrency primitives to avoid a
complete stall in other threads.

2) But I think the motivation for several high performance servers to
use non-block architectures is literally to lower the overhead of thread
scheduling. ie I believe operating system threads will fall apart at
some relatively low number of threads, eg servicing 1 million http
requests it's not feasible to start 1 million operating system threads?
However, it's relatively performant to put 1 million things in a
non-blocking select and wait for something to happen.

I didn't think about it until now, but I guess the whole core of Erlang
is really one large "select"? The whole language really works by
sending a message to some other process and waiting for a message back
again? I presume that's how IO is actually implemented in Erlang, you
send the operating system some message and the VM non-block waits on a
response, waking up the caller once the response is delivered. So it's
really a blocking API on top of an efficient green-thread vm!

For those who haven't given up and wandered off, see Perl's CORO for an
interesting take on this


I guess to try and unify those two points:
- You generally need some kind of concurrency primitives in your
programming language. Non-blocking IO seems to be a very popular
implementation right now.
- You could use threads/fibres instead, either green or operating system
flavoured
- However, for extremely high performance IO loops, it's often
beneficial to write your own concurrency primitives. The fastest
performing primitive that I'm aware of is to run a single thread,
blocking on a "select" (alike, eg epoll), with your own dispatcher on
activity. This gets you right down to the bare metal. Optimisation are
to handle your list of things to wait on over multiple threads, one for
each CPU. Of course nothing stops you dispatching those events back to
higher level primitives, eg multiple operating system or green threads.


I would also point out that Erlang is "lightweight threads", but not
"superlightweight"? I believe we are still looking at 2KB per erlang
"process"? I could write a select based scheduler which managed 1
million tcp connections storing just a few words per connection, but you
would be looking at several gigabytes of ram to start an erlang process
per connection. 10 million erlang processes seems unlikely on typical
desktop and small server machines which are available, but might be
feasibly implemented in a non-blocking select type fashion

Note, I do agree there is a large class of problems which can live with
a "million thread" limitation, especially in return for nicer
concurrency primitives!

Cheers

Ed W

[José Valim]

Two notes:

> The VM is very efficient at keeping processes waiting,

I meant: The VM is very efficient at managing waiting processes. The point is that the VM is not giving work nor polling processes it knows that do not have work to do.

It is also worth pointing out that VM has an async OS thread pool exactly for IO operations. They are moved of the schedulers queue (to not block them) and into the async pool. This is the [async-threads:10] parameter you see when you start iex.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

[Ed W]

On 07/03/2015 00:12, José Valim wrote:

Even though, the other processes are not blocked, but the scheduler still allocates CPU time to the supposedly suspended io process. I think this is where NodeJs has an advantage over Erlang.

This is not true. The Erlang VM has schedulers with run queues per core. Those queues are composed mostly of work to be performed either by running processes and I/O jobs. Once an Erlang process starts waiting for I/O, it no longer does any work, so no work will be put on the queue and therefore no scheduler will allocate work for that process.

I hadn't really thought about it before, but the whole language kind of looks like a wrapper around some functions that are pushing themselves onto a select queue and waiting for an answer? ie everything is about sending a message and waiting for an answer and lends itself to an easy underlying (non-block) implementation.

However, the exciting thing Erlang offers that many other language (when talking non-block) concurrency primitives aren't offering is pre-emptive threads.  So you can actually spin up lots of threads doing "while(true)" and they will all get some CPU time. With most popular non-block languages you have to be painfully careful to never "block" or hog the CPU for more than a short time or you stall everything else... (this is often hard because most of these languages are a mix of block and non block libraries...)

Erlang and Elixir is a very interesting concurrency model.  Thanks Jose for creating Elixir!

Ed W

[Wojciech Kaczmarek]

I guess that since "non-blocking" and "async" became buzzwords there's always some area of confusion around...

On Sat, Mar 7, 2015 at 1:14 AM, Ed W <li...@wildgooses.com> wrote:

[...]

I didn't think about it until now, but I guess the whole core of Erlang is really one large "select"?

No.

 

  The whole language really works by sending a message to some other process and waiting for a message back again? I presume that's how IO is actually implemented in Erlang, you send the operating system some message and the VM non-block waits on a response, waking up the caller once the response is delivered. So it's really a blocking API on top of an efficient green-thread vm!

 

Yup, roughly (you seem to be drawing some technical conclusions but you have a good gut feeling of how it works).

W.

 

[Robert Virding]

It is as José says. Now the BEAM will run up extra threads which are used for IO, the async threadpool. When you do an IO request it will be handed over to one of the async threads which does the IO. While that is happening the erlang/elixir process will be be suspended and that scheduler can then go on and run other erlang/elixir processes. When the IO request has completed then the erlang/elixir process will be rescheduled to handle the IO return value. The system never blocks.

You can set the size of the async threads pool depending on how much IO your system will doing.

That the system should not block on IO has been one of the basic requirements of erlang from the very beginning. If erlang had blocked on IO then it would just not have been interesting for the type of applications we were looking at. Back in the old days before there were OS threads which could handle IO then we used explicit non-blocking IO, and then the system would check with select calls. The non-blocking requirement affects many other things in the system as well.

Robert

On Saturday, March 7, 2015 at 1:23:17 AM UTC+1, José Valim wrote:

Two notes:d

[Robert Virding]

The problem is not to write your own non-blocking select based scheduler, I have done a number a number of them, who hasn't. The problem with doing that is that you then have to make sure that nothing else in your whole system will cause it to block. All you need is *one* thing which can block to break it. One way is to do as Node.js which works but leads to a convoluted and sometimes hard to follow style of writing your code[*]. Being able to write your code as processes with sequential code and hand over the problem of scheduling to the system is an Enormous Win (TM). This is what the Erlang concurrency model and processes, yes they are processes just not OS processes, provide. In the same way as you do with OS processes.

This is not to say that this is always the best way for your application. For example if you just want raw throughput and are willing to sacrifice low-latency then writing a specific hard-wired solution will usually be faster. Of course it will often be difficult to scale, at least automatically as the erlang VM does.

The largest number of concurrent TCP connections I have heard of *in a product* is WhatsApp, http://blog.whatsapp.com/196/1-million-is-so-2011, that had 2 million on one machine running erlang. One guy there told me that they one time peaked at 3 million. So, yes, millions of connections are possible with an erlang system.

I think one of the man benefits of using erlang to do this type of thing is that you don't have to do anything special to be able to do it. OK, that is a slight exaggeration, you need to use the '+P' to set the maximum number of processes, the default is only 262k.

Robert

[*] This style of doing non-blocking systems is the same as the Ericsson AXE switch uses. The AXE is from the '70s. There is nothing new under the sun. This appeals to my slightly twisted sense of humour.

[Booker Bense]

FWIW, it's not very hard to get more cpu time used than wall clock time for elixir programs that are doing nothing 

but I/O. 

I've been playing with writing elixir command line tools for searching file trees and it's pretty simple to get results

like this 

time elixgrep webauth1 ~/work/taylor.hg

  0.65 real         2.00 user         0.52 sys

(i.e. on a 4 core machine , I'm getting 2.52 cpu time in 0.65 wall clock time )

This program is doing nothing but the equivalent of find/grep across about 1000 files. It's not that you can't get this kind of effective use of 

multi-core systems with other languages, but it's no where near as easy. 

- Booker C. Bense

[Jim Freeze]

That's interesting. According to my math, you are utilizing about 3.9 cores. :)

Can you share a code snippet?

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/4598de48-de32-4335-abee-ddbe61d22b0c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Dr. Jim Freeze, Ph.D.

(m) 512 949 9683

[Booker Bense]

The whole code is here

https://github.com/bbense/elixgrep

It was mostly a learning exercise for me. There are a couple key constructs that should 

be replaced by OTP things like Agent and Task, but I wanted to do it "from scratch". 

- Booker C. Bense 

[Pedro Medeiros]

It's a nice number.

I created a small mix app for my current work to simulate some CDR to be stored and manipulated on our high performance C++ cluster. I was also surprised how my Elixir app could use all the cores just out of the box.

--

You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/561fb789-02ab-43a2-a5dd-acd88be2fb4d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Pedro Henrique de Souza Medeiros
----------------------------------
Cel: +55 (61) 9197-0993
Email: pedr...@gmail.com

Beautiful is better than ugly,
Explicit is better than implicit,
Simple is better than complex,
Complex is better than complicated.

The Zen of Python, by Tim Peters