How to switch Elixir to use multicore ?

[mraptor]

How can I make iex/Elexir use multicore CPU ?
Is there some command I can run and see what is the current config for multi-cpu ?

I see in the videos you mentioned as shortcut :

myfun &1

but it seems the current version is using :

&myfun &1

instead, I'm I correct ?

thanks

[mraptor]

One more thing is there a way to time an opration, something like :


time map (list, &fib(&1))

[Vincent Siliakus]

The Erlang VM that Elixir uses should by default use all available CPU cores. You can see this most easily when you startup iex. The first line should look something like this:

Erlang R16B02 (erts-5.10.3) [source] [64-bit] [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false]

Notice the [smp:2:2] info. This tells you that the Erlang VM sees 2 cores and uses 2 schedulers to schedule Erlang/Elixir processes. By default 1 scheduler per core is used, which is something you rarely have to adjust.

If you want to check the number of available cores programmatically, you can use Erlang's system_info function:

iex(1)> :erlang.system_info(:logical_processors)
2

Note that Elixir doesn't automatically make your code using multiple cores. It's your own responsibility to distribute workload over multiple Elixir processes.

To measure execution time, you can use Erlang's timer module like this:

:timer.tc(fn -> Enum.map (list, &fib(&1)) end)

It returns a 2 element tuple. The first value is execution time in microseconds and the second the return value of the code you measure.
  

Op dinsdag 3 december 2013 19:53:27 UTC+1 schreef mraptor:

[mraptor]

Thank you..


Erlang R16B01 (erts-5.10.2) [source] [async-threads:10]

Interactive Elixir (0.11.3-dev) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>  :erlang.system_info(:logical_processors)
4

Yes I made a parallel map to test this , and the reason to suspect something is fishy was that when I ran it was using just one of the CPU's.

Hmm... on my other machine it showed 4:4... and the parallel map worked.
What about the HiPE, I know it is better code generation... is it separate package or I just have to compile Erlang with different compile switches ?

[Alexei Sholik]

Try this:

iex --erl '+S 8' 

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Best regards
Alexei Sholik

[mraptor]

My bad :
 $ eix erlang
[D] dev-lang/erlang
     Available versions:  15.2 15.2.2 15.2.3.1 (~)16 {compat-ethread doc emacs halfword hipe java kpoll odbc sctp smp ssl tk wxwidgets ELIBC="FreeBSD"}
     Installed versions:  16.2(10:16:19 AM 12/03/2013)(ssl wxwidgets -compat-ethread -doc -emacs -halfword -hipe -java -kpoll -odbc -sctp -smp -tk ELIBC="-FreeBSD")
     Homepage:            http://www.erlang.org/
     Description:         Erlang programming language, runtime environment, and large collection of libraries

It was compiled w/o smp support and hipe.. should I also recompile with "-compat-ethread" ?
Now it is better :")
$ ./bin/iex
Erlang R16B01 (erts-5.10.2) [source] [smp:4:4] [async-threads:10] [hipe]

[Vincent Siliakus]

Op woensdag 4 december 2013 17:44:39 UTC+1 schreef mraptor:

What about the HiPE, I know it is better code generation... is it separate package or I just have to compile Erlang with different compile switches ?

Using hipe, code is compiled to native code. This may or may not give a speed boost, depending on the type of code. There is also some overhead involved when switching between native and emulated code, so you should try to avoid this from happening in a tight loop. So using hipe is not necessarily better or faster. I guess you just should carefully benchmark your code if it helps for your particular case.

You can compile an Elixir module to native code with hipe from the command line like this:

ERL_COMPILER_OPTIONS=native elixirc some_module.ex

or compile a complete project with hipe:

ERL_COMPILER_OPTIONS=native mix compile

[mraptor]

good to know ... thanks

[Dave Cottlehuber]

On 04. Dezember 2013 at 12:52:06, Vincent Siliakus (zam...@gmail.com) wrote:
>
> The Erlang VM that Elixir uses should by default use all available
> CPU
> cores. You can see this most easily when you startup iex. The first
> line
> should look something like this:
>
> Erlang R16B02 (erts-5.10.3) [source] [64-bit] [smp:2:2] [async-threads:10]
> [hipe] [kernel-poll:false]
>
> Notice the [smp:2:2] info. This tells you that the Erlang VM sees
> 2 cores
> and uses 2 schedulers to schedule Erlang/Elixir processes.
> By default 1
> scheduler per core is used, which is something you rarely have
> to adjust.
>
> If you want to check the number of available cores programmatically,
> you
> can use Erlang's system_info function:
>
> iex(1)> :erlang.system_info(:logical_processors)
> 2
>
> Note that Elixir doesn't automatically make your code using 
> multiple cores.

Correct - the requirement for using multiple cores is that you have
multiple processes requiring work to be done (run queue). In R15+ workload
is consolidated more strongly than in R14 series to less cores for
improving overall efficiency. More details in links below.

If you are not heavily reliant on multicore you can often get better
performance from a single (non-SMP) runtime. Every core adds increasing
locks within the scheduler & runtime, and may worsen overall cache
performance. AFAICT there’s no user-level info on this (unless you
dig into dtrace) so as always benchmark your code under production
load. Its not sufficient to run just one section of your code in
isolation; Erlang runtime’s strength is in concurrent & low latency
performance.

> It's your own responsibility to distribute workload over multiple
> Elixir
> processes.

The key point here is that multiple processes are able to allocated
to different processors. The ERTS (Erlang Run Time System) will almost
always take care of this for you automatically. Tuning is possible but
in my experience very hard to get right consistently and arguably not
worth doing in the bigger picture. See erl manual for options.

There is a lot of info about this hidden in various places, esp
on Erlang-Questions mailing list. I’ve included a lot of links[1] around
NIFs - you’ll see a couple of different viewpoints here.

The point is made that the BEAM schedulers at present are largely unable
to “see” how much time is spent inside NIFs, and also (from my understanding)
no pre-emptive scheduling power over NIFs — see davisp’s sleepy example
below. If you use tools like fprof and eprof [2] NIFs tend to show up as
0 time spent, which is obviously not the case.

Loïc from 99s and others are in favour of a zero-NIF policy. However in Apache
CouchDB, BigCouch, and CouchBase you’ll see a lot of NIFs used for
raw disk i/o, JSON parsing, disk compression, e.g. external libraries like
Google Snappy or LZ4, yajl or jiffy parser.

A+
Dave

[3] Some good reading is:
http://jlouisramblings.blogspot.com/2013/01/how-erlang-does-scheduling.html
http://ninenines.eu/articles/erlang-scalability
http://www.erlang.se/euc/08/euc_smp.pdf
http://kth.diva-portal.org/smash/record.jsf?searchId=2&pid=diva2:392243
https://github.com/davisp/sleepy
http://www.erlang-factory.com/upload/presentations/708/HitchhikersTouroftheBEAM.pdf
http://www.rigtorp.se/latency.html

https://gist.github.com/gburd/4121795

[1]: read the whole series associated with these threads (over a decade!)
http://www.erlang.org/doc/man/erl_nif.html
http://erlang.org/pipermail/erlang-questions/2001-April/003132.html
http://erlang.org/pipermail/erlang-questions/2010-July/052408.html
http://erlang.org/pipermail/erlang-questions/2012-October/069585.html
http://erlang.org/pipermail/erlang-questions/2013-February/072468.html
http://erlang.org/pipermail/erlang-bugs/2013-May/003545.html

[2]: Useful Erlang doc links:
http://www.erlang.org/doc/efficiency_guide/profiling.html
http://erlang.org/doc/man/erl.html
http://www.erlang.org/doc/efficiency_guide/processes.html
http://www.erlang.org/doc/man/percept.html

[Vincent Siliakus]

Wow, thanks for the comprehensive follow up on my answer!

One small thing I would like to correct in my original answer is the example about programmatically asking for the number of cores used by the Erlang Runtime:

:erlang.system_info(:logical_processors) returns the number of cores in the system, not the number of cores the Erlang Runtime actually uses. If you want to know the number of cores used by the Erlang Runtime, one should use:

:erlang.system_info(:logical_processors_available)

Op donderdag 5 december 2013 10:01:14 UTC+1 schreef Dave Cottlehuber: