[erlang-questions] erlc speed (or lack thereof), Make and emake

[Matthias Lang]

Hi,

I'm looking for tips and tricks to use speed up compiling a bunch of
Erlang modules from within a build system that uses (gnu)Make.

The system uses Erlang, but most of it is actually C code, which is compiled
under four different compilers, plus FPGA code. So ditching 'make'
for something Erlang-centric like 'rebar' doesn't look very attractive.

Background
----------

Doing this on R14B03:

erlc a.erl
erlc b.erl
erlc c.erl
...

for 70 modules on my machine takes 39s. If I do this instead:

erlc a.erl b.erl c.erl ...

it improves to 7s. If I split the job in two parts which run at the
same time, it improves again to 4s wall time, with both cores
busy.

So, a 10x improvement in compilation speed seems attainable, but how do
I explain how to do it to 'make'?


Ideas
-----

I had a few simple ideas for fixing this, but none really work. See below.

The more complicated idea I have is to make a shell script replacement
for 'erlc' which delegates the compilation work to an Erlang node
which is already running. Seems workable, and it won't spring
surprises on 'make', e.g. -j 4 will still do the right thing.

Anyone got a neat idea?

Rest of this mail is just a list of things that don't seem to work.

Matt

======================================================================


Bad Idea #1: straightforward 'make'
-----------

Here's my minimal starting point for a system of three modules, a b c:

beams=(addsuffix .beam, a b c)
%.beam: %.erl nasty_global_stuff.hrl
erlc $<

That causes 'make' to hand one module to erlc at a time, which is slow.


Bad Idea #2: try and use just one rule for all .beams
-----------

Like this:

modules=a b c
beams=$(addsuffix .beam, $(modules))
erls=$(addsuffix .erl, $(modules))
all: $(beams)

$(beams): $(erls) nasty_global_stuff.hrl
erlc $(filter %.erl, $@)

That does almost the right thing, but actually not at all. Change one
.erl and it'll recompile every .beam, because I've told 'make' that
each .beam depends on all .erl files.


Bad Idea #3: use erl -make for all Erlang compilations
-----------

all:
erl -make

That sort-of works, but it only runs one process (Erlang's own 'make'
doesn't seem to attempt running more than one compilation at a time),
passing options is not straightforward (they're not even syntactically
the same as for erlc) and it requires writing 'EMakefiles', so you
sort-of have to understand Erlang in order to be able to fiddle with
the build system.

It's also fiddly explaining dependencies between Erlang stuff and
other stuff to a combination of ordinary 'make' and erlang's own make.

--end of bad ideas---
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Loïc Hoguin]

When removing rebar dependency in the Cowboy makefile to improve my
workflow, I ran into the same issues you did. The normal way of doing
things with make is too damn slow. I'm assuming here that each erlc call
starts an Erlang VM, which isn't made to be fast.

I settled for this:
https://github.com/extend/cowboy/blob/master/Makefile#L33

This recompiles everything all the time, but that's not really an issue
for me. I might steal your bad idea #2 to avoid recompiling deps all the
time though.

--
Loïc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu

[Tim Watson]

Take a look at the Rabbit Makefile - it doesn't recompile all the time *and* it works with -J and is generally really fast.

http://hg.rabbitmq.com/rabbitmq-server/file/default/

Cheers,
Tim

[Paul Davis]

I once wrote a custom erlc frontend for couchdb to see if I could speed up our compilation process. It made a bit of a difference but on a decent machine with -J the improvement wasn't enough to justify adding the complexity.

Script is here:

https://github.com/davisp/couchdb/blob/couch_erlc/bin/couch_erlc

It was working well enough for me to run tests but I didn't spend much time testing it for general bugs and what not.

[Matthias Lang]

On Thursday, January 24, Tim Watson wrote:

> Take a look at the Rabbit Makefile - it doesn't recompile all the
> time *and* it works with -J and is generally really fast.

That Makefile is nice and tidy, but it's compiling the .erl files in
the simplest---and slowest---way possible. It calls 'erlc' about 370
times, i.e. once per file. I.e. exactly as per "Bad Idea #1" in my OP.

Quick measurements on building 3.0.1 from source:

plain 'make': 220s wall time
'make -j 2': 112s

I wanted to get an idea of how much of a speedup was possible, so I
broke out all the calls to 'erlc' from the build process and put
them in a shell script. There are about 370.

Running the script so it does one call to 'erlc' at a time takes 183s.

Changing the script so that each call to 'erlc' compiles many files
drops the time to 40s.

Further changing the script to use multiple (unix) processes gets me to 26s.

That's pretty encouraging. It suggests that a 3x or 4x speedup is
possible for the Erlang part of the Rabbit build process.

Matt

[Dmitry Kolesnikov]

Hello,

I had a following construction in Makefiles

%.beam : %.erl
$(AM_V_ERL)$(ERLC) $(ERL_CFLAGS) -I ./include -b beam \$<

define rules_ERLAPP
$(1)_BEAM=$(subst .in,,$(addprefix ebin/, $(notdir $($(1)_SRC:.erl=.beam))))
nobase_pkgliberl_SCRIPTS += $$($(1)_BEAM)
...
endef

It build only touched file… there was no issue with slow compile.
But I've swapped to rebar just to stream line my workflow and still no issues :-)

- Dmitry

[Erik Søe Sørensen]

One possibility is to let the shell figure out which files need to be recompiled, and invoke erlc on that batch.

Figuring out what needs recompilation could possibly be done by calling make in dry-run mode, to see what it thinks needs to be done, and then processing that output - or, as below, in a more direct manner.

I don't know if all the features necessary to do this are 100% portable, but sticking with (gmake +) bash, this works:

# Iterating through the source files, determine which need recompilation.

# Then - if any do - invoke the compiler on the batch.

fast-compile:

files=""; for i in src/*.erl ; do j=$${i%.erl}.beam; j=ebin/$${j#src/} ; [ "$$j" -nt "$$i" ] || files="$$files $$i" ; done ; \

[ -z "$$files" ] || erlc -o ebin $$files

/Erik

2013/1/24 Matthias Lang <matt...@corelatus.se>

[Scott Lystig Fritchie]

Matthias Lang <matt...@corelatus.se> wrote:

ml> Running the script so it does one call to 'erlc' at a time takes
ml> 183s.

Hrm, well, running on a random OpenSolaris virtual machine that I've got
lying around at the moment....

% time erl -eval "init:stop(0)."
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:0]
[kernel-poll:false] [dtrace]

Eshell V5.9.1 (abort with ^G)
1>
0.12u 0.05s 0:01.29 13.1%

So, almost 1.3 seconds to start up and stop. Some of that time is
probably related to shutting down. The rest is probably startup time.
Some fraction of that is probably the same overhead that starting "erlc"
from scratch each time, plus some overhead that "erlc" has that "erl"
doesn't. Reduce that sum of time, and you'll save a lot of time in your
Makefile recipes. Plus, that reduction would likely make "escript"
startup times lower, which would also please a lot of the Erlang world
also?

-Scott

P.S. Keeping mouth shut since rebar's automatic parallellization doesn't
seem to interest everyone else in the thread.... :-)

[Björn-Egil Dahlberg]

2013/1/25 Scott Lystig Fritchie <frit...@snookles.com>

 

So, almost 1.3 seconds to start up and stop.  Some of that time is
probably related to shutting down.  The rest is probably startup time.
Some fraction of that is probably the same overhead that starting "erlc"
from scratch each time, plus some overhead that "erlc" has that "erl"
doesn't.  Reduce that sum of time, and you'll save a lot of time in your
Makefile recipes.  Plus, that reduction would likely make "escript"
startup times lower, which would also please a lot of the Erlang world
also?

Though we now have parallel code loading, we don't have parallel code fetching during boot. I think most of it is dynamic on-demand fetching in a sequential process anyways.

I think something like spec:ing what modules should be loaded in default boot or app-file is the way to go. Start app -> batch parallel modules loading. We want something reasonable thought through. =)

 

P.S. Keeping mouth shut since rebar's automatic parallellization doesn't
seem to interest everyone else in the thread.... :-)

rebar is awesome!

I wonder why some people seem to shy away from it? 

I don't know if it is feasible to use rebar to build erlang applications within otp. I don't think so. It would be nice to have something like that and Emake needs a lot more attention if it would have the same role. We don't want to make yet another make-tool. it's just silly, we have rebar.

I think if we ever manage to move out applications from otp, that we still maintain, those should use rebar to build them. (I'm a big proponent of splitting down the monolithic otp-repo, let's say its not really prioritized). Within the otp-repo we will use make + erlc for the foreseeable future. 

On another note,

if rebar is the build-tool, what application is the concrete? A repository search and deploy engine?

// Björn-Egil

[ノートン ジョーセフ ウェイ ン]

Bjorn -

Why do you think it isn't feasible to use rebar to build erlang applications within OTP?  

Just curious.

thanks,

Joe N.

p.s. I have been toying with this idea on GitHub.

https://github.com/organizations/otphub

I haven't spent any significant time on this exercise … just out of curiosity.

[Loïc Hoguin]

On topic:

Anthony Ramine gave me a tip on how to build only modified erl files
using make's functionality directly, I'll try it out tomorrow and report
back.

Off topic:

On 01/25/2013 03:05 AM, Björn-Egil Dahlberg wrote:
> P.S. Keeping mouth shut since rebar's automatic parallellization doesn't
> seem to interest everyone else in the thread.... :-)
>
>
> rebar is awesome!
>
> I wonder why some people seem to shy away from it?

My reasons in short:

* Slow (especially for running tests)
* Annoying (it does recursion by default; and might do recursion in
some cases but not in others later, which is as bad)
* Inconvenient (it does funny things to its ct stuff which makes it
much slower and less practical than ct_run, and I run tests all the time)
* Unstable (options change at regular intervals)

I actually removed it initially to improve my development life (where I
do a lot of dialyzing and running tests), then decided to stop using it
entirely. I use Makefiles anyway, and rebar's functionality is
replicated in just a dozen more lines compared to the original Makefile.

I still support rebar for people who use my projects, I think it's the
best current solution to get started using Erlang quickly, I just don't
use it for my own development needs.

As for releases, relcool[1] sounds damn sexy, I have been advising them
a little on how to handle release upgrades, and will be using it
starting next week.

[1] https://github.com/erlware/relcool

--
Loïc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu

[Björn-Egil Dahlberg]

2013/1/25 ノートン ジョーセフ ウェイ ン <nor...@lovely.email.ne.jp>

Bjorn -

Why do you think it isn't feasible to use rebar to build erlang applications within OTP?  

Well, rebar would be a key-component to build otp. That means that rebar probably have to be included in the repo as an application.

Not impossible, but I don't know if it would be ideal.

What I would like, and I don't know if anybody shares this thought:

OTP split down to just a core, namely runtime system, kernel, stdlib, compiler, sasl perhaps a few other applications.

External dependency to rebar. Perhaps rebar could be included in that core. From a technical standpoint, there is ofc no problem. But it would be nice to solve it in another manner.

I don't really know if this is true or not, but I get the feeling when I use rebar that I loose some strictness and it adds some magic. A lack of sense of control. Just the paranoia speaking perhaps. =)

Build the rest of OTP with rebar, pulled down via git or some other means. 

Applications are nowadays normally oriented in separate git repositories. Seems natural to use this scheme for OTP applications as well. 

It has some additional advantages:

* Promotes modularity. 

* Can have separate release cycles.

* Developers can choose different versions for there system, not merely the system version R16*, R15*, R14*, etc ..

I can see many advantages.

I'm pretty sure there are disadvantages too. Cross application updates that otherwise needs strict version-control can be done with one commit in a single repo. Thats pretty neat.

but I digress, to answer your question. I don't think there is a technical problem, but perhaps an administration problem. I would like to see rebar thrive outside OTP. 

We won't go into this any further right now though.

// Björn-Egil

[Björn-Egil Dahlberg]

2013/1/25 Björn-Egil Dahlberg <wallentin...@gmail.com>

2013/1/25 ノートン ジョーセフ ウェイ ン <nor...@lovely.email.ne.jp>

Bjorn -

Why do you think it isn't feasible to use rebar to build erlang applications within OTP?  

Let me just preface everything below by saying that: I speak for myself and not for OTP as a whole when I say this. Some may agree with some might not. Anyways, these are my views.

[Scott Lystig Fritchie]

Loïc Hoguin <es...@ninenines.eu> wrote:

lh> I actually removed it initially to improve my development life
lh> (where I do a lot of dialyzing and running tests), then decided to
lh> stop using it entirely. I use Makefiles anyway, and rebar's
lh> functionality is replicated in just a dozen more lines compared to
lh> the original Makefile.

That's understandable, Loïc, not faulting you for avoiding rebar.

On the matter of reducing startup time, I was quite serious about that.
Both "erlc" and "escript" users would benefit greatly from reduced
startup time.

-Scott

[Florian Schintke]

Hi,

we use the Erlang module 'make' and an Emakefile to specify what and
how to compile if necessary.
See http://erldocs.com/R14B04/tools/make.html

Our Unix makefile entry looks as follows:

---
compile:
@$(ERL) -pa ebin -noinput +B -eval 'case make:all() of up_to_date -> halt(0); error -> halt(1) end.'
---

It starts Erlang once and compiles all touched/depending modules,
which is really fast. Parallel make is missing, but not necessary at
that speed.

See scalaris.googlecode.com for a sample Emakefile.

Florian

[Matthias Lang]

[Daniel Goertzen]

As a variation of bad idea #2, use rebar instead of erlc to update all the beams.  Rebar seems pretty fast to me, so you could even take the brute force approach and unconditionally run rebar on every build:

.PHONY: mybeams

mybeams:

      rebar compile

Dan.

[Loïc Hoguin]

On 01/25/2013 03:28 AM, Loïc Hoguin wrote:
> Anthony Ramine gave me a tip on how to build only modified erl files
> using make's functionality directly, I'll try it out tomorrow and report
> back.

I have finally played with this. It is very nice and works perfectly.

The idea is to recompile only source files newer than the .app file.
This can be done by making source files depend on the .app file, and
using $? to obtain the list of changed files.

This can be seen here:


https://github.com/extend/cowboy/blob/747bfc29ee951841f38d6e8d471041d6a3008d60/Makefile

If no files have changed:

essen@alice (0) % time make
make[1]: Entering directory `/home/essen/extend/ranch'
make[1]: Leaving directory `/home/essen/extend/ranch'
make 0.04s user 0.01s system 77% cpu 0.060 total

If one file has changed:

essen@alice (0) % time make
erlc -v -Werror +debug_info +warn_export_all -o ebin/ -pa ebin/ \
src/cowboy_middleware.erl src/cowboy.erl
make[1]: Entering directory `/home/essen/extend/ranch'
make[1]: Leaving directory `/home/essen/extend/ranch'
make 0.20s user 0.03s system 91% cpu 0.249 total

As you can see I always compile cowboy_middleware, because I need it
compiled before anything else, as it's a behavior.

If you want to rebuild everything:

essen@alice (0) % time make clean app
make[1]: Entering directory `/home/essen/extend/ranch'
rm -rf ebin/
rm -f test/*.beam
rm -f erl_crash.dump
make[1]: Leaving directory `/home/essen/extend/ranch'
rm -rf ebin/
rm -f test/*.beam
rm -f erl_crash.dump
erlc -v -Werror +debug_info +warn_export_all -o ebin/ -pa ebin/ \
src/cowboy_middleware.erl src/cowboy_multipart.erl
src/cowboy_http_handler.erl src/cowboy_handler.erl src/cowboy_router.erl
src/cowboy_http.erl src/cowboy_loop_handler.erl src/cowboy_websocket.erl
src/cowboy_sup.erl src/cowboy_rest.erl src/cowboy_app.erl
src/cowboy_bstr.erl src/cowboy_protocol.erl src/cowboy_req.erl
src/cowboy_middleware.erl src/cowboy.erl src/cowboy_client.erl
src/cowboy_websocket_handler.erl src/cowboy_clock.erl src/cowboy_static.erl
make[1]: Entering directory `/home/essen/extend/ranch'
erlc -v -Werror +debug_info +warn_export_all -o ebin/ -pa ebin/ \
src/ranch_transport.erl src/ranch_transport.erl src/ranch_acceptor.erl
src/ranch_sup.erl src/ranch_listener.erl src/ranch_server.erl
src/ranch_acceptors_sup.erl src/ranch_conns_sup.erl
src/ranch_listener_sup.erl src/ranch_ssl.erl src/ranch_app.erl
src/ranch_protocol.erl src/ranch.erl src/ranch_tcp.erl
make[1]: Leaving directory `/home/essen/extend/ranch'
make clean app 1.39s user 0.10s system 98% cpu 1.512 total

That's pretty fast in my book.

In comparison, rebar:

essen@alice (0) % time rebar clean compile
==> ranch (clean)
==> cowboy (clean)
==> ranch (compile)
Compiled src/ranch_protocol.erl
Compiled src/ranch_transport.erl
Compiled src/ranch_acceptor.erl
Compiled src/ranch_listener.erl
Compiled src/ranch.erl
Compiled src/ranch_tcp.erl
Compiled src/ranch_listener_sup.erl
Compiled src/ranch_conns_sup.erl
Compiled src/ranch_ssl.erl
Compiled src/ranch_acceptors_sup.erl
Compiled src/ranch_server.erl
Compiled src/ranch_sup.erl
Compiled src/ranch_app.erl
==> cowboy (compile)
Compiled src/cowboy_http_handler.erl
Compiled src/cowboy_loop_handler.erl
Compiled src/cowboy_websocket_handler.erl
Compiled src/cowboy_middleware.erl
Compiled src/cowboy.erl
Compiled src/cowboy_bstr.erl
Compiled src/cowboy_router.erl
Compiled src/cowboy_static.erl
Compiled src/cowboy_handler.erl
Compiled src/cowboy_rest.erl
Compiled src/cowboy_protocol.erl
Compiled src/cowboy_sup.erl
Compiled src/cowboy_multipart.erl
Compiled src/cowboy_req.erl
Compiled src/cowboy_clock.erl
Compiled src/cowboy_websocket.erl
Compiled src/cowboy_app.erl
Compiled src/cowboy_client.erl
Compiled src/cowboy_http.erl
rebar clean compile 2.38s user 0.17s system 179% cpu 1.419 total

All this on a state of the art 13" laptop with a good SSD, i7 and
everything, if you wonder.

I probably won't bother trying to make -j to work. Feel free to send a
patch if you want to have fun with it, though.

[Loïc Hoguin]

On 01/25/2013 03:49 AM, Björn-Egil Dahlberg wrote:
>
>
> 2013/1/25 ノートン ジョーセフ ウェイ ン <nor...@lovely.email.ne.jp

> <mailto:nor...@lovely.email.ne.jp>>

>
>
> Bjorn -
>
> Why do you think it isn't feasible to use rebar to build erlang
> applications within OTP?
>
>
>
> Well, rebar would be a key-component to build otp. That means that rebar
> probably have to be included in the repo as an application.
> Not impossible, but I don't know if it would be ideal.

It'd need a lot of work. I love rebar for the simple things, but it's a
pain if you need to do hard things. I'm also not sure you'd gain much
doing this, the only bad thing about compiling OTP right now is that
there's too many applications.

> What I would like, and I don't know if anybody shares this thought:
>
> OTP split down to just a core, namely runtime system, kernel, stdlib,
> compiler, sasl perhaps a few other applications.

Definitely yes.

> External dependency to rebar. Perhaps rebar could be included in that
> core. From a technical standpoint, there is ofc no problem. But it would
> be nice to solve it in another manner.
>
> I don't really know if this is true or not, but I get the feeling when I
> use rebar that I loose some strictness and it adds some magic. A lack of
> sense of control. Just the paranoia speaking perhaps. =)

It adds a lot of magic, makes guesses as to what you want, and gets it
right most of the time. Not as magic with more complex applications.

> Applications are nowadays normally oriented in separate git
> repositories. Seems natural to use this scheme for OTP applications as
> well.
> It has some additional advantages:
> * Promotes modularity.
> * Can have separate release cycles.
> * Developers can choose different versions for there system, not merely
> the system version R16*, R15*, R14*, etc ..
>
> I can see many advantages.

I agree with these points. Core applications should definitely keep the
Rxx version system, but some would perhaps benefit more from being
outside (thinking of common_test particularly). I'm also not sure what
diameter and others are doing there.

> I'm pretty sure there are disadvantages too. Cross application updates
> that otherwise needs strict version-control can be done with one commit
> in a single repo. Thats pretty neat.

The good thing though is that if nobody cares about one such application
you can quickly notice it and drop it or lower the amount of support you
give to it.

> We won't go into this any further right now though.

One step at a time.

--
Loïc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu

[Serge Aleynikov]

Loïc,

There's a little trick you can use in a make file to avoid spawning
external shell to comma delimit files:

empty :=
space := $(empty) $(empty)
comma := $(empty), $(empty)

FILES := $(wildcard src/*.erl)
MODULES := $(subst $(space),$(comma),$(sort $(FILES:src/%.erl=%)))

example:
@echo "UsingShell = $(shell ls src/*.erl | sed
's/src\///;s/\.erl/,/' | sed '$$s/.$$//')"
@echo "UsingMake = $(MODULES)"

And also for the app target you can just do:

ebin/$(PROJECT).app: src/$(PROJECT).app.src
sed 's/{modules, \[\]}/{modules, \[$(MODULES)\]}/' $< > $@

app: deps/ranch ebin/$(PROJECT).app
@$(MAKE) -C $(DEPS_DIR)/ranch


My $2c.

Regards,

Serge

[Loïc Hoguin]

On 01/29/2013 03:32 AM, Serge Aleynikov wrote:
> Loïc,
>
> There's a little trick you can use in a make file to avoid spawning
> external shell to comma delimit files:
>
> empty :=
> space := $(empty) $(empty)
> comma := $(empty), $(empty)
>
> FILES := $(wildcard src/*.erl)
> MODULES := $(subst $(space),$(comma),$(sort $(FILES:src/%.erl=%)))
>
> example:
> @echo "UsingShell = $(shell ls src/*.erl | sed
> 's/src\///;s/\.erl/,/' | sed '$$s/.$$//')"
> @echo "UsingMake = $(MODULES)"

Your "little" trick is a lot more verbose. :) I don't think the external
call slows anything down. Good to know it's possible though.

> And also for the app target you can just do:
>
> ebin/$(PROJECT).app: src/$(PROJECT).app.src
> sed 's/{modules, \[\]}/{modules, \[$(MODULES)\]}/' $< > $@
>
> app: deps/ranch ebin/$(PROJECT).app
> @$(MAKE) -C $(DEPS_DIR)/ranch

I'm using the ebin/$(PROJECT).app rule to compile only the modified .erl
files, and the ebin/$(PROJECT).app needs to be compiled everytime for
that to work properly, not just when .app.src is modified.

Thanks for the interesting tips!

--
Loïc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu