[erlang-questions] "Automatic" recompile app from a console

[John Kemp]

Hi,

I have been writing an application using rebar to build the app. I run
the server and open a console. I then change the source code and
apparently I need to leave the console, 'make rel' and restart the app
and the console.

I'm sure there's a better way to do what I'm doing.

One project I found is https://github.com/rustyio/sync, but this did not
(yet) work for me according to the instructions.

How can I have my rebar (or just Erlang) app automatically recompile
('make rel') when one of the source files changes, or run a command from
within the console to do this?

Cheers,

JohnK
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Garrett Smith]

On Thu, Jan 31, 2013 at 3:08 PM, John Kemp <jo...@jkemp.net> wrote:
> Hi,
>
> I have been writing an application using rebar to build the app. I run the
> server and open a console. I then change the source code and apparently I
> need to leave the console, 'make rel' and restart the app and the console.
>
> I'm sure there's a better way to do what I'm doing.
>
> One project I found is https://github.com/rustyio/sync, but this did not
> (yet) work for me according to the instructions.
>
> How can I have my rebar (or just Erlang) app automatically recompile ('make
> rel') when one of the source files changes, or run a command from within the
> console to do this?

You're asking about a completely different workflow, but on the
outside chance you're simply looking for a streamlined way to
integrate the shell with your programming, this is what I use:

- Edit and compile source explicitly (I have a key binding to run
"make" in Emacs, but I'm sure most editors have an easy equivalent)

- Let mochiweb's reloader, which is started using the "-s reloader"
option, automatically reload the modified beams

I *live* in the Erlang shell during development, always using this
workflow. When I watch others constantly closing and restarting the
Erlang shell in between changes, I wonder how on earth they get
anything done :)

Garrett

[John Kemp]

On 01/31/2013 04:25 PM, Garrett Smith wrote:

> You're asking about a completely different workflow, but on the
> outside chance you're simply looking for a streamlined way to
> integrate the shell with your programming,

I am :)

> this is what I use:
>
> - Edit and compile source explicitly (I have a key binding to run
> "make" in Emacs, but I'm sure most editors have an easy equivalent)

This is the part that was failing for me, actually - I get an error
about reloading a beam file when I ran the make from within emacs. Works
OK from a command line, but not from emacs.

>
> - Let mochiweb's reloader, which is started using the "-s reloader"
> option, automatically reload the modified beams
>
> I *live* in the Erlang shell during development, always using this
> workflow. When I watch others constantly closing and restarting the
> Erlang shell in between changes, I wonder how on earth they get
> anything done :)

I programmed in Erlang a few years ago and gave it up. I don't remember
how I worked before but I'm pretty sure I wouldn't have lived with this
either :) So now I need to get it working again...

Cheers,

John

[Garrett Smith]

On Thu, Jan 31, 2013 at 3:32 PM, John Kemp <jo...@jkemp.net> wrote:
> On 01/31/2013 04:25 PM, Garrett Smith wrote:
>
>> You're asking about a completely different workflow, but on the
>> outside chance you're simply looking for a streamlined way to
>> integrate the shell with your programming,
>
>
> I am :)
>
>
>> this is what I use:
>>
>> - Edit and compile source explicitly (I have a key binding to run
>> "make" in Emacs, but I'm sure most editors have an easy equivalent)
>
>
> This is the part that was failing for me, actually - I get an error about
> reloading a beam file when I ran the make from within emacs. Works OK from a
> command line, but not from emacs.

This is what I have in my emacs.d/init.el:

(global-set-key (kbd "C-x RET") 'compile)

So it's CTRL-C M and I get the compile prompt, which defaults to "make
-k", then enter. Of course I have a Makefile that contains ./rebar
compile in the default target.

Emacs knows how to go to the source line on error with CTRL `

I personally would not want my compiler to auto-compile when I saved
the file, though I realize some IDEs not only do that, but try to
compile as you type! Everyone has a different thing though.

>>
>> - Let mochiweb's reloader, which is started using the "-s reloader"
>> option, automatically reload the modified beams
>>
>> I *live* in the Erlang shell during development, always using this
>> workflow. When I watch others constantly closing and restarting the
>> Erlang shell in between changes, I wonder how on earth they get
>> anything done :)
>
>
> I programmed in Erlang a few years ago and gave it up. I don't remember how
> I worked before but I'm pretty sure I wouldn't have lived with this either
> :) So now I need to get it working again...

Good luck!

[jixiuf]

>> This is the part that was failing for me, actually - I get an error about
>> reloading a beam file when I ran the make from within emacs. Works OK from a
>> command line, but not from emacs.
>
> This is what I have in my emacs.d/init.el:
>
> (global-set-key (kbd "C-x RET") 'compile)
>
> So it's CTRL-C M and I get the compile prompt, which defaults to "make
> -k", then enter. Of course I have a Makefile that contains ./rebar
> compile in the default target.
>
> Emacs knows how to go to the source line on error with CTRL `
>
> I personally would not want my compiler to auto-compile when I saved
> the file, though I realize some IDEs not only do that, but try to
> compile as you type! Everyone has a different thing though.
>

I write erlang-dired-mode.el for emacs

https://raw.github.com/jixiuf/erlang-dired-mode/master/erlang-dired-mode.el

there is a function called `erlang-auto-compile' maybe can help you if you
use emacs.

[Slava Yurin]

Hi John.

 

Maybe  https://github.com/tex/ssync can help you.

 

01.02.2013, 04:09, "John Kemp" <jo...@jkemp.net>:

[Max Lapshin]

Frankly speaking, it is one of the reasons, why I don't use releases at all: it is a horrible mess with file locations.

I always configure Emakefile and run in erlang console  make:all([load]).

It is much faster and more predictable than all your reloaders and syncs.

[Samuel]

make:all([load]) was my usual way of working until not so long ago.
The main problem is that projects usually come without Emakefiles, but
I had this in my erlang default, which worked in most of the cases:

make() -> make([]).

make(Options) ->
make:all(
[load, debug_info | Options]
++ [{i, "../include"}, {outdir, "../ebin"}]).


Lately I have been using EDTS (https://github.com/tjarvstrand/edts)
which is truly awesome, but it breaks my usual workflow as it compiles
every time I save, so make gently refuses to compile anything, thus
not load the new code. But edts automatically loads all changes in the
node shell (emacs wrapper around the erlang shell) which is in many
ways better than the default erlang shell, so I now use that and rely
on edts reloading all my changes automatically. It is working well so
far, and is slightly faster than my previous edit-save-make:all
workflow.


Best

--
Samuel

[Damian Dobroczyński]

A long time ago I used the attached bash script which utilizes
"inotifywait" tool (I think it's Linux only feature) to wait for some
files changed and then it does a command you've given. I used it for TDD
;);) Theoretically you may use "inotifywait" to compile if some sources
changed and then to reload the modules if *.beam files changed (due to
compilation) using some rpc:call to specified node.

-- D.

W dniu 31.01.2013 22:08, John Kemp pisze:

[John Kemp]

This does not seem to work on a Mac FWIW since there is no equivalent of the inotify tools on OS X, for similar reasons to Damian's suggestion. Otherwise (on Linux), it looks promising...

Cheers,

John

[Garrett Smith]

For dev purposes, I bet that a few millisecond delay in change
detection + recompile is okay. inotify facilities is a bit much IMO
for this application.

The Mochiweb reloader polls for changes to the beam files themselves
and there is a delay, but it's not a problem, unless you're not
*really* human.

[Serge Aleynikov]

Another alternative to autoloading of modified beam files the moment
modification is made is to include the following user_default module [1]
in the ERL_LIBS path, which will add a handful of useful commands to the
shell including l() that will reload all modified beams.

Serge

[1] https://github.com/saleyn/util/blob/master/src/user_default.erl

[Scott Lystig Fritchie]

Serge Aleynikov <se...@aleynikov.org> wrote:

sa> [1] https://github.com/saleyn/util/blob/master/src/user_default.erl

Howdy, Serge. I've been using the l() function from that
user_default.erl module for years. It's tremendously handy. There are
times where I want to reload several modules after recompiling but load
them sometime at my choosing rather than immediately after the compiler
is done. l() does that.

-Scott

[Dave Cottlehuber]

On 1 February 2013 16:40, John Kemp <jo...@jkemp.net> wrote:
> This does not seem to work on a Mac FWIW since there is no equivalent of the inotify tools on OS X, for similar reasons to Damian's suggestion. Otherwise (on Linux), it looks promising...

I've used rustyio/sync successfully on Windows and Mac, with notifu
and growl respectively. More info in the README.md at
https://github.com/rustyio/sync

The key things I found:

- run sync *before* you start editing your erlang files
- if the beam doesn't change, nothing will get reloaded

A+
Dave

[Jeremy Ong]

+1 for rustyio/sync.

Haven't had any problems. @John it looks like you are running a released version compiled with rebar? Sync or anything else won't work with that approach. Load the beam files directly with erl -pa path/to/ebin/dirs deps/*/ebin

[Serge Aleynikov]

Hi Scott,

Nice to know I'm not the only one finding it useful! ;-)

Other than for auto-compilation/loading of erlydtl templates, I also
find that having control over the moment of code loading is more
advantageous, as the function shows what modules were loaded, so you
know what version of code is running.

Concerning the user_default, I believe I added a couple more useful
functions there (i.e tc/2, tc/4) after I published this module a while
back. Enjoy! ;-)

Serge

[Diego Llarrull]

Hello everyone,

I'm working on a riak_core based project, that is, one which uses rebar
+ GNU make and where the src files are located in different folders
than the beam files, which are inside inside zipped (.ez) files.

sync + libnotify will do the trick for unzipped files, but some extra
trickery was needed to load the new .beam files

a) compile the modified sources
b) (create and) run a script that generates the .zip files containing
the .beams and deploys those files in their destination directory
c) run the following code:

reload() ->
Modules = [ X || {X,L} <- code:all_loaded(),
is_list(L),string:str(L, ?PATH_TO_CODE_DIR) =/= 0],
lists:map(fun(X) -> c:l(X) end, Modules).

Hope this helps anyone, right now I'm trying out ssync to see if it is
able to handle zipped files.

Best regards,

Diego

[Judson Lester]

On a related note, has anyone written Common Test hooks to compile/reload the CUT modules before running tests?  When I get back to Erlang, it's the top of my list, but I really wouldn't complain if it's been done already.

Judson

[Jesse Gumm]

I too run Rusty's sync.

There are a few situations where sync breaks down a bit:

1) Includes - sync does not detect changes to include files, and so if you were to change an include with records, for example, you could end up with modules that have the old version of the record loaded, and other modules with the new version of the record loaded.

2) parse_transform - From what I understand, sync has some issue with parse_transforms.

This is what, as far as I understand, inspired the development of tex/ssync - which works similarly to sync, but watches all files (even rebar.config, from what it seems) and relies on rebar to recompile everything, then reloads the beams. I haven't played with ssync at all, but I've been meaning to, it seems a worthwhile effort to offload the compiling to rebar and just watch for file changes to trigger the recompile.

-Jesse

--
Jesse Gumm
Owner, Sigma Star Systems
414.940.4866 || sigma-star.com || @jessegumm

[Richard Carlsson]

On 2013-02-01 16:48, Garrett Smith wrote:
> For dev purposes, I bet that a few millisecond delay in change
> detection + recompile is okay. inotify facilities is a bit much IMO
> for this application.
>
> The Mochiweb reloader polls for changes to the beam files themselves
> and there is a delay, but it's not a problem, unless you're not
> *really* human.

There's also this: https://github.com/richcarl/file_monitor

I also had a couple of rather experimental modules called code_monitor
and autoload - I used to keep them under my EUnit development repo, but
I recently removed them and made a new repo for file_monitor. The other
files can be dug out from the history
(https://github.com/richcarl/eunit/commits/master) if you want to play
with them.

/Richard

[Steve Davis]

+1

[Dmitry Klionsky]

We use modified version of sync https://github.com/ten0s/sync that
supports releases, detects new files, and monitors changes in include
files (at least it should, don't use this feature often) as far as the
next requirements are met:
1. Make sync somehow available to your release. Include it into your
release or put it in the erlang libs.
2. Build the project with debug_info.
3. Remove {debug_info, strip} from reltool.config
4. Add {excl_archive_filters, [".*"]} to reltool.config
5. Start sync:start() before making any changes.
6. Disable any growl notifications (optional)

We also edit files in emacs with flymake mode enabled using custom
erlang syntax checker for *.erl, *.hrl, *.config files
https://github.com/ten0s/syntaxerl

In general our workflow is:
1. Build the project with rebar
$ make
2. Start it in the console mode
$ make console
3. Start sync
> sync:start().
4. Now start edit files. Whenever flymake says that the file is OK
pretty soon sync will reload it. This workflow works fine until you need
to make big changes like adding a child to a supervisor or make a new
application. In this case go to point 1 and start over.

Hope this helps.

--
Best regards,
Dmitry Klionsky

[Sergej Jurecko]

We actually were not aware of sync and developed our own similar solution.
It simply works by monitoring ~/ebin, ~/etc, ~/src,~/include,~/apps/../ebin,~/apps/.../src,~/apps/.../dtl,~/deps/../ebin
We have a lot of django template files and we put them in dtl folders. They get compiled with erlydtl.
If a .hrl file gets changed, all source files get compiled for that project as well. For instance hrlname.hrl, will cause all sources that start with hrlname_ to get compiled.
Same with base django template files. if file name is projectname_base.dtl, every .dtl file that starts with projectname_ will get compiled.

We use one large repository for all projects and because there are quite a few apps at the moment, we have simple C/C++ programs that use platform specific APIs for file notifications (FSEvents for osx, inotify for linux). They are pretty trivial one file sources that get compiled automatically on erlang startup if they are not compiled yet. Then they are run with open_port and communicate via stdout.

If anyone wants to make use of those C/C++ file notification programs we can gladly open source them.


Sergej