[erlang-questions] suggestion: shorthand functions

[Ulf Wiger]

Since I've been writing a bunch of rebar.config.script code lately,
I've suffered the agony of trying to write concise and readable
code without having to do tons of copy-paste, weird unwrapping
funs etc.

What I think would make this sort of thing easier, and also
escript programming in general, is if OTP could provide some
modules with concise naming and let-it-fail semantics.

Just off the top of my head, I scribbled down a few functions that
I think would make *my* life easier. I pushed them to github to
get some discussion going.

http://github.com/uwiger/shorthand

The modules are:

f.erl - shorthand functions for file.erl
fn.erl - ditto for filename.erl
e.erl - ditto for erl_eval.erl

The least beneficial is perhaps filename:erl, but my fingers and
eyes ache from all the filename:join(filename:dirname(F), …)
code.

Otherwise, I think the biggest benefit is to stick to let-it-crash
programming, which I find is usually the default when I write
scripts. The original functions are always available if you want
to take a closer look at return values.

(For the file:script() counterparts, I also always pass the name
of the script as a binding).

Comments?

BR,
Ulf W

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
http://feuerlabs.com



_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Thomas O'Dowd]

Ulf,

I'm probably missing something but what's wrong with importing functions
if you are using them so much? Example.

-import(filename, [join/2]).

Tom.

--
Cloudian KK - http://cloudian.jp/
S3 REST API Compliant Cloud Storage with Cloudian®

[Thomas Lindgren]

Another approach to reduce verbosity might be to implement a "DSL" of sorts, a la

%% obviously incomplete, but you get the idea
expand({d, D}) -> filename:dirname(D);
expand(Ds) when list(Ds) -> filename:join([ expand(D) || D <- Ds ]);
expand(F) -> F.

My main gripe with the "expand/1" approach above would be that the resulting caller might look a bit messy. But some experimentation could get the interface right.

(Compiling this into direct calls with a parse transform is left as an exercise.)

Best,
Thomas

[Ulf Wiger]

On 11 Jan 2013, at 10:31, Thomas O'Dowd wrote:

> Ulf,
>
> I'm probably missing something but what's wrong with importing functions
> if you are using them so much? Example.
>
> -import(filename, [join/2]).

This can be done in escripts, but not in file:script() files,
and it only addresses the number of keystrokes.

I realize that historically, few people have written such scripts,
if they even knew the function existed. But now that rebar
supports the notion of rebar.config.script for dynamic configuration,
I have seen them popping up here and there.

But, as I said, the different semantics - return a value or raise
an exception - allow you to eliminate a great deal of code that
checks and unwraps values only to pass them on. That, or you
live with the risk of getting ugly badmatch exceptions.

In order to be a real improvement, these wrapper functions would
need to present more informative exceptions too.

[Ulf Wiger]

Yes, but a bit tricky with escripts and downright impractical
with file:script() files.

A problem with escripts is that, while you can create libraries
with useful helper functions, you then need to pack it all into
an archive, as e.g. rebar does. Of course, when done this way,
ordinary Erlang programming works reasonably well, but it is
too heavy for simple scripts.

For the simple scripts, you tend to rely on the things that exist
in OTP, for better or for worse.

BR,
Ulf W

[Jesper Louis Andersen]

On Jan 11, 2013, at 10:12 AM, Ulf Wiger <u...@feuerlabs.com> wrote:

> Comments?

In the Standard ML module system, in particular with the .MLB file extension you get module-level import renaming:

local
$(OTP_LIB)/R15B % Grab *that* version of the Erlang OTP library
in
module file = f
module filename = fn
module lists = l
end
% From here on out in our build specification, the modules f, fn and l are specified.
your_module.erl

Note that this also solves module clashes:

local
file1.erl
in
module file = f1
end
local
file2.erl
in
module file = f2
end
…

Do note, however, that Standard ML did *not* make the--unfortunately ubiquitous--mistake of conflating a filename with a module name. So it may be hard to pull off in Erlang-land :/

Jesper Louis Andersen
Erlang Solutions Ltd., Copenhagen

[Ulf Wiger]

Yes, this seems a bit harder to emulate in Erlang than my
modest addition. :)

Note also that my primary aim is erlang for scripting, when
it's not always practical to assume the presence of 3rd
party libraries that can be imported.

Also, again, I think that many of the functions most used in
scripts have the wrong semantics, using the {ok,Value} |
{error, Reason} approach.

BR,
Ulf W

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
http://feuerlabs.com

[Raimo Niskanen]

I think it is a nice idea that would improve scripting.

But how to agree on module names and content is harder. There is a limited
number of 1 and 2 character module names, and once in OTP they are written
in stone.

For f.erl I miss e.g is_dir from filelib, which would introduce the notion
of merging old module functionality. Using the name 'fl' for filelib
functions would just be hard to remember.

Aliasing filename:basename to fn:base is a bit unintuitive since the
original Unix command is called 'basename' and for e.g file:list_dir
you have aliased it to f:ls (as for many other) to make them more Unix:ish.
I think it would be better to keep to unix command names where possible.
[Wild idea: f:'-d' for filelib:is_dir, or t:'-d', or f:test(d, Path).]

An alternative approach might be to have a helper module named 'es'
containing all scripting aliases...

>
> BR,
> Ulf W
>
> Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
> http://feuerlabs.com

--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB

[Ulf Wiger]

On 11 Jan 2013, at 15:18, Raimo Niskanen wrote:

> I think it is a nice idea that would improve scripting.
>
> But how to agree on module names and content is harder. There is a limited
> number of 1 and 2 character module names, and once in OTP they are written
> in stone.

This is true.

I don't think keeping the module names within 2 characters is a
top priority. Actually, 'file' is just fine, but the functions in file
consistently return {ok,V} | {error, R}, so mixing it up in there
has its disadvantages.

OTOH, once you string together a few calls to
filename:basename/1 and filename:dirname/1 inside a call
to filename:join/2, 80 character line width doesn't seem
like much. :)

> For f.erl I miss e.g is_dir from filelib, which would introduce the notion
> of merging old module functionality. Using the name 'fl' for filelib
> functions would just be hard to remember.

I also thought about adding is_dir/1, is_file/1, is_symlink/1, but didn't
partly because I had other things to do, and partly because I started
thinking about how the semantics of the original functions is the right
one anyway…

When I added the module fn, it was mainly because e.g. adding
join/2 to f.erl might seem unintuitive, as we're not joining files, but
filenames. OTOH, the functions in filelib could be mapped to f.erl,
I think.

> Aliasing filename:basename to fn:base is a bit unintuitive since the
> original Unix command is called 'basename' and for e.g file:list_dir
> you have aliased it to f:ls (as for many other) to make them more Unix:ish.
> I think it would be better to keep to unix command names where possible.
> [Wild idea: f:'-d' for filelib:is_dir, or t:'-d', or f:test(d, Path).]

Agree.

> An alternative approach might be to have a helper module named 'es'

> containing all scripting aliases…

That's an idea, but on the other hand, the functions may well
catch on outside of scripting too, among those who prefer the
let-it-fail semantics.

BR,
Ulf W

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
http://feuerlabs.com

[Garrett Smith]

Aye. I don't know how critical this use case is (scripting
simplification/improvements) to justify new modules in OTP space. And
I'm not sure there *is* a case that would be sufficiently ubiquitous.

I've resigned myself to writing little project specific
functions/libraries like this to deal with project specific problems.
E.g. this stuff drives me nuts:

{ok, X} = foo:x(),
{ok, Y} = foo:y(),
combine(X, Y)
...

So I'll write foo_x/0 and foo_y/0 with crash-on-error semantics to get this:

combine(foo_x(), foo_y())

But who knew that I needed those particular variants? And what about
changes? It's my project, so I don't mind little wrappers, especially
since functions clarify intent.

E.g. when I see this:

ProjectFile = filename:join(ProjectDir, ProjectName)

I'll almost create a function like this:

project_filename(Dir, Name) -> filename:join(Dir, Name).

> For f.erl I miss e.g is_dir from filelib, which would introduce the notion
> of merging old module functionality. Using the name 'fl' for filelib
> functions would just be hard to remember.
>
> Aliasing filename:basename to fn:base is a bit unintuitive since the
> original Unix command is called 'basename' and for e.g file:list_dir
> you have aliased it to f:ls (as for many other) to make them more Unix:ish.
> I think it would be better to keep to unix command names where possible.
> [Wild idea: f:'-d' for filelib:is_dir, or t:'-d', or f:test(d, Path).]

It seems one could make these arguments ad nauseam. To me it's just
easier (and preferable frankly) to roll my own as the need arises.

> An alternative approach might be to have a helper module named 'es'
> containing all scripting aliases...

I love escripts but find myself intuitively avoiding them for a couple reasons:

- In most cases, bash is more portable and solves system level
problems more directly

- I don't have the escript foo required to wrench my program it into a
single script file

Setting bash aside (not Ulf's use case) the main barrier for me is in
the perceived complexity of getting a release-of-sorts into an
escript. If I could write up one of these wrapper libraries, or pull
it down from somewhere easily, I might use escript everywhere. It's
not hard to write the wrappers (you write them as you need them, it
takes literally less than a minute for each function) and they're
tailored to your requirements.

It may be trivial to package up the required bytes into an escript
today. If it is, I've love to know!

And I don't know how this fits into rebar specific scripting :)

Garrett

[Mahesh Paolini-Subramanya]

I love escripts but find myself intuitively avoiding them for a couple reasons:

- In most cases, bash is more portable and solves system level
problems more directly

- I don't have the escript foo required to wrench my program it into a
single script file

Amen.

perl (bash, python, whatever) is optimally suited for orchestrating between 'application space' and 'business space', as well as orchestration across loosely coupled systems and activities.

Yes, you could force most of this into erlang-world, but to what point?  IMHO, we do live in a polyglot world, and we may as well take advantage of it…

Cheers

p.s. Mind you, there is an entirely different argument to be made about systems where you only have access to erlang (or Ada. or whatever :-)  )

Mahesh Paolini-Subramanya

That Tall Bald Indian Guy...

Google+  | Blog   | Twitter  | LinkedIn

[Ulf Wiger]

On 11 Jan 2013, at 15:55, Garrett Smith wrote:

> Setting bash aside (not Ulf's use case) the main barrier for me is in
> the perceived complexity of getting a release-of-sorts into an
> escript. If I could write up one of these wrapper libraries, or pull
> it down from somewhere easily, I might use escript everywhere. It's
> not hard to write the wrappers (you write them as you need them, it
> takes literally less than a minute for each function) and they're
> tailored to your requirements.

Yes, I agree that this is a barrier. This is also why I started the
discussion of getting a scripting-friendly library into OTP.

The big question is of course how big the gain is, and if it's worth
the redundancy. I thought I'd try to stimulate some discussion
before I invest too much effort into the idea.

While I agree that the wrapper functions are easy to write, they're
still a nuisance, and you have to live with

> It may be trivial to package up the required bytes into an escript
> today. If it is, I've love to know!

Well, what I do in 'setup' is simply to let rebar handle it:

{escript_name, setup_gen}.

in your rebar.config (but with a name of your choosing).

Then call `rebar escriptize`

In 'setup', I've added

{post_hooks, [{compile, "rebar skip_deps=true escriptize"}]}.

since I want the escript to be created even if setup is
included as a dependency and the top app just calls
`rebar compile`

> And I don't know how this fits into rebar specific scripting :)

I've liked the file:script() function ever since I wrote it many
years ago, but failed to recruit others to my religion until I
was able to sneak it into rebar. ;-)

I will say that the biggest problem, if you start chaining scripts
together (which I do sometimes in rebar, and also when using
'setup'), is that when you get e.g. a badmatch exception in a
sub-script, it gets to you via erl_eval, which is not nearly as
nice as an exception from compiled code. Often, you end up
scratching your head trying to figure out which sub-script it was
that actually failed.

Fixable, I'm sure, but I guess it hasn't been a priority.

BR,
Ulf W

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
http://feuerlabs.com

[Yurii Rashkovskii]

Mahesh,

This is certainly a good point about bash/perl/python/whatever, but these become quite useless the moment you need to talk to some erlang node from your escript.

Yurii.

[Mahesh Paolini-Subramanya]

Ah, you misunderstand.

I'm not running around saying Perl Now! Perl Forever!

Instead, what we do is separate the problem domains (to the extent possible, of course).  

There is some set of functionality in the erlang application that needs to be exposed to our business processes - think "add a user", "disable a client", "add billing credits", etc.

These are encapsulated and exposed via escript, and then embedded in perl via 'system' / 'qx' / backquotes / whatever.

The same applies for activities like "rebalance node", "move user to debug node", etc. :-)

cheers

Mahesh Paolini-Subramanya

That Tall Bald Indian Guy...

Google+  | Blog   | Twitter  | LinkedIn

[Garrett Smith]

Hi Mahesh,

Sounds like Yuri's driver is distributed Erlang.

There are a bunch of apps out there that implement their command line
interfaces (CLI) as distributed Erlang nodes that spin up, interact
with another local node, then shut down. RabbitMQ does this.

I *hate* it! (anger issues)

- It's slow -- slower than *Java* for CLI operations

- It's brittle -- distributed Erlang can fail for a number of reasons
that are hard to diagnose

- It introduces a third OS process -- epmd, a little
single-point-of-failure devil that is the devil

I personally advocate run_erl, which exposes a shell to to_erl over
pipes. It's trivial to send commands to an Erlang process from bash or
Python or anything you like.

Some have criticized this approach as also being brittle. All I can
say is that I've used this for years across thousands of servers and
I'm very happy with it.

That said, I often spend my Sunday mornings over coffee wondering how
OS signal support isn't supported in Erlang.

Garrett

[Mahesh Paolini-Subramanya]

Garrett:

This gets perilously close to a "how and where one should be doing distributed erlang in the first place" discussion, which is an entire other can of worms :-) 

I quite agree re: spinning up nodes - adding yet another moving part to what is already perilously close to being a hairball makes my eye twitch… (mind you, I suspect this might be different in simpler scenarios, but I'd rather not go down that road in the first place…)

Cheers

p.s.  The last time I mentioned OS signal support on a Sunday morning, Nicole almost brained me w/ her coffee mug...

Mahesh Paolini-Subramanya

That Tall Bald Indian Guy...

Google+  | Blog   | Twitter  | LinkedIn

[Loïc Hoguin]

Can you give an example of such a run_erl use? I have no idea how the
tool works.

--
Loïc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu

[Bengt Kleberg]

Greetings,

Please start here:
http://www.erlang.org/doc/man/run_erl.html

bengt

[Loïc Hoguin]

No examples there.

Also confusing, Solaris?

Redirect Erlang input and output streams on Solaris®

But mostly because the page you linked has nothing related to how
Garrett uses it.

--
Loïc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu

[Garrett Smith]

Hi Loïc,

I use run_erl/to_erl as a proxy for OS signals. You can also use
to_erl as a tty terminal via pipes, but that's not the case I'm
talking about.

Here's a simple example, requesting an orderly shutdown of the VM:

1. Start the VM as a daemon using run_erl

$ mkdir -p /tmp/myapp
$ run_erl -daemon /tmp/myapp/ tmp/myapp/ erl [your erl start args here]

2. Send a shell command to the VM by way of to_erl and run_erl

$ echo "init:stop()." | to_erl /tmp/myapp/

You can tail the logs on /tmp/myapp/ to see what's going on. run_erl
is basically an Erlang shell that exposes read and write access over
pipes (FIFOs).

$ to_erl /tmp/myapp/

If you want to get values from the process, you can write to the log
(similar to Java's SIGQUIT behavior) or to a file. E.g.

$ echo 'myapp:write_status("/tmp/myapp/status").' | to_erl /tmp/myapp/

If the app requires tighter interactivity with a CLI this approach
starts to break down. Then you're back to distributed Erlang or
roll-your-own interface (REST, simple RPC server, etc.)

My *main* beef with distributed Erlang is epmd. It starts lazily and
when killed (e.g. someone uninstalls the program and user that
*happened* to start it) each of your local nodes goes dark.

I recall seeing some activity around improving epmd somehow (don't
remember) but IMO the problem is that it's a shared resource that
isn't explicitly managed (started by the first Erlang node to need
it).

It'd be nice if you could have some of the distributed Erlang
goodness, but with explicit ports and addressing - no NATing.

Also, IMO Erlang should fail to start in distributed mode if epmd is
*not* already running (with optional override). This would force epmd
to be explicitly managed as an external dependency, which is what it
in fact is.

Garrett