[erlang-questions] Why do we need modules at all?
Joe Armstrong
This is a brain-dump-stream-of-consciousness-thing. I've been
thinking about this for a while.
I'm proposing a slightly different way of programming here
The basic idea is
- do away with modules
- all functions have unique distinct names
- all functions have (lots of) meta data
- all functions go into a global (searchable) Key-value database
- we need letrec
- contribution to open source can be as simple as
contributing a single function
- there are no "open source projects" - only "the open source
Key-Value database of all functions"
- Content is peer reviewed
These are discussed in no particular order below:
Why does Erlang have modules?
There's a good an bad side to modules:
Good: Provides a unit of compilation, a unit of code
distribution. unit of code replacement
Bad: It's very difficult to decide which module to put an individual
function in. Break encapsulation (see later)
Aside: lib_misc.erl
When I'm programming I often get to the point were I say there should
a function foo/2 in lists.erl but their isn't. There should be but
there isn't - foo/2 is a small self contained thing. Why should it be
in lists.erl because it "feels right".
Strings are lists, so why do we have two modules lists.erl and
string.erl how should I decide in which module my new string/list
processing function should go.
To avoid all mental anguish when I need a small function that
should be somewhere else and isn't I stick it in
a module elib1_misc.erl.
My elib1_misc exports the following:
added_files/2 make_challenge/0
as_bits/1 make_response/2
as_bits_test/0 make_response_test/0
bdump/2 make_test_strings/1
bin2hex/1 make_test_strings_test/0
bin2hex_test/0 make_tmp_filename/2
check_io_list/1 merge_kv/1
collect_atom/1 merge_kv_test/0
collect_atom_test/0 mini_shell/0
collect_int/1 module_info/0
collect_int_test/0 module_info/1
collect_string/1 ndots/1
collect_string_test/0 nibble_to_hex_char/1
collect_word/1 nibble_to_hex_char_test/0
complete/2 odd/1
complete_test/0 on_exit/2
dos2unix/1 out_of_date/2
downcase_char/1 outfile/2
dump/2 padd/2
dump_tmp/2 perms/1
duplicates/1 perms_test/0
ensure_started/2 pmap/2
eval_file/1 pmap1/2
eval_file_test/0 pmap1_test/0
eval_string/1 pmap_test/0
eval_string_test/0 priority_receive/0
every/3 random_seed/0
expand_env_vars/1 random_string/1
expand_file_template/3 random_string/2
expand_string_template/2 read_at_most_n_lines/2
expand_tabs/1 read_at_most_n_lines_test/0
expand_tabs_test/0 remove_duplicates/1
expand_template/2 remove_duplicates_test/0
extract_attribute/2 remove_leading_and_trailing_whitespace/1
extract_attribute_test/0 remove_leading_and_trailing_whitespace_test/0
extract_prefix/2 remove_leading_whitespace/1
fetch/2 remove_prefix/2
fetch_test/0 remove_prefix_test/0
file2lines/1 remove_trailing_whitespace/1
file2lines_test/0 replace/3
file2md5/1 root_dir/0
file2numberedlines/1 rpc/2
file2numberedlines_test/0 safe/1
file2paras/1 show_loaded/1
file2stream/1 signed_byte_to_hex_string/1
file2string/1 signed_byte_to_hex_string_test/0
file2template/1 skip_blanks/1
file2term/1 skip_blanks_test/0
file_size_and_type/1 skip_to_nl/1
find_src/1 skip_to_nl_test/0
first/1 sleep/1
flatten_io_list/1 spawn_monitor/3
flush_buffer/0 split_at_char/2
for/3 split_at_char_test/0
force/1 split_list/2
foreach_chunk_in_file/3 split_list_test/0
foreach_word_in_file/2 string2exprs/1
foreach_word_in_string/2 string2exprs_test/0
forever/0 string2html/1
get_erl_section/2 string2latex/1
get_line/1 string2lines/1
get_line/2 string2lines_test/0
have_common_prefix/1 string2stream/1
have_common_prefix_test/0 string2stream_test/0
hex2bin/1 string2template/1
hex2bin_test/0 string2template_test/0
hex2list/1 string2term/1
hex2list_test/0 string2term_test/0
hex_nibble2int/1 string2toks/1
hex_nibble2int_test/0 string2toks_test/0
id/1 sub_binary/3
include_dir/0 template2file/3
include_file/1 term2file/2
interleave/2 term2string/1
is_alphanum/1 test/0
is_blank_line/1 test1_test/0
is_prefix/2 test_function_over_substrings/2
is_prefix_test/0 tex2pdf/1
is_response_correct/3 time_fun/2
keep_alive/2 time_stamp/0
lines2para/1 to_lower/1
list2frequency_distribution/1 to_lower_test/0
list2frequency_distribution_tetrim/1
longest_common_prefix/1 trim_test/0
longest_common_prefix_test/0 unconsult/2
lookup/2 unsigned_byte_to_hex_string/1
lorem/1 unsigned_byte_to_hex_string_test/0
ls/1 which/1
which_added/1
Now I find this very convenient when I write a new small utility function
I stick in in elib1_misc.erl - no mental anguish in choosing a module
name is involved.
The observation that I find this very-convenient is telling me something
about modules - I like my elib1_misc it feels right.
(aside - It seems many development projects have their own private
lib_miscs ...)
Which brings me to the point of my question.
Do we need module's at all? Erlang programs are composed of lots of small
functions, the only place where modules seem useful is to hide a letrec.
The classic example is fibonacci. We want to expose fib/1 but hide the
helper function fib/3. Using modules we say
-module(math).
-export([fib/1]).
fib(N) ->
fib(N, 1, 0).
fib(N, A, B) when N < 2 -> A;
fib(N, A, B) -> fib(N-1, A+B, A).
The downside is we have had to *invent* one module name math - whose *only*
purpose is to hide the definition of fib/3 which we don't want to be made
callable.
If we put a second function into the module math, then this second function
could call fib/3 which breaks the encapsulation of fib/3.
We could say:
let fib = fun(N) -> fib(N, 1, 0) end
in
fib(N, A, B) when N < 2 -> A;
fib(N, A, B) -> fib(N-1, A+B, A).
end.
I hardly dare suggest a syntax for this since I've been following
another thread in this forum where syntax discussion seem to encourage
much comment.
** Please do suggest alternative syntax's here - but do not comment on
other peoples suggestions ...
I would like to just talk about why we have modules.
Another question:
Does the idea of a module come from the idea that functions have to be
stored somewhere, so we store them in a file, and we slurp the
file (as a unit) into the system, so the file becomes a module?
If all the files were store by themselves in a database would this
change things.
I am thinking more and more that if would be nice to have *all* functions in
a key_value database with unique names.
lookup(foo,2) would get the definition foo foo/2 from a database.
The unique names bit is interesting - is this a good idea. Qualified
names (ie names like xxx:foo/2) or (a.b.c.foo/2) sounds like a good
idea but but when I'm programming I have to invent the xxx or the
a.b.c which is very difficult. It also involves the "decision problem"
if the namespaces xxx and a.b.c already exist I have to *choose* which
to put my new function in.
I think there might be a case for alises here joe:foo/2 could be used
while developing "joe" would expand to a horrible random local string the
real name being ab123aZwerasch123123_foo/2 but I would not be able to
publish my code or make it available to a third_part before I had
chosen a sensible name.
(( managing namespaces seems really tricky, a lot of peoople seem
to thing that the problem goes away by adding "." 's to the name
but managing a namespace with namees like foo.bar.baz.z is just as complex
as managing a namespace with names like foo_bar_baz_z or names like
0x3af312a78a3f1ae123 - the problem is that we have to go from a symbolic
name like www.a.b to a reference like 123.45.23.12 - but how do we discover
the initial name www.a.b? - there are two answers - a) we are given the name
(ie we click on a link) - we do not know the name but we search fo it ))
When programs are small we can live with "just the code" in "a few
modules" the ratio of code to meta data is high.
When programs are large we need a lot of meta-data to understand them.
I would like to see all functions with all meta-data in a data base.
I'd like to say:
lookup(foo,2,Attribute) when Attribute =
code|source|documentation|type signatures|revision history|authors|...
The more I think about it the more I think program development should
viewed as changing the state of a Key-Value database.
So I imagine:
1) all functions have unique names
2) there are no modules
3) we discover the name of a function by searching metadata
describing the function in a database
4) all public functions (think open source) are in the same
database
We could make a system to do this.
I think this would make open-source projects easier, since the
granularity of contribution goes down. You could contribute
a single function - not an entire application.
(( A problem with GUT style open source projects is there is
not one database of functions, I often what one function from
this project, another function from another project -- the
granularity of reusable parts should be the individual function.
functions are really easy to reuse
modules are more difficult to reuse
entire applications are very difficult to reuse
(Unless there are isolated through a communication channel))
Possible extensions.
1) Voting for promotion
2) A review process
Given a raw database will *all* functions in it - we could derive an
"approved" functions database.
Popular functions could be moved to the approved database - the
review process would need to be discussed - so kind of peer-review/wiki
stuff.
Comments?
Volunteers?
/Joe
Ralf P. Gans
the ability to replace code in a running environment depends, to my knowledge, on modules.
So modules prevent downtimes and allow 7/24 operation.
cheers, Ralf
> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Amy Lear
Hi,
the ability to replace code in a running environment depends, to my knowledge, on modules.
So modules prevent downtimes and allow 7/24 operation.
cheers, Ralf
... Provided there's a way to, with atomicity, ensure that nothing calls the new version of a function without all of the related changes in associated functions rolling into the VM at the same time, since you may be moving quite a bit around if you find yourself refactoring a few functions to add new capabilities. And that IS something that is easier with modules as a complete unit of dependency.
Is there a way in a clean semantic manner track cross-function dependencies (potentially through some type of version tags?) in such a code database?
Mazen Harake
foo:search(), bar:search(), baz:search()
give
foo_search(), bar_search(), baz_search()
now you want to add a function that does replace; but replace what? say you call it
replace()
now I want to make one because it is specific to my problem so I have a decision problem
foo_replace() ? bar_replace() ?
so I will end up defining a prefix so that my versions of functions (where they collide with the already existing ones) always are unique thus:
mazen_replace()
now everyone starts creating functions with prefixes, now they want to start extending other functions so they build a prefix before the those:
mazen_foo_replace()
and before you know it you will have 20 functions ending with "search()" where search means what ever the prefix hints at...
So you are back to square -1 because now it is worse, you don't have modules and you don't have applications beacuse an application would become what modules was before it.
In fact, this is already a problem, all my opensource applications (and non-opensource as well) have a prefix on the modules because I can't call all my server type modules "server" (there is no namespace).
My 2 cents.
/M
Dmitrii Dimandt
> So I imagine:
>
> 1) all functions have unique names
> 2) there are no modules
> 3) we discover the name of a function by searching metadata
> describing the function in a database
> 4) all public functions (think open source) are in the same
> database
>
> We could make a system to do this.
Hm... you'd still need *some* kind of modules.
For instance, in Webmachine each resource is expected to have some, or all, of predefined public/exported functions: to_html, resource_exists, allowed_methods etc.
So, each resource will have functions with the same name. If we move all functions to a global namespace, then how do we differentiate between these functions? Probably by naming them resource_name_function_name?
Furthermore, what do we do with behaviours? A behaviour is encapsulated within a module, also with a predefined set of exported funcions (and any number of custom exported functions).
Gordon Guthrie
* they are a higher level of abstraction
* they support good working practices
Lets take a module that I use a little bit - and which a colleague
introduced me to:
http://erldocs.com/R14B02/stdlib/digraph.html?i=2&search=digra#get_digraph/1
I can use the couple of exposed functions of that that I know,
confident that if it comes to it I can come back and master the rest
of the interface later. I have confidence that there is a rubustness
there that makes code using it maintainable. (my idea of maintainable
is to be able to ignore most stuff, most of the time...). I can learn
about the code at this higher level of abstraction.
We too have a module with loads of random functions in it, well we
have three (hn_util, util, util2), and each of them is just library
functions. In the utility modules the ratio of exported to
not-exported fns is maybe 7 to 1.
In functional modules the exported/not ratio goes from 1-1 (in modules
that define apis) to 1-3 against.
If we had the (promised) -export-to() directive the number of pure
exported fns would drop.
Given that we build our testing strategy around -export() this
separation is critical...
Gordon
> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
--
Gordon Guthrie
CEO hypernumbers
http://hypernumbers.com
t: hypernumbers
+44 7776 251669
Josh Johnston
So I imagine:1) all functions have unique names2) there are no modules3) we discover the name of a function by searching metadatadescribing the function in a database4) all public functions (think open source) are in the samedatabaseWe could make a system to do this.
Hm... you'd still need *some* kind of modules.
For instance, in Webmachine each resource is expected to have some, or all, of predefined public/exported functions: to_html, resource_exists, allowed_methods etc.
So, each resource will have functions with the same name. If we move all functions to a global namespace, then how do we differentiate between these functions? Probably by naming them resource_name_function_name?
to_html()resource_exists()etc
Gordon Guthrie
following on from what Dimitri said while I was writing me e-mail, we
hang unit test onto modules and design them at that level of
abstraction
Vlad Dumitrescu
Do we need module's at all? Erlang programs are composed of lots of small
functions, the only place where modules seem useful is to hide a letrec.
The classic example is fibonacci. We want to expose fib/1 but hide the
helper function fib/3. Using modules we say
-module(math).
-export([fib/1]).
fib(N) ->
fib(N, 1, 0).
fib(N, A, B) when N < 2 -> A;
fib(N, A, B) -> fib(N-1, A+B, A).
The downside is we have had to *invent* one module name math - whose *only*
purpose is to hide the definition of fib/3 which we don't want to be made
callable.
If we put a second function into the module math, then this second function
could call fib/3 which breaks the encapsulation of fib/3.
We could say:
let fib = fun(N) -> fib(N, 1, 0) end
in
fib(N, A, B) when N < 2 -> A;
fib(N, A, B) -> fib(N-1, A+B, A).
end.
Kresten Krab Thorup
The gen_server module is used for three different things:
- as a collection of API function (left)
- a place to put the code that controls the gen_server process (middle)
- a place to describe the abstract notion of a gen_server behavior (right).
Ideally, I think those three usages should be separate language constructs, but probably not in separate files (or other kinds of units).
When using a gen_server, it would be nice if you did not have to expose the behavior api to the "user" of your module.
Currently, the module is the unit of code loading. I think we could use a bette mechanism, perhaps one that supports an STM-like thing in the VM to control the loading of a set of modules. I don't have much deep experience with code loading in Erlang (but do have it on Java), and it just seems to me that being able to control atomic loading of .. say a new version of an Erlang application would be good to have supported in the VM.
Gilad Bracha (who's working on Newspeak) has an excellent blog on new notions of modues that you should read. You can start here http://gbracha.blogspot.com/2009/06/ban-on-imports.html
One of the things he's arguing is that modules should be VALUES. A "module" in newspeak is kind of a template (function), which take other module instances as arguments. So you hook up a system by applying such templates (possibly using letrec).
If we had a visual programming environment, then many things don't need a name. Self is an example of this (Self, the video: http://video.google.com/videoplay?docid=5776880551404953752#)
Kresten
Xiaopong Tran
>
>
> In a normal language this would be a problem, but in erlang we can get
> the desired behaviour by using pattern matching, right?
>
> So instead of
>
>
> to_html()
> resource_exists()
> etc
>
>
> We have
>
> resource(to_html, ...)
> resource(exists, ...)
> etc
>
> I'm not disagreeing with you - we do need some good way to do this. But
> I don't think modules are necessary in this case.
>
Pattern matching might work here, but do away with modules and using
pattern matching would a heavier load on the programmer.
Modules or not, it does not have much effect on the VM, we can
just make the VM work without modules. But I think programmers
would have to work harder, distinguishing which function is which.
Although I don't really like the Erlang module, but I think it's
better with it than without.
Xiaopong
Jesper Louis Andersen
>
> One of the things he's arguing is that modules should be VALUES. A "module" in newspeak is kind of a template (function), which take other module instances as arguments. So you hook up a system by applying such templates (possibly using letrec).
>
This is a great idea. Why? Because it is the path of existential types :-)
Notice that in Erlang, a module is a Value. It is possible to call
Mod:Fun(..). Second, notice that the parameterized module extension
allows us to parameterize a module over some other module.
Specifically we could have
-module(RBTree, [OrderMod]).
where OrderMod is a module which has a function
-opaque t() :: .....
-export_type([t/0]).
-spec order(t(), t()) -> lt | eq | gt.
and so we have generalized our RedBlack trees to be able to use any
ordering relation, not simply the one that is built it.
The style Gilad is advocating is the "Fully functorial style" from the
Standard ML community. A Functor[1] in SML terminology is a function
from module to module ("functor" stems from category theory where it
designates a "function"[2] one level up). In the style, you build up
programs by stitching together smaller modularized parts building
larger and larger modules in the process.
Example: The gen_server is a functor from a module (which obeys the
gen_server behaviour and has handle_* functions) to a specific
instance of a server.
It also gives you quite the path towards the concept of OOP, as can be
observed from the parameterized module extension and its uses in
various places.
In languages with static types, you can of course enforce that functor
applications are type-correct. If you allow modules as values in such
languages you get existential types which is a way to get something
OOP-like into those languages[3].
So after a little tour around: Erlang already has this :P
Modules are an essential part of building large systems. The very
notion that you can pack up functions in a module and ship them as a
unit is quite important. Also the notion you can replace a
module-component at once. I feel that the problem of function
placement in the right module is less of a problem compared to the
hell it would be with one large global namespace. And it is much more
than just a "bunch of functions thrown together".
Specifically, modules allows me to have an *abstract* view of a data
structure. In the above, I have no idea what t() is. I am only allowed
to compare two t()'s to each other with OrderMod:compare(T1, T2). That
is, the only algebraic operation I am allowed to do is this. Other
parts of the program may have a different view of t() and be able to
do more with it, but to me, it is an opaque term. It matters because
someone can go replace t(), and my code does not have to be changed at
all. This is impossible if t() leaks. And I am grateful the dialyzer
can find places where it is leaking.
Likewise, a module can hide the fact it is implemented with a
gen_server. So I can go replace it with a gen_fsm, and nobody has to
alter their code because the API i exported was exactly the API I will
keep stable.
Processes componentize the heap.
Modules componentize the code.
[1] Beware of connotating the word functor in programming with
something specific in computer science. The meaning is different in
Standard ML, in Haskell and in C++.
[2] I am lying. The basic notion is a morphism of which the function
is a special case when considering the category of Sets.
[3] I still don't get why you can't use static typing in Erlang. The
conservative approach is simply to use polymorphic variants for all
message passing and it would essentially work right out of the bat.
--
J.
Tim Carpenter
I am sticking to the request to not comment on other solutions but
present my own thoughts...
1. Some units of code only make sense as a collection. who would use
part of gen_server in isolation? Well, you could,but to have it
separate exposes it to modification that could distort the focused
purpose of the unit.
2. some functions have intentionally hidden aspects, private calls etc
and these again are best left hidden or considered as a logical whole
otherwise side effects or bloat may occur.
3. I see no reason why we cannot logically group functions into
modules but not have the code in there...i.e. the module contains the
references to functions that make it up. this allows logical handling
of groups of functions for code management and can allow one function
to appear in multiple modules without code duplication. it also allows
one to add/delete/replace a function in a module knowing that the
source code for the other functions have not been touched!
4. naming conventions should be given great thought for simplicity and
intuitiveness. functions could be in n sets, so a strict hierarchy may
not suit.
5. if functions are named as being in modules, the need to provide
abstraction to the full name may be advantageous, so modulename.sort/3
can be used and the module definition translates this to a unique
longhand which can change over time. The programmer should also be
able to use the longhand at will using the module as a convenient bulk
import/compile vehicle that will break at compile time (one would
hope). Some form of 'expects...' notation could allow a form of
protection against inadvertent swapping of functions in shared modules
altering behaviour.
6. we may want 3 elements to naming of functions: shorthand, longhand,
and version, so one can use the latest version of a longhand named
function, lock to a specific version of a longhand named function or
use the shorthand latest version or other such combinations.
7. n- tier modules - modules of modules? possibly.
Tim
Tim Carpenter
Amphibian Ltd
London
ENGLAND
+44(0)7900 888826
amphib...@gmail.com
Bengt Kleberg
It would be very nice to have module renaming/namespacing that allowed
me to do the following:
The modules "misc_lib" and "misc_lib_helper" from developer A should be
able to call each other with their original names.
The modules "misc_lib" and "misc_lib_helper" from developer B should be
able to call each other with their original names.
I should be able to call all 4 at the same time with names that I have
create/renamed.
So, I need to load modules as a set/application/??? that renames them
outside of that set (it would be ok to have the same prefix added to all
of them), and still allows the files in the set/application/??? to keep
using to the old names.
When it comes to improving stdlibs I suggest that we create new, better
modules. Keep the old ones (for ever?) but print a big "deprecated" all
over the man pages.
bengt
On Tue, 2011-05-24 at 12:26 +0200, Jesper Louis Andersen wrote:
> On Tue, May 24, 2011 at 10:06, Joe Armstrong <erl...@gmail.com> wrote:
> > Why do we need modules at all?
>
> I think we need modules, but this post sparked a thought I have had
> for some time now.
>
> I'd like to have module renaming/namespacing!
>
> Specifically, I want a mechanism by which I can clean up the naming
> mess in the stdlib without affecting old programs. That is, I want to
> be able to, at an application/module level, rename function calls in a
> way such a way that can clean up parameter orders and so on. I don't
> care about how the solution is. I am after the goal: Cleaning up the
> stdlib mess.
>
> You may argue this is a messy thing, but I think it is a necessity if
> you want to move the standard library on:
>
> * We can't alter existing functions because people rely on them.
> * Existing functions are not consistent.
>
> This is a social deadlock I want to break. In fact it is a deadlock I
> think we can only break if we allow the coexistence of more than one
> module with the same name (but stemming from different namespaces). I
> don't want nesting. Just two levels, so I can say
>
> -use_namespace(v2_modules).
>
> in modules which use version two of the stdlib in which things are
> sensibly named. At some point you can then flip around and compile old
> code with a compatibility layer.
Gary Hai
In the view of SOP, erlang process is a FSM, and suitable granularity of functions as states.
The finite set of states of a FSM just like erlang module, but it always cross modules boundaries.
So in SOP model, the module concept has been very vague, just for unique distinct name of function.
The model of a state activity is a function as bellow:
act(Entity, Input) -> {ok, Output} | {ok, Output, UpdatedEntity} | {error, Error} | {stop, Output, Entity}
The output of the state may be {NextState, Output}.
The NextState may be the next state activity function or just a directive of next state.
Then FSM engine matches the directive to a state activity then transfers the previous state to the new state with the previous output as input.
The state activity function may be from anywhere, function in a module, binary loaded from database, code dynamically generated by JIT.
--------
Gary Hai
Jesper Louis Andersen
I think we need modules, but this post sparked a thought I have had
for some time now.
I'd like to have module renaming/namespacing!
Specifically, I want a mechanism by which I can clean up the naming
mess in the stdlib without affecting old programs. That is, I want to
be able to, at an application/module level, rename function calls in a
way such a way that can clean up parameter orders and so on. I don't
care about how the solution is. I am after the goal: Cleaning up the
stdlib mess.
You may argue this is a messy thing, but I think it is a necessity if
you want to move the standard library on:
* We can't alter existing functions because people rely on them.
* Existing functions are not consistent.
This is a social deadlock I want to break. In fact it is a deadlock I
think we can only break if we allow the coexistence of more than one
module with the same name (but stemming from different namespaces). I
don't want nesting. Just two levels, so I can say
-use_namespace(v2_modules).
in modules which use version two of the stdlib in which things are
sensibly named. At some point you can then flip around and compile old
code with a compatibility layer.
--
J.
Tim Watson
plan. Having global (mangled) names is a path to misery. In the
languages that do mess with names (python, C++, etc) it creates a
bloody nightmare. I would steer clear of this. I like the idea of
having a global database of code, but not all of it floating in some
etherial top level namespace.
Most of what's cool about http://www.haskell.org/hoogle/ and
http://hackage.haskell.org/packages/hackage.html is made possible by
the combination of good standardised build tools and hosting
repositories, so I'd be more included to focus on improving (and
extending) things like rebar and agner/erlware to provide the same
kind of developer-experience without changing the language so much.
Introducing letrec is a bloody marvelous suggestion. Also making it
easier to do partial application and function composition would
simplify a lot of the re-use cases without messing with the module
system.
I'm not going to stray off too far into alternative-solution land, but
given the OTP team reads this list, I don't want to miss the
opportunity to mention some minor things about the module system that
could do with an overhaul.
- Fully supported parameterised modules
Not because I want to use them as a state bag, but rather that they
make it easy to implement something akin to a functor without having
to reside in a process. This means that all the tools need to support
this concept and it is documented, etc. Extending the module system so
that you can -extend a module to create a new concrete implementation
would be useful:
%% finder.erl
-module(finder).
-functor([M]).
-compile(export_all).
find() -> walk(fun M:match/3).
%% rx_finder.erl
-module(rx_finder).
-extends(finder, [re]).
Perhaps the syntax/approach there isn't quite right, but the point I'm
getting at is that what with parameterised modules and -extends, you
should be able to compose stuff easily.
- Better support for *import*
As the quantity of code you're dealing with increases, you need to be
able to import discrete units of it in simpler and more intuitive
ways. I'm not exactly a fan of Java, but you can see in that world
that the quantity of libraries (often open source ones) being used on
any project means you simply cannot avoid having to import things
(packages, classes, static methods/functions) without going mad.
Although Erlang is still (thankfully) simple enough not to have this
problem, I suspect as the number of projects blossoms it will
eventually approach the critical mass.
Also being able to support something like -import_alias(name, mod)
would be quite handy, although I realise this is fairly trivial with a
parse_transform.
- Fully supporting packages
Personally I think the <app>_<subsystem>_<thing> convention that most
of us follow is fine, but it does get rather tedious. Package,
namespaces or nested modules are all solution that are known to work
and are widely adopted by other languages. The package system in
erlang works fine, but not all the tools properly support it today
(cover, reltool and a few others are broken).
Tim Watson
> Greetings,
>
> It would be very nice to have module renaming/namespacing that allowed
> me to do the following:
> The modules "misc_lib" and "misc_lib_helper" from developer A should be
> able to call each other with their original names.
> The modules "misc_lib" and "misc_lib_helper" from developer B should be
> able to call each other with their original names.
> I should be able to call all 4 at the same time with names that I have
> create/renamed.
>
> So, I need to load modules as a set/application/??? that renames them
> outside of that set (it would be ok to have the same prefix added to all
> of them), and still allows the files in the set/application/??? to keep
> using to the old names.
>
>
I was thinking about this in terms of some kind of special -import or
-alias directive. I think it's possible to do just with a
parse_transform, as long as the set can be fully determined at compile
time. That last point though - therein lies the rub. App-upgrades
change things, and those things can include modules. Dragons.
Nuff-said.
Masklinn
> In the languages that do mess with names (python […]
As far as I know, Python never "messes with names" unless asked explicitly via the "__" prefix, which is rarely used and generally only there to avoid naming conflicts in inheritance cases.
Do you have instances, besides this quite explicit one, where Python performs any name mangling?
Joe Armstrong
If we think of modules as containers of functions, then a module is a "set of functions" -
my problem is with granularity. The smallest unit of reuse is the module. Suppose my application
want just one function from a particular module - I'm forced to load the entire module.
This hampers reuse - I *often* read library code and find a single function and cut-and-paste it into
my new code. It also discourages sharing - since the smallest unit I can share is a module.
The unit of distribution is an application - a set of modules - why this could not be a list of functions
I do not know.
I think I'd like to view an application as something that offered a message based service that internally
was constructed from a set of functions rather than a set of modules.
/Joe
Joe Armstrong
On Tue, May 24, 2011 at 10:06, Joe Armstrong <erl...@gmail.com> wrote:
> Why do we need modules at all?I think we need modules, but this post sparked a thought I have had
for some time now.
I'd like to have module renaming/namespacing!
I agree - I still like the good 'ol Knuthian way - start with source file, then add patch files.
You could make a language could make a language construct for this.
new_foo = alias foo/2
Joe Armstrong
On 24 May 2011 11:39, Bengt Kleberg <bengt....@ericsson.com> wrote:I was thinking about this in terms of some kind of special -import or
> Greetings,
>
> It would be very nice to have module renaming/namespacing that allowed
> me to do the following:
> The modules "misc_lib" and "misc_lib_helper" from developer A should be
> able to call each other with their original names.
> The modules "misc_lib" and "misc_lib_helper" from developer B should be
> able to call each other with their original names.
> I should be able to call all 4 at the same time with names that I have
> create/renamed.
>
> So, I need to load modules as a set/application/??? that renames them
> outside of that set (it would be ok to have the same prefix added to all
> of them), and still allows the files in the set/application/??? to keep
> using to the old names.
>
>
-alias directive. I think it's possible to do just with a
parse_transform, as long as the set can be fully determined at compile
time. That last point though - therein lies the rub. App-upgrades
change things, and those things can include modules. Dragons.
Nuff-said.
It depends upon what -import means.
-import(foo, [bar/2 ]).
Where are two interpretations
a) "when you *run* this program and find a call to foo:bar/2 go and see if you can find the code
for foo, load it then call the function bar/2"
b ) "When you *compile* this program find the code for foo:bar/2 and
extract the code for bar/2 and include it in my code" - in this case you resolve the name at compile time.
I guess when developing a) is useful but when you deploy code it should be b)
/Joe
Tim Watson
> I stick in in elib1_misc.erl - no mental anguish in choosing a module
> name is involved.
But it's not very helpful to someone else who's going to have to
mentally parse your lib_misc library to figure out what they're
looking for. It does sound a bit like a dumping ground.
>
> The observation that I find this very-convenient is telling me something
> about modules - I like my elib1_misc it feels right.
>
> (aside - It seems many development projects have their own private
> lib_miscs ...)
Perhaps if those libraries were broken into their constituent parts
and then those into their constituent projects, they would be easier
to re-use. I see the same thing on many projects - a
<project-name>_config.erl that wraps a gen_server with maybe some
calls to application:get_env and a <project_name>_log.erl and so on. I
nearly wrote a erl-config package on github until I realised gproc
would be a better choice for my needs.
>
> Which brings me to the point of my question.
>
> Do we need module's at all? Erlang programs are composed of lots of small
> functions, the only place where modules seem useful is to hide a letrec.
>
I think that's an oversimplification. Compilation units (e.g., putting
code in different files) is beneficial for humans first and foremost.
I think that categorising functions based on subject-area and/or the
process they relate too is very useful. Once you do that categorising,
putting all the categories in once place is very sensible. OCaml and
Haskell programmers (of which I'm only experiences with the former),
use letrec (and similar concepts such as Haskell's `where`) heavily,
but would not dream of doing away with our module systems just because
we can encapsulate things in this way.
Modules are a useful categorisation for the human mind to remember
things by. Like packages, they help us remember where to look for
things. They're also useful for hiding - breaking encapsulation is a
choice, not something you cannot help when designing a module.
> The downside is we have had to *invent* one module name math - whose *only*
> purpose is to hide the definition of fib/3 which we don't want to be made
> callable.
>
Actually if you put all your mathematical functions in this module, it
would be more useful (and easier to remember).
> If we put a second function into the module math, then this second function
> could call fib/3 which breaks the encapsulation of fib/3.
Not sure that's such a terrible thing really.
>
> We could say:
>
> let fib = fun(N) -> fib(N, 1, 0) end
> in
> fib(N, A, B) when N < 2 -> A;
> fib(N, A, B) -> fib(N-1, A+B, A).
> end.
>
Having letrec (or where) is an awesome idea. Don't actually care much
about the choice of syntax - just another one to learn.
> Does the idea of a module come from the idea that functions have to be
> stored somewhere, so we store them in a file, and we slurp the
> file (as a unit) into the system, so the file becomes a module?
Probably, yes. It is convenient and familiar to do so.
>
> If all the files were store by themselves in a database would this
> change things.
>
Not really, as long as the database is always available and I *never
ever* run into a situation where I can't work because I'm not
connected.
> I am thinking more and more that if would be nice to have *all* functions in
> a key_value database with unique names.
I love the idea of a massive code repo/server, but having everything
in one namespace sounds like a nightmare.
> The unique names bit is interesting - is this a good idea. Qualified
> names (ie names like xxx:foo/2) or (a.b.c.foo/2) sounds like a good
> idea but but when I'm programming I have to invent the xxx or the
> a.b.c which is very difficult. It also involves the "decision problem"
> if the namespaces xxx and a.b.c already exist I have to *choose* which
> to put my new function in.
Still I think it's the right thing to do.
>
> I think there might be a case for alises here joe:foo/2 could be used
> while developing "joe" would expand to a horrible random local string the
> real name being ab123aZwerasch123123_foo/2 but I would not be able to
> publish my code or make it available to a third_part before I had
> chosen a sensible name.
Yuk. ;)
>
> (( managing namespaces seems really tricky, a lot of peoople seem
> to thing that the problem goes away by adding "." 's to the name
> but managing a namespace with namees like foo.bar.baz.z is just as complex
> as managing a namespace with names like foo_bar_baz_z or names like
> 0x3af312a78a3f1ae123 - the problem is that we have to go from a symbolic
> name like www.a.b to a reference like 123.45.23.12 - but how do we discover
> the initial name www.a.b? - there are two answers - a) we are given the name
> (ie we click on a link) - we do not know the name but we search fo it ))
You're right it is hard, but it is possible.
>
>
> When programs are small we can live with "just the code" in "a few
> modules" the ratio of code to meta data is high.
>
> When programs are large we need a lot of meta-data to understand them.
>
> I would like to see all functions with all meta-data in a data base.
>
> I'd like to say:
>
> lookup(foo,2,Attribute) when Attribute =
>
> code|source|documentation|type signatures|revision history|authors|...
>
I like the idea of finding a function based on a wider range of
attributes such as these, but not choosing the route of dispatch based
on a loose predicate. Lookup/3 would have to guarantee that it will
resolve to a single callee every time, which I can't see being
possible.
Tim Watson
>
> a) "when you *run* this program and find a call to foo:bar/2 go and see if
> you can find the code
> for foo, load it then call the function bar/2"
>
> b ) "When you *compile* this program find the code for foo:bar/2 and
> extract the code for bar/2 and include it in my code" - in this case you
> resolve the name at compile time.
>
> I guess when developing a) is useful but when you deploy code it should be
> b)
>
I completely agree. I'm also quite willing to loose the convenience of
(a) providing the compiler is fast enough.
Max Lapshin
I'd like to know if there will be hierarchial modules in Erlang,
because tree of packages is a rather good idea:
erlyvideo.mpegts.encoder
erlyvideo.rtp.encoder
But plain module namespace is also ok. It would be impossible for me
to work with 30K LOC with plain function namespace.
Tim Watson
> to work with 30K LOC with plain function namespace.
I think what Joe is getting at - he'll correct me if I'm wrong about
this - is to refer to functions not only by name, but by other
metadata as well. I'm not sure what this looks like empirically, as
like you I can't see how using only function names you could work with
a flat namespace. Looking up functions by other (additional) metadata
such as type signature, author, date, company, category (mutiples?),
pre/post conditions, safe guarantees, etc - that would work. In fact
if you think about a process engine (such as BPMS solutions and the
like) then process (i.e., function) is often selected dynamically
based on many attributes, not just name.
Just how this will work in practise as a replacement for existing
(shall we say normal) way of referring to a function that is in a
module, I could not say. But it is an interesting concept. Perhaps
using a URI as the name and thereby being able to have multiple URIs
representing the various different category-based paths to identify
the function - the "aliases" could then redirect to the direct,
uniquely named resource which would in turn return the function
metadata (and binary code presumably).
Joe Armstrong
Very strange topic for me.
I'd like to know if there will be hierarchial modules in Erlang,
because tree of packages is a rather good idea:
No it's not - this has been the subject of long and heated discussion and is
why packages are NOT in Erlang - many people - myself included - dislike
the idea of hierarchical namespaces. The *dot* in the name has no semantics
it's just a separator. The name could equally well be encoders.mpg.erlyvideo
or mpg.applications.erlvideo.encoder - there is no logical way to organise the
package name and it does not scale -
erlyvideo.mpegts.encoder
erlyvideo.rtp.encoder
But plain module namespace is also ok. It would be impossible for me
to work with 30K LOC with plain function namespace.
The English language has a flat namespace.
I'd like a drink.alcoholic.beer with my food.unhealthy.hamburger and my food.unhealthy.national.french.fries
I have no problem with flat beer and chips.
/Joe
Joe Armstrong
> But plain module namespace is also ok. It would be impossible for meI think what Joe is getting at - he'll correct me if I'm wrong about
> to work with 30K LOC with plain function namespace.
this - is to refer to functions not only by name, but by other
metadata as well. I'm not sure what this looks like empirically, as
like you I can't see how using only function names you could work with
a flat namespace. Looking up functions by other (additional) metadata
such as type signature, author, date, company, category (mutiples?),
pre/post conditions, safe guarantees, etc - that would work. In fact
if you think about a process engine (such as BPMS solutions and the
like) then process (i.e., function) is often selected dynamically
based on many attributes, not just name.
Yes - how this would work I don't know - my Haskell friends say that searching for
function by type signatures is good.
What do I do today when I suspect that somebody has written some code ?
1) will it take me < ten minutes to write the code?
yes - write it
2) google or ask a friend
3) this gets me to git or sowhere
4) search
5) cut-and-paste
Not a good method - and this is *best practise*
Just how this will work in practise as a replacement for existing
(shall we say normal) way of referring to a function that is in a
module, I could not say. But it is an interesting concept. Perhaps
using a URI as the name and thereby being able to have multiple URIs
representing the various different category-based paths to identify
the function - the "aliases" could then redirect to the direct,
uniquely named resource which would in turn return the function
metadata (and binary code presumably).
Something like this.
Actually for development it would be great to say
-import("http://a.b.c/modname", [foo/2]).
this would just make an rcp to foo/2 on the remote host and save all the pain of locally installing
the necessary code on my machine.
/Joe
Bengt Kleberg
Perhaps it is possible to do parse_transform when loading, but I have
only heard about this at compile time. I meant a way to load modules,
not compile them.
bengt
Frédéric Trottier-Hébert
However, any change to one of the solutions needs to somehow be reflected into the others, even if it's still by convention.
IMO the basic hierarchy of Erlang seems to begin with the OTP release. The name of an OTP application will decide the prefixes used in module names, process names, table names, shared record names, etc. There is nothing to check that application names are unique except the programmer looking out for that, while xref and the compiler can help a bit with the rest.
I think we can generalise OTP to have this given kind of organisation:
release -> OTP applications -> [Modules, Tables, Shared Records, Processes]
An application itself wouldn't have to worry about namespaces (if we forget included applications). The namespace problem comes at the release stage or if we're developing entirely outside of OTP, still using applications or their equivalent.
To me, this makes it feel like namespace management/giving aliases and replacement of values should be managed from the release point of view, but I can't offer any suggestions as to how this could take place without relying on file renaming and strict use of ?NAMESPACE-like macros throughout the code. Nothing easily backwards compatible, and nothing to help people who are not using releases though.
This in no way would solve Joe's problem with being undecided as to where functions should go, but I think getting rid of modules altogether in Erlang wouldn't be doable because of the other dependencies. In places like Javascript (client-side) or schemes without good module functionality, you pretty much just include the files and people then tend to wrap everything in functions or objects to provide an ad-hoc equivalent to namespaces or modules. I can see this behaviour emerging if Erlang were to go that way.
--
Fred Hébert
http://www.erlang-solutions.com
Attila Rajmund Nohl
[...]
> What do I do today when I suspect that somebody has written some code ?
>
>
> 1) will it take me < ten minutes to write the code?
> yes - write it
> 2) google or ask a friend
> 3) this gets me to git or sowhere
> 4) search
> 5) cut-and-paste
>
> Not a good method - and this is *best practise*
This only works if the function in itself useful. Somehow I don't
think this is the case the most time. If I need not a function, but a
functionality, there's a chance there is an ets table behind or a
process, some configuration file setup - then the whole "import a
single function" doesn't help, but just adds complications that even
the last helper function has to have a globally unique name...
Joe Armstrong
The problem of Erlang with regards to namespaces and modules goes further than just module names. In Erlang, the following units can be sharing/clashing namespaces: the process registry, ETS tables, module names, application names, shared records, etc. They're currently entirely disjoint and only held together by convention.
Absolutely - and inconsistently
We can say
Pid = spawn(fun() -> ... end)
Pid ! M
register(name, Pid),
name ! Pid
But not
Mod = load([ .... lists of funs ...])
Mod:reverse(...)
register(lists, Mod)
lists:reverse(..)
same comment for ets tables etc.
However, any change to one of the solutions needs to somehow be reflected into the others, even if it's still by convention.
IMO the basic hierarchy of Erlang seems to begin with the OTP release. The name of an OTP application will decide the prefixes used in module names, process names, table names, shared record names, etc. There is nothing to check that application names are unique except the programmer looking out for that, while xref and the compiler can help a bit with the rest.
I think we can generalise OTP to have this given kind of organisation:
release -> OTP applications -> [Modules, Tables, Shared Records, Processes]
An application itself wouldn't have to worry about namespaces (if we forget included applications). The namespace problem comes at the release stage or if we're developing entirely outside of OTP, still using applications or their equivalent.
To me, this makes it feel like namespace management/giving aliases and replacement of values should be managed from the release point of view, but I can't offer any suggestions as to how this could take place without relying on file renaming and strict use of ?NAMESPACE-like macros throughout the code. Nothing easily backwards compatible, and nothing to help people who are not using releases though.
This in no way would solve Joe's problem with being undecided as to where functions should go, but I think getting rid of modules altogether in Erlang wouldn't be doable because of the other dependencies. In places like Javascript (client-side) or schemes without good module functionality, you pretty much just include the files and people then tend to wrap everything in functions or objects to provide an ad-hoc equivalent to namespaces or modules. I can see this behaviour emerging if Erlang were to go that way.
I agree - it's not just "getting rid of modules" there's a whole load of other stuff.
Tim Carpenter
Hi all,
I am sticking to the request to not comment on other solutions but
present my own thoughts...
1. Some units of code only make sense as a collection. who would use
part of gen_server in isolation? Well, you could,but to have it
separate exposes it to modification that could distort the focused
purpose of the unit.
2. some functions have intentionally hidden aspects, private calls etc
and these again are best left hidden or considered as a logical whole
otherwise side effects or bloat may occur.
3. I see no reason why we cannot logically group functions into
modules but not have the code in there...i.e. the module contains
references to functions that make it up. this allows logical handling
of groups of functions for code management and can allow one function
to appear in multiple modules without code duplication. it also allows
one to add/delete/replace a function in a module knowing that the
source code for the other functions have not been touched!
4. naming conventions should be given great thought for simplicity and
intuitiveness. functions could be in n sets, so a strict hierarchy may
not suit.
5. if functions are named as being in modules, the need to provide
abstraction to the full name may be advantageous, so modulename.sort/3
can be used and the module definition translates this to a unique
longhand which can change over time. The programmer should also be
able to use the longhand at will using the module as a convenient bulk
import/compile vehicle that will break at compile time (one would
hope). Some form of 'expects...' notation could allow a form of
protection against inadvertent swapping of functions in shared modules
altering behaviour.
6. we may want 3 elements to naming of functions: shorthand, longhand,
and version, so one can use the latest version of a longhand named
function, lock to a specific version of a longhand named function or
use the shorthand latest version or other such combinations.
7. n- tier modules - modules of modules? possibly.
Tim
Tim Carpenter
Amphibian Ltd
London
ENGLAND
+44(0)7900 888826
amphib...@gmail.com
_______________________________________________
Tim Watson
> searching for
> function by type signatures is good.
I've only used OCaml commercially, but what little Haskell I've played
with this is a very useful feature (hoogle and hackage combined with
cabal) for finding and using stuff.
>
> What do I do today when I suspect that somebody has written some code ?
>
>
> 1) will it take me < ten minutes to write the code?
> yes - write it
> 2) google or ask a friend
> 3) this gets me to git or sowhere
> 4) search
> 5) cut-and-paste
>
> Not a good method - and this is *best practise*
Not quite. At point (3) you probably get to github, bitbucket or some
such. At that point, you *may* be able to reuse the library and/or
application code in your project, in which case you install it with
epm/sutro/agner or pull it in as a dependency using rebar. Only when
the code is hidden away in the module and not reusable do you have to
copy-paste - I think it's this (latter) case that you're not happy
with and I agree it's poor show.
By way of example, I'm working on a node/cluster monitoring tool
(primarily a web application with web sockets notifications). I built
the library code (for monitoring) by re-using the eper (performance
monitoring) library and a couple of other things (zero-conf for
network discovery and whatnot). I built the web application on
misultin for its websocket support. I also have some common (to the
project) library code which uses the esl/parse_trans library from
github to support better use of records and things like that. For the
database application, I used the esl/setup library from github to
separate the mnesia database/schema setup from the general purpose use
cases and to generate (run once) setup scripts and the like. None of
this is copy and paste re-use.
> Actually for development it would be great to say
>
> -import("http://a.b.c/modname", [foo/2]).
>
> this would just make an rcp to foo/2 on the remote host and save all the
> pain of locally installing
> the necessary code on my machine.
I think it would be worth distinguishing between two disparate concerns here.
1. Where does the (binary or other representation of AST) code come from
2. Where does the/any (runtime) state reside
I don't always want to RPC just to run a library routine, especially
if that call is happing frequently in a tight "loop" and the code
needs to run fast. Much as I love Erlang's location transparency, one
cannot just assume that the network isn't there.
Things I think are potentially awesome here are:
1. Importing from a URI
2. Having the binary code in a repository that is searchable, versioned, etc
In fact this is hardly an unfamiliar model - consider couchdb and/or
riak. When you define a map-reduce function as say javascript, you
distribute/upload this function definition to the server for later
use. I think my two major concerns (that are by no means
unsurmountable) are:
1. wanting to be able to work when I'm not connected to the code server
2. wanting to be able to define the import based on more than just the name
Point (2) is telling. In your example, you imported from
http://a.b.c/modname - you still haven't gotten rid of the module. I
know it was a flippant example and besides the point. Question is, can
you define the search in such a way as to determine exactly where the
dispatch must go? The Haskell folks say searching by type is immensely
useful, and they're right. But they don't compile based on this - they
find what they're looking for and then reference it (explicitly) by
name.
Tim Watson
> only heard about this at compile time. I meant a way to load modules,
> not compile them.
>
You could hack something together to do it at runtime if you're
importing/loading from an AST rather than a blob. I take your point
though - I did misunderstand slightly.
Max Lapshin
How do you imagine this magic function namespace in terms of plain files?
Tim Watson
> I don't understand what are you speaking about.
>
> How do you imagine this magic function namespace in terms of plain files?
>
You forget about files. You're code editor (IDE) is now a browser
based application. You search for things using a sidebar that
ultimately constructs a query to the code server to look for things
based on name/version/type-signature/author/safety/etc which becomes a
URL. You import things based on URLs. Your code is compiled and hosted
on the server. The local file system doesn't have to come into it per
se.
This last point is partly one of my concerns though. I don't always
have an internet connection and something that doesn't work without
one is potentially going to cause me problems. Perhaps if it was easy
for me to copy the entire repository and run it locally, synchronising
when I'm reconnected - that would do nicely. Then the service would be
providing something akin to source version control as well.
My head now hurts. :)
Joe Armstrong
I don't understand what are you speaking about.
How do you imagine this magic function namespace in terms of plain files?
There are no plain files. The functions that you edit and manipulate are stored in a database.
When you need to call a library function you query the database it helps you find the function
you need from the data in the database - you never know the real name of the function or anything about
the namespace it's in.
The problem with plain files is that they become unmanageable when there are lots of them.
A few years ago I had problem with storage - disks weren't big enough - now I have TBs of
raid storage and the next problem arises - how to find stuff. Finding my own code is difficult -
it's just a matter of searching 43 K erlang modules on my local disk - finding other peoples
code is worse - Google and "ask a friend" is the best solution I know of.
As systems get very large the notion of files and modules seems to break down, to be
replaced by "search-able stuff in a database" - I'm suggesting that the smallest unit that
should be searchable/reusable/discoverable should be the function. And that to make it
searchable we need to add a lot of meta data to the functions.
Barry Skidmore
The description of that architecture may be on target?
On May 24, 2011, at 9:10 AM, Max Lapshin wrote:
> I can't imagine myself working with system, that uses non-plain files
> for storing source code.
> Nobody will fix for me Textmate, git or ack to work with this database.
>
> Modules give your clear structure of functions. Without them you will
> end with mpegts_encode, mpegts_decode,
> and will break your head when there will be 3 or 4 underscores in name.
> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions
Thanks,
Barry
Joe Armstrong
(am sending again as this did not seem to get through)3. I see no reason why we cannot logically group functions into modules but not have the code in there...i.e. the module contains references to functions that make it up. this allows logical handling of groups of functions for code management and can allow one function to appear in multiple modules without code duplication. it also allows one to add/delete/replace a function in a module knowing that the source code for the other functions have not been touched!
Hi all,
I am sticking to the request to not comment on other solutions but present my own thoughts...
1. Some units of code only make sense as a collection. who would use part of gen_server in isolation? Well, you could,but to have it separate exposes it to modification that could distort the focused purpose of the unit.
2. some functions have intentionally hidden aspects, private calls etc and these again are best left hidden or considered as a logical whole otherwise side effects or bloat may occur.
I checked some numbers.
The OTP release has 1793 modules and in these there are 56317 functions with different names.
11887 functions are defined in only one module, ie the for 11887 functions the name alone uniquely
identifies the module. The distribution of functions over modules has a long tail with strange spikes,
ie most function names are in very small number of modules. But some function names like handle_call
etc and in a very large number of modules.
Searching for a function using the function name as a key would only lead to a small number of candidate module if we exclude names like handle_call.
it would be interesting to do this with type signatures ...
4. naming conventions should be given great thought for simplicity and intuitiveness. functions could be in n sets, so a strict hierarchy may not suit.
5. if functions are named as being in modules, the need to provide abstraction to the full name may be advantageous, so modulename.sort/3 can be used and the module definition translates this to a unique longhand which can change over time. The programmer should also be able to use the longhand at will using the module as a convenient bulk import/compile vehicle that will break at compile time (one would hope). Some form of 'expects...' notation could allow a form of protection against inadvertent swapping of functions in shared modules altering behaviour.
6. we may want 3 elements to naming of functions: shorthand, longhand, and version, so one can use the latest version of a longhand named function, lock to a specific version of a longhand named function or use the shorthand latest version or other such combinations.
I've been thinking about this for a while - sometimes we need a very precise name (including an MD/SHA
checksum) othertimes a sloppy name - it depends upon the context.
7. n- tier modules - modules of modules? possibly.
I think these are "views" not names - we have functions and then different ways to view them,
in collections of different kinds
Joe Armstrong
I can't imagine myself working with system, that uses non-plain files
for storing source code.
Nobody will fix for me Textmate, git or ack to work with this database.
Why not? - a plain text file is an abstraction - in reality its a collection of magnetised zones
on a disk, or electron spin states or something, this gets turned into a collection of
bits, the bits are rendered in a editor. Plain text doesn't exist - it's just an abstraction.
The underlying representation of the data you see in a browser may come from a file
or as the result of a datebase query, you will never know which it was.
Modules give your clear structure of functions. Without them you will
end with mpegts_encode, mpegts_decode,
and will break your head when there will be 3 or 4 underscores in name.
or should the name be encode_mpegts and decode_mpegts the problem with structured names
is choosing the name - not the fact there is structure in the name.
Imagine the www was just one big structured name and there were no search engines
how would you ever find anything?
/Joe
Cyryl Płotnicki-Chudyk
>
> Why not? - a plain text file is an abstraction - in reality its a collection
> of magnetised zones
> on a disk, or electron spin states or something, this gets turned into a
> collection of
> bits, the bits are rendered in a editor. Plain text doesn't exist - it's
> just an abstraction.
>
How about storing the code in the old school plain text files but with
additional, also plain text, index/metadata files ?
That would satisfy both the searchability and legacy compatibility
--
cyryl
Anthony Ramine
Le 24 mai 2011 à 14:45, Joe Armstrong a écrit :
> On Tue, May 24, 2011 at 1:56 PM, Max Lapshin <max.l...@gmail.com> wrote:
> Very strange topic for me.
>
> I'd like to know if there will be hierarchial modules in Erlang,
> because tree of packages is a rather good idea:
>
> No it's not - this has been the subject of long and heated discussion and is
> why packages are NOT in Erlang - many people - myself included - dislike
> the idea of hierarchical namespaces. The *dot* in the name has no semantics
> it's just a separator. The name could equally well be encoders.mpg.erlyvideo
> or mpg.applications.erlvideo.encoder - there is no logical way to organise the
> package name and it does not scale -
packages are NOT in Erlang? Then the related code should be removed because
erl.lang.number:plus(1, 1) definitely works.
Also, I think the Haskell guys would disagree about packages not scaling.
--
Anthony Ramine
Dev:Extend
http://dev-extend.eu
Jack Moffitt
> for storing source code.
> Nobody will fix for me Textmate, git or ack to work with this database.
I remember when Zope worked like this. I hated it, and to my
knowledge, none of the editors picked up support for it despite it
being pretty popular at the time. I believe they later changed to
uisng the normal filesystem, but it's been a long time since I paid
any attention to that project.
jack.
Robert Raschke
For some reason this reminds me of a cross between Smalltalk and Eiffel.
Robby
Jesper Louis Andersen
>
> packages are NOT in Erlang? Then the related code should be removed because
> erl.lang.number:plus(1, 1) definitely works.
>
The problem is that a package hierarchy enforces a tree-like
structure. And we may not want the tree. You might want the same
module to reside in multiple packages in the hierarchy at the same
time. We are past 2005 now and we know what a tag cloud is. About time
we began thinking of how that would fit into a programming lang. :-)
The only real advantage of a package tree is that you can:
* Shove some packages underneath a common notifier. This limits the
packages currently in scope.
* Shortcut reference to package if they nest.
I think both would be better served by a system different from hierarchy.
--
J.
Frédéric Trottier-Hébert
Max Lapshin
for storing source code.
Nobody will fix for me Textmate, git or ack to work with this database.
Modules give your clear structure of functions. Without them you will
end with mpegts_encode, mpegts_decode,
and will break your head when there will be 3 or 4 underscores in name.
Jack Moffitt
> structure. And we may not want the tree. You might want the same
> module to reside in multiple packages in the hierarchy at the same
> time. We are past 2005 now and we know what a tag cloud is. About time
> we began thinking of how that would fit into a programming lang. :-)
Wouldn't you just use a DAG? Even with plain tags, you'd need to do
something about cycles.
jack.
Vlad Dumitrescu
One issue is that while plain files are abstractions, they are a very common one, compatible with most systems you will see in use, and on which many tools have been developed, including source control, IDEs, e-mail attachments, documentation and conversation.I do not see the idea of a database as inherently bad -- it could be a good supplement to the developer, much like module_info is for modules. Using a DB for everything (with unique IDs) could solve the problem of nested dependencies, where library A needs library B 1.1.2 while library C needs B 1.2.0 and both can not be loaded at once.But the social problems it creates -- how to communicate about functions, how to regroup collections of them in an exportable way, how to version them, how to patch them, how to reload them, etc. are not worth the change, I believe.
Joe Armstrong
Le 24 mai 2011 à 14:45, Joe Armstrong a écrit :
packages are NOT in Erlang? Then the related code should be removed because
> On Tue, May 24, 2011 at 1:56 PM, Max Lapshin <max.l...@gmail.com> wrote:
> Very strange topic for me.
>
> I'd like to know if there will be hierarchial modules in Erlang,
> because tree of packages is a rather good idea:
>
> No it's not - this has been the subject of long and heated discussion and is
> why packages are NOT in Erlang - many people - myself included - dislike
> the idea of hierarchical namespaces. The *dot* in the name has no semantics
> it's just a separator. The name could equally well be encoders.mpg.erlyvideo
> or mpg.applications.erlvideo.encoder - there is no logical way to organise the
> package name and it does not scale -
erl.lang.number:plus(1, 1) definitely works.
Yes - this is an experiment unapproved feature which could be removed at any time.
This is how we do things - we add experimental and undocumented features
and reserve the right to remove them at any time in the future. Packages are not
officially documented and thus not officially supported.
Vlad Dumitrescu
The OTP release has 1793 modules and in these there are 56317 functions with different names.
11887 functions are defined in only one module, ie the for 11887 functions the name alone uniquely
identifies the module. The distribution of functions over modules has a long tail with strange spikes,
ie most function names are in very small number of modules. But some function names like handle_call
etc and in a very large number of modules.
Searching for a function using the function name as a key would only lead to a small number of candidate module if we exclude names like handle_call.
it would be interesting to do this with type signatures ...
Anthony Ramine
>
>
> On Tue, May 24, 2011 at 4:31 PM, Anthony Ramine <n...@dev-extend.eu> wrote:
>
> Le 24 mai 2011 à 14:45, Joe Armstrong a écrit :
>
> > On Tue, May 24, 2011 at 1:56 PM, Max Lapshin <max.l...@gmail.com> wrote:
> > Very strange topic for me.
> >
> > I'd like to know if there will be hierarchial modules in Erlang,
> > because tree of packages is a rather good idea:
> >
> > No it's not - this has been the subject of long and heated discussion and is
> > why packages are NOT in Erlang - many people - myself included - dislike
> > the idea of hierarchical namespaces. The *dot* in the name has no semantics
> > it's just a separator. The name could equally well be encoders.mpg.erlyvideo
> > or mpg.applications.erlvideo.encoder - there is no logical way to organise the
> > package name and it does not scale -
>
> packages are NOT in Erlang? Then the related code should be removed because
> erl.lang.number:plus(1, 1) definitely works.
>
> Yes - this is an experiment unapproved feature which could be removed at any time.
>
> This is how we do things - we add experimental and undocumented features
> and reserve the right to remove them at any time in the future. Packages are not
> officially documented and thus not officially supported.
Well then, forgive me for this little offtopic but you seem to know what is unapproved
and what is unfinished. Are structs - not your proper structs but the commented-out
production found in erl_parse.yrl struct -> atom tuple - an unapproved feature or an
unfinished feature?
Regards,
--
Anthony Ramine
Dev:Extend
http://dev-extend.eu
Joe Armstrong
/Joe
Parnell Springmeyer
Hash: SHA1
I actually came here to suggest Python's implementation of packages and
modules - it confuses a lot of people at first, but it's actually quite
powerful and I wish Erlang would do something more like it.
Packages are a group of modules (modules are files in python, just like
erlang) that can be sub-divided into sub-packages within that package
using the filesystem, __init__.py files and naming. My apologies if this
is already understood.
The language I would point at as an example of a "mess" would be
something like PHP.
To comment on your suggestion: I much like the idea being able to
inherit modules, to do it the "Erlang way", I think that may be one of
the better and least painful-to-change routes.
- --
Parnell "ixmatus" Springmeyer (http://ixmat.us)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJN2886AAoJEPvtlbpI1POLZn0IAL2f62A5CrSroCi5XVwVACRX
Ecbq0Dzesqls61wK9gtX2lpEA16DEuxubz2m5FrGoKB18h4olrUi7fF7JCHcyEl6
+tc/rLwsGtvkE3bHrSRjbB7KTOEP1Dfa76i9xNvyPJXMZ8qdA3TBwKwiIhAx72a/
V1XkzEy4Z30UL7gWpysLkMc48p1LpoPnaTA20A5AXgcFd3SkzcuARRheWoEnuT1u
L6N3FfR3glELYaNjQYiBxbOOuR8vS4kxVxk78NLgOGY87t4JGdNvmcXt/64lhxkN
3zNJnclr8zA+JLyernUPNpGDWV9fAuzpzkDExI0+57v3MN4TpnN1GEblj8l65Ao=
=XGb2
-----END PGP SIGNATURE-----
Parnell Springmeyer
Hash: SHA1
I really like the idea of packages - I'm not quite sure why everyone is
against them... Even Racket has extended the syntax of Scheme to the
point where they support packages and modules now; it's a very natural
abstraction for the human mind IMHO.
I would, personally, stop using (or fork) the language if I were forced
to program connected to a DB. If it were a local ets-like DB that
started when the Erlang VM started, cool. But not an over-the-internet
type of thing, this is a particularly bad idea too because some
applications really do need to be network efficient; if my language adds
overhead to my network because it needs to request the definition of a
function over the internet, that is A Bad Thing.
I don't like the idea loading everything when the VM starts. I much
prefer (and wish Erlang did it this way) the concept of calling an
import statement. I like being able to see what packages, modules, and
functions another module is referencing.
Additionally this solves the issue of pre-loading the entire namespace
when the VM starts; you only import when you /need/ it.
I am intrigued, however, by the idea of type signatures and finding
functions based on their metadata - cool idea.
In practice though, I would really hate maintaining the metadata for
every function I've wrote! I much rather group common functions into a
module and then annotate the module with metadata.
Anthony Ramine <n...@dev-extend.eu> writes:
- --
Parnell "ixmatus" Springmeyer (http://ixmat.us)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJN29UaAAoJEPvtlbpI1POLpSAH/ijpOgfq1F+AF140/aXZke6C
EXrWaiFBOgna0lkKu4lmwusOebLbyoiPs9D9vstlVff7npZ4Zdw32fp9LRuBjwfY
rgGesMBoUGNMtpzXuuduCH7CkZNCIrBXj8uvKOBlDxt5csOz8FCpxvKDOXZg/z0I
bFMP0HTYj1pzC2ixk2RmlUdKdbMLCCriIU+n0R5f4lIEck1FcoDm233ZHv1Sv9QC
9PlQ4kF91H/CavPAdD/JX4sDp6NRTCjV/VWCUzDTOyK/11/fl4R7nAB61TXzkVmz
tv3y5z5RCkfjlatgvGSQ3d9vRbSn5fQneguxMX54JHI0WTEcWZ3xCQZZ8QKPuhY=
=Faf+
-----END PGP SIGNATURE-----
Tim Watson
Yes, smalltalk had/has everything bootstrapped into the one
environment, including the editor, stdlib, object system, etc.
Tim Watson
> I really like the idea of packages - I'm not quite sure why everyone is
> against them... Even Racket has extended the syntax of Scheme to the
> point where they support packages and modules now; it's a very natural
> abstraction for the human mind IMHO.
I've tried using them, but the lack of support makes it difficult. The
bugs in the cover tool you can get around by doing -import(ets) in
every module that needs to be cover compiled and as a work-around that
fixes it. But reltool doesn't work so beam files in package
sub-directories beneath ebin do not get copied into the release. There
are probably other bugs too.
> I don't like the idea loading everything when the VM starts. I much
> prefer (and wish Erlang did it this way) the concept of calling an
> import statement. I like being able to see what packages, modules, and
> functions another module is referencing.
You can already (today) choose whether everything loads on startup or
on demand (i.e., when you call Mod:Func(Args)) by choosing to run the
VM as an embedded system or not. To see what packages, modules and
functions your modules are referencing, you can use the xref tool
which comes with the standard distribution and is supported by tools
like rebar.
> I am intrigued, however, by the idea of type signatures and finding
> functions based on their metadata - cool idea.
Yes, this works particularly well for haskell (e.g., hoogle) and
combined with cabal/hackage makes it easy to find what you need based
mainly on type signatures. Sadly type signatures are often *all* the
documentation you get of a Haskell library/application, which unless
you're a type guru can be just as frustrating as
write-yourself/cut+paste and often more so than
google=>github=>rebar-deps.
>
> In practice though, I would really hate maintaining the metadata for
> every function I've wrote! I much rather group common functions into a
> module and then annotate the module with metadata.
>
If extra metadata were to be attached to functions, this should/would
be generated by the compiler anyway.
Personally, I think a step in the right direction would be to preserve
type signatures at runtime - something that has been discussed
elsewhere on this list not that long ago IIRC.
Dave Challis
of functions within (though maybe this can be replaced with good
function metadata).
Let's say we've got a sets module which happens to implement set data
structures as lists.
This sets module might define some generally useful list manipulation
functions, which others want to use.
However, the intent of those functions is to serve the sets module. If
the module owner wants to change one of the functions to be faster for
certain types of list and slower for others, it would make sense to do
so if it improved performance of the sets module as a whole.
If that function was in a global namespace of functions, it would need
metadata which indicated something like: "Although this is a general
list processing function, it is intended to support set data structures,
and as such may change internally to best serve this set of functions here."
I guess what I'm trying to say is that I can see the need for code
duplication when you've got:
X, a function intended to do X
Y, a function intended to do Y, which happens to be implemented as X
Though the code for them might look identical, there's the implied
contract that X will always do X, whereas Y could change at any time.
Modules (or namespaces in other languages) seem to be a reasonable way
of describing this intent/contract (albeit in vague terms, though most
modules seem to have an understandable purpose or theme).
--
Dave Challis
d...@ecs.soton.ac.uk
Tim Watson
> Personally, I think a step in the right direction would be to preserve
> type signatures at runtime - something that has been discussed
> elsewhere on this list not that long ago IIRC.
>
Sorry - that wasn't what I meant to say. Rather, I think that it would
be good to preserve type specifications in the beam code for a
compiled module, which doesn't happen at present.
Tom Murphy
> In practice though, I would really hate maintaining the metadata for
> every function I've wrote! I much rather group common functions into a
> module and then annotate the module with metadata.
Ideally, it wouldn't just be just you maintaining the metadata - if
the functions were in the centralized DB, others could suggest or make
changes, wikipedia-style.
Tom
Joe Armstrong
-----BEGIN PGP SIGNED MESSAGE-----I really like the idea of packages - I'm not quite sure why everyone is
Hash: SHA1
against them... Even Racket has extended the syntax of Scheme to the
point where they support packages and modules now; it's a very natural
abstraction for the human mind IMHO.
I would, personally, stop using (or fork) the language if I were forced
to program connected to a DB.
Why? Who cares how the data is stored - the requirements on storage
would be never-loose-anything fast latency access form any machine
I'd imaging a fast local db, that synced with some cloud storage and
where you could work on-or off-line with periods of no net access.
If your machine were struck by lightening you should be ably to
resume on any other machine, possibly loosing a very small ammount of data.
I imagine a local copy of couchDB with off-site replicas would be ok for this.
If it were a local ets-like DB that
started when the Erlang VM started, cool.
Ets is too fast - we don't need to be ets fast :-)
But not an over-the-internet
type of thing, this is a particularly bad idea too because some
applications really do need to be network efficient; if my language adds
overhead to my network because it needs to request the definition of a
function over the internet, that is A Bad Thing.
It's a trade off - if you always store modified functions in the net the
most you can loose is one function. If you don't you might loose
a lot of work if your machine suddenly disintegrates.
Umm let's think - in a good day I might write 500 lines of code
say 500 * 50 bytes in 86400 seconds that's 0.3 bits/second
I might look at ten times as much code - but I'd cache that locally.
/Joe
Joe Armstrong
On 5/24/11, Parnell Springmeyer <ixm...@gmail.com> wrote:Ideally, it wouldn't just be just you maintaining the metadata - if
> In practice though, I would really hate maintaining the metadata for
> every function I've wrote! I much rather group common functions into a
> module and then annotate the module with metadata.
the functions were in the centralized DB, others could suggest or make
changes, wikipedia-style.
This did occur to me - the wikipedia allows large numbers of people
to make small contributions. Think of what I want as a wiki where each
entry is a single erlang function.
Anthony Molinaro
those available for javascript libs like YUI/JQuery), wouldn't get you most
of what you want.
So I imagine being able to share individual functions from your own
misc lib, and then constructing misc libs with the set of functions
you want, maybe when the addition of a module name.
This means you can still use the module name as a method of encapsulation
but you can construct a module from a collection of functions.
You might even make it such that you can start to define modules in a
local file via a description language or set of compiler directives as
well as providing a web interface for constructing these modules.
So maybe you could do something like
-module(lists2).
-remote("http://erlfshare.com/?author=joe&func=remove_duplicates",
dedup/2).
-remote("http://erlfshare.com/?author=joe&func=list2frequency_distribution",
freq/1).
And that would be the definition of the lists2 module in your codebase,
it could pull the functions from the shared repo at compile time. If
you want to be safe you could version everything and pull in particular
versions or hide the url construction, and instead have a proplist you
set values in.
At the very least this allows you to share functions and work with
pre-existing code.
Just a thought,
-Anthony
--
------------------------------------------------------------------------
Anthony Molinaro <anth...@alumni.caltech.edu>
Tim Watson
the whole hog in terms of what Joe is proposing, but would certainly
move us in (what IMHO is) a good direction and potentially towards
Joe's vision if he wants to go that far. I know it isn't the whole
hog, but maybe it's a useful increment towards where we'd like to be.
Why don't we start by building a search engine for finding Erlang
"stuff" such as OTP applications/libraries, modules and functions.
I would propose that we allow users to search for things by
- kind (e.g., process/gen_thing, modules, functions, whole applications)
- category (set of arbitrary tags?)
- type signature (i.e., spec)
I'm guessing this will require patching the compiler to preserve the
type specs in the beam code. That is something I would need help with,
but may be trivial for others.
The other things I think you might want are (in no particular order):
1. binary (code) artefact repository (presumably with a REST/web interface?)
2. an means of uploading artefacts (tarballs? .ez archives?) to the repository
3. library code for introspecting module metadata (e.g., pulling type
specs from beam code)
4. some means of annotating packages/modules/functions with metadata
outside of the beam code
5. search (indexing, etc) capabilities
6. a way to obtain the *things* that you've identified as being useful to you
Not everything will need to be built from scratch. Binary (code)
artefact repositories already exist (to some extent) in the form of
CEAN (which has version 2.0 coming out soon IIRC) and the Erlware
repository. Those tools may not be immediately fit for re-use as they
come with baggage, but they may be a good base upon which to build
something for starters.
Other things/ideas might be:
- Integration with build tools (such as rebar) would be useful.
- Integration with existing search/package-mgmt tools such as agner
You could easily write a plugin that uses epm/agner/sutro to search
github for something that isn't in the repo, pulls it, builds it and
adds it. So many possibilities.
I do have other projects on the go, and a job, a family and even a
social life! Nonetheless, I would be happy to contribute to an
initiative such as this (or something similarly incremental).
Tim Watson
thoughts were that once you've got all that beam code indexed and/or
in memory, it's not a huge step to do what you want because the
underlying infrastructure is already there.
Also I see the approach you're proposing as being a complimentary
and/or alternative to the current status quo, so folks wanting to
carry on as-is will simply make use of the search+download
capabilities without utilising the *edit my code like its a wiki page*
features and so on.
Tim Carpenter
Sent from my iPhone
Vlad Dumitrescu
Joe Armstrong
I'd like to propose a "step in the right direction", which won't go
the whole hog in terms of what Joe is proposing, but would certainly
move us in (what IMHO is) a good direction and potentially towards
Joe's vision if he wants to go that far. I know it isn't the whole
hog, but maybe it's a useful increment towards where we'd like to be.
Why don't we start by building a search engine for finding Erlang
"stuff" such as OTP applications/libraries, modules and functions.
YES - I've made a start on this - the first program is
a segmenter that attempts to destruct a module.
I've just bough a synology DS411 slim with 3TB of RAID 5 storage
so am (finally) writing a backup program, this has first priority
in the hobby dept. Then I have to attack my legacy collection of 40K
erlang modules :-)
Unfortunately information about individual functions is
scattered all over the place. I want the source, the type signature,
the documentation. These are available in a variety of different
formats in different places. Documentation is in untagged comments
in @tagged edoc format or in XML in a completely different file.
I'd like this in one place with defined abstract types for everything
Volunteers ....
Silly design question: if multiple people can edit (say) a wiki
of functions what formatting standards should be enforced?
- the last person who edited the text
- some "moderator" (who)
- by a pretty printer
(is there a *really good* pp? - with color syntax marking)
I would propose that we allow users to search for things by
- kind (e.g., process/gen_thing, modules, functions, whole applications)
- category (set of arbitrary tags?)
- type signature (i.e., spec)
I'm guessing this will require patching the compiler to preserve the
type specs in the beam code. That is something I would need help with,
but may be trivial for others.
Easy - it is there already (I think) poke around in beam_lib.erl
this analyses the .beam files.
The other things I think you might want are (in no particular order):
1. binary (code) artefact repository (presumably with a REST/web interface?)
2. an means of uploading artefacts (tarballs? .ez archives?) to the repository
3. library code for introspecting module metadata (e.g., pulling type
specs from beam code)
4. some means of annotating packages/modules/functions with metadata
outside of the beam code
5. search (indexing, etc) capabilities
6. a way to obtain the *things* that you've identified as being useful to you
Not everything will need to be built from scratch. Binary (code)
artefact repositories already exist (to some extent) in the form of
CEAN (which has version 2.0 coming out soon IIRC) and the Erlware
repository. Those tools may not be immediately fit for re-use as they
come with baggage, but they may be a good base upon which to build
something for starters.
Other things/ideas might be:
- Integration with build tools (such as rebar) would be useful.
- Integration with existing search/package-mgmt tools such as agner
You could easily write a plugin that uses epm/agner/sutro to search
github for something that isn't in the repo, pulls it, builds it and
adds it. So many possibilities.
I do have other projects on the go, and a job, a family and even a
social life! Nonetheless, I would be happy to contribute to an
initiative such as this (or something similarly incremental).
I'll be back ...
/Joe
Joe Armstrong
A flow is a list of flows or grains
A grain is an atomic thing
A module is basically a flow where the grains are functions.
A novel is a flow of " chapter flows" a "chapter flow" is a flow
of paragraph grains etc.
An image gallery is a flow of image grains.
Everything is a flow or a grain.
I'm wriiting a folding editor in javascript to support editing/organising
flows
I made a weak minded attempt at this
see
https://github.com/joearms/elib1/tree/master/unsupported/folding
In this directory are some rudimentary folding editors
test_js6.html and experiment.erl are fun.
(( If anybody reading this is an ace javascript hacker and wants to help
please mail me -- I know *exactly* what help I need))
My first goal is to segement modules into flows and
put them into a web based flow editor.
All the flows will be in a global database.
One thing I missed saying - associated with each function is
*research* (NOT documentation) I want to store all reserach notes
necessary to write the function along with the function in a hidden fold
this should contain text, PDFs images etc. - the lot.
Often the research to write 100 lines of codes takes weeks
and we throw all this away - this must be stored
with the code (NOT linked, store a copy, the link might be
dead years later)
/Joe
Raoul Duke
> When I'm programming I often get to the point were I say there should
> a function foo/2 in lists.erl but their isn't. There should be but
> there isn't - foo/2 is a small self contained thing. Why should it be
> in lists.erl because it "feels right".
that Open-Closed thing, and suck elk, is just never that easy no
matter what the paradigm?
sincerely.
jm
>
> If we think of modules as containers of functions, then a module is a
> "set of functions" -
> my problem is with granularity. The smallest unit of reuse is the
> module. Suppose my application
> want just one function from a particular module - I'm forced to load
> the entire module.
>
> This hampers reuse - I *often* read library code and find a single
> function and cut-and-paste it into
> my new code. It also discourages sharing - since the smallest unit I
> can share is a module.
>
> The unit of distribution is an application - a set of modules - why
> this could not be a list of functions
> I do not know.
>
> I think I'd like to view an application as something that offered a
> message based service that internally
> was constructed from a set of functions rather than a set of modules.
>
Here's a "mad hatter" suggestion. Introduce the concept of a "shadow"
module. For example, say your using the list module and your after a
lists:randomise/1. In this proposal you'd have your own lists module,
say joes_lists,
-module(joes_lists)
randomise(List) ->
%% code to randomise the order of the list
In you code you would then say,
-shadows(joes_lists, lists).
then use lists:randomise(AList) in your code. The compiler would first
check the lists module and if the required function didn't exist it
would use your modules. This would avoid you accidentially overriding
the main modules definitions which may cause confusion. In such a case,
the clash should be treated as a compilation error.
The second part would be distribution and release management. There's
several ways this could be done one would be to have a script which
would build a set of "patch modules" which would be the cut down
versions of your own development modules containing only the code and
associated documentation your using in the current project.
Actually, it might be better to include this as part of the project
development: execute a script which scans your code repository and the
-shadow/2 compiler directives and generates the shadow modules as
needed. This could mean that your custom modules could be kept in a git
or mercurial repository, on a web server, or in a key-value store. The
URLs would then be part of the configuration for this script along with
version tags, etc. Yes, I going to ignore how the modules get in there
in the first place.
Jeff.
Edmond Begumisa
I've personally wanted an easy way to load modules from different places.
If you started with making modules easy to load from different places,
then later provide support for functions, I imagine this would be easier
to implement immediately and easier for people to make use of immediately
... where "people" refers to me :)
I've got two suggestions (mostly inspired by the Mozilla Framework)...
1. How about letting the programmer choose where the functions come from
(kv store, fs, web-server, etc)...
For instance, a module could start...
-import(["file://misc/collect_int", "http://erlang.org/lists/*",
"mnesia://db1/misc/merge_kv"]).
By default, the code loader would know what to do when it encounters
"file://", "http://", "https://", "ftp://", "sftp://" and "mnesia://" URI
schemes. Programmers could be allowed define their own URI schemes by
providing callbacks for loading code. For instance, one would be able to
define "ubf://" and have the code loader call his/her code when it
encounters "ubf://foo/bar"
2. Expanding on this: letting the programmer centralise the possible
locations of modules/functions and use this to define namespaces...
One possible way of doing this is to steal from the "chrome registration"
system that the Mozilla Framework uses. A programmer could define
one-level deep namespaces in a configuration file which points to targets
that could come from anywhere, then use that in her modules. A module
could then start...
-import(["erlang://misc/collect_int", "erlang://couch/mapreduce"]).
Where "misc" and "couch" are namespaces and "erlang://" is a special URI
scheme telling the code loader to refer to the configuration file
"erlang.manifest" which would contain...
[{"misc", "file://./ebin/misc.beam"},
{"couch", "http://localhost:7777/ebin/"}]
Where "couch" points to a yaws server serving couch beams, or something
along those lines. Then she can easy switch between different code sources
using one file.
A code-signing system could also be introduced.
- Edmond -
On Tue, 24 May 2011 18:06:19 +1000, Joe Armstrong <erl...@gmail.com> wrote:
> Why do we need modules at all?
>
> This is a brain-dump-stream-of-consciousness-thing. I've been
> thinking about this for a while.
>
> I'm proposing a slightly different way of programming here
> The basic idea is
>
> - do away with modules
> - all functions have unique distinct names
> - all functions have (lots of) meta data
> - all functions go into a global (searchable) Key-value database
> - we need letrec
> - contribution to open source can be as simple as
> contributing a single function
> - there are no "open source projects" - only "the open source
> Key-Value database of all functions"
> - Content is peer reviewed
>
> These are discussed in no particular order below:
>
> Why does Erlang have modules?
>
> There's a good an bad side to modules:
>
> Good: Provides a unit of compilation, a unit of code
> distribution. unit of code replacement
>
> Bad: It's very difficult to decide which module to put an individual
> function in. Break encapsulation (see later)
>
> Aside: lib_misc.erl
>
> When I'm programming I often get to the point were I say there should
> a function foo/2 in lists.erl but their isn't. There should be but
> there isn't - foo/2 is a small self contained thing. Why should it be
> in lists.erl because it "feels right".
>
> Strings are lists, so why do we have two modules lists.erl and
> string.erl how should I decide in which module my new string/list
> processing function should go.
>
> To avoid all mental anguish when I need a small function that
> should be somewhere else and isn't I stick it in
> a module elib1_misc.erl.
>
> My elib1_misc exports the following:
>
> added_files/2 make_challenge/0
> as_bits/1 make_response/2
> as_bits_test/0 make_response_test/0
> bdump/2 make_test_strings/1
> bin2hex/1 make_test_strings_test/0
> bin2hex_test/0 make_tmp_filename/2
> check_io_list/1 merge_kv/1
> collect_atom/1 merge_kv_test/0
> collect_atom_test/0 mini_shell/0
> collect_int/1 module_info/0
> collect_int_test/0 module_info/1
> collect_string/1 ndots/1
> collect_string_test/0 nibble_to_hex_char/1
> collect_word/1 nibble_to_hex_char_test/0
> complete/2 odd/1
> complete_test/0 on_exit/2
> dos2unix/1 out_of_date/2
> downcase_char/1 outfile/2
> dump/2 padd/2
> dump_tmp/2 perms/1
> duplicates/1 perms_test/0
> ensure_started/2 pmap/2
> eval_file/1 pmap1/2
> eval_file_test/0 pmap1_test/0
> eval_string/1 pmap_test/0
> eval_string_test/0 priority_receive/0
> every/3 random_seed/0
> expand_env_vars/1 random_string/1
> expand_file_template/3 random_string/2
> expand_string_template/2 read_at_most_n_lines/2
> expand_tabs/1 read_at_most_n_lines_test/0
> expand_tabs_test/0 remove_duplicates/1
> expand_template/2 remove_duplicates_test/0
> extract_attribute/2 remove_leading_and_trailing_whitespace/1
> extract_attribute_test/0
> remove_leading_and_trailing_whitespace_test/0
> extract_prefix/2 remove_leading_whitespace/1
> fetch/2 remove_prefix/2
> fetch_test/0 remove_prefix_test/0
> file2lines/1 remove_trailing_whitespace/1
> file2lines_test/0 replace/3
> file2md5/1 root_dir/0
> file2numberedlines/1 rpc/2
> file2numberedlines_test/0 safe/1
> file2paras/1 show_loaded/1
> file2stream/1 signed_byte_to_hex_string/1
> file2string/1 signed_byte_to_hex_string_test/0
> file2template/1 skip_blanks/1
> file2term/1 skip_blanks_test/0
> file_size_and_type/1 skip_to_nl/1
> find_src/1 skip_to_nl_test/0
> first/1 sleep/1
> flatten_io_list/1 spawn_monitor/3
> flush_buffer/0 split_at_char/2
> for/3 split_at_char_test/0
> force/1 split_list/2
> foreach_chunk_in_file/3 split_list_test/0
> foreach_word_in_file/2 string2exprs/1
> foreach_word_in_string/2 string2exprs_test/0
> forever/0 string2html/1
> get_erl_section/2 string2latex/1
> get_line/1 string2lines/1
> get_line/2 string2lines_test/0
> have_common_prefix/1 string2stream/1
> have_common_prefix_test/0 string2stream_test/0
> hex2bin/1 string2template/1
> hex2bin_test/0 string2template_test/0
> hex2list/1 string2term/1
> hex2list_test/0 string2term_test/0
> hex_nibble2int/1 string2toks/1
> hex_nibble2int_test/0 string2toks_test/0
> id/1 sub_binary/3
> include_dir/0 template2file/3
> include_file/1 term2file/2
> interleave/2 term2string/1
> is_alphanum/1 test/0
> is_blank_line/1 test1_test/0
> is_prefix/2 test_function_over_substrings/2
> is_prefix_test/0 tex2pdf/1
> is_response_correct/3 time_fun/2
> keep_alive/2 time_stamp/0
> lines2para/1 to_lower/1
> list2frequency_distribution/1 to_lower_test/0
> list2frequency_distribution_tetrim/1
> longest_common_prefix/1 trim_test/0
> longest_common_prefix_test/0 unconsult/2
> lookup/2 unsigned_byte_to_hex_string/1
> lorem/1 unsigned_byte_to_hex_string_test/0
> ls/1 which/1
> which_added/1
>
> Now I find this very convenient when I write a new small utility function
> I stick in in elib1_misc.erl - no mental anguish in choosing a module
> name is involved.
>
> The observation that I find this very-convenient is telling me something
> about modules - I like my elib1_misc it feels right.
>
> (aside - It seems many development projects have their own private
> lib_miscs ...)
>
> Which brings me to the point of my question.
>
> Do we need module's at all? Erlang programs are composed of lots of small
> functions, the only place where modules seem useful is to hide a letrec.
>
> The classic example is fibonacci. We want to expose fib/1 but hide the
> helper function fib/3. Using modules we say
>
> -module(math).
> -export([fib/1]).
>
> fib(N) ->
> fib(N, 1, 0).
>
> fib(N, A, B) when N < 2 -> A;
> fib(N, A, B) -> fib(N-1, A+B, A).
>
> The downside is we have had to *invent* one module name math - whose
> *only*
> purpose is to hide the definition of fib/3 which we don't want to be made
> callable.
>
> If we put a second function into the module math, then this second
> function
> could call fib/3 which breaks the encapsulation of fib/3.
>
> We could say:
>
> let fib = fun(N) -> fib(N, 1, 0) end
> in
> fib(N, A, B) when N < 2 -> A;
> fib(N, A, B) -> fib(N-1, A+B, A).
> end.
>
> I hardly dare suggest a syntax for this since I've been following
> another thread in this forum where syntax discussion seem to encourage
> much comment.
>
> ** Please do suggest alternative syntax's here - but do not comment on
> other peoples suggestions ...
>
> I would like to just talk about why we have modules.
>
> Another question:
>
> Does the idea of a module come from the idea that functions have to be
> stored somewhere, so we store them in a file, and we slurp the
> file (as a unit) into the system, so the file becomes a module?
>
> If all the files were store by themselves in a database would this
> change things.
>
> I am thinking more and more that if would be nice to have *all*
> functions in
> a key_value database with unique names.
>
> lookup(foo,2) would get the definition foo foo/2 from a database.
>
> The unique names bit is interesting - is this a good idea. Qualified
> names (ie names like xxx:foo/2) or (a.b.c.foo/2) sounds like a good
> idea but but when I'm programming I have to invent the xxx or the
> a.b.c which is very difficult. It also involves the "decision problem"
> if the namespaces xxx and a.b.c already exist I have to *choose* which
> to put my new function in.
>
> I think there might be a case for alises here joe:foo/2 could be used
> while developing "joe" would expand to a horrible random local string the
> real name being ab123aZwerasch123123_foo/2 but I would not be able to
> publish my code or make it available to a third_part before I had
> chosen a sensible name.
>
> (( managing namespaces seems really tricky, a lot of peoople seem
> to thing that the problem goes away by adding "." 's to the name
> but managing a namespace with namees like foo.bar.baz.z is just as
> complex
> as managing a namespace with names like foo_bar_baz_z or names like
> 0x3af312a78a3f1ae123 - the problem is that we have to go from a symbolic
> name like www.a.b to a reference like 123.45.23.12 - but how do we
> discover
> the initial name www.a.b? - there are two answers - a) we are given the
> name
> (ie we click on a link) - we do not know the name but we search fo it ))
>
>
> When programs are small we can live with "just the code" in "a few
> modules" the ratio of code to meta data is high.
>
> When programs are large we need a lot of meta-data to understand them.
>
> I would like to see all functions with all meta-data in a data base.
>
> I'd like to say:
>
> lookup(foo,2,Attribute) when Attribute =
>
> code|source|documentation|type signatures|revision
> history|authors|...
>
> The more I think about it the more I think program development should
> viewed as changing the state of a Key-Value database.
>
> So I imagine:
>
> 1) all functions have unique names
> 2) there are no modules
> 3) we discover the name of a function by searching metadata
> describing the function in a database
> 4) all public functions (think open source) are in the same
> database
>
> We could make a system to do this.
>
> I think this would make open-source projects easier, since the
> granularity of contribution goes down. You could contribute
> a single function - not an entire application.
>
> (( A problem with GUT style open source projects is there is
> not one database of functions, I often what one function from
> this project, another function from another project -- the
> granularity of reusable parts should be the individual function.
>
> functions are really easy to reuse
> modules are more difficult to reuse
> entire applications are very difficult to reuse
> (Unless there are isolated through a communication channel))
>
> Possible extensions.
>
> 1) Voting for promotion
> 2) A review process
>
> Given a raw database will *all* functions in it - we could derive an
> "approved" functions database.
>
> Popular functions could be moved to the approved database - the
> review process would need to be discussed - so kind of peer-review/wiki
> stuff.
>
> Comments?
>
> Volunteers?
>
> /Joe
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Richard O'Keefe
> Why do we need modules at all?
To hide data types,
and to provide short names (we can import a function
and not have to keep on repeating a long long name for it).
I note, without recommendation either way,
the existence of
local
<declarations>
in
<declarations>
end
in Standard ML, where the second lot of declarations
defines things visible outside but the first lot
defines things visible only inside this form.
> fib(N) ->
> fib(N, 1, 0).
>
> fib(N, A, B) when N < 2 -> A;
> fib(N, A, B) -> fib(N-1, A+B, A).
could be
local
fun fib_aux n a b = if n < 2 then a else fib_aux (n-1) (a+b) a
in
fun fib n = fib_aux n 1 0
end
in SML. I also note that this is NOT something that SML uses instead
of modules.
>Does the idea of a module come from the idea that functions have to be
>
> stored somewhere, so we store them in a file, and we slurp the
> file (as a unit) into the system, so the file becomes a module?
Probably not. Smalltalk has classes, but the unit of compilation is a
single method, and the unit of storage is classically a "change set"
(which may involve any number of classes and methods, and may include
revisions to existing classes).
I also note that in the days when Interlisp-D was just that and did
not include Xerox Common Lisp, you slurped a file (as a unit) into the
system, but there were no modules.
There are plenty of languages where a file may contain many modules,
possibly nested. (SML, Ada, to name just two.)
> If all the files were store by themselves in a database would this
> change things.
>
> I am thinking more and more that if would be nice to have *all* functions in
> a key_value database with unique names.
In Erlang as it stands, a source file is the natural unit of version
control and documentation. The idea of coping with a sea of functions
all being busily revised by hundreds if not thousands of programmers
acting asynchronously troubles me.
Haskell has a large package library, and people can update things often,
and I keep on seeing version issues in the Haskell-café mailing list.
Can
- scope of names
- grain of compilation and reloading
- version control
- documentation
- units of (program) storage
be teased apart?
Smalltalk seems to suggest that something along these lines might be
doable. (Smalltalk methods are version controlled individually.)
However, Smalltalk documentation, especially internal documentation,
varies between woeful and dire.
> When programs are large we need a lot of meta-data to understand them.
The Xerox slogan: a program is a data base, not a listing.
Interlisp-D did not strictly speaking _have_ source files.
There were files with source code in them, but they were
randomly accessed data bases. Thanks to macros, this did
not work as well as one might hope, but it did work after
a fashion.
Hey, maybe this could be a way to get rid of the preprocessor!
Joe, you've invented one great language, this is crazy enough to
be the concept for a second one.
Richard O'Keefe
On 24/05/2011, at 8:39 PM, Dmitrii Dimandt wrote:
>>
>> So I imagine:
>>
>> 1) all functions have unique names
>> 2) there are no modules
>> 3) we discover the name of a function by searching metadata
>> describing the function in a database
>> 4) all public functions (think open source) are in the same
>> database
>>
>> We could make a system to do this.
>
> Hm... you'd still need *some* kind of modules.
>
> For instance, in Webmachine each resource is expected to have some, or all, of predefined public/exported functions: to_html, resource_exists, allowed_methods etc.
>
> So, each resource will have functions with the same name. If we move all functions to a global namespace, then how do we differentiate between these functions? Probably by naming them resource_name_function_name?
Meta-data.
You say "I need a function about <topic1> with a ping cousin about <topic2>
and a pong friend about <topic3>"
>
> Furthermore, what do we do with behaviours? A behaviour is encapsulated within a module, also with a predefined set of exported funcions (and any number of custom exported functions).
In SML, a behaviour would be a functor.
In some other languages, it would be a single function
that took a record of anonymous functions &c and returned
a record of anonymous functions. Inventing an ad hoc
syntax:
local
<{start ~ Start, stop ~ Stop, send ~ Send, explode ~ Explode}> =
some¥wonderful¥new¥behaviour(<{
initial_state ~ 42,
shut_down ~ fun (State) -> ... end,
restart_policy ~ ad_infinitum,
update_state ~ fun (State, Info) -> ... end
}>)
in
what you please using Start, Stop, Send, Explode
end
The trick is to combine this with hot loading, and I have too many things
to do today to think of something. That's what Joe is for...
Richard O'Keefe
On 24/05/2011, at 8:49 PM, Gordon Guthrie wrote:
> ...and also...
>
> following on from what Dimitri said while I was writing me e-mail, we
> hang unit test onto modules and design them at that level of
> abstraction
>
Right now that is clear a Really Good Thing to do.
The question is whether it has to be that way, or whether it is
possible to devise something better.
Do the unit of documentation
and the unit of testing
and the unit of loading
*have* to be the same thing?
How would a revised system *work* in a distributed world?
Back in the early days of the language Pop, the people at
the Edinburgh AI department had a game. The whole machine
they were using was programmed in Pop, on top of bare metal.
The game goes like this:
how much of the system can you zap before you have
to reboot the machine?
How much of a shared sea-of-functions can you update before
having to revert to backups?
Michael Turner
Vance Shipley
I love the idea of ditching modules files and storing functions
in a data store. What I'm picturing though is that the functions
are stored in their abstract syntax form (i.e. data) so that the
issue of formatting standards is stripped away. You need tools
which translate it in and out so that we can continue to use the
editors we are familiar with but you can edit in emacs using it's
bizarre indentation and I can use vim and indent with tabs. ROK
can put seperators at the begging of lines, I'll see them at the
end.
I have long wanted to put a toolchain together which allowed me
to easily store abstract syntax in the source code control system
repository. The check in and check out processes would convert
(pretty print) it to my personal prefernce settings.
The problem with that plan seems to be with the preprocessor. It
seems to me though that your plan suffers the same problem. If
macros are defined at a module level, or in an include file, you'll
have to deal with them when you pull the functions out of the module
and stuff it into the data store. The extended syntax tree used in
the erl_syntax application might be the answer.
How do you deal with record definitions?
--
-Vance
On Tue, May 24, 2011 at 10:08:40PM +0200, Joe Armstrong wrote:
} Silly design question: if multiple people can edit (say) a wiki
} of functions what formatting standards should be enforced?
}
} - the last person who edited the text
} - some "moderator" (who)
} - by a pretty printer
} (is there a *really good* pp? - with color syntax marking)
--
-Vance
Edmond Begumisa
wrote:
> On 24/05/2011, at 8:06 PM, Joe Armstrong wrote:
>
>> Why do we need modules at all?
>
> To hide data types,
What are the possible ways of dealing with this? When you break modules up
into individual functions?
I presume such a system would therefore need a way of importing types
(complete with versioning) in addition to functions. Which would then
introduce additional complexity for the programmer than just importing a
whole module (the C header-file mess). Maybe the proposed system could
support both -- modules and individual functions?
- Edmond -
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
jm
>
> Back in the early days of the language Pop, the people at
> the Edinburgh AI department had a game. The whole machine
> they were using was programmed in Pop, on top of bare metal.
> The game goes like this:
> how much of the system can you zap before you have
> to reboot the machine?
>
> How much of a shared sea-of-functions can you update before
> having to revert to backups?
>
This is an aside, but one of those things I've been wondering about.
Is there a language in existence which scales from "here's a piece of
bare metal" and scales up to declarative distributed programming of the
likes of Erlang? At the moment, it seems that the accepted way is to
change from assembler to C to something else. Is there a language that
progresses from the low level machine code basics up by adding language
features and improving the runtime environment which is written in
itself, ie turtles all the way down, and is more than an AST written in
ascii (I'm looking at you lisp)?
Jeff.
Richard O'Keefe
On 24/05/2011, at 10:26 PM, Jesper Louis Andersen wrote:
> On Tue, May 24, 2011 at 10:06, Joe Armstrong <erl...@gmail.com> wrote:
>> Why do we need modules at all?
>
> I think we need modules, but this post sparked a thought I have had
> for some time now.
>
> I'd like to have module renaming/namespacing!
This has been proposed in some detail.
Can't remember who by, but I think his initials were R.A.O'K. (:-)
That proposal was inspired by Eiffel's LACE configuration language.
Richard O'Keefe
On 25/05/2011, at 12:45 AM, Joe Armstrong wrote:
>
>
> On Tue, May 24, 2011 at 1:56 PM, Max Lapshin <max.l...@gmail.com> wrote:
> Very strange topic for me.
>
> I'd like to know if there will be hierarchial modules in Erlang,
> because tree of packages is a rather good idea:
>
> No it's not - this has been the subject of long and heated discussion and is
> why packages are NOT in Erlang - many people - myself included - dislike
> the idea of hierarchical namespaces. The *dot* in the name has no semantics
> it's just a separator. The name could equally well be encoders.mpg.erlyvideo
> or mpg.applications.erlvideo.encoder - there is no logical way to organise the
> package name and it does not scale -
I should point out that Haskell has dotted names,
uses them to indicate semantics,
and has a *managed* hierarchical name space;
Data.* and Control.* and Net.* and so on are
tolerably clear.
I should also admit that there are occasional complaints
about it not being clear where things have been put, and
other indexing and searching tools are often used to find
things.
Richard O'Keefe
On 25/05/2011, at 3:34 PM, Edmond Begumisa wrote:
> Hello Joe,
>
> I've personally wanted an easy way to load modules from different places.
>
> If you started with making modules easy to load from different places, then later provide support for functions, I imagine this would be easier to implement immediately and easier for people to make use of immediately ... where "people" refers to me :)
>
> I've got two suggestions (mostly inspired by the Mozilla Framework)...
>
> 1. How about letting the programmer choose where the functions come from (kv store, fs, web-server, etc)...
>
> For instance, a module could start...
>
> -import(["file://misc/collect_int", "http://erlang.org/lists/*", "mnesia://db1/misc/merge_kv"]).
Why does this have to be done inside the module?
Why cannot the mapping from module names to sources
be *outside*, like an SGML (or XML) catalogue,
or a .lace file?
Bengt Kleberg
modules, having the same name, being used at the same time. Since they
are already compiled we must do the mapping outside of the modules.
bengt
Edmond Begumisa
> ... 2. Expanding on this: letting the programmer centralise the possible
> locations ...
That "erlang.manfest" contains is a prop_list that maps "erlang://" URIs
to actual locations. Sorry it wasn't clear enough.
- Edmond -
On Wed, 25 May 2011 17:07:04 +1000, Richard O'Keefe <o...@cs.otago.ac.nz>
wrote:
>
> On 25/05/2011, at 3:34 PM, Edmond Begumisa wrote:
>
>> Hello Joe,
>>
>> I've personally wanted an easy way to load modules from different
>> places.
>>
>> If you started with making modules easy to load from different places,
>> then later provide support for functions, I imagine this would be
>> easier to implement immediately and easier for people to make use of
>> immediately ... where "people" refers to me :)
>>
>> I've got two suggestions (mostly inspired by the Mozilla Framework)...
>>
>> 1. How about letting the programmer choose where the functions come
>> from (kv store, fs, web-server, etc)...
>>
>> For instance, a module could start...
>>
>> -import(["file://misc/collect_int", "http://erlang.org/lists/*",
>> "mnesia://db1/misc/merge_kv"]).
>
> Why does this have to be done inside the module?
> Why cannot the mapping from module names to sources
> be *outside*, like an SGML (or XML) catalogue,
> or a .lace file?
>
>
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Michael Truog
> On 25/05/2011, at 12:45 AM, Joe Armstrong wrote:
>>
>> On Tue, May 24, 2011 at 1:56 PM, Max Lapshin <max.l...@gmail.com> wrote:
>> Very strange topic for me.
>>
>> I'd like to know if there will be hierarchial modules in Erlang,
>> because tree of packages is a rather good idea:
>>
>> No it's not - this has been the subject of long and heated discussion and is
>> why packages are NOT in Erlang - many people - myself included - dislike
>> the idea of hierarchical namespaces. The *dot* in the name has no semantics
>> it's just a separator. The name could equally well be encoders.mpg.erlyvideo
>> or mpg.applications.erlvideo.encoder - there is no logical way to organise the
>> package name and it does not scale -
> I should point out that Haskell has dotted names,
> uses them to indicate semantics,
> and has a *managed* hierarchical name space;
> Data.* and Control.* and Net.* and so on are
> tolerably clear.
>
> I should also admit that there are occasional complaints
> about it not being clear where things have been put, and
> other indexing and searching tools are often used to find
> things.
If a large collaborative effort was formed to generate source code, it is important that code is reused to reduce complexity. A hierarchy would be our most natural way of determining source code that can be referenced to build larger systems, and that would make the dependency more likely to receive maintenance. That would helps us avoid the current situation with things like jungerl. I don't think it is important whether we use a '.' character or a '_' character to create a hierarchy with the names in our source code. However, the '.' character used by the undocumented Erlang package functionality could be used to provide better support for unmanageable hierarchies, perhaps with search and documentation benefits.
The other part to imposing order on a large collaborative effort seems to be determining the quality of the code for reuse. A rating system could be used and doesn't require a hierarchy. However, with many different common segments of source code available, a hierarchy can provide a way of focusing on relevant source code and should promote code reuse.
Edmond Begumisa
<bengt....@ericsson.com> wrote:
> Moreover, to solve the problem this should work for two different beam
> modules, having the same name, being used at the same time. Since they
> are already compiled we must do the mapping outside of the modules.
>
>
> bengt
>
Yes, in my second suggestion the mappings would be outside the modules, in
a single "manifest" file. This idea comes from the way "chrome.manifest"
and the "chrome://" URI works in Mozilla (those who've written
Firefox/Thunderbird extensions would be familiar with it.)
The code loader would have to consult the manifest file for the "real"
locations of the modules. That's what I was trying to describe here...
> 2. Expanding on this: letting the programmer centralise the possible
> locations of modules/functions and use this to define namespaces...
>
> One possible way of doing this is to steal from the "chrome
> registration" system that the Mozilla Framework uses. A programmer could
> define one-level deep namespaces in a configuration file which points to
> targets that could come from anywhere, then use that in her modules. A
> module could then start...
>
> -import(["erlang://misc/collect_int", "erlang://couch/mapreduce"]).
>
> Where "misc" and "couch" are namespaces and "erlang://" is a special URI
> scheme telling the code loader to refer to the configuration file
> "erlang.manifest" which would contain...
>
> [{"misc", "file://./ebin/misc.beam"},
> {"couch", "http://localhost:7777/ebin/"}]
>
> Where "couch" points to a yaws server serving couch beams, or something
> along those lines. Then she can easy switch between different code
> sources using one file.
>
> A code-signing system could also be introduced.
>
- Edmond -
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Richard O'Keefe
On 25/05/2011, at 6:16 PM, jm wrote:
>
> Is there a language in existence which scales from "here's a piece of bare metal" and scales up to declarative distributed programming of the likes of Erlang? At the moment, it seems that the accepted way is to change from assembler to C to something else. Is there a language that progresses from the low level machine code basics up by adding language features and improving the runtime environment which is written in itself, ie turtles all the way down, and is more than an AST written in ascii (I'm looking at you lisp)?
The original Smalltalk ran on bare metal.
There is or was a "SqueakNOS" (no operating system) port
that ran on bare metal, but with a glue layer in C.
Interlisp-D ran on bare metal.
There were some low level primitives, but you quickly
got into Lisp, and everything from the garbage collector
to the network and file system stacks as in Lisp.
Lisp machines ditto.
In both the MIT and Xerox lisp machines, you could twiddle
bits in registers or work on natural language translation.
Lisp is definitely more than an AST written in ASCII, and
there are several Algol-ish front ends for it. Reduce
has (had?) RLisp and Macsyma used CGOL, I think.
The Poplog system ran under VMS, and because they were
serving lots of undergraduates, they really wanted a rock
solid time sharing system underneath. But there's a
language layer called Syspop that gets you quite low,
with Pop11, Prolog, Common Lisp, and ML layered on top.
My experience with Interlisp-D was that you really want
to hide the bottom level pretty quickly.
Joe Armstrong
> Unfortunately information about individual functions is
> scattered all over the place. I want the source, the type signature,
> the documentation. These are available in a variety of different
> formats in different places. Documentation is in untagged comments
> in @tagged edoc format or in XML in a completely different file.
Yes, this is a really important part of it. Being able to slurp in
Erlang projects from the file system (and github, etc), access all
this information and provide it in a searchable fashion would
*already* put us way ahead of the crowd in terms of discovery.
> Silly design question: if multiple people can edit (say) a wiki
> of functions what formatting standards should be enforced?
>
> - the last person who edited the text
> - some "moderator" (who)
> - by a pretty printer
> (is there a *really good* pp? - with color syntax marking)
I think there are reasonable ones around. Take a look at
http://tidier.softlab.ntua.gr:20000/tidier/getstarted. I would also
propose that you let the viewer configure the pp to their own
preferences. Given that you're actually storing abstract code (or AST
or something like that) rather than lines of source code anyway, this
is just human readable representation after all. Let people view with
whatever (potentially hideous) whitespace &c convention they like.
Very interesting idea - there was once a pascal pretty printer that analysed your code
and inferred the style that you preferred - "Umm this guy likes semi-colons at the end of
line ..." they in applied it to code you hadn't written.
>> I'm guessing this will require patching the compiler to preserve the
>> type specs in the beam code. That is something I would need help with,
>> but may be trivial for others.
>
> Easy - it is there already (I think) poke around in beam_lib.erl
> this analyses the .beam files.
>
I was pretty sure this wasn't the case. I'll dig in to beam_lib at
lunch time and have another look.
Joe Armstrong
Joe,
I love the idea of ditching modules files and storing functions
in a data store. What I'm picturing though is that the functions
are stored in their abstract syntax form (i.e. data) so that the
issue of formatting standards is stripped away. You need tools
which translate it in and out so that we can continue to use the
editors we are familiar with but you can edit in emacs using it's
bizarre indentation and I can use vim and indent with tabs. ROK
can put seperators at the begging of lines, I'll see them at the
end.
I have long wanted to put a toolchain together which allowed me
to easily store abstract syntax in the source code control system
repository. The check in and check out processes would convert
(pretty print) it to my personal prefernce settings.
YES
The problem with that plan seems to be with the preprocessor. It
seems to me though that your plan suffers the same problem. If
macros are defined at a module level, or in an include file, you'll
have to deal with them when you pull the functions out of the module
and stuff it into the data store. The extended syntax tree used in
the erl_syntax application might be the answer.
YES
How do you deal with record definitions?
YES :-) - compile their hash into the function?
Edmond Begumisa
If I can easily load modules or Joe's functions from different locations,
then I can deploy my Erlang applications similarly to web-applications.
I'm using Erlang at both the client and the server. Currently, reltool and
appup I've found a too tedious. web-app style deployment would have big
advantages...
* In corporate environments: clients can have a bare-bones installation,
launch that, then get most of the Erlang code from a central server.
Upgrading everyone is replacing a few files on the server. I could write
some simple code to upgrade modules already loaded by clients.
* On the web: With code-signing supported, I can safely deploy all
applications on my startup's web-server. I get all users (using different
completely applications) to download and install the *same* single
bare-bones installation. Erlang code for the different apps comes from my
web-server. Users whose licenses run dry can't access newer code.
- Edmond -
Vlad Dumitrescu
I love the idea of ditching modules files and storing functions
in a data store. What I'm picturing though is that the functions
are stored in their abstract syntax form (i.e. data) so that the
issue of formatting standards is stripped away.
<snip>
The problem with that plan seems to be with the preprocessor. It
seems to me though that your plan suffers the same problem. If
macros are defined at a module level, or in an include file, you'll
have to deal with them when you pull the functions out of the module
and stuff it into the data store. The extended syntax tree used in
the erl_syntax application might be the answer.
Joe Armstrong
On 24/05/11 9:25 PM, Joe Armstrong wrote:Here's a "mad hatter" suggestion. Introduce the concept of a "shadow" module. For example, say your using the list module and your after a lists:randomise/1. In this proposal you'd have your own lists module, say joes_lists,
If we think of modules as containers of functions, then a module is a "set of functions" -
my problem is with granularity. The smallest unit of reuse is the module. Suppose my application
want just one function from a particular module - I'm forced to load the entire module.
This hampers reuse - I *often* read library code and find a single function and cut-and-paste it into
my new code. It also discourages sharing - since the smallest unit I can share is a module.
The unit of distribution is an application - a set of modules - why this could not be a list of functions
I do not know.
I think I'd like to view an application as something that offered a message based service that internally
was constructed from a set of functions rather than a set of modules.
-module(joes_lists)
randomise(List) ->
%% code to randomise the order of the list
In you code you would then say,
-shadows(joes_lists, lists).
then use lists:randomise(AList) in your code. The compiler would first check the lists module and if the required function didn't exist it would use your modules. This would avoid you accidentially overriding the main modules definitions which may cause confusion. In such a case, the clash should be treated as a compilation error.
The second part would be distribution and release management. There's several ways this could be done one would be to have a script which would build a set of "patch modules" which would be the cut down versions of your own development modules containing only the code and associated documentation your using in the current project.
This is not mad hatter :-)
Knuth mad a patch system years ago. You start off with a file - freeze it so it's read only
then write patches (replace lines N1,N2 with ...) - he did this with no tool support.
One problems today is how we work. We make file1
and edit it to file2, of course we can compute a patch P that transforms file1 to file2.
P works like this:
P = diff(file1, file2) such that
patch(file, P) => file2
The problem with P is that is bears no meaningful semantic information - there is no intentionality.
many years ago I thought we shuld have thingks like this:
-module(foo).
-export([bar/2], ...).
...
This defines foo.erl (as usual)
Now we define a patch bar.patch
-patch(bar).
-insert(foo).
-insert(baz).
-delete(foo:bar/2).
-delete(baz:foo/4).
-rename(baz:zxy/2, new/2).
code as before
...
A preprocessor takes bar.patch and makes bar.erl from the patch instructions.
This defines a compile time editing operation - insert actually inserts code from foo.erl, delete
deletes code ...
There are just actually simple refactoring commands.
The *nice* thing about bar.patch is we can preserves intentionality (if there is such a word) - the
patches produced by diff are usless - this is why a revision control system want you to comment
every diff. Here the place for comments is in the patch file.
The above is really trivial to implement
it would be relatively easy to generate a list of such commands from my (proposed) folding editor
-insert(foo, "http://.....") would be very nice
/Joe
Joe Armstrong
Hello Joe,
I've personally wanted an easy way to load modules from different places.
If you started with making modules easy to load from different places, then later provide support for functions, I imagine this would be easier to implement immediately and easier for people to make use of immediately ... where "people" refers to me :)
I've got two suggestions (mostly inspired by the Mozilla Framework)...
1. How about letting the programmer choose where the functions come from (kv store, fs, web-server, etc)...
For instance, a module could start...
-import(["file://misc/collect_int", "http://erlang.org/lists/*", "mnesia://db1/misc/merge_kv"]).
By default, the code loader would know what to do when it encounters "file://", "http://", "https://", "ftp://", "sftp://" and "mnesia://" URI schemes. Programmers could be allowed define their own URI schemes by providing callbacks for loading code. For instance, one would be able to define "ubf://" and have the code loader call his/her code when it encounters "ubf://foo/bar"
2. Expanding on this: letting the programmer centralise the possible locations of modules/functions and use this to define namespaces...
One possible way of doing this is to steal from the "chrome registration" system that the Mozilla Framework uses. A programmer could define one-level deep namespaces in a configuration file which points to targets that could come from anywhere, then use that in her modules. A module could then start...
-import(["erlang://misc/collect_int", "erlang://couch/mapreduce"]).
Where "misc" and "couch" are namespaces and "erlang://" is a special URI scheme telling the code loader to refer to the configuration file "erlang.manifest" which would contain...
[{"misc", "file://./ebin/misc.beam"},
{"couch", "http://localhost:7777/ebin/"}]
Where "couch" points to a yaws server serving couch beams, or something along those lines. Then she can easy switch between different code sources using one file.
A code-signing system could also be introduced.
Did this a while back - you can also xform everything - all private modules can have their
names changed to MD5SUM(Content) of the module IFYSWIM
/Joe
Joe Armstrong
On Wed, May 25, 2011 at 07:58, Vance Shipley <van...@motivity.ca> wrote:I love the idea of ditching modules files and storing functions
in a data store. What I'm picturing though is that the functions
are stored in their abstract syntax form (i.e. data) so that the
issue of formatting standards is stripped away.<snip>
The problem with that plan seems to be with the preprocessor. It
seems to me though that your plan suffers the same problem. If
macros are defined at a module level, or in an include file, you'll
have to deal with them when you pull the functions out of the module
and stuff it into the data store. The extended syntax tree used in
the erl_syntax application might be the answer.There are more things that need to be preserved by the parser, which make existing pretty-printers less than good. For example:- integer values can be shown using different bases, from case to case, so one can't use a generic setting. So the parser needs to keep track of that.
The new parser/tokeniser does I'm not sure if its worked its way in the pretty printers - I do have
some unpublished code that does exact colorization of the source program
Joe Armstrong
How about
public
Functions etc.
private
Functions etc.
end.
Functions etc is just a regular sequence of function defs macros, records etc.
The use of the words public and private would attract hoards of programmers to our humble little
language...
/Joe
Vlad Dumitrescu
On Wed, May 25, 2011 at 6:29 AM, Richard O'Keefe <o...@cs.otago.ac.nz> wrote:
On 24/05/2011, at 8:06 PM, Joe Armstrong wrote:To hide data types,
> Why do we need modules at all?
and to provide short names (we can import a function
and not have to keep on repeating a long long name for it).
I note, without recommendation either way,
the existence of
local
<declarations>
in
<declarations>
end
in Standard ML, where the second lot of declarations
defines things visible outside but the first lot
defines things visible only inside this form.
How about
public
Functions etc.
private
Functions etc.
end.
Functions etc is just a regular sequence of function defs macros, records etc.
The use of the words public and private would attract hoards of programmers to our humble little
language...
Tim Watson
> Very interesting idea - there was once a pascal pretty printer that analysed
> your code
> and inferred the style that you preferred - "Umm this guy likes semi-colons
> at the end of
> line ..." they in applied it to code you hadn't written.
<grin>
>> >> I'm guessing this will require patching the compiler to preserve the
>> >> type specs in the beam code. That is something I would need help with,
>> >> but may be trivial for others.
>> >
>> > Easy - it is there already (I think) poke around in beam_lib.erl
>> > this analyses the .beam files.
>> >
>>
>> I was pretty sure this wasn't the case. I'll dig in to beam_lib at
>> lunch time and have another look.
>
>
So you're sort of correct. The specs are present in the abstract_code
chunk. I'm guessing this will only be the case if (a) you've compiled
with debug_info and/or (b) you've used the abstract_code
parse_transform (from github?) to ensure it's present either way. You
would need to make sure it is preserved in order to get the type info
by inspecting the beam. Perhaps slurping dialyzers generated plt
file(s) would be a better (more general) approach? Maybe we should
talk to Kostis - I suspect the idea of a type-based search engine
might whet his appetite somewhat.
Robert Virding
----- "Anthony Ramine" <n...@dev-extend.eu> wrote:
>
> Well then, forgive me for this little offtopic but you seem to know what is unapproved
> and what is unfinished. Are structs - not your proper structs but the commented-out
> production found in erl_parse.yrl struct -> atom tuple - an unapproved feature or an
> unfinished feature?
>
It is the remains of an unfinished feature which no one ever removed. There are more commented out remains in erl_expand_records, erl_lint and erl_pp. They should really all be removed as if/when the new structs/frames are implemented they will be wrong and in the way.
Robert