Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Message from discussion Turning rebar into a package manager
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Joe Armstrong  
View profile  
 More options Oct 23 2012, 3:46 am
From: Joe Armstrong <erl...@gmail.com>
Date: Tue, 23 Oct 2012 09:46:18 +0200
Local: Tues, Oct 23 2012 3:46 am
Subject: Re: [erlang-questions] Turning rebar into a package manager

On Fri, Oct 19, 2012 at 2:33 PM, Fred Hebert <monon...@ferd.ca> wrote:
> For anyone interested, there's been an interesting blog post on the topic of
> package managers written at
> http://hyperthunk.wordpress.com/2012/05/28/does-erlangotp-need-a-new-...

That's very interesting. Thanks for the reference

I got to thinking a bit more. ...

The above link has an example command

> git fetch-pack --include-tag -v g...@github.com:hyperthunk/gitfoo.git refs/tags/lager-0.9.4

This appears to fetch a tagged tree without all all other versions -
just what I wanted

At some level of abstraction we need to tag versions of applications.
Using the latest
version is fine for development but not for shipping a product.

If we adopt a common tagging convention then we can do two things.

     1) express dependencies (ie foo-1.4.5 depends upon bar-3.2.6)
         and
     2) make a "release" ie say that a certain set of tagged
applications has been tested and is known
         to work together.

At the moment I guess very few Erlang git projects made by rebar have
vI.J.K style tags.

Two more problems remain:

     A) Weakening the dependencies

        I know what foo-3.4.2 depends upon bar-4.2.5 means - this is
very precise

        Should we allow foo-3.* depends upon bar-4.*

        There must be some language of dependencies, and conventions
(ie odd major version numbers are experimental,
         even stable)

         Does anybody have any pointers to a discussion of this.

         I'd like a few simple rules that cover the common use cases
rather than fancy rules that cover everything

     B) The *one* module name problem

          We can't load modules in foo-1.2.3 and foo-2.1.6 into the
same node so if A depends upon
          foo-1.2.3 and B depends upon foo-2.1.6 we can't run A and B
in the same node

         There are a number of solutions possible:

         i)  name_munging -> convert xx.erl to xx_foo_1.2.3.erl

            It *is* possible - but you have to take care of registered
process names etc. there are a lot of
            edge cases that make this tricky.

        ii) Don't solve the problem at all. Run A and B in different nodes

        My feeling is ii) is correct - we could then define a node as
a release (ie a set of tagged applications)
        that are know to work together.

        I think problems of /consistency of
modules/security/authentication should be solved at the node level
        and not the process/module level

How could one make a release?

     a) All contributors must tag versions of their applications and
follow common packaging conventions
     b) we need an improved manifest

Maybe volunteers could make releases - actually a release is little
more than a set of applications
that are know to.

Seems we also need to standardize a new an improved manifest file - we
need two types
one that describe applications, the other which describes releases.
These should be super-sets of
the existing descriptions.

What should be in the manifest?

How should things be packaged?

I rather like the "put everything in a zip file (with a changed
extension) and add a top-level
manifest that describes what will be in the zip file" approach.

Again design of the manifest should strike a balance between simple
and intuitive and
"can do everything" -
(a zip file with some code that you do apply on :-) would be too
simple and dangerous)

Cheers

/Joe

> They mention a few very interesting issues that are likely to be brought up
> again here, and I would recommend reading it as a nice entry point to the
> discussion of what problems common solutions have, and what could possibly
> be done to fix it.

> On 12-10-19 6:36 AM, Joe Armstrong wrote:

>> Here is a programming exercise ...

>> I think we can turn rebar into a package manager with 4 simple and one not
>> so
>> simple tweaks.

>> This would turn rebar from being something that is very useful to
>> something
>> that is very very useful :-)

>> (aside - I'm revising my Erlang book, and I'd like to describe
>> how package management works, but there is no package manager
>> which is a shame - so I got to thinking about package managers
>> and wondered why still don't have a decent package manager.

>> The one I've seen that I like a lot is npm (node package manager)
>> so we could do worse than follow their design)

>> So this is how I think rebar could become a package manager

>> Tweak 1
>> =======

>>     > rebar get-deps

>> Fetches the dependencies in rebar.conf and stores them in
>> the current directory, in a directory called deps.

>> After a while you get dependencies scattered all over the place,
>> I've seen many examples of this.

>> This means that there is no central place to go and look if you
>> want to find code.

>> NPM sticks ALL dependencies under ${HOME}/.npm

>> Suggestion:

>>     All dependencies should be stored in ${HOME}/.erl_packages

>>     (Aside - if we followed npm the directory structure would be
>>      something like)

>>     ${HOME}/.erl_packages/cowboy/6.2.1/src
>>                                                                 /ebin
>>                                                       /5.2.3
>>                          /stdlib/12.3.1
>>                                 /14.6.8

>>     etc. ie a package/Vsn/src ... structure

>> Tweak 2
>> =======

>>     move rebar.config to ${HOME}/.erl_packages/rebar.config
>>     and add commands to automatically edit it

>>     > rebar add_package foo

>>     might add a line like
>>     {foo, ".*", "git://...."}

>>     to rebar.config

>>     The point here is to change the content of rebar.config with
>>     shell commands rather that editing it by hand

>> Tweak 3
>> =======
>>     fix rebar so it just downloads a particular version of a
>>     program and not all the .git stuff - most people will only
>>     want to use other peoples code, not hack it.

>> Tweak 4
>> =======

>> Fix the erlang load paths to find all the dependencies

>> I do this now. Put all my dependencies in
>> ${HOME}/nobackup/erlang_imports/deps and put the following
>> code in my .erlang startup file

>>    Home = os:getenv("HOME").
>>    Dir = Home ++ "/nobackup/erlang_imports/deps",
>>    case file:list_dir(Dir) of
>>       {ok, L} ->
>>           lists:foreach(fun(I) ->
>>                                Path = Dir ++ "/" ++ I ++ "/ebin",
>>                                code:add_path(Path)
>>                        end, L);
>>      _ ->
>>         void
>>    end.

>> Tweak 5
>> =======
>>     This can't be done by rebar
>>     Make an index of all erlang apps on github that follow
>>     the erlang packaging structure

>>     Somebody has to write a program to do this.

>>     The package index should be at a well know URL

>>     > rebar get_index

>>     This should fetch the  index from the well-know URL and store in
>>     ${HOME}/.erl_packages

>>     > rebar search XXX

>>     would search the fetched index

>>     > rebar add XXXX

>>     would take the index entry add it to the config fill fetch the code
>>     and compile it

>> Note the following:

>> 1) The trust model.

>>     We trust the supplier of a program not the program.

>>     So for example on github we might trust programs
>>     published by the user joearms (me) or erlang (the OTP release)
>>     but not by some_guy_ive_never_hear_of

>>     It would be quite easy to add a trust section to rebar.config

>>     {trust, ["git://joearms", git://erlang", ...]}

>> 2) There is no "publish" step

>>     We put our code on github (or somewhere) and hope that it gets indexed
>>     by the indexer.

>>     We might mail the indexer if it is published in an obscure place.

>>    Comments?

>>     Cheers

>> /Joe
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questi...@erlang.org
>> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
erlang-questi...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.