[erlang-questions] Turning rebar into a package manager

[Joe Armstrong]

Here is a programming exercise ...

I think we can turn rebar into a package manager with 4 simple and one not so
simple tweaks.

This would turn rebar from being something that is very useful to something
that is very very useful :-)


(aside - I'm revising my Erlang book, and I'd like to describe
how package management works, but there is no package manager
which is a shame - so I got to thinking about package managers
and wondered why still don't have a decent package manager.

The one I've seen that I like a lot is npm (node package manager)
so we could do worse than follow their design)

So this is how I think rebar could become a package manager


Tweak 1
=======

> rebar get-deps

Fetches the dependencies in rebar.conf and stores them in
the current directory, in a directory called deps.

After a while you get dependencies scattered all over the place,
I've seen many examples of this.

This means that there is no central place to go and look if you
want to find code.

NPM sticks ALL dependencies under ${HOME}/.npm

Suggestion:

All dependencies should be stored in ${HOME}/.erl_packages

(Aside - if we followed npm the directory structure would be
something like)

${HOME}/.erl_packages/cowboy/6.2.1/src
/ebin
/5.2.3
/stdlib/12.3.1
/14.6.8

etc. ie a package/Vsn/src ... structure


Tweak 2
=======

move rebar.config to ${HOME}/.erl_packages/rebar.config
and add commands to automatically edit it

> rebar add_package foo

might add a line like
{foo, ".*", "git://...."}

to rebar.config

The point here is to change the content of rebar.config with
shell commands rather that editing it by hand

Tweak 3
=======
fix rebar so it just downloads a particular version of a
program and not all the .git stuff - most people will only
want to use other peoples code, not hack it.

Tweak 4
=======

Fix the erlang load paths to find all the dependencies

I do this now. Put all my dependencies in
${HOME}/nobackup/erlang_imports/deps and put the following
code in my .erlang startup file

Home = os:getenv("HOME").
Dir = Home ++ "/nobackup/erlang_imports/deps",
case file:list_dir(Dir) of
{ok, L} ->
lists:foreach(fun(I) ->
Path = Dir ++ "/" ++ I ++ "/ebin",
code:add_path(Path)
end, L);
_ ->
void
end.

Tweak 5
=======
This can't be done by rebar
Make an index of all erlang apps on github that follow
the erlang packaging structure

Somebody has to write a program to do this.

The package index should be at a well know URL

> rebar get_index


This should fetch the index from the well-know URL and store in
${HOME}/.erl_packages


> rebar search XXX

would search the fetched index

> rebar add XXXX

would take the index entry add it to the config fill fetch the code
and compile it

Note the following:

1) The trust model.

We trust the supplier of a program not the program.

So for example on github we might trust programs
published by the user joearms (me) or erlang (the OTP release)
but not by some_guy_ive_never_hear_of

It would be quite easy to add a trust section to rebar.config

{trust, ["git://joearms", git://erlang", ...]}

2) There is no "publish" step

We put our code on github (or somewhere) and hope that it gets indexed
by the indexer.

We might mail the indexer if it is published in an obscure place.

Comments?

Cheers

/Joe
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Ulf Wiger]

Joe,

You should be able to prototype tweak 1 using a rebar.config.script file.

https://github.com/basho/rebar/wiki/Dynamic-configuration

Tweak 2, you could accomplish with an alias rebar='rebar --config …', and commands to edit the config could be put in a special escript.

Tweak 3 could also be done through a rebar.config.script.

BR,
Ulf W

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
http://feuerlabs.com

[José Valim]

I believe one of the issues that would need to be addressed first in rebar is the one of repeatability.

rebar does not ensure that developers in the same project are going to use the same snapshot of a dependency. This means developers can depend on the same git repository but different commits leading to the famous "the tests pass on my machine".

I am aware this could be fixed by the team enforcing a commit sha when declaring the dependency but many simply don't do it. Besides it puts the work on you to manage the commit shas. If you want to update the dependency, you need to remove the commit sha, update the repo, write the new commit sha.

This could be easily fixed by having a lock file per project with the commit shas of the dependencies. "rebar get-deps" should retrieve the shas in such lock, if one is available, or simply get the latest and write to the lock. A command like "rebar unlock-deps" could be available when developers want to update their dependencies, regardless of the current lock.

Also, regarding Joe's proposal, picking up a version from a git repository may be tricky. The best approach is to index git tags only, hoping the publisher won't modify an existing tag. If someone is depending on a branch though, it should not be considered a version or a part of .erl_packages. I believe it should still belong to the repo.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

[Peter Lemenkov]

Hello All.

2012/10/19 Joe Armstrong <erl...@gmail.com>:

> Here is a programming exercise ...
>
> I think we can turn rebar into a package manager with 4 simple and one not so
> simple tweaks.

As a simple programming exercise - that's ok. As a something which
people would use - that's a horribly horribly bad.

Almost every language went through that stage - "we have a build tool
which is capable of fetching deps - let's replace package manager with
this shiny thing". So far we've got many different language-specific
build tools with an option to behave like a poor error-prone package
managers (for R, Haskell, Node,js, Ruby, Python, Perl, Go, Scala,
etc). None of them works reliably in a something more complex than a
developer's environment (which is messy by definition and that's
absolutely ok).

I briefly covered this topic during Moscow Erlang Factory Lite (if
someone is brave enough to stand my accent then you may even google
for the video) - here is a link to PDF with my presentation:

* http://peter.fedorapeople.org/presentations/Moscow_Erlang_Factory_Lite_2012.pdf

Before investing your free time into this project (again, if it's a
simply a mind exercise then it's absoluely ok, and I don't have any
objections in this very case) I strongly advise you to spend some time
studying features of a currently available general package managers -
deb and rpm. Perhaps integrating rebar with them could be a more
viable alternative than reimplementing their features.
--
With best regards, Peter Lemenkov.

[Max Lapshin]

Joe.

I don't like the idea to store project dependencies outside project tree.

There may be same versions of software downloaded from different
github accounts.

I've worked with Ruby on Rails for several years and looked how
packaging was developed. Now it is just like rebar: packaging inside
project tree, but rebar is better: it doesn't require to pack into
some archive.

So I really consider, that rebar is the best package manager for
erlang right now and it doesn't need any additions.

[Max Bourinov]

Hi guys,

I think is what checking out ideas of maven tool (1, 2) will be very useful too. Those guys did a perfect tool for java which does packet and life-cycle management very easy and predictable. I had a great experience with it and it did saved a lot of my time.

----

1. http://en.wikipedia.org/wiki/Apache_Maven

2. http://maven.apache.org/

Best regards,

Max

[Max Lapshin]

On Fri, Oct 19, 2012 at 3:48 PM, Max Bourinov <bour...@gmail.com> wrote:
> Hi guys,
>
> I think is what checking out ideas of maven tool (1, 2) will be very useful
> too. Those guys did a perfect tool for java which does packet and life-cycle
> management very easy and predictable. I had a great experience with it and
> it did saved a lot of my time.
>

What do you consider good in Maven?
It is impossible to see any good things in provided links because
there are only tons of XML.

[Thomas Allen]

On Fri, October 19, 2012 6:36 am, Joe Armstrong wrote:
> Tweak 1
> =======
>
> > rebar get-deps
>
> Fetches the dependencies in rebar.conf and stores them in
> the current directory, in a directory called deps.
>
> After a while you get dependencies scattered all over the place,
> I've seen many examples of this.
>
> This means that there is no central place to go and look if you
> want to find code.
>
> NPM sticks ALL dependencies under ${HOME}/.npm
>
> Suggestion:
>
> All dependencies should be stored in ${HOME}/.erl_packages

To be clear, `npm' only will install packages in $HOME/node_modules if no
`package.json' file is found in $PWD. Otherwise, packages are installed at
$PWD/node_modules, much like `rebar's behavior when a `rebar.config' is
present.

Thomas Allen

[Max Lapshin]

I think that this topic will reinncarnate again and again.


Package managers are ok only for box-products. For example:
apt-get install rabbitmq (depends on erlang-nox and something else)

When we speak about in-house development, it is impossible to use
package manager for frequent deploy, because it is inconvenient. git
pull is much faster and better.

Erlang is used primarily for in-house development when you can't rely
on repository maintainers.

Damn! There is no default Erlang R15B02 neither in Ubuntu, neither in
Centos/Fedora. What are we speaking about? How can someone seriously
speak about using packages, when it takes a year to update erlang in
distributives???

[Max Bourinov]

Max,

Please follow this link: http://en.wikipedia.org/wiki/Apache_Maven

There is only one scary XML.

Best regards,

Max

[Max Lapshin]

On Fri, Oct 19, 2012 at 4:04 PM, Max Bourinov <bour...@gmail.com> wrote:
> Max,
>
> Please follow this link: http://en.wikipedia.org/wiki/Apache_Maven
>

It is not an answer. What good have _you_ found in Maven? XML config
as everything in Java?

[Max Bourinov]

http://www.quickmeme.com/meme/3rexpa/

Best regards,

Max

[Matthew Evans]

We are using Maven for both java and erlang development. The erlang plugin works well, but I find Maven's "convention over configuration" painful to say the least. You want to do something slightly out of the ordinary (as is often the case with a multi language project) and your life is spent in XML hell as you are trying to work around roadblocks. A good example was getting it to spew out thrift code for both languages.

On top of that I'm personally not a fan of separating source control and package management. All you need is for someone to deploy without doing a source update and you're left trying to figure out why your build is broken.

Just my 2c...

---

From: bour...@gmail.com
Date: Fri, 19 Oct 2012 16:04:05 +0400
To: max.l...@gmail.com
CC: erlang-q...@erlang.org; diz...@gmail.com
Subject: Re: [erlang-questions] Turning rebar into a package manager

[Max Bourinov]

Max, do you want me re-type article from wikipedia?

So, I try: It is very good package manager that is de-facto standard in Java world. Sorry guys I said Java here. So, the idea was to check ideas behind it because I believe those Java guys have good ideas.. Apache guys... So, we can avoid some unnecessary design pitfalls... For example maven has idea od snapshot/release repositories, and this idea could be worth to implement it in rebar packet manager.

But, I suggest you to not be scared by XML. It is only one little XML. All the rest is text: http://en.wikipedia.org/wiki/Apache_Maven

Best regards,

Max

[Ondřej Černoš]

Hi,

as someone coming to Erlang after many years of doing enterprisey
Java, I feel like answering your question.

>> I think is what checking out ideas of maven tool (1, 2) will be very useful
>> too. Those guys did a perfect tool for java which does packet and life-cycle
>> management very easy and predictable. I had a great experience with it and
>> it did saved a lot of my time.
>>
>
> What do you consider good in Maven?
> It is impossible to see any good things in provided links because
> there are only tons of XML.

- repeatable builds
- declarative version management
- it is a de facto standard for dependency management in Java world,
therefore thousands of java libs and other artifacts are available for
maven users
- continuous integration tools integration (jenkins, bamboo and others)
- ability to integrate traditional build tools into maven workflow (ant, make)
- ability to combine public/private repositories
- automatic deployment of build artefacts into repository, if desired
- extensibility through plugins

It is true that XML is not the best format to configure a build tool,
but hey, maven was created at the times when XML seemed like a good
idea and sincerely, it is not The Issue with the tool that is worth
using as a way to refuse it.

Inventing a package manager/dependency manager without taking a
serious look on maven experience and lessons learned would be very
shortsighted, imho.

regards,
ondrej cernos

[Max Bourinov]

True. Maven is a hell when you need something special.

But I suggested to only check concepts behind it. Just think: Java guys are very productive with maven - there are reasons why. So, if we want to build a package manager, maybe before we start, we can enrich its design by checking what others did?

Best regards,

Max

[Max Bourinov]

Ondřej Černoš, thank you for your mail. You just wrote what I thought bro.

There are many valuable features in maven. If we go for rebar package-manager - we should consider them too.

Best regards,

Max

[Max Lapshin]

> - repeatable builds

it exists in rebar

> - declarative version management

it exists in rebar

> - it is a de facto standard for dependency management in Java world,
> therefore thousands of java libs and other artifacts are available for
> maven users

is it a pro? I really don't care about Java users. It is their problem
that they still have to use Java with its horrible infrastructure.

> - ability to integrate traditional build tools into maven workflow (ant, make)

Great. You have two megabytes of XML for Maven that allows you to
include 20 megabytes of XML for Ant.
Good tool.

> - ability to combine public/private repositories

it exists in rebar.

> - automatic deployment of build artefacts into repository, if desired

it exists in rebar

> - extensibility through plugins

it exists in rebar.

>
> It is true that XML is not the best format

In fact it is the worsests declarative format.

> but hey, maven was created at the times when XML seemed like a good
> idea and sincerely, it is not The Issue with the tool that is worth
> using as a way to refuse it.

Yes, but rebar can do _everything_ that maven do with a simple and
easy-to-edit config file.

[Ondřej Černoš]

If rebar does everything maven does, then why this thread even exists?

ondrej

ps. Could we please stop bashing Java and XML and discuss on topic?

[Fred Hebert]

For anyone interested, there's been an interesting blog post on the
topic of package managers written at
http://hyperthunk.wordpress.com/2012/05/28/does-erlangotp-need-a-new-package-management-solution/

They mention a few very interesting issues that are likely to be brought
up again here, and I would recommend reading it as a nice entry point to
the discussion of what problems common solutions have, and what could
possibly be done to fix it.

[Max Lapshin]

On Fri, Oct 19, 2012 at 4:25 PM, Ondřej Černoš <cer...@gmail.com> wrote:
> If rebar does everything maven does, then why this thread even exists?
>
> ondrej
>
> ps. Could we please stop bashing Java and XML and discuss on topic?
>

Ok. Max told that maven is a wonderful tool that can do many things
and we've found that rebar already can do everything that maven can
do, but much better because a man can edit rebar.config.

This thread have appeared due to specific approach of managing
dependencies that rebar brings.
It implies that you dependency control is glued together with source
control and all program dependencies are isolated together inside your
deploy.

It is good for deploying your software on your servers, but it is not
convenient when you are a admin of PHP server that wants to install
ejabberd.

These are different clauses and if you want to make a package for any
modern distributive you will have to make a serious preparation of
your source code, that will can even make it incompatible with further
development.

This is why sometimes source for debian package is generated from current tree.

[Max Lapshin]

binary artefacts...
And then I deploy jiffy on debian 5 32 bit and... Wow! GLIBC incompatible? =)

everything could be much, much easier if there were no drivers and no nifs.

[Alex Shneyderman]

> On Fri, Oct 19, 2012 at 4:04 PM, Max Bourinov <bour...@gmail.com> wrote:
>> Max,
>>
>> Please follow this link: http://en.wikipedia.org/wiki/Apache_Maven
>>
>
> It is not an answer. What good have _you_ found in Maven? XML config
> as everything in Java?

At least java got strings correctly from day 1. Sorry, I have low
tolerance for sarcasm, especially un-substantiated. Maybe their notion
of concurrency is way off, but overall the language is not that bad.

Good maven:

1. build script that can build and produce a package. So, POM (a lot
of XML that you dislike so much is actually one of the best things
about maven, sorry).
2. centralized repository.
3. unified process. Makes it easy to start with mavenized projects.
4. transitive dependencies.

Bad points:

1. maven's use of snapshot dependencies - it makes builds
non-reproducible ("works on my machine" problem)
2. maven's use of plugins (the reason for this being a bad point is
actually exactly as point 1, since plugins are dependencies)
3. POM is too verbose. Maven's obsession with not using attributes on
the elements is quite annoying. Although if you use an IDE you rarely
have to see any XML.
4. centralized repository - maven's reliance on repository is at times
crippling (especially when you are in a hurry and internet link is
down). Some artifacts in the repository are simply incorrectly
defined, which makes dealing with maven builds a big PITA if you
happened to come across such an artifact.
5. Maven has limited number of scopes (scopes are needed to decide
which transitive dependencies are needed). This often makes it hard to
package products that have optional things enabled/disabled. Although,
this is can be partially mitigated by a plugin (forgot the name of it,
sorry). For example, ivy has pre-defined scopes and allows for custom
ones.

As you can see most of my good points are also bad points (or
partially bad), that is why I am avoiding maven like a plague. That is
not to say it is not a good product. Like with everything else you
learn its good and avoid the bad ones.

Cheers,
Alex.

[Robert Virding]

----- Original Message -----
> From: "Alex Shneyderman" <a.shne...@gmail.com>
> Sent: Friday, 19 October, 2012 3:37:01 PM
>
> > On Fri, Oct 19, 2012 at 4:04 PM, Max Bourinov <bour...@gmail.com>
> > wrote:
> >> Max,
> >>
> >> Please follow this link: http://en.wikipedia.org/wiki/Apache_Maven
> >>
> >
> > It is not an answer. What good have _you_ found in Maven? XML
> > config
> > as everything in Java?
>
> At least java got strings correctly from day 1. Sorry, I have low
> tolerance for sarcasm, especially un-substantiated. Maybe their
> notion
> of concurrency is way off, but overall the language is not that bad.

No, they didn't. They made the size of each character 16 bits which they still can't fit one unicode codepoint into a string character so they have had to add encodings and an extra set of functions for working with encoded strings. So it is messy.

Robert

[Dmitry Belyaev]

This reminds me of cabal. And if I remember correctly Haskell folks found it hard to use by developers and invented cabal-dev that is a lot like rebar.
I personally don't like the idea of managing dependencies outside of concrete application.

The great idea is "Tweak 5" to create index of erlang libs and apps.

--
Dmitry Belyaev

[Max Bourinov]

True. There is kind of unicode, but when it comes to the coding you have to deal with code pages...

Best regards,

Max

[Max Lapshin]

> 1. build script that can build and produce a package. So, POM (a lot
> of XML that you dislike so much is actually one of the best things
> about maven, sorry).

Can it produce package that will run on any platform? No, of course,
because there may be C extensions.
So we either fallback to debian way of building on server farm, or
distribute in source.
In any case there is no explicit need of packages except current
practice of repo on github.

> 2. centralized repository.

Is it good? Why is it good? Why should I go through procedure of
registration of my package when I can just point to proper repo on
github?

> 3. unified process. Makes it easy to start with mavenized projects.

rebar already has it

> 4. transitive dependencies.

rebar already has it.

> 1. maven's use of snapshot dependencies - it makes builds
> non-reproducible ("works on my machine" problem)

rebar has this problem if you don't specify proper version. But you
_can_ specify exact commit
and everything will work ok.

> As you can see most of my good points are also bad points (or
> partially bad), that is why I am avoiding maven like a plague. That is
> not to say it is not a good product. Like with everything else you
> learn its good and avoid the bad ones.
>

So I don't see anything that rebar can take from maven. Really.
So why do you, java guys tell about it if rebar already does everything ok?

[Tim Watson]

I'm glad *someone* reads my (infrequently updated) blog! :)

Ironically, now that I'm looking into whether or not larger projects (like RabbitMQ) can be transitioned to rebar, I'm leaning back towards project local dependency management *precisely because* of the need to tweak version numbers and tags and the like. For example, Rabbit has many numerous components and when working on a feature you might create branches in several projects that need to be tracked in synch with one another. Whilst rebar *doesn't* do that out of the box, adding the functionality is going to be a lot simpler if the dependencies and their config files are bootstrapped locally rather than trying to use $HOME/.rebar_deps or whatever.

[Manuel A. Rubio "Bombadil"]

Hi,

it's possible to use Maven with another languages as Erlang. There
aren't need to modify rebar.

Regads.

El 2012-10-19 14:17, Max Bourinov escribió:
> Max, do you want me re-type article from wikipedia?
>
> So, I try: It is very good package manager that is de-facto standard
> in Java world. Sorry guys I said Java here. So, the idea was to check
> ideas behind it because I believe those Java guys have good ideas..
> Apache guys... So, we can avoid some unnecessary
> design pitfalls... For example maven has idea od snapshot/release
> repositories, and this idea could be worth to implement it in rebar
> packet manager.
>
> But, I suggest you to not be scared by XML. It is only one little
> XML.

> All the rest is text: http://en.wikipedia.org/wiki/Apache_Maven [3]

>
> Best regards,
> Max
>
> On Fri, Oct 19, 2012 at 4:10 PM, Max Lapshin <max.l...@gmail.com
> [4]> wrote:
>
>> On Fri, Oct 19, 2012 at 4:04 PM, Max Bourinov <bour...@gmail.com
>> [1]> wrote:
>> > Max,
>> >
>> > Please follow this link:

>> http://en.wikipedia.org/wiki/Apache_Maven [2]

>> >
>>
>> It is not an answer. What good have _you_ found in Maven?  XML
>> config
>> as everything in Java?
>
>
>

> Links:
> ------
> [1] mailto:bour...@gmail.com
> [2] http://en.wikipedia.org/wiki/Apache_Maven
> [3] http://en.wikipedia.org/wiki/Apache_Maven
> [4] mailto:max.l...@gmail.com

--
Manuel A. Rubio "Bombadil"
Usuario de GNU/Linux #323628 acorde a http://counter.li.org/
Técnico en Admin. Sistemas Informáticos

[Dmitry Kolesnikov]

Hello,

Couple of comments here

1. package manager is a platform dependent. This is a beauty of rebar it works out of box on CentOs/Ubuntu/MacOS/etc.

2. Max said "I don't like the idea to store project dependencies outside project tree". This is true. However I do have number of "common" libraries. I'd like to keep them at central place. Joe's tweak 1 would be useful.

- Dmitry

[Yurii Rashkovskii]

Joe,

The real question is what a "package manager" is supposed to do. What are the problems that it should solve?

My current take is:

1. Library (package) discovery (search)

2. Library location/retrieval

3. Dependencies

4. Library publishing (enabling 1)

After my experience with Agner (http://erlagner.org) I've moved onto a much more lightweight approach I am using in EXPM (http://expm.co) [http://rashkovskii.com/2012/10/01/expm-or-meet-agner-2/]. 

It is essentially an index and, technically speaking, EXPM does not know how to retrieve or build libraries in that index. It does know, however, how to instruct rebar, mix or an scm of choice (git only at the moment) to get the library. If you look at some package, for example http://expm.co/cowboy/head, at the bottom you'll see those automatically generated "instructions". The command line tool is capable of exactly that as well.

I believe that EXPM currently addresses 1 & 2 to a reasonable extent and it has an incomplete support for 3 (dependencies). Not by design, just because it is young. It can figure out the exact versions of dependencies given constraints, but it doesn't do dependencies' dependencies (yet). This can be easily circumvented by writing a simple shell script that'll run expm deps over expm deps output (recursion!)

EXPM also does a fairly good job at 4 — you can easily publish a package with no need to go through a registration procedure (yet your package updates are protected). It isn't without minor flaws, but it works pretty well so far.

I personally think that either rebar or mix are good tools for retrieving dependencies and EXPM or another similar tool to assist with finding what libraries to use and where are they located is a good way to go.

Just my 2c

[Joe Armstrong]

On Fri, Oct 19, 2012 at 2:33 PM, Fred Hebert <mono...@ferd.ca> wrote:
> For anyone interested, there's been an interesting blog post on the topic of
> package managers written at
> http://hyperthunk.wordpress.com/2012/05/28/does-erlangotp-need-a-new-package-management-solution/

That's very interesting. Thanks for the reference

I got to thinking a bit more. ...

The above link has an example command

> git fetch-pack --include-tag -v g...@github.com:hyperthunk/gitfoo.git refs/tags/lager-0.9.4

This appears to fetch a tagged tree without all all other versions -
just what I wanted

At some level of abstraction we need to tag versions of applications.
Using the latest
version is fine for development but not for shipping a product.

If we adopt a common tagging convention then we can do two things.

1) express dependencies (ie foo-1.4.5 depends upon bar-3.2.6)
and
2) make a "release" ie say that a certain set of tagged
applications has been tested and is known
to work together.

At the moment I guess very few Erlang git projects made by rebar have
vI.J.K style tags.


Two more problems remain:

A) Weakening the dependencies

I know what foo-3.4.2 depends upon bar-4.2.5 means - this is
very precise

Should we allow foo-3.* depends upon bar-4.*

There must be some language of dependencies, and conventions
(ie odd major version numbers are experimental,
even stable)

Does anybody have any pointers to a discussion of this.

I'd like a few simple rules that cover the common use cases
rather than fancy rules that cover everything

B) The *one* module name problem

We can't load modules in foo-1.2.3 and foo-2.1.6 into the
same node so if A depends upon
foo-1.2.3 and B depends upon foo-2.1.6 we can't run A and B
in the same node

There are a number of solutions possible:

i) name_munging -> convert xx.erl to xx_foo_1.2.3.erl

It *is* possible - but you have to take care of registered
process names etc. there are a lot of
edge cases that make this tricky.

ii) Don't solve the problem at all. Run A and B in different nodes

My feeling is ii) is correct - we could then define a node as
a release (ie a set of tagged applications)
that are know to work together.

I think problems of /consistency of
modules/security/authentication should be solved at the node level
and not the process/module level

How could one make a release?

a) All contributors must tag versions of their applications and
follow common packaging conventions
b) we need an improved manifest

Maybe volunteers could make releases - actually a release is little
more than a set of applications
that are know to.

Seems we also need to standardize a new an improved manifest file - we
need two types
one that describe applications, the other which describes releases.
These should be super-sets of
the existing descriptions.

What should be in the manifest?

How should things be packaged?

I rather like the "put everything in a zip file (with a changed
extension) and add a top-level
manifest that describes what will be in the zip file" approach.

Again design of the manifest should strike a balance between simple
and intuitive and
"can do everything" -
(a zip file with some code that you do apply on :-) would be too
simple and dangerous)

Cheers

/Joe

[Tuncer Ayaz]

On Tue, Oct 23, 2012 at 9:46 AM, Joe Armstrong <erl...@gmail.com> wrote:
> On Fri, Oct 19, 2012 at 2:33 PM, Fred Hebert <mono...@ferd.ca> wrote:
>> For anyone interested, there's been an interesting blog post on the topic of
>> package managers written at
>> http://hyperthunk.wordpress.com/2012/05/28/does-erlangotp-need-a-new-package-management-solution/
>
> That's very interesting. Thanks for the reference
>
> I got to thinking a bit more. ...
>
> The above link has an example command
>
>> git fetch-pack --include-tag -v g...@github.com:hyperthunk/gitfoo.git
>> refs/tags/lager-0.9.4
>
> This appears to fetch a tagged tree without all all other versions -
> just what I wanted

You can also use "git-archive --remote=<repo>":
http://www.kernel.org/pub/software/scm/git/docs/git-archive.html
$ git archive --remote=git://repo.or.cz/cgit.git --prefix=cgit-0.9.3/ \
-o cgit-0.9.3.tar.gz v0.9.0.3

[Tim Watson]

Dude - long time no speak! :)

That doesn't work when the repository is hosted on github - sadly they have disabled support for `git archive --remote` presumably because it interferes with gathering download statistics for repositories.

Cheers,
T

[Loïc Hoguin]

On 10/19/2012 03:53 PM, Dmitry Belyaev wrote:
> This reminds me of cabal. And if I remember correctly Haskell folks found it hard to use by developers and invented cabal-dev that is a lot like rebar.
> I personally don't like the idea of managing dependencies outside of concrete application.
>
> The great idea is "Tweak 5" to create index of erlang libs and apps.

An index of Erlang projects is really all that's missing. Better if the
index can be queried through rebar.

$ rebar search http
cowboy http://example.org/cowboy
mochiweb http://example.org/mochiweb
...

$ rebar show cowboy
(detailed infos about cowboy project)

--
Loïc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu

[Gleb Peregud]

On Tue, Oct 23, 2012 at 3:14 PM, Loïc Hoguin <es...@ninenines.eu> wrote:
> An index of Erlang projects is really all that's missing. Better if the
> index can be queried through rebar.
>
> $ rebar search http
> cowboy http://example.org/cowboy
> mochiweb http://example.org/mochiweb
> ...
>
> $ rebar show cowboy
> (detailed infos about cowboy project)

http://expm.co/ solves server side part. expm may add a plugin for
rebar to add "search" command. The slight problem with rebar is that
it is mostly used as a local per-repo binary, not as a global one.

[Loïc Hoguin]

On 10/23/2012 07:33 PM, Gleb Peregud wrote:
> On Tue, Oct 23, 2012 at 3:14 PM, Loïc Hoguin <es...@ninenines.eu> wrote:
>> An index of Erlang projects is really all that's missing. Better if the
>> index can be queried through rebar.
>>
>> $ rebar search http
>> cowboy http://example.org/cowboy
>> mochiweb http://example.org/mochiweb
>> ...
>>
>> $ rebar show cowboy
>> (detailed infos about cowboy project)
>
> http://expm.co/ solves server side part. expm may add a plugin for
> rebar to add "search" command. The slight problem with rebar is that
> it is mostly used as a local per-repo binary, not as a global one.

It doesn't solve the server side part. The package information available
is for the most part entirely useless when you are looking for packages.
It does the strict minimum for this purpose: small one-line description
+ keyword list. Plus you have to leave the package list and scroll down
to find the github link where the useful information can be found. It's
trying to do packages, when what I need is an index of projects. The
index page there with a github/project homepage link instead of an
internal link on the name would do wonders.

As for rebar being per-repo, that's not that true anymore. And since
it's pretty much the de facto standard now, it's not really needed now.

--
Loïc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu

[Yurii Rashkovskii]

> http://expm.co/ solves server side part. expm may add a plugin for
> rebar to add "search" command. The slight problem with rebar is that
> it is mostly used as a local per-repo binary, not as a global one.

It doesn't solve the server side part. The package information available
is for the most part entirely useless when you are looking for packages.
It does the strict minimum for this purpose: small one-line description
+ keyword list. Plus you have to leave the package list and scroll down
to find the github link where the useful information can be found. It's
trying to do packages, when what I need is an index of projects. The
index page there with a github/project homepage link instead of an
internal link on the name would do wonders.

All fair points. However, not all projects would have GitHub repositories. I guess what I can do is: in addition to the internal link, show the homepage link if specified, otherwise github if specified, otherwise nothing. Would that make sense?

[Loïc Hoguin]

homepage|'scm repo page' exclusively, projects with neither of these
likely don't exist. :)

Was just discussing with Jose that it would be good to also add links to
documentation pages when they exist.

[Yurii Rashkovskii]

>> All fair points. However, not all projects would have GitHub
>> repositories. I guess what I can do is: in addition to the internal
>> link, show the homepage link if specified, otherwise github if
>> specified, otherwise nothing. Would that make sense?
>
>
> homepage|'scm repo page' exclusively, projects with neither of these likely
> don't exist. :)

SCM repo pages known to me are only an option with github and may be
bitbucket any other? I don't want to support bitbucket yet because
expm doesn't know much about hg yet :)

>
> Was just discussing with Jose that it would be good to also add links to
> documentation pages when they exist.

Good point!

[Loïc Hoguin]

On 10/23/2012 10:41 PM, Yurii Rashkovskii wrote:
>>> All fair points. However, not all projects would have GitHub
>>> repositories. I guess what I can do is: in addition to the internal
>>> link, show the homepage link if specified, otherwise github if
>>> specified, otherwise nothing. Would that make sense?
>>
>>
>> homepage|'scm repo page' exclusively, projects with neither of these likely
>> don't exist. :)
>
> SCM repo pages known to me are only an option with github and may be
> bitbucket any other? I don't want to support bitbucket yet because
> expm doesn't know much about hg yet :)

Yeah but you can fill it as a "homepage" instead of "scm repo" and
people can still get there even if the tool doesn't support it.

--
Loïc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu

[Yurii Rashkovskii]

Loïc,

> Yeah but you can fill it as a "homepage" instead of "scm repo" and people
> can still get there even if the tool doesn't support it.

Deployed :) I found no projects on bitbucket yet so I didn't add
support for it, but homepage and github should work now!

Yurii.

[Tim McNamara]

Hi Joe, looks neat. Some thoughts:

Tweak 3 concerns me slightly. I would like to be able to ask for git
HEAD if required.

Coloured output (and Unicode characters, e.g. ticks) would be another
thing that Erlang/OTP take inspiration from npm, It makes terminal use
surprisingly pleasant.

I feel that the community should take ownership of having its own
index, rather than relying on Github. That would be somewhat more
neutral of different source repositories. See also
http://pypi.python.org/pypi?%3Aaction=index (warning, huge)

[Tim Watson]

Loic

On 23 Oct 2012, at 18:55, Loïc Hoguin <es...@ninenines.eu> wrote:

>>
>> http://expm.co/ solves server side part. expm may add a plugin for
>> rebar to add "search" command. The slight problem with rebar is that
>> it is mostly used as a local per-repo binary, not as a global one.
>
> It doesn't solve the server side part. The package information available is for the most part entirely useless when you are looking for packages. It does the strict minimum for this purpose: small one-line description + keyword list. Plus you have to leave the package list and scroll down to find the github link where the useful information can be found. It's trying to do packages, when what I need is an index of projects. The index page there with a github/project homepage link instead of an internal link on the name would do wonders.
>

That would certainly be helpful.

> As for rebar being per-repo, that's not that true anymore.

How so? I'm still under the impression that this remains true:

- it gets dependencies locally
- resolving dependencies outside base_dir can break things

Maybe I've or become a bit out of touch with the bleeding edge changes. I'm a huge rebar fan/user so please correct me if I'm wrong.

> And since it's pretty much the de facto standard now, it's not really needed now.
>

Are you saying rebar is the de facto standard? I wish it was, and do wonder if inclusion in otp would make it more likely to be.

And what's not really needed? This expm thing, or a package manager in general?

Cheers
Tim

[Loïc Hoguin]

On 10/24/2012 05:10 PM, Tim Watson wrote:
>> As for rebar being per-repo, that's not that true anymore.
>
> How so? I'm still under the impression that this remains true:
>
> - it gets dependencies locally
> - resolving dependencies outside base_dir can break things
>
> Maybe I've or become a bit out of touch with the bleeding edge changes. I'm a huge rebar fan/user so please correct me if I'm wrong.
>
>> And since it's pretty much the de facto standard now, it's not really needed now.
>>
>
> Are you saying rebar is the de facto standard? I wish it was, and do wonder if inclusion in otp would make it more likely to be.
>
> And what's not really needed? This expm thing, or a package manager in general?

I was talking about including rebar inside repositories. It's not really
needed anymore. Haven't had it in cowboy for maybe a year and nobody
seemed to complain. I think it's OK today to tell people to install
rebar (or update their rebar if you need a more recent feature).

[Tim Watson]

Ah ok! Yes I agree with this.

[Ignas Vyšniauskas]

On 10/23/2012 10:41 PM, Yurii Rashkovskii wrote:

> SCM repo pages known to me are only an option with github and may be
> bitbucket any other? I don't want to support bitbucket yet because
> expm doesn't know much about hg yet :)

bitbucket supports git since ages[1]?

http://gitorious.org/ is also quite used, in particular by (very) FOSS-y
people.

Ignas

[1]: http://blog.bitbucket.org/2011/10/03/bitbucket-now-rocks-git/

[Yurii Rashkovskii]

All I am saying is that at the moment, none of packages on expm use
neither gitorious, nor bitbucket (with either git or hg option). They
are welcome to use it, as soon as somebody will use bitbucket: or hg:
option for a package on expm.co, I'll add better support for these...

[Ignas Vyšniauskas]

On 10/23/2012 09:46 AM, Joe Armstrong wrote:
> The above link has an example command
>
>>> git fetch-pack --include-tag -v g...@github.com:hyperthunk/gitfoo.git
refs/tags/lager-0.9.4

Wouldn't --depth=1 make it even better? Do we need all of the history here?

Ignas

[Tim Watson]

On 30 Oct 2012, at 09:41, Ignas Vyšniauskas wrote:

> On 10/23/2012 09:46 AM, Joe Armstrong wrote:
>> The above link has an example command
>>
>>>> git fetch-pack --include-tag -v g...@github.com:hyperthunk/gitfoo.git
> refs/tags/lager-0.9.4
>
> Wouldn't --depth=1 make it even better? Do we need all of the history here?
>

Indeed we do not, though I've not tried that. Someone also mentioned to me offline (a while ago) that they were concerned about using git as a backend rather than adopting something that is not only distributed but actually has mirrors in place, e.g., apt/yum or whatever. Stashing things using git simply seemed like a *free* way to get started when I was first looking at it.

So, I'm fairly sure we're not going to figure out a way to do this that pleases everybody. And *even* if the package management and distribution problem was neatly solved, there would be cases where you might actually *want* source code dependencies instead. Another reason we looked at git was because once you've built everything, if you're resolving dependencies from a central location (like $HOME/repo or whatever) then you've got to deploy them as a snapshot each time they're built and so on. This stuff actually gets quite involved - using say maven, which is actually very good at managing this, it can still make the development process over complicated at times.

Take an example project: RabbitMQ. You're building multiple projects that depend on each other, and the bug you're fixing involves making changes to three different components, each of which lives in its own repository. How're you going to figure out which projects need to be built if they're all completely discrete by the time they go into the repository? Of course its easy to solve dependencies with make and rebar will compile *including* dependencies by default, which goes a long way to making this situation workable in practise.

Of course what Joe is really talking about is the ability to install *other people's work* for which you have absolutely no (or maybe just very little) interest in the source code. I typically don't care how a particular bit of cowboy or webmachine or whatever work, as long as it does I just ignore it.

And this problem cannot be *that hard* to solve, because you know what - stuff like rubygems/bundler is really damn useful, as it PyPI. I'm sure the node thing is great, though I've never looked. Even OCaml Godi comes in useful and despite the vitriol I hear about it, cabal+hackage is invaluable. Point is, these things make it easy to get stuff done, thought they have gaps, flaws and whatnot - they still add a lot of value.

Oh and BTW if someone does come up with a system to do this, then it *should* just be packaged as a rebar plugin, as there's no need to change rebar at all in order to override things like these.

Cheers,
Tim

[Andrew Berman]

I come from the Java world and I'm sorta blown away that Erlang doesn't have something comparable to JAR files, which work just fine.  Everyone needs to agree on a standard for which to package the applications.  JARs are just zips of class files with config files, so why not start there and just have a zip file with the BEAMs and priv directory with configs.  Then we need erl to be able to read these zip files and use them as libraries without having to unzip them.  I think we need to just start simple and then evolve over time if we need to.

--Andrew

[Joe Armstrong]

On Tue, Oct 30, 2012 at 6:31 PM, Joe Armstrong <erl...@gmail.com> wrote:

> On Tue, Oct 30, 2012 at 6:17 PM, Andrew Berman <rex...@gmail.com> wrote:
>> I come from the Java world and I'm sorta blown away that Erlang doesn't have
>> something comparable to JAR files, which work just fine.
>

> Well it does actually - but this is not stunning well documented.
>
> In the manual page for code there is a section entited
>
> LOADING OF CODE FROM ARCHIVE FILES

My mail got sent before I'd completed it ...

programs like rebar use these features

/Joe

[Andrew Berman]

Perfect, however, I see it's still experimental.  I guess people just need to start using .ez files then so it's more used and tested.  We should probably have rebar be able to read a rebar.config file within the ez file to determine dependencies, no?  

[Loïc Hoguin]

What I don't understand is what you would use these for? It's useful for
tools like rebar because it's more a script with dependencies (all
packaged in one file) than a full system, but it doesn't make sense for
a full system which is generally done using releases (and these bundle
the erl executable).

I suppose releases could be packaging the applications they contain but
what for? That's just an extra unnecessary step there.

Some way to package releases could be cool for some things, but that
wouldn't be the same as what is done in the Java world. Production
systems don't need Erlang installed to run Erlang systems, the Erlang
system is included in your release.

> <watson....@gmail.com <mailto:watson....@gmail.com>>

> >>> erlang-q...@erlang.org <mailto:erlang-q...@erlang.org>

> >>> http://erlang.org/mailman/listinfo/erlang-questions
> >>>
> >>
> >>
> >> _______________________________________________
> >> erlang-questions mailing list

> >> erlang-q...@erlang.org <mailto:erlang-q...@erlang.org>

> >> http://erlang.org/mailman/listinfo/erlang-questions
> >>
>
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions
>

--
Loïc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu

[Ola Andersson A]

Hi,

I've been experimenting a little with escripts including code archives for test purposes.
It's a really useful feature that seems to work nicely at least this far.

The only stumbling block I have seen is when an application included in the archive uses C drivers.
When the path (code:priv_dir(App)) provided to e.g. erl_ddll points into a code archive it will result in an error.

I fixed it with a hack that extracted the driver to a temp directory and patched the app to read the driver from the temp directory and then remove it.
Not something I would use in production code.

Are there any plans to add the possibility to load a driver directly from an archive?

BR
/OLA.

[Tim Watson]

I think you'll find that constraint is enforced by the operating system, rather than the erlang runtime. If operating systems allowed you to dynamically load a library from a process address space, that would open up a massive security hole.

[Ola Andersson A]

Yes, I suppose that would allow for a few interesting possibilities. ;-)

An improvement would be to make it configurable in OTP apps where to find the included drivers.
application:set_env could be used for that purpose.
Of course it's probably not realistic to implement this change in every OTP app with C drivers.

Any other ideas?
/OLA.

[Tim Watson]

Hi

On 31 Oct 2012, at 13:57, Ola Andersson A wrote:

> Yes, I suppose that would allow for a few interesting possibilities. ;-)
>
> An improvement would be to make it configurable in OTP apps where to find the included drivers.
> application:set_env could be used for that purpose.
> Of course it's probably not realistic to implement this change in every OTP app with C drivers.
>
> Any other ideas?

There's nothing to stop applications from doing the same workaround you're already using. If a common library added support for this, then everyone could share the same code to achieve it as well. One thing we discussed on the erlware mailing list when this idea was being knocked around was using .ez as one of the distribution formats. Certainly https://github.com/hyperthunk/rebar_dist_plugin supports this packaging method, although that plugin may not work with the latest version of rebar as I've not had time to bring it up to date. Now that rebar is moving properly over to semver, it'll be easier to manage that sort of thing.

[Igor Karymov]

Lock files it's good idea. We are using simple script for this purpose.

When you want create release branch just type create release. And script do recursive walk and lock dependency current version in root rebar.config.

All another time moving head most usable for our development process.

пятница, 19 октября 2012 г., 15:10:52 UTC+4 пользователь José Valim написал:

I believe one of the issues that would need to be addressed first in rebar is the one of repeatability.

rebar does not ensure that developers in the same project are going to use the same snapshot of a dependency. This means developers can depend on the same git repository but different commits leading to the famous "the tests pass on my machine".

I am aware this could be fixed by the team enforcing a commit sha when declaring the dependency but many simply don't do it. Besides it puts the work on you to manage the commit shas. If you want to update the dependency, you need to remove the commit sha, update the repo, write the new commit sha.

This could be easily fixed by having a lock file per project with the commit shas of the dependencies. "rebar get-deps" should retrieve the shas in such lock, if one is available, or simply get the latest and write to the lock. A command like "rebar unlock-deps" could be available when developers want to update their dependencies, regardless of the current lock.

Also, regarding Joe's proposal, picking up a version from a git repository may be tricky. The best approach is to index git tags only, hoping the publisher won't modify an existing tag. If someone is depending on a branch though, it should not be considered a version or a part of .erl_packages. I believe it should still belong to the repo.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer