Re: [erlang-questions] Rebar dependency recursion

[Tuncer Ayaz]

On Thu, Nov 1, 2012 at 5:47 PM, Eric Merrit wrote:
> Guys,
>
> I have been talking with Tuncer and seeing the various bits on going
> on about handling dependency recursion in rebar. That is `skip_deps`
> vs `-r`. When you mix into that things like config inheritance it
> becomes a mess. I thought I would see if I can get some consensus on
> the matter. Just as a short FYI, here are some issues on this topic.
>
> * https://github.com/basho/rebar/issues/303
> * https://github.com/basho/rebar/issues/275
> * https://github.com/basho/rebar/pull/293 - (this has been merged)
>
> I think the default case in rebar is that you *do not want it to
> recurse*. Only in the case of a `get-deps` followed by a `compile`
> do you want to do things recursively. This is almost always true in
> my case, other`s experiences may vary.
>
> That being the case I think the default behavior should be the
> non-recursive option and you explicitly tell rebar to recurse with
> the `-r`. As a variation it will probably be worthwhile to support a
> list of applications to talk about app specific recursion. In this
> model `-r`/`--recursive` would do recursion as it normally does
> while `--recursive=app1,app2,app3` would only recurse into app1,
> app2 and app3 respectively. There is a branch implementing basic
> -r/--recursive linked in rebar/issues/303 (dss-fix-skip-deps).
>
> This seems to be the most simplest solution to the problem and
> doesn't really introduce new semantics though it does inverse
> semantics. The downside to this is that its backwards incompatible.
> However, we are coming up on a new major version so it's a good
> opportunity to do backwards incompatible changes. I don't see a
> solid way for rebar to automatically decide which tasks to carry out
> recursively and which not considering you can have many variations
> of commands to run in a single rebar invocation.
>
> In any case, it would be nice to settle the issue, with either
> `skip_deps` is the way going forward or we will be moving to the
> `-r`. Obviously, I would like to see the `-r` option, but settling
> the issue on way or the other is best.

All solutions considered so far, I think introducing an explicit
-r/--recursive as found in dizzy's dss-fix-skip-deps branch is a good
solution.

If someone can come up with a solid way to make rebar aware of what
commands are going to be run and use that as a way to dynamically
enable/disable recursion, I'd be very interested to read that. I hope
there is a better solution which keeps the ease of use for the initial
'get-deps compile' step while not forcing us to use skip_deps and/or
apps=/skip_apps= afterwards. Config inheritance is a related problem
and discussed in the above ticket #275.
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Seth Falcon]

On Nov 1, 2012, at 10:07 AM, Tuncer Ayaz wrote:

> On Thu, Nov 1, 2012 at 5:47 PM, Eric Merrit wrote:
>> That being the case I think the default behavior should be the
>> non-recursive option and you explicitly tell rebar to recurse with
>> the `-r`. As a variation it will probably be worthwhile to support a
>> list of applications to talk about app specific recursion. In this
>> model `-r`/`--recursive` would do recursion as it normally does
>> while `--recursive=app1,app2,app3` would only recurse into app1,
>> app2 and app3 respectively. There is a branch implementing basic
>> -r/--recursive linked in rebar/issues/303 (dss-fix-skip-deps).

I'm +1 on a move to -r over skip_deps.

Has there been discussion on the idea of commands defining recursive behavior themselves?

There are some commands where recursion is never correct (e.g. creating a template or generating a release). And commands where I think lack recursion needs to be the default like get-deps. The challenge is that there are a number of commands where both behaviors are reasonable depending on your use (compile, clean, and eunit in particular).

One idea for keeping some of the ease for initial build would be to provide two versions of some commands so that they can be composed:

rebar get-deps rcompile eunit

recursive: get-deps rcompile
non-recursive: eunit

If we don't arrive at a compromise and have to pick either skip_deps or -r, then I strongly favor -r. In either case, you may not get what you wanted. With default being recursive you will sometimes get a completely broken thing (create template) whereas with default non-recursive, some of the right thing will have happened.

+ seth

[Tim Watson]

This is all fine, but it's mighty annoying to have run rebar repeatedly because you want to get-deps first then do something else.

Modules already have a way to indicate that they want recursion - preprocess/2. Can this be extended so that modules can say 'I want recursion' or [predirs] or ok/nothing? Then you'd get a nice interplay where rebar_deps says [predirs] and rebar_compile says 'yes please' to the recursion into those predirs but rebar_templater says 'no thanks' to them and you can chain commands cleanly without invoking rebar multiple times.

In that scheme, -r means 'override the guys who said no to recrusion'

Sent from my iPhone.

[Fredrik Linder]

-r: +1

/Fredrik

[Jared Morrow]

I'd argue that get-deps & compile are the most common functions, and if this proposal is to make things more consistent, making those commands (and others that should almost always have recursion) magically recurse without the '-r' passed in would be bad.  So all the "-r +1's" people are saying should count towards people who really want to write, "rebar -r compile" everytime they compile.  Also, you still won't be able to mix recursive and non recursive commands in one call.   

-J

[Loïc Hoguin]

So we need two executables.

On 11/01/2012 10:02 PM, Jared Morrow wrote:
> I'd argue that get-deps & compile are the most common functions, and if
> this proposal is to make things more consistent, making those commands
> (and others that should almost always have recursion) magically recurse
> without the '-r' passed in would be bad. So all the "-r +1's" people
> are saying should count towards people who really want to write, "rebar
> -r compile" everytime they compile. Also, you still won't be able to
> mix recursive and non recursive commands in one call.
>
> -J
>
> On Thu, Nov 1, 2012 at 1:10 PM, Fredrik Linder <fredrik...@gmail.com
> <mailto:fredrik...@gmail.com>> wrote:
>
> -r: +1
>
> /Fredrik
>
> On 1 nov 2012, at 12:00, Tim Watson <watson....@gmail.com

> <mailto:watson....@gmail.com>> wrote:
>
> > This is all fine, but it's mighty annoying to have run rebar
> repeatedly because you want to get-deps first then do something else.
> >
> > Modules already have a way to indicate that they want recursion -
> preprocess/2. Can this be extended so that modules can say 'I want
> recursion' or [predirs] or ok/nothing? Then you'd get a nice
> interplay where rebar_deps says [predirs] and rebar_compile says
> 'yes please' to the recursion into those predirs but rebar_templater
> says 'no thanks' to them and you can chain commands cleanly without
> invoking rebar multiple times.
> >
> > In that scheme, -r means 'override the guys who said no to recrusion'
> >
> > Sent from my iPhone.
> >
> > On 1 Nov 2012, at 17:07, Tuncer Ayaz <tunce...@gmail.com

> >> erlang-q...@erlang.org <mailto:erlang-q...@erlang.org>

> >> http://erlang.org/mailman/listinfo/erlang-questions
> > _______________________________________________
> > erlang-questions mailing list

> > erlang-q...@erlang.org <mailto:erlang-q...@erlang.org>

> > http://erlang.org/mailman/listinfo/erlang-questions
> _______________________________________________
> erlang-questions mailing list

> erlang-q...@erlang.org <mailto:erlang-q...@erlang.org>

> http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions
>

--
Loïc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu

[Jared Morrow]

Do you mean two calls to the executable or literally two executables?

I would strongly -1 the latter.  

-J

On Thu, Nov 1, 2012 at 3:03 PM, Loïc Hoguin <es...@ninenines.eu> wrote:

So we need two executables.

On 11/01/2012 10:02 PM, Jared Morrow wrote:

I'd argue that get-deps & compile are the most common functions, and if
this proposal is to make things more consistent, making those commands
(and others that should almost always have recursion) magically recurse
without the '-r' passed in would be bad.  So all the "-r +1's" people
are saying should count towards people who really want to write, "rebar
-r compile" everytime they compile.  Also, you still won't be able to
mix recursive and non recursive commands in one call.

-J

On Thu, Nov 1, 2012 at 1:10 PM, Fredrik Linder <fredrik...@gmail.com

<mailto:fredrikelinder@gmail.com>> wrote:

    -r: +1

    /Fredrik

    On 1 nov 2012, at 12:00, Tim Watson <watson....@gmail.com

     >> erlang-q...@erlang.org <mailto:erlang-questions@erlang.org>

     >> http://erlang.org/mailman/listinfo/erlang-questions
     > _______________________________________________
     > erlang-questions mailing list

     > erlang-q...@erlang.org <mailto:erlang-questions@erlang.org>

     > http://erlang.org/mailman/listinfo/erlang-questions
    _______________________________________________
    erlang-questions mailing list

    erlang-q...@erlang.org <mailto:erlang-questions@erlang.org>

    http://erlang.org/mailman/listinfo/erlang-questions




_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

--
Loďc Hoguin

[Tuncer Ayaz]

On Thu, Nov 1, 2012 at 10:02 PM, Jared Morrow <ja...@basho.com> wrote:
> I'd argue that get-deps & compile are the most common functions, and if this
> proposal is to make things more consistent, making those commands (and
> others that should almost always have recursion) magically recurse without
> the '-r' passed in would be bad. So all the "-r +1's" people are saying
> should count towards people who really want to write, "rebar -r compile"
> everytime they compile. Also, you still won't be able to mix recursive and
> non recursive commands in one call.

The idea is that you only want to recursively compile on get-deps,
update-deps, or if you have sub_dirs and want to compile all
apps.

[Tuncer Ayaz]

On Thu, Nov 1, 2012 at 8:00 PM, Tim Watson wrote:
> This is all fine, but it's mighty annoying to have run rebar
> repeatedly because you want to get-deps first then do something
> else.

The idea with dss-fix-skip-deps in its current form is that you only
initially would do 'rebar -r g-d com'.

> Modules already have a way to indicate that they want recursion -
> preprocess/2. Can this be extended so that modules can say 'I want
> recursion' or [predirs] or ok/nothing? Then you'd get a nice
> interplay where rebar_deps says [predirs] and rebar_compile says
> 'yes please' to the recursion into those predirs but rebar_templater
> says 'no thanks' to them and you can chain commands cleanly without
> invoking rebar multiple times.
>
> In that scheme, -r means 'override the guys who said no to
> recrusion'

So your idea is that commands should explicitly announce if they want
to be used recursively and otherwise ignore non-base_dir applications
with -r meaning "force all commands to process deps/sub_dirs", right?
How would you handle the case that compile is happy to process
sub_dirs/deps but you want to compile just base_dir?

[Tim Watson]

I'd argue that that's a case of consistency over practicality. Now what was it Oscar Wilde was saying... ;)

Sent from my iPhone.

[Tim Watson]

On 1 Nov 2012, at 21:50, Tuncer Ayaz <tunce...@gmail.com> wrote:

>>
>
> So your idea is that commands should explicitly announce if they want
> to be used recursively and otherwise ignore non-base_dir applications
> with -r meaning "force all commands to process deps/sub_dirs", right?
> How would you handle the case that compile is happy to process
> sub_dirs/deps but you want to compile just base_dir?

I think we need to go back to 'it just works' in that instance. Compile wants to recurse because headers in sub dirs might have changed and dependency checking and compiling based on need/dependency is 'the right thing to do' always.

My thought was that some actions have to take place recursively on order to be correctly applied, and it's no good letting people override them. For something like test runs, doc building, xref, and so on, it can and should be optional. But compile/resolve shouldn't be. Adding that API to core would mean that internal and plugin modules get a means to enforce the right behaviour.

[Tuncer Ayaz]

On Fri, Nov 2, 2012 at 2:55 AM, Tim Watson wrote:

>
> On 1 Nov 2012, at 21:50, Tuncer Ayaz wrote:
>
>> So your idea is that commands should explicitly announce if they
>> want to be used recursively and otherwise ignore non-base_dir
>> applications with -r meaning "force all commands to process
>> deps/sub_dirs", right? How would you handle the case that compile
>> is happy to process sub_dirs/deps but you want to compile just
>> base_dir?
>
> I think we need to go back to 'it just works' in that instance.
> Compile wants to recurse because headers in sub dirs might have
> changed and dependency checking and compiling based on
> need/dependency is 'the right thing to do' always.
>
> My thought was that some actions have to take place recursively on
> order to be correctly applied, and it's no good letting people
> override them. For something like test runs, doc building, xref, and
> so on, it can and should be optional. But compile/resolve shouldn't
> be. Adding that API to core would mean that internal and plugin
> modules get a means to enforce the right behaviour.

This could work. I'll let you all know once the prototype is ready
for testing.

[Tim Watson]

Dude...

On 4 Nov 2012, at 23:56, Tuncer Ayaz wrote:
> On Fri, Nov 2, 2012 at 2:55 AM, Tim Watson wrote:

>> [snip]

> This could work. I'll let you all know once the prototype is ready
> for testing.

You rock! :)

Once you give the say so, I'll test all my plugins + projects against your branch.

Cheers,
Tim

[Tuncer Ayaz]

Here's a quick prototype you can test:
https://github.com/tuncer/rebar/compare/recursion
It's not ready for merging and some of the internals are subject to
change.

We also have to discuss config inheritance and whether it should
be made explicit or removed.

[Tim Watson]

On 5 Nov 2012, at 23:24, Tuncer Ayaz wrote:

> On Mon, Nov 5, 2012 at 12:13 PM, Tim Watson wrote:
>> Dude...
>>
>> On 4 Nov 2012, at 23:56, Tuncer Ayaz wrote:
>>> On Fri, Nov 2, 2012 at 2:55 AM, Tim Watson wrote:
>>>> [snip]
>>> This could work. I'll let you all know once the prototype is ready
>>> for testing.
>>
>> You rock! :)
>>
>> Once you give the say so, I'll test all my plugins + projects
>> against your branch.
>
> Here's a quick prototype you can test:
> https://github.com/tuncer/rebar/compare/recursion
> It's not ready for merging and some of the internals are subject to
> change.
>
> We also have to discuss config inheritance and whether it should
> be made explicit or removed.

Great, I'll have a look at that after work today. I'm not married to the config inheritance either way, though the ability to override the config in sub_dirs using plugins is important to me.

[Tim Watson]

Works nicely for me. I didn't find bugs in any projects I tested again and wrote a minimal plugin to test of the two behaviours an they seemed to work as expected. Nice one!

Sent from my iPhone.

[Tuncer Ayaz]

On Tue, Nov 13, 2012 at 8:45 AM, Tim Watson wrote:
> Works nicely for me. I didn't find bugs in any projects I tested
> again and wrote a minimal plugin to test of the two behaviours an
> they seemed to work as expected. Nice one!

Great, thanks for testing!

> On 5 Nov 2012, at 23:24, Tuncer Ayaz wrote:
>
>> On Mon, Nov 5, 2012 at 12:13 PM, Tim Watson wrote:
>>> Dude...
>>>
>>> On 4 Nov 2012, at 23:56, Tuncer Ayaz wrote:
>>>> On Fri, Nov 2, 2012 at 2:55 AM, Tim Watson wrote:
>>>>> [snip]
>>>> This could work. I'll let you all know once the prototype is ready
>>>> for testing.
>>>
>>> You rock! :)
>>>
>>> Once you give the say so, I'll test all my plugins + projects
>>> against your branch.
>>
>> Here's a quick prototype you can test:
>> https://github.com/tuncer/rebar/compare/recursion
>> It's not ready for merging and some of the internals are subject to
>> change.
>>
>> We also have to discuss config inheritance and whether it should be
>> made explicit or remove