combining core-dev and contrib-apps-dev

[Tobias McNulty]

I'm trying to create a proof-of-concept patch for a big change in rapidsms (the route process).  It spans both rapidsms-core-dev and rapidsms-contrib-apps-dev.

With the current setup there's no easy way to track these changes as a singular unit.  Instead, I have to create a feature branch in each repository and push changes to each, and keep my submodule reference up to date in the meantime.  It effectively doubles all the commits I make to contrib-apps-dev.

In other words, it means I waste more time on logistics and making sure everything's pointing to the right place, instead of actually writing code.

And when it comes time to review my patch, there are two places to look, instead of just one.

Is it really worth it?  I'm still not convinced that the benefits of keeping them separate outweigh the cost.  It's great that I can push directly to contrib-apps-dev and all, but I'm going to ask before I push anything anyways, and honestly I'd much rather just send someone a patch and have them push it than keep dealing with git submodules.  The fact that it's even an issue suggests that they're more of a single unit anyways; after all, they /are/ part of the same Python package and we're basically required to keep them in sync (rapidsms.contrib is not 3rd party software).

Thanks.

Tobias

--
Tobias McNulty
Caktus Consulting Group, LLC
P.O. Box 1454
Carrboro, NC 27510
USA: +1 (919) 951-0052
http://www.caktusgroup.com

[Kevin Samuel]

Maybe we should have a clear separation between "official" contrib apps,
from the dev team. Apps that are not required, but that most of the
users will use (like auth in Django). They could lie in the repo.

And the other ones will be kept in the separate repo.

> --
> You received this message because you are subscribed to the Google
> Groups "rapidsms" group.
> To post to this group, send email to rapi...@googlegroups.com.
> To unsubscribe from this group, send email to
> rapidsms+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/rapidsms?hl=en.

[Tobias McNulty]

On Mon, Aug 16, 2010 at 4:51 AM, Kevin Samuel <info.k...@googlemail.com> wrote:

Maybe we should have a clear separation between "official" contrib apps,
from the dev team. Apps that are not required, but that most of the
users will use (like auth in Django). They could lie in the repo.

And the other ones will be kept in the separate repo.

I'm not exactly sure what you mean, but there is rapidsms-community-apps-dev.

Tobias

[Tobias McNulty]

Oh, another logical flaw in the separation of core-dev and contrib-apps-dev:

I can commit code to contrib-apps-dev, but I can't update the submodule reference in core-dev.  So, I need an actual core developer to step in and effectively "authorize" the commits anyways (by updating the submodule reference for me).

Tobias

[Evan Wheeler]

On Mon, Aug 16, 2010 at 2:38 AM, Tobias McNulty <tob...@caktusgroup.com> wrote:
> I'm trying to create a proof-of-concept patch for a big change in rapidsms
> (the route process).  It spans both rapidsms-core-dev and
> rapidsms-contrib-apps-dev.
> With the current setup there's no easy way to track these changes as a
> singular unit.  Instead, I have to create a feature branch in each
> repository and push changes to each, and keep my submodule reference up to
> date in the meantime.  It effectively doubles all the commits I make to
> contrib-apps-dev.
> In other words, it means I waste more time on logistics and making sure
> everything's pointing to the right place, instead of actually writing code.
> And when it comes time to review my patch, there are two places to look,
> instead of just one.
> Is it really worth it?  I'm still not convinced that the benefits of keeping
> them separate outweigh the cost.  It's great that I can push directly to
> contrib-apps-dev and all, but I'm going to ask before I push anything
> anyways, and honestly I'd much rather just send someone a patch and have
> them push it than keep dealing with git submodules.  The fact that it's even
> an issue suggests that they're more of a single unit anyways; after all,
> they /are/ part of the same Python package and we're basically required to
> keep them in sync (rapidsms.contrib is not 3rd party software).

I am strongly in favor of keeping them separate. I think that any
features that require synchronous changes to both will continue to be
rare and we should be strengthening the division between app-land and
core-land rather than encouraging the lack of separation that has
plagued the project in the past (perhaps because everything was in a
single repository?).

I agree that developing with submodules is very annoying. Personally,
I find it easier to clone the repositories separately (you can clone
one within the other or symlink them side-by-side) and only deal with
submodules periodically.

evan

[Tobias McNulty]

On Tue, Aug 17, 2010 at 10:53 AM, Evan Wheeler <ewhe...@unicef.org> wrote:

I am strongly in favor of keeping them separate. I think that any
features that require synchronous changes to both will continue to be
rare and we should be strengthening the division between app-land and
core-land rather than encouraging the lack of separation that has
plagued the project in the past (perhaps because everything was in a
single repository?).

I'm not sure I understand how it changes the code if they're split into separate repositories.  For that matter, I'm not sure I see what separation you're talking about: the project skeleton (in core) includes pointers to a number of contrib apps, and all the contrib apps (obviously) depend on core-dev.  Furthermore, they're part of the same Python package, so we have no choice but to keep them in sync.

What are the specifics of the separation we're trying to achieve by keeping them in separate repositories?  What kind of lack of separation has caused issues in the past?

Thanks,

[Evan Wheeler]

On Tue, Aug 17, 2010 at 11:10 AM, Tobias McNulty <tob...@caktusgroup.com> wrote:
> On Tue, Aug 17, 2010 at 10:53 AM, Evan Wheeler <ewhe...@unicef.org> wrote:
>>
>> I am strongly in favor of keeping them separate. I think that any
>> features that require synchronous changes to both will continue to be
>> rare and we should be strengthening the division between app-land and
>> core-land rather than encouraging the lack of separation that has
>> plagued the project in the past (perhaps because everything was in a
>> single repository?).
>
> I'm not sure I understand how it changes the code if they're split into
> separate repositories.  For that matter, I'm not sure I see what separation
> you're talking about: the project skeleton (in core) includes pointers to a
> number of contrib apps, and all the contrib apps (obviously) depend on
> core-dev.  Furthermore, they're part of the same Python package, so we have
> no choice but to keep them in sync.
> What are the specifics of the separation we're trying to achieve by keeping
> them in separate repositories?  What kind of lack of separation has caused
> issues in the past?

In the past, we've had a silly proliferation of rapidsms forks. Part
of the reason is that in order to make a change to something like
locations templates or the old reporters app -- you had to fork the
entire repository (core code, app code and all). Since you've already
forked the whole thing and you are in the field hacking on a
deployment, it was very easy to hack on other apps and hack on the
core. Apps are no longer treated as separate pieces of functionality
that work together -- they become a large interconnected stew. Very
little is reusable. Afterwards, many useful little changes may be
bundled with deployment-specific hackery in large commits and never
get shared. Now, one only needs to fork the contrib repository and can
avoid the temptation of mucking things up.

Even though its a subtle and in some ways meaningless separation, as
you point out, the repository division helps promote thinking about
the rapidsms framework and its app ecosystem as different things --
not just a big hot mess called rapidsms. Hopefully this will subtly
encourage us to focus on making apps that are not unnecessarily
dependent on other apps so that they might be reusable.

Finally, the annoyance of separate repositories only affects a small
portion of the community. The pip installer is a single step whether
it downloads one repository or two behind the scenes. Most rapidsms
app developers will be authoring and maintaining their own apps in
their own repositories. Some may contribute improvements to the
contrib apps. Fewer still will work on core framework code. Each of
these constituent groups (app developers, contrib developers, core
developers) have a different interests and, if the community continues
to grow, each will have its own champions and contributors independent
of the others.

Best,
Evan

[Tobias McNulty]

On Wed, Aug 18, 2010 at 7:32 PM, Evan Wheeler <ewhe...@unicef.org> wrote:

In the past, we've had a silly proliferation of rapidsms forks. Part

of the reason is that in order to make a change to something like
locations templates or the old reporters app -- you had to fork the
entire repository (core code, app code and all). Since you've already
forked the whole thing and you are in the field hacking on a
deployment, it was very easy to hack on other apps and hack on the
core. Apps are no longer treated as separate pieces of functionality
that work together -- they become a large interconnected stew. Very
little is reusable. Afterwards, many useful little changes may be
bundled with deployment-specific hackery in large commits and never
get shared. Now, one only needs to fork the contrib repository and can
avoid the temptation of mucking things up.

You shouldn't be forking anything, unless you really want to contribute something back to whatever you forked.  It sounds to me like what's happened in the past is a product of tight timelines, bad habits, or both, and not necessarily something that will be changed by arbitrarily separating a given Python package into two separate code repositories.

Even though its a subtle and in some ways meaningless separation, as
you point out, the repository division helps promote thinking about
the rapidsms framework and its app ecosystem as different things --
not just a big hot mess called rapidsms. Hopefully this will subtly
encourage us to focus on making apps that are not unnecessarily
dependent on other apps so that they might be reusable.

If you really want people to think about them as separate things (they're not right now, they have a number of interdependencies), then put the contrib apps in a separate Python package altogether and remove /all/ links to them from the core.  Honestly I don't think this is a very productive approach, because the contrib apps are useful, but it is a very straightforward way of achieving what you're talking about.

 

Finally, the annoyance of separate repositories only affects a small
portion of the community. The pip installer is a single step whether
it downloads one repository or two behind the scenes. Most rapidsms
app developers will be authoring and maintaining their own apps in
their own repositories. Some may contribute improvements to the
contrib apps. Fewer still will work on core framework code. Each of
these constituent groups (app developers, contrib developers, core
developers) have a different interests and, if the community continues
to grow, each will have its own champions and contributors independent
of the others.

It affects a huge portion of the people working on RapidSMS itself, and I think anything we can do to make volunteers' time better spent will go a long way to ensuring a cohesive product that people enjoy contributing to.

Cheers,

Tobias

--
Tobias McNulty, Managing Partner

Caktus Consulting Group, LLC
http://www.caktusgroup.com

[Tobias McNulty]

On Fri, Aug 20, 2010 at 2:11 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:

It affects a huge portion of the people working on RapidSMS itself, and I think anything we can do to make volunteers' time better spent will go a long way to ensuring a cohesive product that people enjoy contributing to.

Maybe I'm just an idiot, but I wasted another half hour this evening trying to figure out how to resolve submodule conflicts in git.  In the end I just reverted my change and re-merged, because I couldn't figure out a clean way to do it otherwise.  All the solutions I found seemed geared towards giving the change in one's local branch priority, when I wanted the inverse.

What do others think?  In general, I'm still of the opinion that the contrib submodule adds little value to the project at the expense of a lot of pain for both developers and patch reviewers.  If avoiding a "big hot mess" is what we need, I think that falls more on the core devs (and the rest of the community) to do in-depth code reviews, rather than any particular method of slicing up the code.

Yours,

Tobias

[Colin Copeland]

On Fri, Aug 27, 2010 at 12:44 AM, Tobias McNulty <tob...@caktusgroup.com> wrote:

What do others think?  In general, I'm still of the opinion that the contrib submodule adds little value to the project at the expense of a lot of pain for both developers and patch reviewers.  If avoiding a "big hot mess" is what we need, I think that falls more on the core devs (and the rest of the community) to do in-depth code reviews, rather than any particular method of slicing up the code.

I agree, I believe that the core/contrib separation actually makes it harder to work on a patch. At least, I find that I have to jump through more hoops to actually test and submit the changes. I tried to record the method I used to submit my last patch here:

http://docs.rapidsms.org/Development/Contributing/Example

There are a lot of steps and, possibly due to my unfamiliarity with git, changes to submodules can easily be lost or put you in a weird state unless you do everything correctly the first time. I don't know if this is the best method, but I figured I'd document it as a possible set of steps to submit a patch via GitHub. What does everyone else do? For me, the separation makes it harder to follow trunk (unless I want to introduce git submodules into my project dependency list), which is fairly easy with Django [1].

Further, the Hudson build server won't pick up contrib changes until a core dev updates the submodules, right? So contrib may receive several breaking pushes (from core or non-core devs), but the build server wont see the break until a core dev updates the reference in rapidsms-core-dev. Then the build breaks and emails core devs about failures introduced not in the most recent core commit, but rather commits made a while back in contrib.

Regards,

Colin

[1] http://docs.djangoproject.com/en/dev/topics/install/?#installing-the-development-version

 

Yours,

Tobias

--
Tobias McNulty, Managing Partner

Caktus Consulting Group, LLC
http://www.caktusgroup.com

[Evan Wheeler]

On Fri, Aug 27, 2010 at 11:44 AM, Colin Copeland
<cop...@caktusgroup.com> wrote:
> On Fri, Aug 27, 2010 at 12:44 AM, Tobias McNulty <tob...@caktusgroup.com>
> wrote:
>>
>> What do others think?  In general, I'm still of the opinion that the
>> contrib submodule adds little value to the project at the expense of a lot
>> of pain for both developers and patch reviewers.  If avoiding a "big hot
>> mess" is what we need, I think that falls more on the core devs (and the
>> rest of the community) to do in-depth code reviews, rather than any
>> particular method of slicing up the code.

> I agree, I believe that the core/contrib separation actually makes it harder
> to work on a patch. At least, I find that I have to jump through more hoops
> to actually test and submit the changes. I tried to record the method I used
> to submit my last patch here:
> http://docs.rapidsms.org/Development/Contributing/Example

I'm in agreement here too. Over the last two years we core dev's
failed at preventing a "big hot mess."
Adam and I had a nice talk about this the other night and I think that
the lack of extensibility in the old architecture had a lot to do with
this.

Colin, thanks for taking the time to write up your example. The
process is indeed ridiculous.

> There are a lot of steps and, possibly due to my unfamiliarity with git,
> changes to submodules can easily be lost or put you in a weird state unless
> you do everything correctly the first time. I don't know if this is the best
> method, but I figured I'd document it as a possible set of steps to submit a
> patch via GitHub. What does everyone else do? For me, the separation makes
> it harder to follow trunk (unless I want to introduce git submodules into my
> project dependency list), which is fairly easy with Django [1].

Yes it is very easy to get things very screwy and weird with git.
I'm thinking that whatever questionable value there is in using
submodules will never be worth the tradeoff of raising the barrier to
entry by requiring so much monkeying around with git.

> Further, the Hudson build server won't pick up contrib changes until a core
> dev updates the submodules, right? So contrib may receive several breaking
> pushes (from core or non-core devs), but the build server wont see the break
> until a core dev updates the reference in rapidsms-core-dev. Then the build
> breaks and emails core devs about failures introduced not in the most recent
> core commit, but rather commits made a while back in contrib.
> Regards,
> Colin
> [1] http://docs.djangoproject.com/en/dev/topics/install/?#installing-the-development-version

Another great point. A functional build server will probably be a
better way to help us prevent a hot mess.

Looking at the contrib apps:
ajax, djangoadmin, export, httptester, messagelog, registration,
search, default, echo, handlers, locations, messaging, scheduler,
training

I'm comfortable with all of them going into the main repository with
the exception of search and training. The training app, while awesome,
isn't designed for ongoing use during production -- and is a great
example of how app authors can modify rapidsms' core routing system
from the comfort of a discrete app. As for search, I think that we
should have some kind of search API offered to app authors, but until
this implementation is more widely used I believe it should live with
the community apps.

Color me convinced. I say we move ajax, djangoadmin, export,
httptester, messagelog, registration, default, echo, handlers,
locations, messaging, and scheduler into the main repository, and move
search and training into the community repository.

I think that timing these changes with the shifting of
rapidsms/rapidsms-core-dev code into rapidsms/rapidsms is a perfect
opportunity to do so.

Cheers,
Evan

[Adam Mckaig]

On Fri, Aug 27, 2010 at 12:23 PM, Evan Wheeler <ewhe...@unicef.org> wrote:
> Color me convinced. I say we move ajax, djangoadmin, export,
> httptester, messagelog, registration, default, echo, handlers,
> locations, messaging, and scheduler into the main repository, and move
> search and training into the community repository.

+1 on that. let's merge contrib back into core.

</adam>

[Tobias McNulty]

On Fri, Aug 27, 2010 at 12:23 PM, Evan Wheeler <ewhe...@unicef.org> wrote:

> I agree, I believe that the core/contrib separation actually makes it harder
> to work on a patch. At least, I find that I have to jump through more hoops
> to actually test and submit the changes. I tried to record the method I used
> to submit my last patch here:
> http://docs.rapidsms.org/Development/Contributing/Example

 

Colin, thanks for taking the time to write up your example. The
process is indeed ridiculous.

 

Indeed.  That page is awesome.  I think it'll be easy to update to reflect the process post-merge, too.  Thanks for doing what I failed to do!

 

Looking at the contrib apps:

ajax, djangoadmin, export, httptester, messagelog, registration,
search, default, echo, handlers, locations, messaging, scheduler,
training

I'm comfortable with all of them going into the main repository with
the exception of search and training. The training app, while awesome,
isn't designed for ongoing use during production -- and is a great
example of how app authors can modify rapidsms' core routing system
from the comfort of a discrete app. As for search, I think that we
should have some kind of search API offered to app authors, but until
this implementation is more widely used I believe it should live with
the community apps.

This is a great point and I'm thrilled you're thinking about what belongs where.  Originally I was thinking we'd just copy over contrib wholesale, but now that you mention it, I agree that this is the perfect time to set the precedent for what belongs in contrib/core and what doesn't.

django.contrib is made up of "optional, de facto standard implementations of common patterns" [1] and I'd propose we adopt a similar policy for rapidsms.contrib (unless we already have one that I don't know about).

Color me convinced. I say we move ajax, djangoadmin, export,
httptester, messagelog, registration, default, echo, handlers,
locations, messaging, and scheduler into the main repository, and move
search and training into the community repository.

*dances for joy*  You just made my week.  Now I just can't wait to work on a patch for the newly unified version of RapidSMS.  :-)

Your plan for splitting things up sounds good to me.  Personally I'm not very familiar with the code in there, but I've added a rapidsms.contrib.* to my list of things to review.  I'm also very curious to hear others' thoughts on which apps belong where.

 

I think that timing these changes with the shifting of
rapidsms/rapidsms-core-dev code into rapidsms/rapidsms is a perfect
opportunity to do so.

+1

Tobias

[1] http://jacobian.org/writing/what-is-django-contrib/

[Cory Zue]

On Fri, Aug 27, 2010 at 1:20 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:

I think that timing these changes with the shifting of
rapidsms/rapidsms-core-dev code into rapidsms/rapidsms is a perfect
opportunity to do so.

+1

Sounds good guys, you've got my +1 as well. 

Cory