Repo experiment [was : Git submodule proof of concept]

[Fred Lin]

Dear all,

To followup yesterday discussion, I've setup a 'repo' which can fetch both gaia.git and vendor customization.git via 'repo sync' command.

vendor customization would be fetched and saved automatically into gaia/distribution folder.

https://github.com/gasolin/gaia-custom-repo


With this initial prompt, I think repo is capable to organize gaia modules.
Also, I found the benefit for developer is its easy to change default gaia(mozilla-b2g/gaia) to our own repository(gasolin/gaia) by set your own 'repo' default.xml. And stay mostly the same process that we are familiar with.

Simple guide is documented in gaia-custom-repo/README


regards
--
Fred Lin

----- Origin -----
Topic: Re: Git submodule proof of concept

Though I'm not a repo expert, I'd support give repo a try for all projects

1. B2G & Chrome OS use it (inherit from Android), we can benefit from share the same tool while they might have met the same problem years ago

[1] repo vs git submodule discussion in linaro maillist: http://lists.linaro.org/pipermail/linaro-dev/2011-June/004941.html
[2] 'repo is built to manage an OS distribution, in Git' repo announcement mail http://lwn.net/Articles/304488/

2. Repo is not bound to gaia submodules, thus vendors can set their own .repo, do their customization build, independent their work without mixed gaia itself and make contribution back easier. And again vendors from android background might already familiar with it.

3. repo does support git submodules (since its also a git)


Some Refs:
* http://www.instructables.com/id/Using-Googles-repo-command-in-your-own-projects/?ALLSTEPS
* http://source.android.com/source/developing.html#git-and-repo-cheatsheet

regards
--
Fred Lin

[Tim Chien]

I felt that |repo| would introduce another level of complexity, I feel
that for devs the flexibility outweighs the learning curve. Obviously
people other than devs (ResEng, A-team, QA etc.) need to be on-board
too, and they might think otherwise.

As Gaia gets more complex, our way of organize the code would have to
adopt too -- before we fall into invertible double-land, or
triple-land hell; as I said before, that's not a scalable way to reuse
code.

Quick question: does |repo| supports a hybrid approach? Say if I get a
repo w/ submodule and I have |repo| to pull it, would the submodules
got pulled too?

> _______________________________________________
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-gaia



--
Tim Guan-tin Chien, Engineering Manager and Front-end Lead, Firefox
OS, Mozilla Corp. (Taiwan)

[Kevin Grandon]

IMO whichever approach we use should be nearly invisible developers. If we can script repo with our Makefile then I don't see anything wrong with that :)

I do think that git submodules have an easier learning curve, and some benefits when it comes to third party developers. E.g., third party devs will simply see the linked repo in github if they are browsing the source, and will have a pretty good idea of how to include the scripts in their projects.

Perhaps it would be comparing workflows in a variety of situations for both approaches?

Best,
Kevin

[Andrew Sutherland]

On 05/16/2013 12:03 AM, Tim Chien wrote:
> Quick question: does |repo| supports a hybrid approach? Say if I get a
> repo w/ submodule and I have |repo| to pull it, would the submodules
> got pulled too?

From:
https://gerrit.googlesource.com/git-repo/+/master/docs/manifest-format.txt

===
Element project
---------------

One or more project elements may be specified. Each element
describes a single Git repository to be cloned into the repo
client workspace. You may specify Git-submodules by creating a
nested project. Git-submodules will be automatically
recognized and inherit their parent's attributes, but those
may be overridden by an explicitly specified project element.
===

I've looked at the source to some of repo (from
https://gerrit.googlesource.com/git-repo/+/master) and the code seems to
agree with this. Specifically, project.py has logic that parses
.gitmodules and explicitly has a concept of whether a project
"is_derived": "False if the project was explicitly defined in the
manifest; True if the project is a discovered submodule."

Andrew

[Fred Lin]

Thanks for bring this topic back,

I've write an article to summarize my exploration about build a 'repo' configure file to manage a project with multiple git repositories
http://blog.gasolin.idv.tw/2013/05/how-to-build-your-own-repo-to-manage.html

'Repo' script solved fetching and organizing multiple git problem. But I've not go through the commit code process yet. Since 'repo' will host all .git within .repo folder, 'git status' command not worked as usual, need to replaced to the 'repo status' command.
http://source.android.com/source/developing.html#viewing-client-status

regards
--
Fred Lin

----- Origin -----

sender: "Andrew Sutherland" <asuth...@asutherland.org>

[Tim Chien]

On Sat, Jun 15, 2013 at 6:55 AM, Fred Lin <fl...@mozilla.com> wrote:
> I've write an article to summarize my exploration about build a 'repo' configure file to manage a project with multiple git repositories
> http://blog.gasolin.idv.tw/2013/05/how-to-build-your-own-repo-to-manage.html
>
> 'Repo' script solved fetching and organizing multiple git problem. But I've not go through the commit code process yet. Since 'repo' will host all .git within .repo folder, 'git status' command not worked as usual, need to replaced to the 'repo status' command.
> http://source.android.com/source/developing.html#viewing-client-status

Thanks Fred for looking into this.

> sender: "Andrew Sutherland" <asuth...@asutherland.org>
> On 05/16/2013 12:03 AM, Tim Chien wrote:
>> Quick question: does |repo| supports a hybrid approach? Say if I get a
>> repo w/ submodule and I have |repo| to pull it, would the submodules
>> got pulled too?

> I've looked at the source to some of repo (from
> https://gerrit.googlesource.com/git-repo/+/master) and the code seems to
> agree with this. Specifically, project.py has logic that parses
> .gitmodules and explicitly has a concept of whether a project
> "is_derived": "False if the project was explicitly defined in the
> manifest; True if the project is a discovered submodule."

Thanks. The reason I asked this question is because I want to know if
we could get the best of the both worlds.
We certainly have use cases in Gaia itself where it would make sense
to use submodule some times and use |repo| some other times.
(hint: use submodule when you don't need the flexibility given by
|repo| to avoid complexity)

On Sat, Jun 15, 2013 at 3:34 AM, Andrew Sutherland
<asuth...@asutherland.org> wrote:
> Now that 1.1 is done-ish and we have some breathing space, this seems like a
> good time to pursue a cleanup like this.
>
> Vivien, Tim, you are the module co-owners of Gaia; is this something you
> would approve if we pursued this more aggressively? The gaia productivity
> team is meeting early next week in SF to focus on making the e-mail app
> something that does not require major activation energy to hack on; this is
> something we can mini-sprint on as an important first step.

I will not speak for Vivien, but I agree with Andrew that it's time to
fire up and move on.
As of |repo| v.s. submodule, I guess the choices is not important
because the question answered means they could co-exist.

We need an action plan here before the actual work.
Which directory we are currently looking at to turn into submodules,
and which directory should be turn into |repo| repos?
On top my head, I could think of a few shared libraries (building
blocks, webL10n, etc.) which we should turn into submodules, but I
couldn't think of any that should be turn into |repo| repos given the
flexibility required (maybe app-to-build config?) I don't think I
understand |repo| well enough to answer that.

For e-mail libraries, given the fact we run Makefile of every app, it
could certainly be set-up in the app-specific Makefile.

PS. Will we eventually require |repo| command for building Gaia just
like we require |make|? It shouldn't be hard to get than gjslint right
:P?

--
Tim Guan-tin Chien, Engineering Manager and Front-end Lead, Firefox
OS, Mozilla Corp. (Taiwan)

[Mathieu Rochette]

Hi there,

a few remarks as a third party dev, I would love to see a (up-to-date)
building block repository. I could use them as submodule or (more
likely in my case) install them with bower.
additionally I'd like to pull only a subset of the building block, I'd
like to be able to use any of those commands depending on my needs:
bower install mozilla/firefoxos-bb
bower install mozilla/firefoxos-bb-stable
bower install mozilla/firefoxos-bb-unstable
bower install mozilla/firefoxos-bb-header
bower install mozilla/firefoxos-bb-list
etc.


regards,

--
Mathieu Rochette

[Kevin Grandon]

> We need an action plan here before the actual work.

Totally agree. My proposal would be to externalize all files in the shared/ directory into "modules" first. Then follow-up with individual applications where needed.

I'm actually really happy with my git submodule POC. Since the old conversation died out I haven't been keeping it up to date. I'd be more than willing to update the pull request and go through and externalize shared code for all of our apps if we agree on doing so.

>From what I understand most folks seem to think that repo is more complex than a submodule implementation. Being so, I'd recommend implementing 'git submodule' first, and if problems arise or we need additional control - migrate to repo then.

Best,
Kevin

----- Original Message -----
From: "Tim Chien" <timd...@mozilla.com>
To: "Fred Lin" <fl...@mozilla.com>
Cc: "Andrew Sutherland" <asuth...@asutherland.org>, dev-...@lists.mozilla.org
Sent: Saturday, June 15, 2013 10:50:08 AM
Subject: Re: Repo experiment [was : Git submodule proof of concept]

[Tim Chien]

On Sun, Jun 16, 2013 at 4:05 AM, Kevin Grandon <kgra...@mozilla.com> wrote:
>> We need an action plan here before the actual work.
>
> Totally agree. My proposal would be to externalize all files in the shared/ directory into "modules" first. Then follow-up with individual applications where needed.
>

That make sense. Many shared files already on comes from external
repo; make them submodule is saner than what we do right now.
As of the naming, I would suggest |shared/ext/| with a
|shared/ext/README.md| to explain how submodule works and how to patch
these files from upstream (if these are necessary to ease the
migration).

Note that I don't intend to have git submodules to replace shared file
copying in Makefile. We could have each app references the same
external repo itself, but that would make |git submodule init| clone
the same repo N times.

[Axel Hecht]

Hi,

jumping in here in the middle of it:

We're already struggling with hg mirrors and automation of tests etc.

Finding localizable content and deciding where to put the localized
files is hard, too.

I'm afraid that splitting up our work into yet more repos is going to
make those problems exponentially harder to solve.

Axel

[Andrew Sutherland]

On 06/16/2013 11:58 AM, Axel Hecht wrote:
> We're already struggling with hg mirrors and automation of tests etc.
>
> Finding localizable content and deciding where to put the localized
> files is hard, too.
>
> I'm afraid that splitting up our work into yet more repos is going to
> make those problems exponentially harder to solve.

Is your concern about the contents of gaia/shared/locales or other paths?

Andrew

[Axel Hecht]

The l10n setup is rather delicate at this point, and I don't understand
all the details between what happens on git and what happens on hg to be
able to drill down to "oh, if you don't touch this path I'll be fine".

Axel

[James Lal]

I don't claim to be an expert on the l10n build process (my knowledge ends
at its already in another repo) but the point of repo would be to simplify
the use of any external repo. Typically in the larger projects I have been
involved in splitting up the logic that generally is reusable into its own
logic repo with tests / documentation as only resulted in good things...
That is probably not true with the l10n repo's but at worst whatever we do
now should be interoperable with repo.

On Sun, Jun 16, 2013 at 2:21 PM, Axel Hecht <l1...@mozilla.com> wrote:

> On 6/16/13 9:34 PM, Andrew Sutherland wrote:
>

> The l10n setup is rather delicate at this point, and I don't understand
> all the details between what happens on git and what happens on hg to be
> able to drill down to "oh, if you don't touch this path I'll be fine".
>
> Axel
>

> ______________________________**_________________
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/**listinfo/dev-gaia<https://lists.mozilla.org/listinfo/dev-gaia>
>

[Fred Lin]

I'm not familiar with the usage of git submodule though. So my suggestion about repo might has some bias.

Maybe we can work together and come out a proposal/small script to clearly describe/generate which part of gaia could be organized in different git repositories (both git submodule or repo need that). I think /shared directory is a good place to kick off.

the proposal could be a base that help communities and managers do the wise decision.
If all agree the whole picture, we can go to next step and adopt either git submodule or repo that helps us organize gaia.

The change in a long term might not only effect how we separate the directory, but may also influence the build process(ex: how we copy /shared resources when its an separate git repository), or how we shared findings from gaia to the open web.


regards
--
Fred Lin

----- origin -----
寄件人: "Kevin Grandon" <kgra...@mozilla.com>
收件人: "Tim Chien" <timd...@mozilla.com>
副本: "Andrew Sutherland" <asuth...@asutherland.org>, dev-...@lists.mozilla.org, "Fred Lin" <fl...@mozilla.com>
寄件箱: 2013年6 月16日, 星期日 上午 4:05:05
標題: Re: Repo experiment [was : Git submodule proof of concept]

> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-gaia

--
Tim Guan-tin Chien, Engineering Manager and Front-end Lead, Firefox
OS, Mozilla Corp. (Taiwan)

_______________________________________________
dev-gaia mailing list
dev-...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-gaia

[Axel Hecht]

On 6/17/13 2:31 AM, James Lal wrote:
> I don't claim to be an expert on the l10n build process (my knowledge ends
> at its already in another repo) but the point of repo would be to simplify
> the use of any external repo. Typically in the larger projects I have been
> involved in splitting up the logic that generally is reusable into its own
> logic repo with tests / documentation as only resulted in good things...
> That is probably not true with the l10n repo's but at worst whatever we do
> now should be interoperable with repo.

This isn't just l10n builds, this is also the steps we do to maintain
http://hg.mozilla.org/gaia-l10n/en-US/

My ask would be for these experiments to include l10n, and not just in a
"yeah, could possibly work". It needs to be included in a way that makes
it at least easier to localize Firefox OS. If it got harder or more
cumbersome, we'll have yet more chances to fail. And we're in no good
shape right now. I have a hard time to give localizers what they need,
and they have a hard time localizing and testing.

We should make things easier to get the product out that we ship, thus
improving the development of the localized product.

Also see my comments on git vs hg at
https://groups.google.com/forum/#!msg/mozilla.dev.platform/k1MW06xRYPo/T49Wb5wifP8J.

Axel

[Fred Lin]

I've created bug 883711 to track initiative to separate gaia module for reuse
https://bugzilla.mozilla.org/show_bug.cgi?id=883711

The init take start from 4 directories (I do that under my github account,
these are for experiment and wont sync with gaia-master):

- gaia-exp branch (gaia remove shared folder)
- gaia-shared (shared folder remove building block)
- gaia-shared-style (stable building block)
- gaia-shared-style-unstable (unstable building block)


https://github.com/gasolin/gaia-repo contains the experimental 'repo' for test purpose.

Besides, you can make your own directories by using attached script from bug 883711.
Once we agreed to separate some gaia directories for facilitating public reuse,
we can use this script to automate the process.


I think this proposal wont be happen until more devs involved and see if those tools(repo or git submodule) does benefit our development process.

regards
--
Fred Lin

----- origin -----

title: Re: Repo experiment [was : Git submodule proof of concept]