String files in mozilla-central

[flod]

Hi guys,
I'm doing some experiments with Mercurial and I want to be sure about
this, considering how much time it takes to clone mozilla-central and
mess with it. As far as I know all files containing en-US strings in
mozilla-central are stored in these paths, am I missing something obvious?

browser/locales/en-US
dom/locales/en-US
mobile/android/base/locales/en-US
mobile/android/locales/en-US
mobile/locales/en-US
mobile/xul/locales/en-US
netwerk/locales/en-US
services/sync/locales/en-US
security/manager/locales/en-US
toolkit/locales/en-US

Francesco

[Alexander L. Slovesnik]

You've missed "embedding/android/locales/en-US/"

> _______________________________________________
> dev-l10n mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-l10n
>


--
Sincerely yours,
Alexander L. Slovesnik a.k.a. Unghost
==>Jabber ID: ung...@mozilla-russia.org
==>Gmail Talk ID: ung...@gmail.com
==>Skype: alexander.slovesnik
==>Twitter: http://twitter.com/unghost

[flod]

Il 10/02/13 10:47, Alexander L. Slovesnik ha scritto:
> You've missed "embedding/android/locales/en-US/"
Right, I completely forgot that file.

Updated list

browser/locales/en-US
dom/locales/en-US
embedding/android/locales/en-US/

[Axel Hecht]

Generally speaking, the directories are those references from the file
set in
http://mxr.mozilla.org/comm-central/find?text=&string=l10n.*ini&regexp=1, in
the 'dirs' entry in the [compare] section,
http://mxr.mozilla.org/comm-central/source/mozilla/browser/locales/l10n.ini#9.

Not sure if you should also pick up the [extras] for the en-US spellchecker?

A nice little stress test for your thoughts might be "what would happen
if the metro guys wouldn't pave-over their history, but actually merge
https://hg.mozilla.org/projects/elm", which is going to add another dir.

Note: You forgot browser/branding/official so far, the other two


11 extensions/reporter
12 other-licenses/branding/firefox

are only in there to notify localizers of obsolete files.

Axel

[flod]

Il 10/02/13 18:08, Axel Hecht ha scritto:

> A nice little stress test for your thoughts might be "what would
> happen if the metro guys wouldn't pave-over their history, but
> actually merge https://hg.mozilla.org/projects/elm", which is going to
> add another dir.

That's interesting, considering that a first run of "hg convert" on
mozilla-central took over 5 hours. Now I'm running it again with a new
set of filters to see what happens – e.g. this time I chose to import
the entire locales folder instead of locales/en-US - and it started from
scratch (121.000 changesets to go...).
I guess tomorrow I'll see if everything exploded ;-)

Francesco

P.S. for those curious, I'm trying to understand if it's possible to
create a string only hg repository with a meaningful history and how
difficult this would be to maintain.

[flod]

Il 10/02/13 19:08, flod ha scritto:

> Now I'm running it again with a new set of filters to see what happens
> – e.g. this time I chose to import the entire locales folder instead
> of locales/en-US - and it started from scratch (121.000 changesets to
> go...).

Never mind this: it started from scratch because I misspelled the
destination repository (smart).

"hg convert" keeps track of the last changeset converted in a previous
run, so not sure what could happen with a merge that changes previous
history (I guess you have to examine again the entire repository to end
up with something meaningful).

I'll keep experimenting but I'm starting to see a lot of limits in my
approach.

Francesco

[Axel Hecht]

On 10.02.13 21:17, flod wrote:
> Il 10/02/13 19:08, flod ha scritto:
>> Now I'm running it again with a new set of filters to see what happens

>> � e.g. this time I chose to import the entire locales folder instead

>> of locales/en-US - and it started from scratch (121.000 changesets to
>> go...).
> Never mind this: it started from scratch because I misspelled the
> destination repository (smart).
>
> "hg convert" keeps track of the last changeset converted in a previous
> run, so not sure what could happen with a merge that changes previous
> history (I guess you have to examine again the entire repository to end
> up with something meaningful).
>
> I'll keep experimenting but I'm starting to see a lot of limits in my
> approach.
>
> Francesco

Great, thanks for the investigation. That's exactly one of the potential
problems that I wanted to get figured out.

Axel

[flod]

2013/2/10 Axel Hecht <l1...@mozilla.com>

> Great, thanks for the investigation. That's exactly one of the potential
> problems that I wanted to get figured out.

I decided to go a little deeper and recreated this situation:

- Started a new repository called *main-rep* (this would be
mozilla-central), made 3 commits in it.
- Cloned in a second repository called *child-rep* (e.g. elm repository)
and converted to a third repository called *converted-rep* (e.g.
mozilla-central string only)

At this point all three repositories have the same content with 3 commits.

Then I made 2 changes on both main-rep and child-rep, overlapping in time,
run again hg convert so that main-rep and converted-rep have the same
changesets. As I wrote in my previous mail, hg convert consider only the
new changesets (2) so this is a very quick process even on an elephant like
mozilla-central

This is the content of converted-rep (and main-rep) at this point
http://www.yetanothertechblog.com/wp-content/uploads/2013/02/rep_beforemerge.png

Then I merged child-rep into main-rep, run hg convert again from main-rep
to converted-rep and all new changesets are copied (something I wasn't sure
about)
http://www.yetanothertechblog.com/wp-content/uploads/2013/02/mainrep_aftermerge.png

Unless I'm missing something obvious, maybe because of the oversimplified
repositories, this is good. Merges don't mess conversions, the real problem
is if you add a new folder to the inclusion filter since it won't be run
against changesets already imported.

For those interested, in my first test - partial, considering that I missed
some folders and didn't import the entire locales folder - these are the
numbers:

- mozilla-central: 1.8 GB, 114631 files, 121454 changesets, 675603
revisions
- string-only repository: 15 MB, 551 files, 3290 changesets, 4347
revisions.

Francesco

[flod]

2013/2/11 flod <fl...@lodolo.net>

> Unless I'm missing something obvious, maybe because of the oversimplified
> repositories, this is good. Merges don't mess conversions, the real problem
> is if you add a new folder to the inclusion filter since it won't be run
> against changesets already imported.

Probably last update on the matter: after each conversion there's an
automatic commit "update tags".
This kind of mess everything, because new branches are created in the
mozilla-strings repository and you can't just update but merge+commit (not
sure if this can be safely automated).

This is an example of what happened with the last update
http://www.yetanothertechblog.com/wp-content/uploads/2013/02/need_merge.png

Francesco

P.S. judging by the around 300 strings missing on central I guess that
metro just landed ;-)

[Axel Hecht]

The "and then we'll add it to the list of exposed files" is one scenario
I'm concerned about.

I.e.:

hg add foo/locales/en-US/foo.dtd
... history for foo ...
vi browser/locales/l10n.ini to pick up foo
convert.

Also, some way to trick the conversion is

hg add browser/content/foo.dtd
... history for foo
hg mv browser/content/foo.dtd browser/locales/en-US/chrome/foo.dtd

Both are actually good in practice, as they allow developer to hack on
early stages of the UI without having to bother with the final strings.
See metro.

Also, which version of hg are you using? I don't have any updates of
tags. I do see updates of bookmarks, though.

Axel

[flod]

Il 15/02/13 16:46, Axel Hecht ha scritto:

> Also, which version of hg are you using? I don't have any updates of
> tags. I do see updates of bookmarks, though.

(smart face) version 1.6.4 :-\
Didn't thought that Debian repositories use versions this old. I'll have
to update to a recent version and retest.

Francesco

[Iacopo Benesperi]

flod ha scritto:

> (smart face) version 1.6.4 :-\
> Didn't thought that Debian repositories use versions this old. I'll have
> to update to a recent version and retest.

FYI, only Debian Stable uses a version so old, and there are backports
anyway:
http://packages.debian.org/search?keywords=mercurial&searchon=names&suite=all&section=all
In general, stable = out of date anyway.

Iacopo

[flod]

Il 15/02/13 20:01, Iacopo Benesperi ha scritto:

> FYI, only Debian Stable uses a version so old, and there are backports
> anyway:
> http://packages.debian.org/search?keywords=mercurial&searchon=names&suite=all&section=all
> In general, stable = out of date anyway.

As I said, I didn't thought it could that old (now using backports
package, which is still quite old anyway, version 2.2.2 vs 2.5.1)

Francesco

[flod]

Il 15/02/13 16:46, Axel Hecht ha scritto:

Also, which version of hg are you using? I don't have any updates of tags.
I do see updates of bookmarks, though.

Tried again with 2.2.2, I still see an automatic commit at the end ("update
tags" by convert-repo) and because of this I always need to merge after a
new run of hg convert.

See the graph here
https://bitbucket.org/flod/mozilla-central-strings/commits/all

Also, I'm not sure I'm following you in your examples about picking up new
contents. Practical example: I completely forgot about metro, so I didn't
add it to the list of includes before restarting the conversion from
scratch.
So I copied the files from mozilla-central to my repository, added them and
committed the change. Obviously I have no history for those files, just the
initial commit I did to add them.
https://bitbucket.org/flod/mozilla-central-strings/commits/f358da207b61ea74bc78695ddaee54b08a419653

Francesco

[Axel Hecht]

What's convert-repo? I'm using the 'convert' extension of mercurial [1],
with a

[extensions]
convert=

in my ~/.hgrc, and then just

hg convert inbound outbound

with a filemap etc.

Axel

[1] http://mercurial.selenic.com/wiki/ConvertExtension

[flod]

>
> What's convert-repo? I'm using the 'convert' extension of mercurial [1],
> with a
>

Same here

Content of ~/.hgrc

*[extensions]
hgext.convert=
*
Command to launch the import

*hg convert --filemap ~/filemap.txt /srv/hg/mozilla-central
/srv/hg/mozilla-strings*

filemap.txt contains a series of includes like these

*# From /browser/locales/l10n.ini
include "browser/locales"
etc.*

At the end of each conversion I found that automatic commit.

Francesco

[Axel Hecht]

Odd. I can't reproduce that.

Axel

[flod]

>
> Odd. I can't reproduce that.
>

I found a couple of references around but no real solutions
http://mercurial.808500.n3.nabble.com/Multiple-heads-after-convert-td803514.html
http://stackoverflow.com/questions/7866379/renaming-tags-while-converting-a-mercurial-repository

Now the repository on BitBucket is empty because I'm running a new
conversion excluding .hgtags (not sure it'll work).

Francesco

[Axel Hecht]

Hrm. maybe convert.hg.tagsbranch helps if you put that to a different
branch than the default branch? Then it'd be a bit out of the way.

Axel