Exited sync due to gc errors

[Troy Perales]

I just updated to repo launcher version 1.18 after doing repo sync and copying the new script to my ~/bin folder. Now I'm getting the following:

error: Exited sync due to gc errors

It looks like it's happening at the end of the sync. Any suggestions?

[Shawn Pearce]

I would start with trying `repo --trace sync -j1 -n` to identify where
the gc error happens.

[Shawn Pearce]

Also, what version of Git is this?

[Troy Perales]

When I did repo --trace sync -j1 -n, everything went smoothly. As soon as I removed -j1, the default -j4 kicked in and gave me the same error. Here's the end of the output:

: export GIT_DIR=/home/tperales/dev/repos/maple/platform/.repo/projects/system/vold.git

: git fetch esi --tags +refs/heads/*:refs/remotes/esi/*

: load refs /home/tperales/dev/repos/maple/platform/.repo/projects/system/core.git

: export GIT_DIR=/home/tperales/dev/repos/maple/platform/.repo/projects/system/wlan/ti.git

: git fetch esi --tags +refs/heads/*:refs/remotes/esi/*

: load refs /home/tperales/dev/repos/maple/platform/.repo/projects/system/extras.git

: export GIT_DIR=/home/tperales/dev/repos/maple/platform/.repo/projects/vendor/esi.git

: git fetch esi --tags +refs/heads/*:refs/remotes/esi/*

: load refs /home/tperales/dev/repos/maple/platform/.repo/projects/system/netd.git

: load refs /home/tperales/dev/repos/maple/platform/.repo/projects/system/vold.git

: load refs /home/tperales/dev/repos/maple/platform/.repo/projects/system/wlan/ti.git

: load refs /home/tperales/dev/repos/maple/platform/.repo/projects/vendor/esi.git

: export GIT_DIR=/home/tperales/dev/repos/maple/platform/.repo/projects/bionic.git

: git -c pack.threads=2 gc --auto 1>| 2>|

: export GIT_DIR=/home/tperales/dev/repos/maple/platform/.repo/projects/boot.git

: git -c pack.threads=2 gc --auto 1>| 2>|

: export GIT_DIR=/home/tperales/dev/repos/maple/platform/.repo/projects/bootable/bootloader/legacy.git

: git -c pack.threads=2 gc --auto 1>| 2>|

: export GIT_DIR=/home/tperales/dev/repos/maple/platform/.repo/projects/bootable/diskinstaller.git

: git -c pack.threads=2 gc --auto 1>| 2>|

: export GIT_DIR=/home/tperales/dev/repos/maple/platform/.repo/projects/bootable/recovery.git

: git -c pack.threads=2 gc --auto 1>| 2>|

error: Exited sync due to gc errors

Doh! I'm running version 1.7.0.4. I'm guessing I need to upgrade to 1.8. I'll wait until the update hits my repository and just use -j1 for now.

Thanks, Shawn.

[Shawn Pearce]

On Fri, Oct 26, 2012 at 11:03 AM, Troy Perales <tper...@esi-estech.com> wrote:
> When I did repo --trace sync -j1 -n, everything went smoothly. As soon as I
> removed -j1, the default -j4 kicked in and gave me the same error. Here's
> the end of the output:
>
> : export

> GIT_DIR=/home/tperales/dev/repos/maple/platform/.repo/projects/bootable/recovery.git
> : git -c pack.threads=2 gc --auto 1>| 2>|
>
> error: Exited sync due to gc errors
>
>
> Doh! I'm running version 1.7.0.4. I'm guessing I need to upgrade to 1.8.
> I'll wait until the update hits my repository and just use -j1 for now.

The latest version of repo now requires Git 1.7.2 or newer, as that is
when the -c flag was introduced.

[Troy Perales]

Just in case anyone else runs into this...

I was so far back a version because I was using the default "git-core" package for Ubuntu Lucid Lynx without realizing it hand't been updated since 2011. I ended up adding the Ubuntu Git Maintainers PPA from here https://launchpad.net/~git-core/+archive/ppa.

[Pursehouse, David]

The change mentioned below is backwards compatible with Python versions older than 2.6 because the multiprocessing module is only used if it’s available:

 

try:

  import multiprocessing

except ImportError:

  multiprocessing = None

 

The problem is caused by the `-c` option on the git command, which was introduced in this change:

 

commit 091f893625269fd12adadf7d1f60c21b4b83e779

Author: Dave Borowitz <dbor...@google.com>

Date:   Tue Oct 23 17:01:04 2012 -0700

 

    project: Support config args in git command callables

   

    Change-Id: I9d4d0d2b1aeebe41a6b24a339a154d258af665eb

 

The solution is to use git version 1.7.2 or newer as mentioned by Shawn [1]

 

[1] https://groups.google.com/d/msg/repo-discuss/tnIWemm8M6k/qYq9pr3uwf8J

 

 

 

From: repo-d...@googlegroups.com [mailto:repo-d...@googlegroups.com] On Behalf Of myungsung kwak
Sent: Monday, October 29, 2012 3:01 PM
To: repo-d...@googlegroups.com
Subject: Re: Exited sync due to gc errors

 

Hi, my name Myungsung Kwak.

 

I guess that the cause of the problem occurred by python version.

We can be found at the following commit in repo(git project) 

 

commit 188572170e8cdf28df55a5ca90ed70d14b72b804

Author: Dave Borowitz <dbor...@google.com>

Date:   Tue Oct 23 17:02:59 2012 -0700

 

    sync: Run gc --auto in parallel

 

    We can't just let this run wild with a high (or even low) -j, since

    that would hose a system. Instead, limit the total number of threads

    across all git gc subprocesses to the number of CPUs reported by the

    multiprocessing module (available in Python 2.6 and above).

 

    Change-Id: Icca0161a1e6116ffa5f7cfc6f5faecda510a7fb9

 

This commit contains modification about gc subprocess.

Please, check the python version in your server. 

 

Thank you.

2012년 10월 28일 일요일 오후 1시 40분 31초 UTC+9, Sung-tae Hong 님의 말:

Oh! thanks. i also have a problem gc errors. it works well.

2012년 10월 28일 일요일 오전 1시 24분 53초 UTC+9, James Hall 님의 말:

@Pablo Poll' run these commands in a terminal:

sudo add-apt-repository ppa:git-core/ppa

sudo apt-get update

sudo apt-get install git-core

 

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

[Pursehouse, David]

> Shouldn't we update python version to 2.6(or above) if we use latest repo?

 

Right, the latest version of repo requires 2.6 or higher.

 

Python 2.5 support was removed in a couple of other changes ([1] [2]), but those have nothing to do with the error message that is being discussed here J

 

[1] https://gerrit-review.googlesource.com/37761

[2] https://gerrit-review.googlesource.com/38730

 

 

 

From: repo-d...@googlegroups.com [mailto:repo-d...@googlegroups.com] On Behalf Of myungsung kwak
Sent: Monday, October 29, 2012 5:40 PM
To: repo-d...@googlegroups.com
Cc: myungsung kwak
Subject: Re: Exited sync due to gc errors

 

Thank you for your response. 

I understand that the git version needs to be updated. 

 

I have one question.

If so, Shouldn't we update python version to 2.6(or above) if we use latest repo?

 

As you said, multiproccessing api is available in python version 2.6(or above)

http://docs.python.org/2.6/library/multiprocessing.html?highlight=multiprocessing#module-multiprocessing

2012년 10월 29일 월요일 오후 5시 17분 24초 UTC+9, David Pursehouse 님의 말:

[Dave Borowitz]

I could also go back and make this work with older versions of git. I didn't realize -c was such a new addition.

[Conley Owens]

I think it'd be best if we didn't. Apparently, git-core is an
obsolete package and Ubuntu users should be using git instead (which
is at a sufficient version on 10.04).

I can update the AOSP docs so that AOSP users will run into this confusion less.

~cco3

[Brad Larson]

On Monday, October 29, 2012 11:51:05 AM UTC-5, Conley Owens wrote:

I think it'd be best if we didn't.  Apparently, git-core is an
obsolete package and Ubuntu users should be using git instead (which
is at a sufficient version on 10.04).

I can update the AOSP docs so that AOSP users will run into this confusion less.

It has been a long time since I've had the opportunity to work with a repo-based project, but I think there is a min git version in the top-level repo script?  It seems like updating this to a version of git which supports -c would clear up a lot of confusion as well.

Brad

[Conley Owens]

On Mon, Oct 29, 2012 at 12:15 PM, Brad Larson <bkla...@gmail.com> wrote:
>
>
> On Monday, October 29, 2012 11:51:05 AM UTC-5, Conley Owens wrote:
>>
>> I think it'd be best if we didn't. Apparently, git-core is an
>> obsolete package and Ubuntu users should be using git instead (which
>> is at a sufficient version on 10.04).

OK, it looks like I'm wrong on this. While the "git" package exists
in 12.04, in 10.04 it only seems to be in Google's internal mirror of
the main Ubuntu repository so it's not that simple for people to
install. It would probably by best to stop using -c for the time
being. Is it an easy fix?

>>
>> I can update the AOSP docs so that AOSP users will run into this confusion
>> less.
>
>
> It has been a long time since I've had the opportunity to work with a
> repo-based project, but I think there is a min git version in the top-level
> repo script? It seems like updating this to a version of git which supports
> -c would clear up a lot of confusion as well.

It's already in:
https://gerrit-review.googlesource.com/#/c/38903/

[Shawn Pearce]

On Wed, Oct 31, 2012 at 10:39 AM, Conley Owens <cc...@android.com> wrote:
> On Mon, Oct 29, 2012 at 12:15 PM, Brad Larson <bkla...@gmail.com> wrote:
>>
>>
>> On Monday, October 29, 2012 11:51:05 AM UTC-5, Conley Owens wrote:
>>>
>>> I think it'd be best if we didn't. Apparently, git-core is an
>>> obsolete package and Ubuntu users should be using git instead (which
>>> is at a sufficient version on 10.04).
>
> OK, it looks like I'm wrong on this. While the "git" package exists
> in 12.04, in 10.04 it only seems to be in Google's internal mirror of
> the main Ubuntu repository so it's not that simple for people to
> install. It would probably by best to stop using -c for the time
> being. Is it an easy fix?

The other way to supply the data in -c is to use the
GIT_CONFIG_PARAMETERS environment variable. This needs to be a series
of single quoted values, e.g.:

env["GIT_CONFIG_PARAMETERS"] = "'foo.bar=1' 'this.that=2'"

Note the string beings with " ' and ends with ' " because of the
quoting required in the value. :-)

This exists in Git 1.7.3 and newer, but unlike -c doesn't break older
Git. Older Git would just ignore the unrecognized environment
variable.

[Dave Borowitz]

Yes, it's an easy fix. Older git clients will just not be parallelized.

We shouldn't use the environment variable as Shawn suggested, because this option is necessary to avoid completely hosting the system.

[Shawn Pearce]

On Wed, Oct 31, 2012 at 11:08 AM, Dave Borowitz <dbor...@google.com> wrote:
> Yes, it's an easy fix. Older git clients will just not be parallelized.

Good point, we can version test git and only run parallel gc if its
new enough to understand the -c flag.

> We shouldn't use the environment variable as Shawn suggested, because this
> option is necessary to avoid completely hosting the system.

Uhm, yes. This is correct. Thanks Dave.

[Shawn Pearce]

On Mon, Nov 5, 2012 at 3:09 AM, hagenp <hagen....@gmail.com> wrote:

So for repo v1.11.1, the "quick and easy" workaround is

repo sync -j1

Correct?

If you can't upgrade Git, yes, an easy work around is to use -j1.