PSA: --force-sync needed once

[Don Garrett]

Someone landed a CL to reuse the same checkout directory for a different git repo into the manifest yesterday. Once you've sync'd the new stuff once, behavior should be back to normal.

Full disclaimer: I reviewed it and knew this might be an issue, so it's my fault.

[Don Garrett]

Some people have reported errors, even with "--force-sync".

Has anyone had problems and worked around them?

[Don Garrett]

+Vadim Bendebury 

From Vadim:

Syncing work tree:  97% (398/410)  fatal: This operation must be run in a work tree

…

Traceback (most recent call last):

  File "/mnt/host/source/.repo/repo/main.py", line 531, in <module>

    _Main(sys.argv[1:])

  File "/mnt/host/source/.repo/repo/main.py", line 507, in _Main

    result = repo._Run(argv) or 0

  File "/mnt/host/source/.repo/repo/main.py", line 180, in _Run

    result = cmd.Execute(copts, cargs)

  File "/mnt/host/source/.repo/repo/subcmds/sync.py", line 853, in Execute

    project.Sync_LocalHalf(syncbuf, force_sync=opt.force_sync)

  File "/mnt/host/source/.repo/repo/project.py", line 1406, in Sync_LocalHalf

    self._InitWorkTree(force_sync=force_sync)

  File "/mnt/host/source/.repo/repo/project.py", line 2565, in _InitWorkTree

    raise GitError("cannot initialize work tree")

error.GitError: cannot initialize work tree

My only thought is to clean out the files for the problematic projects inside .repo/projects and .repo/project-objects/, but if anything goes wrong, there is a strong potential for wiping out the entire repo checkout.

[Julius Werner]

Do you have more detailed instructions? Last time I just ran 'repo
sync --force-sync' in response to an email like this I lost my local
branches in most repos. Is it enough to run it for a specific repo?

> --
> --
> Chromium OS Developers mailing list: chromiu...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en
>

[Mike Frysinger]

if you need to run --force-sync for a repo, there's no way to avoid it.  doing it one-by-one doesn't change things.

please see my insns from the last thread if you have local branches you want to preserve:

  https://groups.google.com/a/chromium.org/d/msg/chromium-os-dev/2-0oCy_CX5s/gnuAREChAwAJ

-mike

[Mike Frysinger]

repo is not great at recovering from a wedged sync.  if you need to force a specific repo, please see my insns from the last thread:

  https://groups.google.com/a/chromium.org/d/msg/chromium-os-dev/2-0oCy_CX5s/bzzrp7ToAwAJ

-mike

--

[Julius Werner]

> if you need to run --force-sync for a repo, there's no way to avoid it.
> doing it one-by-one doesn't change things.

Okay, but do I need to run it for all repos right now? Or just for
one? So if Don could just tell me which repo to run it on (which is
most like one I don't care about and don't have local work for), can I
just do that and not disturb my other stuff? Or is --force-sync a
global thing that doesn't apply to individual repos?

[Vadim Bendebury]

BTW, I was not able to recover and just am recreating my chroot.

Also, is there a bug to fix the command which the tool suggests the user runs.

 'repo sync --force-sync src/third_party/toolchain-utils' always fails as it shows the local path name instead of the project name.

This seems to be going on for quite a while.

-v

[Achuith Bhandarkar]

Fwiw, repo --force-sync worked for me, and I didn't lose any work in my unsubmitted branches.

[Mike Frysinger]

On Tue, Jan 17, 2017 at 7:49 PM, Julius Werner wrote:
>> if you need to run --force-sync for a repo, there's no way to avoid it.
>> doing it one-by-one doesn't change things.
>
> Okay, but do I need to run it for all repos right now? Or just for
> one?

only you can decide that. if you don't want to sync your checkout,
then don't. but partial repo syncs are not supported, so if you try
syncing any repos, you need to sync them all if you want help if
something fails.

> So if Don could just tell me which repo to run it on (which is
> most like one I don't care about and don't have local work for), can I
> just do that and not disturb my other stuff?

Don doesn't know the state of your local checkout. which ones are
broken are shown when you try to run `repo sync`. look at the error
messages and repo will tell you exactly which are failing.

if you're running it in parallel and the intermixed output is
confusing, then force -j1 on the command line. the sync should halt
at the first failing repo and then you can resolve them one by one.

the git repo that triggered Don's PSA here is
src/third_party/toolchain-utils if that's what you're asking.

> Or is --force-sync a
> global thing that doesn't apply to individual repos?

--force-sync applies to the command you ran. if you didn't specify a
repo, then repo sync will sync all repos. if you specify specific
repos, then repo sync will operate on those.

sorry if the response sound general, but that's because we're talking
about a tool that has optional arguments. it's kind of like asking
"does the -l option affect all files when i run `ls` ?".
-mike

[Vadim Bendebury]

On Tue, Jan 17, 2017 at 8:22 PM, Mike Frysinger <vap...@chromium.org> wrote:

Don doesn't know the state of your local checkout.  which ones are
broken are shown when you try to run `repo sync`.  look at the error
messages and repo will tell you exactly which are failing.

if you're running it in parallel and the intermixed output is
confusing, then force -j1 on the command line.  the sync should halt
at the first failing repo and then you can resolve them one by one.

But for that matter Don sure knows what directory is broken, it would definitely be helpful to include this information in PSAs like this. It sure beats hundreds of people running 'repo sync' and watching the output for errors.

--vb

 

the git repo that triggered Don's PSA here is
src/third_party/toolchain-utils if that's what you're asking.

> Or is --force-sync a
> global thing that doesn't apply to individual repos?

--force-sync applies to the command you ran.  if you didn't specify a
repo, then repo sync will sync all repos.  if you specify specific
repos, then repo sync will operate on those.

sorry if the response sound general, but that's because we're talking
about a tool that has optional arguments.  it's kind of like asking
"does the -l option affect all files when i run `ls` ?".
-mike

[Mike Frysinger]

On Tue, Jan 17, 2017 at 11:59 PM, Vadim Bendebury <vbe...@chromium.org> wrote:

On Tue, Jan 17, 2017 at 8:22 PM, Mike Frysinger <vap...@chromium.org> wrote:

Don doesn't know the state of your local checkout.  which ones are
broken are shown when you try to run `repo sync`.  look at the error
messages and repo will tell you exactly which are failing.

if you're running it in parallel and the intermixed output is
confusing, then force -j1 on the command line.  the sync should halt
at the first failing repo and then you can resolve them one by one.

But for that matter Don sure knows what directory is broken, it would definitely be helpful to include this information in PSAs like this. It sure beats hundreds of people running 'repo sync' and watching the output for errors.

if your repo was already in a bad state, listing the one repo wouldn't help -- you already need to go through things one-by-one to fix them

if your repo was in a good state, using the one repo wouldn't matter because specifying --force-sync as Don suggested would fix the issue and there would be no need to watch the output

the repo in question (toolchain_utils) is developed by very few devs (like 5), so the likelihood of losing work is pretty low

-mike

[Julius Werner]

> if your repo was in a good state, using the one repo wouldn't matter because
> specifying --force-sync as Don suggested would fix the issue and there would
> be no need to watch the output

Are you suggesting that --force-sync will not drop local branches from
repos that are already "in a good state"? Because that's the issue
we're all trying to avoid here, and I could swear I've also lost stuff
from repos that had previously been fine in the past. (Maybe there was
some minor detail not "in a good state" in them, but if there was it
wasn't apparent to me in any way... they were still perfectly usable,
including getting updated by a normal 'repo sync' without errors.)

[Mike Frysinger]

--force-sync should only drop data from repos where a normal `repo
sync` fails. reading the repo source confirms this -- the only time
force_sync is checked is in a scenario where an error occurred and
repo is going to raise it if force is disabled.

so when i say "good state", i mean `repo sync` completes w/out error.
-mike

[ggg]

On Friday, January 13, 2017 at 3:38:09 PM UTC-8, Don Garrett wrote:

Some people have reported errors, even with "--force-sync".

Has anyone had problems and worked around them?

Yes...partly with help from this thread.

Here's what I had to do:

cd ~/trunk/.repo/

# I didn't do this exactly - but the same thing manually:

find project* -name tooltoolchain-utils\* | xargs rm -rf

cd ../src/

repo sync --force-sync

and I only had one other turd to clean up that was my own fault.

cheers,

grant

[Vadim Bendebury]

This fixed my problem (I used toolchain-utils instead of tooltoolchain-utils however ;)

thank you!

--vb

[Grant Grundler]

On Wed, Jan 18, 2017 at 9:04 PM, Vadim Bendebury <vbe...@chromium.org> wrote:
> This fixed my problem (I used toolchain-utils instead of tooltoolchain-utils
> however ;)

Excellent - apologies for the typo. :D

I was able to build last week but after running repo sync today, I get
this failure:
...
=== Start output for job feedback-0.0.1-r1683 (0m2.3s) ===
feedback-0.0.1-r1683: >>> Emerging (1 of 1)
chromeos-base/feedback-0.0.1-r1683::chromiumos for /build/gale/
feedback-0.0.1-r1683: * Running stacked hooks for pre_pkg_setup
feedback-0.0.1-r1683: * sysroot_build_bin_dir ...
feedback-0.0.1-r1683: [ ok ]
feedback-0.0.1-r1683: * Running stacked hooks for pre_src_unpack
feedback-0.0.1-r1683: * python_multilib_setup ...
feedback-0.0.1-r1683: [ ok ]
feedback-0.0.1-r1683: >>> Unpacking source...
feedback-0.0.1-r1683: Cloning into
'/build/gale/tmp/portage/chromeos-base/feedback-0.0.1-r1683/work/feedback-0.0.1/platform2'...
feedback-0.0.1-r1683: done.
feedback-0.0.1-r1683: fatal: Not a git repository (or any parent up to
mount point /var/cache/chromeos-cache)
feedback-0.0.1-r1683: Stopping at filesystem boundary
(GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
feedback-0.0.1-r1683: error: could not lock config file .git/config:
No such file or directory
feedback-0.0.1-r1683: fatal: Not a git repository (or any parent up to
mount point /var/cache/chromeos-cache)
feedback-0.0.1-r1683: Stopping at filesystem boundary
(GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
feedback-0.0.1-r1683: * ERROR:
chromeos-base/feedback-0.0.1-r1683::chromiumos failed (unpack phase):
feedback-0.0.1-r1683: * git-2_update_repo: can't update from
https://chromium.googlesource.com/chromium/src/components/feedback.git

Anyone have a clue WTH happened now? :(

cheers,
grant

[Brian Norris]

FWIW, I followed Grant's instructions with Vadim's suggested fix, and I somehow ended up with a corrupted toolchain-utils repository similar to that noted by Grant below (except his is for platform2/ instead). I had to go in and manually re-clone the toolchain-utils.git into the right place in my .repo/ directory, but I ended up getting it working.

But I would not recommend the above to anyone, if they can avoid it. I may have done something else wrong, but that's my experience.

For Grant: maybe you killed the wrong things in your .repo/? Can you check out that the symlinks aren't broken in src/platform2/.git/ ?

cheers,
grant

[Mike Frysinger]

that error doesn't look related to --force-sync

-mike

cheers,
grant

[ggg]

On Monday, January 23, 2017 at 1:53:35 PM UTC-8, Brian Norris wrote:

FWIW, I followed Grant's instructions with Vadim's suggested fix, and I somehow ended up with a corrupted toolchain-utils repository similar to that noted by Grant below (except his is for platform2/ instead). I had to go in and manually re-clone the toolchain-utils.git into the right place in my .repo/ directory, but I ended up getting it working.

But I would not recommend the above to anyone, if they can avoid it. I may have done something else wrong, but that's my experience.

That's fair. But given things are "spontaneously combusting", I haven't seen better advice than what I gave.

 

For Grant: maybe you killed the wrong things in your .repo/? Can you check out that the symlinks aren't broken in src/platform2/.git/ ?

Good idea - but those seem to all be ok. :/

(cr) (BARE:(54b2da9...)) grundler@firesword ~/trunk/src/platform2/.git $ ls -l

total 248

lrwxrwxrwx 1 grundler eng     48 Jul  8  2015 config -> ../../../.repo/projects/src/platform2.git/config

lrwxrwxrwx 1 grundler eng     67 Jul  8  2015 description -> ../../../.repo/project-objects/chromiumos/platform2.git/description

-rw-r--r-- 1 grundler eng     41 Jan 23 13:26 HEAD

lrwxrwxrwx 1 grundler eng     61 Jul  8  2015 hooks -> ../../../.repo/project-objects/chromiumos/platform2.git/hooks

-rw-r--r-- 1 grundler eng 226407 Jan 23 13:26 index

lrwxrwxrwx 1 grundler eng     60 Jul  8  2015 info -> ../../../.repo/project-objects/chromiumos/platform2.git/info

lrwxrwxrwx 1 grundler eng     46 Jul  8  2015 logs -> ../../../.repo/projects/src/platform2.git/logs

lrwxrwxrwx 1 grundler eng     63 Jul  8  2015 objects -> ../../../.repo/project-objects/chromiumos/platform2.git/objects

lrwxrwxrwx 1 grundler eng     53 Jul  8  2015 packed-refs -> ../../../.repo/projects/src/platform2.git/packed-refs

lrwxrwxrwx 1 grundler eng     46 Jul  8  2015 refs -> ../../../.repo/projects/src/platform2.git/refs

lrwxrwxrwx 1 grundler eng     64 Jul  8  2015 rr-cache -> ../../../.repo/project-objects/chromiumos/platform2.git/rr-cache

lrwxrwxrwx 1 grundler eng     49 Jul  8  2015 shallow -> ../../../.repo/projects/src/platform2.git/shallow

lrwxrwxrwx 1 grundler eng     59 Jul  8  2015 svn -> ../../../.repo/project-objects/chromiumos/platform2.git/svn

(cr) (BARE:(54b2da9...)) grundler@firesword ~/trunk/src/platform2/.git $ grep -R feedback *

Binary file index matches

Binary file objects/pack/pack-7c7ac257da3bba32e3d5ae67b2a706700713478a.pack matches

grep: shallow: No such file or directory

I don't understand the "BARE:" tag here but that goes away when I "cd ../feedback".

But I agree with Mike's assessment: since "repo sync" ran successfully, this is likely a different problem.

Is there the equivalent of "repo sanity_check" to validate all sync links/etc are "good"?

thanks!

grant

[ggg]

Just to follow up...

On Monday, January 23, 2017 at 1:36:27 PM UTC-8, ggg wrote:

I was able to build last week but after running repo sync today, I get
this failure:
...

... 

feedback-0.0.1-r1683: fatal: Not a git repository (or any parent up to
mount point /var/cache/chromeos-cache)

running "cros clean --clobber" allowed the build to proceed successfully.

cheers,

grant