git (cl upload) is painfully slow on Windows

[Gabriel Charette]

Hello Infra & Bruce :),

On Windows, git cl upload is painfully slow (over 3 minutes for a small CL)

(and it's not git cl presubmit -- which takes < 10s)

Here's an ETW trace of an upload : https://drive.google.com/file/d/1jSevO9VCANcU25QpAYK05h7QskG9cdQY/view (I gave access to Bruce; please request access if you need; didn't make public by default for privacy concerns).

Thanks,

Gab

[Erik Staab]

[+ajp, +tandrii]

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAJTZ7L%2BdAQbc7hSEsh5LBgWGKrBKcxehzF8-mLuWtbWDVeetBQ%40mail.gmail.com.

[Bruce Dawson]

The slowdown is sometimes because "git gc" has not been run for a while, and sometimes because of slow responses from the network. I'll take a look at the trace but if you have auto-gc disabled and haven't run a git gc lately then I recommend doing that.

[Andy Perelson]

FYI - Right now we collect metrics on git cl upload time but it's very high level and we can't tell what's due to Gerrit and what portion of the time is spent in our scripts. We'll be adding some monitoring soon to track the latency of our requests to Gerrit specifically so we can better understand the distribution and escalate to Gerrit any needed performance improvements with some real concrete data behind the ask.

Andy

On Tue, Oct 16, 2018 at 11:14 AM, Bruce Dawson <bruce...@chromium.org> wrote:

The slowdown is sometimes because "git gc" has not been run for a while, and sometimes because of slow responses from the network. I'll take a look at the trace but if you have auto-gc disabled and haven't run a git gc lately then I recommend doing that.

On Tue, Oct 16, 2018 at 9:32 AM Erik Staab <est...@chromium.org> wrote:

[+ajp, +tandrii]

On Tue, Oct 16, 2018 at 7:56 AM Gabriel Charette <g...@chromium.org> wrote:

Hello Infra & Bruce :),

On Windows, git cl upload is painfully slow (over 3 minutes for a small CL)

(and it's not git cl presubmit -- which takes < 10s)

Here's an ETW trace of an upload : https://drive.google.com/file/d/1jSevO9VCANcU25QpAYK05h7QskG9cdQY/view (I gave access to Bruce; please request access if you need; didn't make public by default for privacy concerns).

Thanks,

Gab

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

[Bruce Dawson]

We don't have symbols for git.exe which complicates this somewhat, but there is still hope. Here's what I can see:

The trace shows that git.exe (15160) runs for about 101 seconds and its behavior is bimodal. It spends about 37 s in a CPU heavy phase (including a period where it ramps up multiple threads) and then it goes into a low CPU usage IPC phase.

During the second phase it is mostly idle and is repeatedly calling msvcrt.dll!_write to communicate with git.exe (15072) which is calling msvcrt.dll!_read. This communication takes about 61 s during which time git.exe (15160) consumes 13.2 s of CPU time and git.exe (15072) consumes just 0.643 s of CPU time. git.exe (15072) spends most of its time waiting on git-remote-https.exe (11076).

The ~2 seconds that is unaccounted for that is 15160 waiting on 15072 in an msvcrt.dll!_read call.

Some relevant processes together with their command lines:

git.exe (13628): git.exe  cl upload -d -t "fix test compile"

git.exe (15160): git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset -q

git.exe (15072): git send-pack --stateless-rpc --helper-status --thin --no-progress https://chromium.googlesource.com/chromium/src.git/ --stdin

So... 37 s to pack objects, then 61 s to send the packed objects. svchost.exe and WmiPrvSE.exe were also running in the background but I see no reason to suspect that they were affecting anything.

P.S. It would be nice to get symbols for git.exe, and it would also be nice to get git.exe to use the regular CRT instead of the system's msvcrt.dll.

[Robert Iannucci]

Fwiw, I'm reworking how we build third party packages (including git).
I'm hoping to at least make the symbols available as a secondary
package for the new binaries.

Right now we get git directly from https://git-scm.com/download/win
and repackage it; if you build git locally and put it in front of
depot_tools in %PATH%, you may be able to gain additional insight that
way.

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAE5mQiPcX5jYfkQMZJSMfNNanPfGJsYS-hCnaEscxSL-Yt0SOA%40mail.gmail.com.

[Bruce Dawson]

Last time I tried to build git it was a huge pain, and building it with symbols was uncharted territory. The availability of clang-cl and other changes may have improved that. So, I probably won't build it myself but I do look forward to having symbols. If they can be put on Chrome's symbol server like clang-cl does then that would be brilliant because it would make them automatically available to all, with no special packaging or instructions needed.

In this case I would monitor your network as that appeared to be the bottleneck for much of the time. Running "git gc" might help with the earlier stages, as would having more memory and more CPU cores (two hyperthreaded CPUs is weak enough to cause some slowdowns, especially when other processes are running).

[Alan Cutter]

I've only tried git cl upload on Windows a couple of times but I aborted them after 10 minutes so these days I copy paste the diff into Keep and git cl upload on my Linux box.

>>>> To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

>>>> To post to this group, send email to infr...@chromium.org.
>>>> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAJTZ7L%2BdAQbc7hSEsh5LBgWGKrBKcxehzF8-mLuWtbWDVeetBQ%40mail.gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups "infra-dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

[Dirk Pranke]

That sounds really, horribly broken. Please file bugs when you see that sort of behavior. 

For a normal size CL, if the actual upload takes more than, say, 10s, there's probably something wrong. However, upload might also be running slow presubmit hooks, so it's important to figure out how much of the delay is in the hooks versus the actual upload.

Really we should be aiming for uploads to be well under 10s regardless. @ajp - do you know if we have enough numbers now to know what the distribution tends to look like, so we can start talking about SLOs here?

-- Dirk

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/c613ba2e-64f8-449d-a100-ecb9ac0a5f08%40chromium.org.

[Steve Kobes]

Bit of a shot in the dark but have you tried 64-bit git instead of the 32-bit depot_tools version?  This fixed OOM issues during gc for me.  (I filed a bug to switch depot_tools which got ignored.)

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAEoffTCfLDJ%2BH6Tn%2BChtHQiPkegyLW5SJGVF1CBRoxN%2B8aFMBA%40mail.gmail.com.

[Andy Perelson]

+ Edward Lesmes

I think the short answer is we have numbers but they aren't useful for setting SLOs yet. 

You can look at total time for git cl upload but:

• It includes non-SLO time such as user editing CL description
• As I have just discovered from looking at our data for this thread, we apparently are getting no data from Windows :(
• Data collection can be opted out of, not sure how significant a skew that causes in results, if any. Hopefully nothing significant.

Edward and I are in the middle of picking some finer grained metrics to collect, at the very least the timing for all our requests to Gerrit but it sounds like individual git command timing might be useful as well. Now we also have a task to look into why we aren't getting Windows metrics. As we improve this we will be collecting totally new metrics so we might have to re-ask for permission to collect more data.

Andy

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAEoffTCfLDJ%2BH6Tn%2BChtHQiPkegyLW5SJGVF1CBRoxN%2B8aFMBA%40mail.gmail.com.

[Robert Iannucci]

Hm... I'm pretty sure we use the 64 bit git on 64 bit platforms
(https://chromium.googlesource.com/chromium/tools/depot_tools/+/master/bootstrap/win/manifest.txt).
Is it possible that your `cipd` is the 32bit version? (cipd figures
out how to resolve ${platform} based on its own architecture).

>>>>> >>>> To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

>>>>> >>>> To post to this group, send email to infr...@chromium.org.
>>>>> >>>> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAJTZ7L%2BdAQbc7hSEsh5LBgWGKrBKcxehzF8-mLuWtbWDVeetBQ%40mail.gmail.com.
>>>>> >
>>>>> > --
>>>>> > You received this message because you are subscribed to the Google Groups "infra-dev" group.

>>>>> > To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

>>>>> > To post to this group, send email to infr...@chromium.org.
>>>>> > To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAE5mQiPcX5jYfkQMZJSMfNNanPfGJsYS-hCnaEscxSL-Yt0SOA%40mail.gmail.com.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "infra-dev" group.

>>> To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

>>> To post to this group, send email to infr...@chromium.org.
>>> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/c613ba2e-64f8-449d-a100-ecb9ac0a5f08%40chromium.org.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "infra-dev" group.

>> To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

[Daniel Bratell]

10 seconds sounds fast. A 1 line patch just took me 35 seconds on Linux and I can't say it felt that much slower than usual, though I've seen faster. Is sub 10 seconds common/normal? Most of the time is usually spent with that new "progress bar" of remote output.

/Daniel

>>>> To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

>>>> To post to this group, send email to infr...@chromium.org.
>>>> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAJTZ7L%2BdAQbc7hSEsh5LBgWGKrBKcxehzF8-mLuWtbWDVeetBQ%40mail.gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups "infra-dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

> To post to this group, send email to infr...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAE5mQiPcX5jYfkQMZJSMfNNanPfGJsYS-hCnaEscxSL-Yt0SOA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/c613ba2e-64f8-449d-a100-ecb9ac0a5f08%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAEoffTCfLDJ%2BH6Tn%2BChtHQiPkegyLW5SJGVF1CBRoxN%2B8aFMBA%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

[Bruce Dawson]

depot_tools git is 64-bit.

Normally I would run dumpbin to see whether a binary was 32-bit or 64-bit but this fails on git.exe:

>dumpbin c:\src\depot_tools\win_tools-2_7_6_bin\git\cmd\git.exe /headers

Microsoft (R) COFF/PE Dumper Version 14.16.26830.0

Copyright (C) Microsoft Corporation.  All rights reserved.

Dump of file c:\src\depot_tools\win_tools-2_7_6_bin\git\cmd\git.exe

PE signature found

File Type: EXECUTABLE IMAGE

c:\src\depot_tools\win_tools-2_7_6_bin\git\cmd\git.exe : fatal error LNK1106: invalid file or disk full: cannot seek to 0x6509A

So, I tried running git.exe under 32-bit windbg and that failed. Running under 64-bit windbg succeeded. So the depot_tools git.exe is 64-bit.

[Dirk Pranke]

I don't know what normal is these days, but I think we should be aiming for < 10s. I wouldn't be surprised if we're not that close now, but 35s on Linux sounds slow to me.

-- Dirk

>>>> To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

>>>> To post to this group, send email to infr...@chromium.org.
>>>> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAJTZ7L%2BdAQbc7hSEsh5LBgWGKrBKcxehzF8-mLuWtbWDVeetBQ%40mail.gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups "infra-dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

> To post to this group, send email to infr...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAE5mQiPcX5jYfkQMZJSMfNNanPfGJsYS-hCnaEscxSL-Yt0SOA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/c613ba2e-64f8-449d-a100-ecb9ac0a5f08%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAEoffTCfLDJ%2BH6Tn%2BChtHQiPkegyLW5SJGVF1CBRoxN%2B8aFMBA%40mail.gmail.com.

[Bruce Dawson]

Minor aside: I noticed a few days ago that one of the git.exe child processes was printing "Invalid parameter passed to C runtime function." to the OutputDebugString stream. That means that a secure CRT function was being misused. I managed to catch the failure in a debugger but since git.exe ships with no symbols this turns out to not give us a lot of extra information. So, yet another reason to make sure we get symbols for git.exe. I have a crash dump but I doubt it will be much use unless somebody really feels like decompiling.

>>>> To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

>>>> To post to this group, send email to infr...@chromium.org.
>>>> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAJTZ7L%2BdAQbc7hSEsh5LBgWGKrBKcxehzF8-mLuWtbWDVeetBQ%40mail.gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups "infra-dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

> To post to this group, send email to infr...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAE5mQiPcX5jYfkQMZJSMfNNanPfGJsYS-hCnaEscxSL-Yt0SOA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/c613ba2e-64f8-449d-a100-ecb9ac0a5f08%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAEoffTCfLDJ%2BH6Tn%2BChtHQiPkegyLW5SJGVF1CBRoxN%2B8aFMBA%40mail.gmail.com.

[Gabriel Charette]

I do "git gc" regularly and just did one again last night. I also rebooted (had a ton of svchost.exe processes around, not sure why). Neither helped, my last upload this morning was 4 minutes...

Re. getting more cores and memory, this is from my laptop. I understand it's not high-performance, but it should still be able to upload a text diff fairly efficiently.... It was faster for me to upload the 452MB ETW trace to Google Drive then it was to upload my 28 LOCs CL... goma also compiles efficiently so network is fine.

Le mer. 17 oct. 2018 17 h 24, Dirk Pranke <dpr...@chromium.org> a écrit :

That sounds really, horribly broken. Please file bugs when you see that sort of behavior. 

Done https://crbug.com/896642, hopefully this can help prioritize this investigation.

For a normal size CL, if the actual upload takes more than, say, 10s, there's probably something wrong. However, upload might also be running slow presubmit hooks, so it's important to figure out how much of the delay is in the hooks versus the actual upload.

Like I said in the OP, presubmit is not the issue here. I just had a 4 minutes upload and presubmit was 9.8s (albeit that's slow for the 10s goal but it's still not nearly the main issue yet for me).

>>>> To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

>>>> To post to this group, send email to infr...@chromium.org.
>>>> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAJTZ7L%2BdAQbc7hSEsh5LBgWGKrBKcxehzF8-mLuWtbWDVeetBQ%40mail.gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups "infra-dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

> To post to this group, send email to infr...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAE5mQiPcX5jYfkQMZJSMfNNanPfGJsYS-hCnaEscxSL-Yt0SOA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

To post to this group, send email to infr...@chromium.org.

[Takuto Ikuta]

Curious, what is shown when you specify GIT_TRACE=1 during `git cl upload`?

https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAJTZ7LKZSFuNZB3qKdFRJXLDhjOe7pZvkO7BCchkyn2A0yHS4g%40mail.gmail.com.

[Gabriel Charette]

Here's the GIT_TRACE log : https://paste.googleplex.com/5358887750336512

During the long step, the machine is barely using any CPU and I/O :

[Takuto Ikuta]

This output of git cl upload on my windows Z840 with GIT_TRACE=1 GIT_TRACE_PERFORMANCE=1 GIT_TRACE_SETUP=1 (not include GIT_TRACE_PACKET=1) 

https://paste.googleplex.com/5072076410978304

It took around a minute but not several minutes long.

I wonder whether re-cloning chromium repository fix the issue or not.

Also this may be the issue related to GoB server side considering your machine is not busy during the operation.

On Fri, Oct 19, 2018 at 1:55 AM Ryan Tseng <hin...@google.com> wrote:

Can you run this again with more detailed traces? Include:

GIT_TRACE_PACKET=1 GIT_TRACE_PERFORMANCE=1 GIT_TRACE_SETUP=1

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAJTZ7L%2BnEtkzzmQ2E2p%3D3rzFJuY2CWRGVA%3Dr1Wsan7xLdo-wzg%40mail.gmail.com.

[Gabriel Charette]

Interesting!

The trace shows a *lot* of items like these :

15:34:48.065516 pkt-line.c:80           packet:          git< 55baafa88b38cf49f4f0c41e1a57b4bc6f11212c refs/tags/62.0.3200.2

15:34:48.072535 pkt-line.c:80           packet:          git< e4d4d2b89dfcfe92b7daef2de518e579045d4a0e refs/tags/62.0.3200.3

15:34:48.082561 pkt-line.c:80           packet:          git< 0d5cc51d20f846e3cc85cb336be9c2e7f6e4bbd2 refs/tags/62.0.3200.4

15:34:48.089580 pkt-line.c:80           packet:          git< cad0904f262c3b70751a305fcec8d5196d12f942 refs/tags/62.0.3200.5

15:34:48.098603 pkt-line.c:80           packet:          git< 4615456063c3f7e3f13f10b8b9242b886f9f1426 refs/tags/62.0.3200.6

15:34:48.107629 pkt-line.c:80           packet:          git< f579f720f045b619ba77b2baeafb3be4588543ea refs/tags/62.0.3201.0

15:34:48.119661 pkt-line.c:80           packet:          git< a63eff9ed666678ea7357986ef3d73dbc726b315 refs/tags/62.0.3201.1

15:34:48.131692 pkt-line.c:80           packet:          git< 08790ec61f307db31244649dfb895b8552af7c0e refs/tags/62.0.3201.2

15:34:48.140715 pkt-line.c:80           packet:          git< f5ecd50e81808c901217d0925580f72d9e70c9c9 refs/tags/62.0.3201.3

15:34:48.151744 pkt-line.c:80           packet:          git< 9393a72d105ae9163523c72f0ec87b1023f6f630 refs/tags/62.0.3201.4

15:34:48.161773 pkt-line.c:80           packet:          git< d98311d679492c50380db8a6186d4da69e2c8221 refs/tags/62.0.3202.0

So it's basically sending a packet for every refs/branch-heads/* and refs/tags/*..! That would explain why I'm seeing worse performance (I did gclient sync --with_branch_heads)

That definitely seems unnecessary to upload a diff..

[Robert Iannucci]

I think this is fixed in the newer versions of git; we're going to roll out the new version soon.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAJTZ7L%2Bfs67e6C5UkYwoHUHz8zSFETscrB3KjWtT7Ajg4iz%3DQg%40mail.gmail.com.

[Bruce Dawson]

Do we have any plans to moving to git releases from here? I ask because they have symbols which would be handy for profiling and debugging ([git.exe] Invalid parameter passed to C runtime function.)

[Robert Iannucci]

Yes, actually, that's where we usually pull them from. If they have symbols now, then that's great!

[Gabriel Charette]

Le ven. 19 oct. 2018 18 h 29, Robert Iannucci <iann...@chromium.org> a écrit :

I think this is fixed in the newer versions of git; we're going to roll out the new version soon.

Cool! How soon? Will I break the world if I try to get ahead of the official infra release (and if not, which version should I upgrade to?)

[Robert Iannucci]

You shouldn't break; Grab the latest from https://github.com/git-for-windows/git/releases. Should be 2.19.1

[Bruce Dawson]

I tracked down symbols for git.exe (yay!) so now I can see what is going on in Gabriel's trace. It's weird. Of the 84,199 samples in the main git.exe process I see 23,620 in git.exe!create_delta and 19,533 in git.exe!create_delta_index. And, to be clear, I don't mean "with those functions on the call stack" I mean "in those functions". It looks like 5 or 6 instructions account for almost 25% of the CPU time in that process. I'm not sure what that means.

The majority of the time is still spent in the "mostly idle" phase though. The really busy time is "only" twenty seconds.

[Gabriel Charette]

Interesting, my two git cl uploads tonight from hotel room (20:46 GMT on Sunday) were very fast (still on the same git version as before).

So either it has to do with Gerrit's server being overwhelmed on weekdays or a problem when on Google-A..?!

[Andrii Shyshkalov]

I have a theory that speed of upload depends heavily on how many commits behind your branch is wrt target branch.

ie, if I don't "gclient sync" for a couple of days, then make 1 line change to cq.cfg, upload easily takes 1+ minutes (from Linux workstation!). Usually, interrupting upload and running "git rebase-update" and re-uploading is faster. I promised to trace this to GoB team, but I haven't had time to do it yet.

So, next time your upload takes long time, please check commit position of your branch's base and then compare that to origin/master

(or run "git fetch origin && git status" and git will print how many commits you are behind)

--

Andrii

[Gabriel Charette]

On Mon, Oct 22, 2018 at 12:31 PM Andrii Shyshkalov <tan...@chromium.org> wrote:

I have a theory that speed of upload depends heavily on how many commits behind your branch is wrt target branch.

Looks like this is correct. I had much faster git cl uploads throughout the holidays and it started being slower this week with this morning hitting peak unbearable again... Having to sync just to upload is super annoying (having to rebuild the world is no fun).