Win10 build stuck on STAMP step

[Jonah Ryan-Davis]

Since yesterday I've been unable to build chrome. Gn gets stuck for an indefinite time on a STAMP step. I built successfully on Wed, but broken since Thurs. It's happened with a few different files, 2 different examples are:

[53990/61207] STAMP obj/chrome/browser_dependencies.stamp

[15410/28284] STAMP obj/content/public/browser/browser_sources.stamp

Steps I've tried:

1. Changing my gn args, following exactly the gn args of a dbg bot, is_debug both true/false, is_component_build both true/false

2. Restarting goma

3. Cleaning my out directory

4. Restarting my workstation

5. Rebasing on ToT

Unfortuantely I don't see any sort of error messages. It just takes a long time (2hr+ on one single step) and I end up killing the process.

Are there any arguments I can pass to see more verbose info?

Or, does anyone have a solution?

Thanks,

Jonah

[Peter Kasting]

On Fri, Mar 19, 2021 at 11:38 AM 'Jonah Ryan-Davis' via Chromium-dev <chromi...@chromium.org> wrote:

Since yesterday I've been unable to build chrome. Gn gets stuck for an indefinite time on a STAMP step. I built successfully on Wed, but broken since Thurs. It's happened with a few different files, 2 different examples are:

[53990/61207] STAMP obj/chrome/browser_dependencies.stamp

[15410/28284] STAMP obj/content/public/browser/browser_sources.stamp

Multiple people on my team are stuck on this as well.

PK 

[Peter Kasting]

Filed an initial bug at https://bugs.chromium.org/p/chromium/issues/detail?id=1190269 , folks can add more info there.

PK 

[Lei Zhang]

When it hangs, abort. Then run ninja with -j1 -v to see what command
is hanging. Is it actually the stamp step or something else?

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
> ---
> You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/9b6ae2e1-bfe7-4424-b1ff-3f7404992cb9n%40chromium.org.

[Peter Kasting]

On Fri, Mar 19, 2021 at 11:49 AM Lei Zhang <the...@chromium.org> wrote:

When it hangs, abort. Then run ninja with -j1 -v to see what command
is hanging. Is it actually the stamp step or something else?

I just get:

ninja: Entering directory `out/Debug'

... and no further output.

PK

[Nico Weber]

Omit the -v, else ninja will print the command only after it has completed

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAHOzFA0RTnDfgNZB7mW6%2B%2BNyxWSi2R%3DByrj_7%2B51eymrqN4wQ%40mail.gmail.com.

[Dirk Pranke]

We've been debugging this in crbug.com/1190269; the current theory is that there's a bug in Python3's `multiprocessing` library that shows up on machines with > 64 cores (life's rough, right?).

[ Also, shout out to @Stephen McGruer for noticing the similarity to a bug he/they hit in the WPT tests! ]

The CL to switch from Python2 to Python3 has been reverted (in https://crrev.com/c/2774784). This iteration of the flip did expose a few more bugs, which is useful. I expect to try again early next week, but I'll make sure some of the big-machine-having people test it out first :).

[ We'll also try to get the Python folks to take a look at this. This might already be known in https://bugs.python.org/issue40263 with a proposed fix in https://github.com/python/cpython/pull/19501 but for some reason that pull request hasn't been accepted yet ... ]

-- Dirk

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAMGbLiFhEzEHDr8ppPshXmWB-ac-o0Ry6aAi4yiCDxuL%2BHs97g%40mail.gmail.com.

[Bruce Dawson]

This issue is understood now and a fix on the way, so that's great.

For future reference, if you want to find out what command is being run when the build hangs or just takes a long time on a single step there is tools\buildstate.py (runnable from Windows as tools\buildstate.bat). This will print the command-lines of all currently running build commands. In this case it printed:

> call python3 E:\src\tools\buildstate.py

Gathering process data...

Tracking the children of these PIDs:

40328

34096: E:/depot_tools/bootstrap-3_8_0_chromium_8_bin/python3/bin/python3.exe ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database

Found 1 children

You can also use sysinternals' procexp.exe to view the process tree and see what the child processes are.