sheriffs: Google Chrome Win compile failures on protoc.exe

[Primiano Tucci]

[Apologies for the mass-spam, couldn't find a more appropriate audience for this message]

TL;DR

If you are sheriffing (or just watching the waterfall) and see the Google Chrome Win compile step failing with some protoc error (more below), can you please archive the output folder somewhere before reopening the tree and restarting the bot?

Longer version

We have an extremely odd bug (crbug.com/644525) where, once per week, on one bot only, protoc.exe invocations fail with an access violation error, which looks like this.

FAILED: gen/chrome/browser/profile_resetter/profile_reset_report.pb.h gen/chrome/browser/profile_resetter/profile_reset_report.pb.cc pyproto/chrome/browser/profile_resetter/profile_reset_report_pb2.py 

C:/b/depot_tools/python276_bin/python.exe ../../tools/protoc_wrapper/protoc_wrapper.py profile_reset_report.proto --protoc ./../Release/protoc --proto-in-dir ../../chrome/browser/profile_resetter --cc-out-dir gen/chrome/browser/profile_resetter --py-out-dir pyproto/chrome/browser/profile_resetter

Protoc has returned non-zero status: -1073741819 .

• Tried reproducing locally without any luck
• Tried running the binary with DrMemory but nothing interesting came out.
• Checked the .ninja.log of the failing steps and the sequencing / deps seem okay (all proto invocations happen after the protoc.exe linking)
• Any speculative fix didn't make any difference

I requested infra to replace the slave, as at this point we suspect this to be some HW failure.

In the meantime, if anybody should see this failure, having a copy of the output folder could help us figuring out what's going wrong in the case this is not a HW failure.

Thanks,

Primiano

[Bruce]

The build machines are all configured to save Windows Error Reporting crashes locally. That should let you grab protoc.exe crash dumps after this bug happens, from %LOCALAPPDATA%\CrashDumps. The PE and PDB for protoc.exe will be gone but we could rebuild them from the same hash and then (I believe) force windbg to load the crash using these mismatched-but-equivalent-binaries and then the cause of the crash should be clear, or at least much clearer.

I started saying that we should enable WER local dumps but then I remembered having that thought a few months ago, and indeed they are already enabled:

https://codereview.chromium.org/1825163003/

See https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx for more details on how this works.

[Primiano Tucci]

Ah thanks Bruce.

in the meantime, infra just replaced the bot (crbug.com/649702). They confirmed that the slave was running on a pretty old machine, which gives some hope to the hw failure theory (although very odd that happens always to protoc.exe).

On the other side, a new machine means that the old CrashDumps directory is just gone.

Let's see if this happens again.   

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Paweł Hajdan, Jr.]

+infra-dev

Whoa, if I remember correctly we were trying to disable Windows Error Reporting for https://bugs.chromium.org/p/chromium/issues/detail?id=522396 . On the other hand, https://codereview.chromium.org/1825163003/ tries to enable it. I'm not sure what's the result of two conflicting settings/scripts interacting with each other.

Let's sort this out. I think WER is known to cause hangs in layout tests (see above bug). :-/

Paweł

[Bruce Dawson]

The two goals are not necessarily at odds. That bug (522396) is reporting hangs due to the WER UI coming up, and I also don't want that to come up.

The CL I landed turns on local saving of crash dumps, and attempts to turn off the UI:

    winreg.SetValueEx(key, "DontShowUI", 0, winreg.REG_DWORD, 1)

But, apparently that wasn't working. I hope that we can find a way to let WerFault.exe do its job (crash dumps can be useful) without bringing up any UI (hanging the build machines is not useful).

[Primiano Tucci]

So, apparently the same bot had another instance of that failure after the machine has been replaced and reimaged.

I managed to rdesktop into the bot but there is no %LOCALAPPDATA%\CrashDumps.

I got a bit lost about what the final state of WER is supposed to be after the discussion between Bruce and Paweł.

[Nektarios Paisios]

This bug is happening today on my local machine.

Nektarios.

 

[Bruce Dawson]

Can you share more details? In addition it would be very helpful to get the .exe that is crashing and a crash dump. The crash dump recording can be configured by following these instructions:

https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396

Also, it would be helpful to know what point you are synced to. A mitigation for the original protoc.exe issue was landed two weeks ago and is believe to be effective. Either it is failing in some cases or you are hitting a different bug and it would be helpful to know which.

If the binaries still crash when you run them manually then this is a different bug - still worth investigating of course.

[Bruce Dawson]

I just filed crbug.com/812421. It looks like a bug in Microsoft's incremental linker.