Incorrect symbols in stack trace from Windows cross build.

[Kyle Charbonneau]

I've built Windows on Linux following the cross build instructions and copying all of the required files to a Windows machine. It runs fine but symbolized stack traces look wrong. Here is an example stack trace from a crash in the GPU process:

[7644:3128:0604/162412.414:FATAL:weak_ptr.h(256)] Check failed: get() != nullptr.
Backtrace:
        base::debug::CollectStackTrace [0x00007FFCBD078482+18]
        base::debug::StackTrace::StackTrace [0x00007FFCBD0777F2+18]
        logging::LogMessage::~LogMessage [0x00007FFCBD09784C+108]
        gpu::ImageDecodeAcceleratorStub::ProcessCompletedDecode [0x00007FFCAC5DC1E4+9554]
        gpu::ImageTransportSurface::CreateNativeSurface [0x00007FFCAC5E22F1+333]
        viz::CreateSoftwareOutputDeviceWin [0x00007FFC9A06BAAF+9807]
        viz::SkiaOutputSurfaceImplNonDDL::WaitSyncToken [0x00007FFC9A07BF3C+6876]
        viz::SkiaOutputSurfaceImplNonDDL::WaitSyncToken [0x00007FFC9A07BDBF+6495]
        viz::SkiaOutputSurfaceImpl::InitializeOnGpuThread [0x00007FFC9A06EE08+808]
        gpu::Scheduler::RunNextTask [0x00007FFCBB096310+676]
        base::TaskAnnotator::RunTask [0x00007FFCBD10B6EC+412]
        base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl [0x00007FFCBD13074E+654]
        base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoSomeWork [0x00007FFCBD130381+97]
        base::MessagePumpDefault::Run [0x00007FFCBD0A7EFC+188]
        base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run [0x00007FFCBD1311EF+271]
        base::RunLoop::RunWithTimeout [0x00007FFCBD0E3E51+801]
        Ordinal0 [0x00007FFCB349513C+20796]
        content::ContentMain [0x00007FFCB47F1F5C+2748]
        service_manager::Main [0x00007FFCA9AA195E+1954]
        content::ContentMain [0x00007FFCB47F14E0+64]
        ChromeMain [0x00007FFCB5AD11CF+311]
        Ordinal0 [0x00007FF6E6A337E7+14311]
        Ordinal0 [0x00007FF6E6A3188B+6283]
        IsSandboxedProcess [0x00007FF6E6B034DA+719629]
        BaseThreadInitThunk [0x00007FFCE6F97974+20]
        RtlUserThreadStart [0x00007FFCE904A271+33]

I believe the crash is happening in ImageTransportSurface::CreateNativeSurface() so that part is correct but both ImageDecodeAcceleratorStub::ProcessCompletedDecode() and CreateSoftwareOutputDeviceWin() are pretty much impossible. I've tried causing a crash in other places and get the same type of wrong stack traces. This a release build with the following GN args, but stack traces from debug cross builds are also broken for me

target_os = "win"
is_component_build = true
use_goma = true
is_debug = false
symbol_level = 1
dcheck_always_on = true

 

Is this a known issue with cross builds? Am I doing something wrong to cause it?

Kyle

[Alexei Svitkine]

Is it possible that the machine code for the correct function is the same as the machine code for the one being shown? This can be the case with short functions. If so, it might be a case of deduping by the linker where multiple functions have the same impl.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAERSRDCj8frOWnHEFczG9Ah5ZuVK0s%3DBOg%3DM%2BerHj5u5_QvbVA%40mail.gmail.com.

[Reid Kleckner]

When you say you "copied all of the required files", did that include the PDB files? This looks like an "exports-only" stack trace, so chrome's stack dumper must not be able to load any PDBs. Alternatively, maybe chrome can't find dbghelp.dll.

In general, if you want stack traces with line numbers, I would suggest using symbol_level=2. If that is too slow to link, try finding a way to sneak "-gline-tables-only" into cflags. There doesn't seem to be a corresponding gn setting for that, although maybe there should be . It should produce small, line table only debug info that doesn't slow down links as much as symbol_level=2.

My suggestion for debugging it would be build a small test executable, say base_unittests.exe, with symbol_level=2. Add a small test to it that does CHECK(false), and iterate on the file copying and configuration until the stack trace has line numbers. If the stack trace has no line numbers, then you know that PDB loading failed. Then you can go back to symbol_level=1 if you still want faster links.

--

[Peter Collingbourne]

On Tue, Jun 4, 2019 at 3:28 PM 'Reid Kleckner' via Chromium-dev <chromi...@chromium.org> wrote:

When you say you "copied all of the required files", did that include the PDB files? This looks like an "exports-only" stack trace, so chrome's stack dumper must not be able to load any PDBs. Alternatively, maybe chrome can't find dbghelp.dll.

In general, if you want stack traces with line numbers, I would suggest using symbol_level=2. If that is too slow to link, try finding a way to sneak "-gline-tables-only" into cflags. There doesn't seem to be a corresponding gn setting for that, although maybe there should be . It should produce small, line table only debug info that doesn't slow down links as much as symbol_level=2.

That's what symbol_level=1 means on non-Windows platforms, I believe.

Maybe we should consider removing the "using_sanitizer" part from here?

https://cs.chromium.org/chromium/src/build/config/compiler/BUILD.gn?l=2349

That would make Windows consistent with other platforms.

Peter 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACs%3DtyKbw1m_QcmtJgqBvOhR9g5nU2G-ZWEwAS0p0QRBA3rNvw%40mail.gmail.com.

[Kyle Charbonneau]

I was not copying over the PDB files and that was totally the problem. The stack trace looks correct afterwards. Thanks for the help! I'll update the cross build instructions to mention that mini_installer target doesn't include PDB files and those are required to get proper stack traces.

[Reid Kleckner]

On Wed, Jun 5, 2019 at 7:05 AM Kyle Charbonneau <kyle...@chromium.org> wrote:

I was not copying over the PDB files and that was totally the problem. The stack trace looks correct afterwards. Thanks for the help! I'll update the cross build instructions to mention that mini_installer target doesn't include PDB files and those are required to get proper stack traces.

Excellent. :)

 On Tue, Jun 4, 2019 at 4:52 PM Peter Collingbourne <p...@chromium.org> wrote:

That's what symbol_level=1 means on non-Windows platforms, I believe.

Maybe we should consider removing the "using_sanitizer" part from here?

https://cs.chromium.org/chromium/src/build/config/compiler/BUILD.gn?l=2349

That would make Windows consistent with other platforms.

I hadn't realized (or had forgotten) the other platforms had converged on that behavior! I got some numbers and fired of a CL to make Windows consistent here:

https://chromium-review.googlesource.com/c/chromium/src/+/1644784