Symbol files for Chromium DLLs not on symbol server

[Joe Laughlin]

Hello Everyone,

 

I am using the path for the Google Symbol server as show below, but I can not resolve symbols for Canary, Dev, Beta, and now Stable builds.

 

Set _NT_SYMBOL_PATH= SRV*c:\symbols*https://chromium-browser-symsrv.commondatastorage.googleapis.com;

I first noticed the issue for 98.0.4758.54 (Official Build) beta (64-bit) (cohort: Beta), but now it seems to be everywhere.

Is this a known issue? Is there an ETA for a fix?

[Joe Laughlin]

Here’s some additional info. When I look in my symbols folder when opening a stable build ETW trace with WPA  I see this:

 

I have the symbol server timeout set as follows.

set SYMSRV_HTTP_RECEIVE_TIMEOUT=120

[Joe Laughlin]

I just increased the SYMSRV_HTTP_RECEIVE_TIMEOUT to 240 seconds and was able to receive the stale symbol files. Seems excessive.

 

Canary/Dev still don’t resolve.

[Henrik Andreasson]

Just FYI, I have the same issue and noticed today that symbols for
Chrome Canary on Windows (100.0.4854.0) were not downloaded as
expected.

This has always worked before and I have not changed my symbol serverpath.

Seems like a regression on the server side.

On Wed, Jan 26, 2022 at 10:13 PM 'Joe Laughlin' via Chromium-dev
<chromi...@chromium.org> wrote:
>
> I just increased the SYMSRV_HTTP_RECEIVE_TIMEOUT to 240 seconds and was able to receive the stale symbol files. Seems excessive.
>
>
>
> Canary/Dev still don’t resolve.
>
>
>
> From: Joe Laughlin
> Sent: Wednesday, January 26, 2022 12:05 PM
> To: 'chromium-dev' <chromi...@chromium.org>
> Subject: RE: Symbol files for Chromium DLLs not on symbol server
>
>
>
> Here’s some additional info. When I look in my symbols folder when opening a stable build ETW trace with WPA I see this:
>
>
>

> I have the symbol server timeout set as follows.
>
> set SYMSRV_HTTP_RECEIVE_TIMEOUT=120
>
>
>
> From: Joe Laughlin
> Sent: Wednesday, January 26, 2022 11:55 AM
> To: chromium-dev <chromi...@chromium.org>
> Subject: Symbol files for Chromium DLLs not on symbol server
>
>
>
> Hello Everyone,
>
>
>
> I am using the path for the Google Symbol server as show below, but I can not resolve symbols for Canary, Dev, Beta, and now Stable builds.
>
>
>
> Set _NT_SYMBOL_PATH= SRV*c:\symbols*https://chromium-browser-symsrv.commondatastorage.googleapis.com;
>
>
> I first noticed the issue for 98.0.4758.54 (Official Build) beta (64-bit) (cohort: Beta), but now it seems to be everywhere.
>
> Is this a known issue? Is there an ETA for a fix?
>

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
> ---
> You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/SJ0PR00MB098945F7F3E409A158CA3916CD209%40SJ0PR00MB0989.namprd00.prod.outlook.com.

[Bruce Dawson]

I just checked on 100.0.4854.0 and it worked for me. I tested with chrome.exe and chrome.dll. symchk is handy because you can share the command and the output (making it clearer what exactly you are trying and what is failing) and it has a -v option that can give insights into failures. Recommended.

There were a couple of symbol upload problems earlier this month. The current situation is either a change in the behavior of Google storage or else it is generic network flakiness that is being reported more frequently due to the upload bugs of a couple of weeks ago. In all the case that I have checked the symbols are actually there, in the correct location.

Here is my symchk and related output, critically - PASSED + IGNORED files = 1:

C:\Users\brucedawson\AppData\Local\Google\Chrome SxS\Application\100.0.4854.0>set _NT
_NT_SYMBOL_PATH=SRV*C:\symbols*https://msdl.microsoft.com/download/symbols;SRV*C:\symbols*https://chromium-browser-symsrv.commondatastorage.googleapis.com

C:\Users\brucedawson\AppData\Local\Google\Chrome SxS\Application\100.0.4854.0>set path=c:\Program Files (x86)\Windows Kits\10\Debuggers\x64;%path%

C:\Users\brucedawson\AppData\Local\Google\Chrome SxS\Application\100.0.4854.0>symchk chrome.dll

SYMCHK: FAILED files = 0
SYMCHK: PASSED + IGNORED files = 1

[Henrik Andreasson]

Thanks Bruce, using symchk is a great idea.

In my case, I had downloaded 100.0.4849.0 locally (it was not the official Canary build) and symbols were not found initially.
I then checked the Symbols Hub in WPA for the Id of e.g chrome.dll and manually copied the already downloaded PDP to that location:

c:\symbols\chrome.dll.pdb\A81F0FE8757A3D7F4C4C44205044422E1\chrome.dll.pdb

symchk succeeds now but it might be since I've done this manual steps.

Will use it next time I'm hit by a similar issue and report back. Adding the complete output below just in case. Again, note that I had manually copied PDBs to my drive before running symchk

C:\Users\henrika\Downloads\Chrome-bin\100.0.4849.0>symchk -v chrome.dll
[SYMCHK] Searching for symbols to C:\Users\henrika\Downloads\Chrome-bin\100.0.4849.0\chrome.dll in path SRV*c:\symbols*https://msdl.microsoft.com/download/symbols;SRV*c:\symbols*https://chromium-browser-symsrv.commondatastorage.googleapis.com
DBGHELP: Symbol Search Path: SRV*c:\symbols*https://msdl.microsoft.com/download/symbols;SRV*c:\symbols*https://chromium-browser-symsrv.commondatastorage.googleapis.com
[SYMCHK] Using search path "SRV*c:\symbols*https://msdl.microsoft.com/download/symbols;SRV*c:\symbols*https://chromium-browser-symsrv.commondatastorage.googleapis.com"
DBGHELP: No header for C:\Users\henrika\Downloads\Chrome-bin\100.0.4849.0\chrome.dll.  Searching for image on disk (IS THIS OK?)
DBGHELP: C:\Users\henrika\Downloads\Chrome-bin\100.0.4849.0\chrome.dll - OK
SYMSRV:  BYINDEX: 0x1
         c:\symbols*https://msdl.microsoft.com/download/symbols
         chrome.dll.pdb
         A81F0FE8757A3D7F4C4C44205044422E1
SYMSRV:  PATH: c:\symbols\chrome.dll.pdb\A81F0FE8757A3D7F4C4C44205044422E1\chrome.dll.pdb
SYMSRV:  RESULT: 0x00000000
DBGHELP: chrome - private symbols & lines
        c:\symbols\chrome.dll.pdb\A81F0FE8757A3D7F4C4C44205044422E1\chrome.dll.pdb
[SYMCHK] MODULE64 Info ----------------------
[SYMCHK] Struct size: 1680 bytes
[SYMCHK] Base: 0x180000000
[SYMCHK] Image size: 185155584 bytes
[SYMCHK] Date: 0x61ee2552
[SYMCHK] Checksum: 0x00000000
[SYMCHK] NumSyms: 0
[SYMCHK] SymType: SymPDB
[SYMCHK] ModName: chrome
[SYMCHK] ImageName: C:\Users\henrika\Downloads\Chrome-bin\100.0.4849.0\chrome.dll
[SYMCHK] LoadedImage: C:\Users\henrika\Downloads\Chrome-bin\100.0.4849.0\chrome.dll
[SYMCHK] PDB: "c:\symbols\chrome.dll.pdb\A81F0FE8757A3D7F4C4C44205044422E1\chrome.dll.pdb"
[SYMCHK] CV: RSDS
[SYMCHK] CV DWORD: 0x53445352
[SYMCHK] CV Data:  chrome.dll.pdb
[SYMCHK] PDB Sig:  0
[SYMCHK] PDB7 Sig: {A81F0FE8-757A-3D7F-4C4C-44205044422E}
[SYMCHK] Age: 1
[SYMCHK] PDB Matched:  TRUE
[SYMCHK] DBG Matched:  TRUE
[SYMCHK] Line nubmers: TRUE
[SYMCHK] Global syms:  TRUE
[SYMCHK] Type Info:    TRUE
[SYMCHK] ------------------------------------
SymbolCheckVersion  0x00000002
Result              0x001f0001
DbgFilename
DbgTimeDateStamp    0x61ee2552
DbgSizeOfImage      0x0b094000
DbgChecksum         0x00000000
PdbFilename         c:\symbols\chrome.dll.pdb\A81F0FE8757A3D7F4C4C44205044422E1\chrome.dll.pdb
PdbSignature        {A81F0FE8-757A-3D7F-4C4C-44205044422E}
PdbDbiAge           0x00000001
[SYMCHK] [ 0x00000000 - 0x001f0001 ] Checked "C:\Users\henrika\Downloads\Chrome-bin\100.0.4849.0\chrome.dll"

SYMCHK: FAILED files = 0
SYMCHK: PASSED + IGNORED files = 1

[Henrik Andreasson]

Also tried with 100.0.4855.0 (latest Canary) and it worked like a charm.

[Bruce Dawson]

The "No header" warning seems to be normal - you can ignore that.

Running symchk after you have placed the .pdb in the right location in your local symbol server will always succeed because symchk (and VS, windbg, and WPA) will look in the local cache first.

BTW, if you have somehow downloaded PDBs (how did you do that? on the build archives site?) and you want to add them to your local symbol-server cache then you can use this syntax:

symstore add /f chrome.exe.pdb /s c:\symbols /t test

This is generally easier than manually constructing the destination path. The /t option is required but unimportant. The /s option points to your local symbol-server cache. You can use this same syntax for .exe, .dll, and .pdb files. Handy. More details are here.

It does seem that you and Joe hit some sort of network weirdness. If it keeps happening then we'll have to investigate more. Somehow.

--

Bruce Dawson, he/him