gclient vpython picks incorrect python on Windows

[Tekman]

Hi,

I'm trying to build in an Azure DevOps environment. The build works fine on my local machine but things go wrong on the Azure agent. The agent appears to have several python installations in the path; I followed the instructions to prepend depot_tools to %PATH%, but vpython still appears to pick the wrong location.

Output of "where python":

d:\a\1\s\build\depot_tools\python.bat
C:\hostedtoolcache\windows\Python\2.7.16\x64\python.exe
C:\hostedtoolcache\windows\Python\3.6.8\x64\python.exe
C:\hostedtoolcache\windows\Python\3.7.5\x64\python.exe
C:\ProgramData\Chocolatey\bin\python.exe

Output of "fetch --no-history v8":

[1808 0 venv.go:969] Command (cwd=C:\Users\VSSADM~1\AppData\Local\Temp\vpython_bootstrap518473594\packages\virtualenv-15.1.0): [C:\ProgramData\Chocolatey\bin\python2.7.exe -B -E -s virtualenv.py --no-download C:\Users\VssAdministrator\.vpython-root\4f66e4]
Process output:
Traceback (most recent call last):
 File "C:/Users/VssAdministrator/.vpython-root/4f66e4/lib/python2.7/site.py", line 703, in <module>
 main()
 File "C:/Users/VssAdministrator/.vpython-root/4f66e4/lib/python2.7/site.py", line 692, in main
 aliasmbcs()
 File "C:/Users/VssAdministrator/.vpython-root/4f66e4/lib/python2.7/site.py", line 515, in aliasmbcs
 import locale, codecs
 File "C:\Users\VssAdministrator\.vpython-root\4f66e4\lib\python2.7/locale.py", line 19, in <module>
 import functools
ImportError: No module named functools
New python executable in C:\Users\VssAdministrator\.vpython-root\4f66e4\Scripts\python2.7.exe
Also creating executable in C:\Users\VssAdministrator\.vpython-root\4f66e4\Scripts\python.exe
ERROR: The executable C:\Users\VssAdministrator\.vpython-root\4f66e4\Scripts\python2.7.exe is not functioning
ERROR: It thinks sys.prefix is u'c:\\users\\vssadministrator\\appdata\\local\\temp\\vpython_bootstrap518473594\\packages\\virtualenv-15.1.0' (should be u'c:\\users\\vssadministrator\\.vpython-root\\4f66e4')
ERROR: virtualenv is not compatible with this system or executable
Note: some Windows users have reported this error when they installed Python for "Only this user" or have multiple versions of Python installed. Copying the appropriate PythonXX.dll to the virtualenv Scripts/ directory may fix this problem.

Environment:
...

Path=d:\a\1\s\build\depot_tools;C:\Users\VssAdministrator\AppData\Roaming\Python\Python27\Scripts;C:\hostedtoolcache\windows\Python\2.7.16\x64\Scripts;C:\hostedtoolcache\windows\Python\2.7.16\x64;C:\agents\2.160.1\externals\git\cmd;C:/hostedtoolcache/windows\Python\3.6.8\x64;C:/hostedtoolcache/windows\Python\3.6.8\x64\Scripts;C:\Program Files\Mercurial\;C:\ProgramData\kind;C:\vcpkg;C:\cf-cli;C:\Program Files (x86)\NSIS\;C:\Program Files\Mercurial\;C:\Program Files\Boost\1.69.0;C:\Program Files\dotnet;C:\mysql-5.7.21-winx64\bin;C:\Program Files\Java\zulu-8-azure-jdk_8.40.0.25-8.0.222-win_x64\bin;C:\npm\prefix;C:\Rust\.cargo\bin;C:\hostedtoolcache\windows\Ruby\2.5.5\x64\bin;C:\Go1.12.7\bin;C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\mingw64\bin;C:\hostedtoolcache\windows\Python\3.7.5\x64\Scripts;C:\hostedtoolcache\windows\Python\3.7.5\x64;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\OpenSSH\;C:\ProgramData\Chocolatey\bin;C:\Program Files\Docker;C:\Program Files\PowerShell\6\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files\dotnet\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;c:\tools\php;C:\Program Files (x86)\Subversion\bin;C:\Program Files\nodejs\;C:\ProgramData\chocolatey\lib\maven\apache-maven-3.6.2\bin;C:\Program Files\CMake\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\Program Files\OpenSSL\bin;C:\Users\VssAdministrator\.dotnet\tools;C:\Program Files\Microsoft SQL Server\Client ;d:\a\1\s\build\depot_tools


...

[1808 0 annotate.go:241] original error: exit status 100
[1808 0 annotate.go:241]
[1808 0 annotate.go:241] goroutine 1:
[1808 0 annotate.go:241] #0 go.chromium.org/luci/vpython/venv/venv.go:613 - venv.(*Env).installVirtualEnv()
[1808 0 annotate.go:241] reason: failed to create VirtualEnv
[1808 0 annotate.go:241]
[1808 0 annotate.go:241] #1 go.chromium.org/luci/vpython/venv/venv.go:529 - venv.(*Env).createLocked.func2()
[1808 0 annotate.go:241] reason: failed to install VirtualEnv
[1808 0 annotate.go:241]
[1808 0 annotate.go:241] #2 go.chromium.org/luci/common/system/filesystem/tempdir.go:55 - filesystem.(*TempDir).With()
[1808 0 annotate.go:241] #3 go.chromium.org/luci/vpython/venv/venv.go:103 - venv.withTempDir()
[1808 0 annotate.go:241] #4 go.chromium.org/luci/vpython/venv/venv.go:515 - venv.(*Env).createLocked()
[1808 0 annotate.go:241] #5 go.chromium.org/luci/vpython/venv/venv.go:272 - venv.(*Env).ensure.func1()
[1808 0 annotate.go:241] reason: failed to create new VirtualEnv
[1808 0 annotate.go:241]
[1808 0 annotate.go:241] #6 go.chromium.org/luci/vpython/venv/venv.go:989 - venv.mustReleaseLock()
[1808 0 annotate.go:241] #7 go.chromium.org/luci/vpython/venv/venv.go:258 - venv.(*Env).ensure()
[1808 0 annotate.go:241] #8 go.chromium.org/luci/vpython/venv/venv.go:154 - venv.With()
[1808 0 annotate.go:241] reason: failed to create empty probe environment
[1808 0 annotate.go:241]
[1808 0 annotate.go:241] #9 go.chromium.org/luci/vpython/run.go:62 - vpython.Run()
[1808 0 annotate.go:241] #10 go.chromium.org/luci/vpython/application/application.go:312 - application.(*application).mainImpl()
[1808 0 annotate.go:241] #11 go.chromium.org/luci/vpython/application/application.go:397 - application.(*Config).Main.func1()
[1808 0 annotate.go:241] #12 go.chromium.org/luci/vpython/application/support.go:46 - application.run()
[1808 0 annotate.go:241] #13 go.chromium.org/luci/vpython/application/application.go:396 - application.(*Config).Main()
[1808 0 annotate.go:241] #14 vpython/main.go:105 - main.mainImpl()
[1808 0 annotate.go:241] #15 vpython/main.go:111 - main.main()
[1808 0 annotate.go:241] #16 runtime/proc.go:203 - runtime.main()
[1808 0 annotate.go:241] #17 runtime/asm_amd64.s:1357 - runtime.goexit()
##[error]Cmd.exe exited with code '1'.

Any tips for how to debug the issue? I can't uninstall python on the agent unfortunately; is there a way to force vpython to pick the first python in the path (which would be python.bat in depot_tools in this case, pointing to the downloaded bootstrap executable)?

Thank you!

[Edward Lesmes]

Can you please file a bug in crbug.com?
I'm not sure why this happens, but we can probably modify vpython in Depot Tools to force it to use the bootstrapped version.

[Matthew Einhorn]

I opened a bug report at https://bugs.chromium.org/p/chromium/issues/detail?id=1037849&q=Azure&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified.

I experienced a similar issue trying to compile angle on github actions, which is running on Azure DevOps.