Siso: Optimize libc++ header tracking to reduce Windows lstat calls [chromium/src : main]
Junji Watanabe (Gerrit)
Junji Watanabe added 2 comments![]( "Open in Gerrit")
CC: ms...@google.com
The context is that https://crrev.com/c/7809476 caused a build performance regression on Windows.
"third_party/libc++/src/include:headers",Does this CL mean `third_party/libc++/src/include:headers` wasn't necessary...?
Related details
- Ryosuke Miyake
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Junji Watanabe (Gerrit)
Junji Watanabe added 1 comment![]( "Open in Gerrit")
"third_party/libc++/src/include:headers",Does this CL mean `third_party/libc++/src/include:headers` wasn't necessary...?
I guess it should rely on the Siso's internal handling for tree inputs rather than explicitly specifying ":headers" filegroup.
https://source.chromium.org/chromium/build/+/main:siso/build/deps_gcc.go;l=91-100;drc=633997ad07f17a700a6822c5dbca050ec5ef26e6
@uk...@google.com Can you confirm if this config was misuse?
Related details
- Fumitoshi Ukai
- Ryosuke Miyake
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Fumitoshi Ukai (Gerrit)
Fumitoshi Ukai added 2 comments![]( "Open in Gerrit")
inputs to all compiler targets unconditionally. This increases thehmm, this sounds like hashfs bug. it should not call lstat so many times?
"third_party/libc++/src/include:headers",Junji WatanabeDoes this CL mean `third_party/libc++/src/include:headers` wasn't necessary...?
I guess it should rely on the Siso's internal handling for tree inputs rather than explicitly specifying ":headers" filegroup.
https://source.chromium.org/chromium/build/+/main:siso/build/deps_gcc.go;l=91-100;drc=633997ad07f17a700a6822c5dbca050ec5ef26e6@uk...@google.com Can you confirm if this config was misuse?
`:headers` are specially handled if this directory is used as sysroot/include dirs on command line (as used as precomputed headers).
other suffix are handled as just filegroup, so always expand.
Related details
- Ryosuke Miyake
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Junji Watanabe (Gerrit)
Junji Watanabe added 1 comment![]( "Open in Gerrit")
"third_party/libc++/src/include:headers",Junji WatanabeDoes this CL mean `third_party/libc++/src/include:headers` wasn't necessary...?
Fumitoshi UkaiI guess it should rely on the Siso's internal handling for tree inputs rather than explicitly specifying ":headers" filegroup.
https://source.chromium.org/chromium/build/+/main:siso/build/deps_gcc.go;l=91-100;drc=633997ad07f17a700a6822c5dbca050ec5ef26e6@uk...@google.com Can you confirm if this config was misuse?
`:headers` are specially handled if this directory is used as sysroot/include dirs on command line (as used as precomputed headers).
other suffix are handled as just filegroup, so always expand.
I wanted to ask if you specify :headers filegroup as an input like this, does it expand it repeatedly, which ended up having many lstats rather than reusing the precomputed tree?
Junji Watanabe (Gerrit)
Junji Watanabe added 1 comment![]( "Open in Gerrit")
inputs to all compiler targets unconditionally. This increases thehmm, this sounds like hashfs bug. it should not call lstat so many times?
I guess if multiple compiles start running in parallel with the filegroup. All compiles try to do the existence check in parallel competing each other?
Matt Stark (Gerrit)
Matt Stark voted Code-Review+1![]( "Open in Gerrit")
| Code-Review | +1 |
Related details
- Ryosuke Miyake
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Fumitoshi Ukai (Gerrit)
Fumitoshi Ukai added 1 comment![]( "Open in Gerrit")
inputs to all compiler targets unconditionally. This increases theJunji Watanabehmm, this sounds like hashfs bug. it should not call lstat so many times?
I guess if multiple compiles start running in parallel with the filegroup. All compiles try to do the existence check in parallel competing each other?
I think precomputed tree compute digest once.
https://chromium.googlesource.com/build/+/refs/heads/main/siso/build/tree_input.go#127
Junji Watanabe (Gerrit)
Junji Watanabe added 1 comment![]( "Open in Gerrit")
inputs to all compiler targets unconditionally. This increases theJunji Watanabehmm, this sounds like hashfs bug. it should not call lstat so many times?
Fumitoshi UkaiI guess if multiple compiles start running in parallel with the filegroup. All compiles try to do the existence check in parallel competing each other?
I think precomputed tree compute digest once.
https://chromium.googlesource.com/build/+/refs/heads/main/siso/build/tree_input.go#127
That is only when the filegroup is treated via [treeInput](https://source.chromium.org/chromium/build/+/main:siso/build/tree_input.go;l=82;bpv=1;bpt=1?q=treeInput%20-file:%5Eout%20-file:%5Egen), right?
If you add `foo:headers` to `inputs`, it's expanded as a normal filegroup, IIUC.
Junji Watanabe (Gerrit)
Junji Watanabe added 1 comment![]( "Open in Gerrit")
"third_party/libc++/src/include:headers",Probably, you can just remove this filegroup from the libcxx_inputs.
Fumitoshi Ukai (Gerrit)
Fumitoshi Ukai added 1 comment![]( "Open in Gerrit")
inputs to all compiler targets unconditionally. This increases theJunji Watanabehmm, this sounds like hashfs bug. it should not call lstat so many times?
Fumitoshi UkaiI guess if multiple compiles start running in parallel with the filegroup. All compiles try to do the existence check in parallel competing each other?
Junji WatanabeI think precomputed tree compute digest once.
https://chromium.googlesource.com/build/+/refs/heads/main/siso/build/tree_input.go#127
That is only when the filegroup is treated via [treeInput](https://source.chromium.org/chromium/build/+/main:siso/build/tree_input.go;l=82;bpv=1;bpt=1?q=treeInput%20-file:%5Eout%20-file:%5Egen), right?
If you add `foo:headers` to `inputs`, it's expanded as a normal filegroup, IIUC.
yeah, but then it would cause hashfs contention, but not call lstat so much?
Junji Watanabe (Gerrit)
Junji Watanabe added 1 comment![]( "Open in Gerrit")
inputs to all compiler targets unconditionally. This increases theJunji Watanabehmm, this sounds like hashfs bug. it should not call lstat so many times?
Fumitoshi UkaiI guess if multiple compiles start running in parallel with the filegroup. All compiles try to do the existence check in parallel competing each other?
Junji WatanabeI think precomputed tree compute digest once.
https://chromium.googlesource.com/build/+/refs/heads/main/siso/build/tree_input.go#127
Fumitoshi UkaiThat is only when the filegroup is treated via [treeInput](https://source.chromium.org/chromium/build/+/main:siso/build/tree_input.go;l=82;bpv=1;bpt=1?q=treeInput%20-file:%5Eout%20-file:%5Egen), right?
If you add `foo:headers` to `inputs`, it's expanded as a normal filegroup, IIUC.
yeah, but then it would cause hashfs contention, but not call lstat so much?
it would cause hashfs contention,
What symptoms may happen under hashfs contentions?
Fumitoshi Ukai (Gerrit)
Fumitoshi Ukai added 1 comment![]( "Open in Gerrit")
inputs to all compiler targets unconditionally. This increases theJunji Watanabehmm, this sounds like hashfs bug. it should not call lstat so many times?
Fumitoshi UkaiI guess if multiple compiles start running in parallel with the filegroup. All compiles try to do the existence check in parallel competing each other?
Junji WatanabeI think precomputed tree compute digest once.
https://chromium.googlesource.com/build/+/refs/heads/main/siso/build/tree_input.go#127
Fumitoshi UkaiThat is only when the filegroup is treated via [treeInput](https://source.chromium.org/chromium/build/+/main:siso/build/tree_input.go;l=82;bpv=1;bpt=1?q=treeInput%20-file:%5Eout%20-file:%5Egen), right?
If you add `foo:headers` to `inputs`, it's expanded as a normal filegroup, IIUC.
Junji Watanabeyeah, but then it would cause hashfs contention, but not call lstat so much?
it would cause hashfs contention,
What symptoms may happen under hashfs contentions?
hashfs operation becomes slightly slow?
Junji Watanabe (Gerrit)
Junji Watanabe added 1 comment![]( "Open in Gerrit")
inputs to all compiler targets unconditionally. This increases theJunji Watanabehmm, this sounds like hashfs bug. it should not call lstat so many times?
Fumitoshi UkaiI guess if multiple compiles start running in parallel with the filegroup. All compiles try to do the existence check in parallel competing each other?
Junji WatanabeI think precomputed tree compute digest once.
https://chromium.googlesource.com/build/+/refs/heads/main/siso/build/tree_input.go#127
Fumitoshi UkaiThat is only when the filegroup is treated via [treeInput](https://source.chromium.org/chromium/build/+/main:siso/build/tree_input.go;l=82;bpv=1;bpt=1?q=treeInput%20-file:%5Eout%20-file:%5Egen), right?
If you add `foo:headers` to `inputs`, it's expanded as a normal filegroup, IIUC.
Junji Watanabeyeah, but then it would cause hashfs contention, but not call lstat so much?
it would cause hashfs contention,
What symptoms may happen under hashfs contentions?
I don't know if I understand your replies. But, we observe slow lstat operations via [hashfs's lstat](https://source.chromium.org/chromium/build/+/main:siso/hashfs/osfs/osfs.go;l=147;drc=2b3c0dd099c66a67da5dc2c54ff37c80225432d2) call. This is why the size for LstatSemaphore was discussed in another CL.
Do you think that's a separate issue?
Fumitoshi Ukai (Gerrit)
Fumitoshi Ukai added 1 comment![]( "Open in Gerrit")
inputs to all compiler targets unconditionally. This increases theJunji Watanabehmm, this sounds like hashfs bug. it should not call lstat so many times?
Fumitoshi UkaiI guess if multiple compiles start running in parallel with the filegroup. All compiles try to do the existence check in parallel competing each other?
Junji WatanabeI think precomputed tree compute digest once.
https://chromium.googlesource.com/build/+/refs/heads/main/siso/build/tree_input.go#127
Fumitoshi UkaiThat is only when the filegroup is treated via [treeInput](https://source.chromium.org/chromium/build/+/main:siso/build/tree_input.go;l=82;bpv=1;bpt=1?q=treeInput%20-file:%5Eout%20-file:%5Egen), right?
If you add `foo:headers` to `inputs`, it's expanded as a normal filegroup, IIUC.
Junji Watanabeyeah, but then it would cause hashfs contention, but not call lstat so much?
Junji Watanabeit would cause hashfs contention,
What symptoms may happen under hashfs contentions?
I don't know if I understand your replies. But, we observe slow lstat operations via [hashfs's lstat](https://source.chromium.org/chromium/build/+/main:siso/hashfs/osfs/osfs.go;l=147;drc=2b3c0dd099c66a67da5dc2c54ff37c80225432d2) call. This is why the size for LstatSemaphore was discussed in another CL.
Do you think that's a separate issue?
hashfs keeps stat result (that's the main purpose of hashfs), so if hashfs stat repeatedly calls os lstat, it would be hashfs bug.
Ryosuke Miyake (Gerrit)
Ryosuke Miyake voted Commit-Queue+1![]( "Open in Gerrit")
- Fumitoshi Ukai
- Junji Watanabe
- Matt Stark
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Ryosuke Miyake (Gerrit)
Ryosuke Miyake voted and added 2 comments![]( "Open in Gerrit")
Votes added by Ryosuke Miyake
| Commit-Queue | +1 |
2 comments
inputs to all compiler targets unconditionally. This increases theJunji Watanabehmm, this sounds like hashfs bug. it should not call lstat so many times?
Fumitoshi UkaiI guess if multiple compiles start running in parallel with the filegroup. All compiles try to do the existence check in parallel competing each other?
Junji WatanabeI think precomputed tree compute digest once.
https://chromium.googlesource.com/build/+/refs/heads/main/siso/build/tree_input.go#127
Fumitoshi UkaiThat is only when the filegroup is treated via [treeInput](https://source.chromium.org/chromium/build/+/main:siso/build/tree_input.go;l=82;bpv=1;bpt=1?q=treeInput%20-file:%5Eout%20-file:%5Egen), right?
If you add `foo:headers` to `inputs`, it's expanded as a normal filegroup, IIUC.
Junji Watanabeyeah, but then it would cause hashfs contention, but not call lstat so much?
Junji Watanabeit would cause hashfs contention,
What symptoms may happen under hashfs contentions?
Fumitoshi UkaiI don't know if I understand your replies. But, we observe slow lstat operations via [hashfs's lstat](https://source.chromium.org/chromium/build/+/main:siso/hashfs/osfs/osfs.go;l=147;drc=2b3c0dd099c66a67da5dc2c54ff37c80225432d2) call. This is why the size for LstatSemaphore was discussed in another CL.
Do you think that's a separate issue?
hashfs keeps stat result (that's the main purpose of hashfs), so if hashfs stat repeatedly calls os lstat, it would be hashfs bug.
Reference: http://crrev.com/c/7951179.
"third_party/libc++/src/include:headers",Probably, you can just remove this filegroup from the libcxx_inputs.
Done
Related details
- Fumitoshi Ukai
- Junji Watanabe
- Matt Stark
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Junji Watanabe (Gerrit)
Junji Watanabe voted and added 2 comments![]( "Open in Gerrit")
Votes added by Junji Watanabe
| Code-Review | +1 |
2 comments
inputs to all compiler targets unconditionally. This increases theJunji Watanabehmm, this sounds like hashfs bug. it should not call lstat so many times?
Fumitoshi UkaiI guess if multiple compiles start running in parallel with the filegroup. All compiles try to do the existence check in parallel competing each other?
Junji WatanabeI think precomputed tree compute digest once.
https://chromium.googlesource.com/build/+/refs/heads/main/siso/build/tree_input.go#127
Fumitoshi UkaiThat is only when the filegroup is treated via [treeInput](https://source.chromium.org/chromium/build/+/main:siso/build/tree_input.go;l=82;bpv=1;bpt=1?q=treeInput%20-file:%5Eout%20-file:%5Egen), right?
If you add `foo:headers` to `inputs`, it's expanded as a normal filegroup, IIUC.
Junji Watanabeyeah, but then it would cause hashfs contention, but not call lstat so much?
Junji Watanabeit would cause hashfs contention,
What symptoms may happen under hashfs contentions?
Fumitoshi UkaiI don't know if I understand your replies. But, we observe slow lstat operations via [hashfs's lstat](https://source.chromium.org/chromium/build/+/main:siso/hashfs/osfs/osfs.go;l=147;drc=2b3c0dd099c66a67da5dc2c54ff37c80225432d2) call. This is why the size for LstatSemaphore was discussed in another CL.
Do you think that's a separate issue?
Ryosuke Miyakehashfs keeps stat result (that's the main purpose of hashfs), so if hashfs stat repeatedly calls os lstat, it would be hashfs bug.
Reference: http://crrev.com/c/7951179.
(unblocking this CL.)
"third_party/libc++/src/include:headers",Does this CL mean `third_party/libc++/src/include:headers` wasn't necessary...?
Fumitoshi UkaiI guess it should rely on the Siso's internal handling for tree inputs rather than explicitly specifying ":headers" filegroup.
https://source.chromium.org/chromium/build/+/main:siso/build/deps_gcc.go;l=91-100;drc=633997ad07f17a700a6822c5dbca050ec5ef26e6@uk...@google.com Can you confirm if this config was misuse?
Junji Watanabe`:headers` are specially handled if this directory is used as sysroot/include dirs on command line (as used as precomputed headers).
other suffix are handled as just filegroup, so always expand.
I wanted to ask if you specify :headers filegroup as an input like this, does it expand it repeatedly, which ended up having many lstats rather than reusing the precomputed tree?
- Fumitoshi Ukai
- Matt Stark
- Ryosuke Miyake
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Ryosuke Miyake (Gerrit)
Ryosuke Miyake voted Commit-Queue+1![]( "Open in Gerrit")
- Fumitoshi Ukai
- Matt Stark
- Takuto Ikuta
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Takuto Ikuta (Gerrit)
Takuto Ikuta added 1 comment![]( "Open in Gerrit")
This change optimizes how Siso tracks libc++ headers to reduce the
number of filesystem operations (lstat) on Windows.
Previously, origin/main attached third_party/libc++/src/include:headers
(~240 files) directly to compiler targets via libcxx_inputs. Because
filegroups attached to executable dependencies are unconditionally
expanded on every compile step rather than being optimized as
precomputed sysroot trees, this generated excessive concurrent lstat
calls on Windows.
This change resolves the issue by removing libcxx_inputs from executable
dependencies and instead mapping third_party/libc++/src/include via
__input_deps directly to literal custom files (__config_site,
__assertion_handler). Standard libc++ headers route dynamically through
precomputed tree caching (tree_input.go) with zero filegroup expansion.
Verified on win-build-perf-developer builder, confirming build success
and eliminating regression.This description doesn't match with current patchset?
Related details
- Fumitoshi Ukai
- Matt Stark
- Ryosuke Miyake
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Ryosuke Miyake (Gerrit)
Ryosuke Miyake voted Commit-Queue+1![]( "Open in Gerrit")
- Fumitoshi Ukai
- Matt Stark
- Takuto Ikuta
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Ryosuke Miyake (Gerrit)
Ryosuke Miyake added 1 comment![]( "Open in Gerrit")
This change optimizes how Siso tracks libc++ headers to reduce the
number of filesystem operations (lstat) on Windows.
Previously, origin/main attached third_party/libc++/src/include:headers
(~240 files) directly to compiler targets via libcxx_inputs. Because
filegroups attached to executable dependencies are unconditionally
expanded on every compile step rather than being optimized as
precomputed sysroot trees, this generated excessive concurrent lstat
calls on Windows.
This change resolves the issue by removing libcxx_inputs from executable
dependencies and instead mapping third_party/libc++/src/include via
__input_deps directly to literal custom files (__config_site,
__assertion_handler). Standard libc++ headers route dynamically through
precomputed tree caching (tree_input.go) with zero filegroup expansion.
Verified on win-build-perf-developer builder, confirming build success
and eliminating regression.This description doesn't match with current patchset?
Done
Related details
- Fumitoshi Ukai
- Matt Stark
- Takuto Ikuta
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Takuto Ikuta (Gerrit)
Takuto Ikuta voted and added 1 comment![]( "Open in Gerrit")
Votes added by Takuto Ikuta
| Code-Review | +1 |
1 comment
Verified on win-build-perf-developer builder, confirming build success
and eliminating regression.Could you leave link to this build as a reference?
Related details
- Fumitoshi Ukai
- Matt Stark
- Ryosuke Miyake
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Ryosuke Miyake (Gerrit)
Ryosuke Miyake voted and added 1 comment![]( "Open in Gerrit")
Votes added by Ryosuke Miyake
| Commit-Queue | +1 |
1 comment
Verified on win-build-perf-developer builder, confirming build success
and eliminating regression.Could you leave link to this build as a reference?
Done
Related details
- Fumitoshi Ukai
- Matt Stark
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Ryosuke Miyake (Gerrit)
Ryosuke Miyake voted Commit-Queue+2![]( "Open in Gerrit")
| Commit-Queue | +2 |
Chromium LUCI CQ (Gerrit)
Chromium LUCI CQ submitted the change![]( "Open in Gerrit")
Unreviewed changes
12 is the latest approved patch-set.
No files were changed between the latest approved patch-set and the submitted one.
Change information
Siso: Optimize libc++ header tracking to reduce Windows lstat calls
This change optimizes how Siso tracks libc++ headers to reduce the
number of filesystem operations (lstat) on Windows.
Previously, third_party/libc++/src/include:headers (~240 files) was
included in libcxx_inputs, which is mapped as input dependencies to
compiler executables (clang++, clang-cl, etc.). Because filegroups
attached to executable dependencies are unconditionally expanded on
every compile step, this generated excessive concurrent lstat calls on
Windows.
This change resolves the issue by removing
third_party/libc++/src/include:headers from libcxx_inputs. Standard
libc++ headers are already handled dynamically by Siso via precomputed
tree caching with zero filegroup expansion, making the explicit
filegroup dependency redundant.
Verified on win-build-perf-developer builder, confirming build success
and eliminating regression.
- M build/config/siso/clang_all.star
Code-Review: +1 by Junji Watanabe, +1 by Takuto Ikuta
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |