PSA: Turn on Git’s fsmonitor!

[Avi Drissman]

Not too long ago, Git got a feature called "fsmonitor" which significantly helps with large git repos and with platforms like Mac and Win where filesystem performance is an issue. It's pretty simple as long as you have an up-to-date git:

> git config --global core.untrackedCache true

> git config --global core.fsmonitor true

The speed gains are astonishing. I highly recommend that you enable this feature on git.

Avi

[dan...@chromium.org]

Thanks Avi

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACWgwAYAfMOaQmO6n1C6Ft9zEaTiPrHoYWu5rYBbOiN9TEi0SA%40mail.gmail.com.

[Dale Curtis]

+1, this is amazing. I didn't realize this was now built into git. On macOS git status went from ~4s to 0.6s.

$ git status
On branch ms
Your branch is ahead of 'main' by 1 commit.
  (use "git push" to publish your local commits)


It took 3.19 seconds to enumerate untracked files,
but the results were cached, and subsequent runs may be faster.
See 'git help status' for information on how to improve this.

nothing to commit, working tree clean

$ time git status
On branch ms
Your branch is ahead of 'main' by 1 commit.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean
git status  0.61s user 0.07s system 84% cpu 0.802 total

- dale

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHtyhaRW8iucQd8ECmZpZnMNo%2BW1FeOqhvo_O%2B3yr0fmOotsBg%40mail.gmail.com.

[dan...@chromium.org]

This should go in

https://chromium.googlesource.com/chromium/src/+/main/docs/windows_build_instructions.md#get-the-code

https://chromium.googlesource.com/chromium/src/+/main/docs/mac_build_instructions.md#Get-the-code

I see the mac one does mention untrackedCache though not on the well lit path, and doesn't have fsmonitor yet.

Any takers?

[Avi Drissman]

I'll add it. Just wanted to scream from the top of the mountain how good this is.

[dan...@chromium.org]

Thanks, without the email only new folks would ever know!

[Feras Aldahlawi]

This cannot be real!!! 

FYI, if you are using Mac-M1 you might need to run `git fsmonitor--daemon start` before you see gains. Check the status using `git fsmonitor--daemon status`.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHtyhaQ-VtdOuo%3D1obKrvbyDM9wr9Owft1xw96%2BPyL7J-0%3DU0Q%40mail.gmail.com.

[Peter Kasting]

On Fri, May 26, 2023 at 7:44 AM 'Avi Drissman' via Chromium-dev <chromi...@chromium.org> wrote:

> git config --global core.untrackedCache true

For me this one is the big win -- I've had fsmonitor on for awhile without too much speed gain, but this setting really cuts `git status` times on Windows.

PK 

[Steve Kobes]

Wow, core.untrackedCache speeds up Linux too.  For me git status went from 2s to 0.8s.

Interestingly it seems slightly slower with both core.untrackedCache and core.fsmonitor, compared to setting only core.untrackedCache.

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAHOzFBDLk7ZnP0-y0SV0kXi82g441W_ksx0tPi1_qQ-ii3cwg%40mail.gmail.com.

[Avi Drissman]

You should not need to explicitly start it.

https://git-scm.com/docs/git-fsmonitor--daemon:

When core.fsmonitor is set to true (see git-config[1]) commands, such as git status, will ask the daemon for changes and automatically start it (if necessary).

$ git fsmonitor--daemon status

for me shows the daemon running even though I did not explicitly start it.

[Bruce Dawson]

The speedup for me is particularly noticeable on "git cl status". I often try to avoid this command because it is too slow, and now I don't think I'll need to.

[Bruce Dawson]

While experimenting with this feature on Windows and monitoring for any possible issues I came across this article which then led me to notice that I now have 178 fsmonitor--daemon processes running on my laptop, measured with this command:

    wmic process where "commandline like 'git fsmon%'" get processid, commandline | find /c "fsmonitor--daemon"

The count doesn't seem to grow over time and it appears to be an implementation detail, with each process monitoring a different directory. You can see the directory being monitored by running this command:

    handle -p %%d | find "RWD"

where %%d is the PID.

I haven't tested on Linux or OSX so I don't know if the behavior is similar. The process count does increase if I run "git status" in another repo, which makes sense. So, FYI.

--

Bruce Dawson, he/him

[Avi Drissman]

On my macOS box, I do have a bunch as well:

~> ps a | grep [f]smonitor | wc -l
     169

I'm ok with it; the daemons are quiescent and are blocked when nothing is happening, so there's no system impact.

fsmonitor does not function on Linux; inotify support has not been added.

Nearly every aspect of git is sped up by fixing status; the article that I linked in the first post of this thread illustrates why. Happy to hear that it's helped you!

Avi

[K. Moon]

I suspect the usual caveats apply to anything performing file system monitoring: Since operations are now eager, if you touch a lot of the file system at once, you'll probably incur high resource usage during the updates. Otherwise, I can't see much downside. (That said, I'm also too lazy to change anything, as I don't really spend that much of my day on Git operations.)

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACWgwAbnOsRhoXv-pnm-PU4SXu5JFCjn3%3D%3DGxwREoZYae6bZ4g%40mail.gmail.com.

[Mark Mentovai]

It’s one git fsmonitor--daemon process per repository. 169 and 178 are entirely plausible numbers of git repositories in a full gclient-based Chrome checkout.

If you don’t like it, you could set core.fsmonitor = true just for the repository where you need it (git config --local inside Chrome, for example) instead of git config --global.

Since the daemon watching the Chrome checkout can (and does) see changes in other repositories, technically there should be a way to reduce this to just a single daemon and single filesystem watcher for all of Chrome. This is essentially the “it doesn’t understand submodules” limitation from the git-fsmonitor--daemon man page, although the man page focuses on the “outer repository’s watch picks up changes to inner repositories but has no use for them” aspect, and not the “watches on inner repositories duplicate the watch on the outer” aspect. It seems like enhancing this would make a suitable Summer of Code-type project.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACWgwAbnOsRhoXv-pnm-PU4SXu5JFCjn3%3D%3DGxwREoZYae6bZ4g%40mail.gmail.com.

[Caleb Raitto]

For some reason, on a gLinux cloud machine, I'm not sure this works?

$ git fsmonitor--daemon status
fatal: fsmonitor--daemon not supported on this platform

$ git --version
git version 2.41.0.rc0.172.g3f132b7071-goog

Seems like the version should be new enough, although the Google internal version may be customized...

I didn't notice any speedup either after running the 2 git config commands.

-Caleb

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

[Christian Biesinger]

Yes, this does not work on Linux.

You could try the approach described in https://www.spinics.net/lists/git/msg458447.html

Christian

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACWgwAbnOsRhoXv-pnm-PU4SXu5JFCjn3%3D%3DGxwREoZYae6bZ4g%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/179c4525-c10a-4628-b39b-486f7ac0a0d6n%40chromium.org.

[Bruce Dawson]

> Since operations are now eager, if you touch a lot of the file system at once, you'll probably incur high resource usage during the updates

I took a brief look at this on Windows by doing a build and monitoring accumulated CPU time across all of the fsmonitor processes. One of them accumulated about eight seconds of CPU time and the others accumulated none. Compared to the cost of a build, and compared to the number of files generated, this is tiny so it looks like (on Windows anyway) the overhead is not a problem.

After a couple of days of use the busiest fsmonitor process has now accumulated 22 seconds of CPU time and the others are still sitting at zero. This seems fine, but I recommend checking the "CPU time" column after a few days if you think your usage patterns might be more problematic.

[K. Moon]

I was thinking something more like, "You decided to change every single copyright header in the source tree," rather than a build. But I agree that it's probably negligible, unless you're doing the rewrites in a tight loop for some reason.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/0f7f5f4b-ff1b-46be-a972-0ae2dafe2810n%40chromium.org.

[Avi Drissman]

The amount of CPU and usage used by tracking changes as they occur is far less than the CPU and usage of repeatedly scrubbing through the git directory that it prevents.

No matter how much CPU the fsmonitor would take, it would be worth the tradeoff.

Avi

[K. Moon]

I don't think the overall conclusion changes, but I don't think that's strictly true: ~Infinity times a small number is still bigger than ~1 times a big number. The actual answer is still workload-dependent.

[Henrik Andreasson]

FYI, see Improve Git monorepo performance with a file system monitor for more details

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACwGi-4nz-jOdCNmLBne%2B8M6XK3t7Ajv%2BUHJ5YthyTreszZpNQ%40mail.gmail.com.

[Mason Freed]

I just wanted to say thanks to Avi for calling this out! It is so much better than before.

On Tuesday, May 30, 2023 at 10:22:12 AM UTC-7 Christian Biesinger wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACWgwAbnOsRhoXv-pnm-PU4SXu5JFCjn3%3D%3DGxwREoZYae6bZ4g%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

[Philipp Wollermann]

I wanted to enable this on Linux and it seems that this made it work.

curl -o ~/bin/query-watchman https://raw.githubusercontent.com/git/git/master/templates/hooks--fsmonitor-watchman.sample

chmod +x ~/bin/query-watchman

git config --global core.fsmonitor $HOME/bin/query-watchman
git config --global core.untrackedcache true

Benchmark:

philwo@philwo:chromium/src ‹7ed625e6e8f54›$ hyperfine \
    --shell /bin/bash \
    --warmup 1 \
    --prepare "git config --global core.untrackedcache false; git config --global core.fsmonitor false" \
    --command-name "no untrackedcache, no fsmonitor" \
    "git status" \
    --prepare "git config --global core.untrackedcache true; git config --global core.fsmonitor false" \
    --command-name "untrackedcache only" \
    "git status" \
    --prepare "git config --global core.untrackedcache false; git config --global core.fsmonitor $HOME/bin/query-watchman" \
    --command-name "fsmonitor only" \
    "git status" \
    --prepare "git config --global core.untrackedcache true; git config --global core.fsmonitor $HOME/bin/query-watchman" \
    --command-name "untrackedcache + fsmonitor" \
    "git status"

Benchmark 1: no untrackedcache, no fsmonitor
  Time (mean ± σ):      1.149 s ±  0.008 s    [User: 0.605 s, System: 4.074 s]
  Range (min … max):    1.135 s …  1.160 s    10 runs
 
Benchmark 2: untrackedcache only
  Time (mean ± σ):     573.5 ms ±   2.1 ms    [User: 367.4 ms, System: 3747.5 ms]
  Range (min … max):   569.6 ms … 576.2 ms    10 runs
 
Benchmark 3: fsmonitor only
  Time (mean ± σ):      1.030 s ±  0.006 s    [User: 0.582 s, System: 0.445 s]
  Range (min … max):    1.022 s …  1.040 s    10 runs
 
Benchmark 4: untrackedcache + fsmonitor
  Time (mean ± σ):     408.4 ms ±   4.3 ms    [User: 372.6 ms, System: 37.9 ms]
  Range (min … max):   400.3 ms … 415.0 ms    10 runs
 
Summary
  'untrackedcache + fsmonitor' ran
    1.40 ± 0.02 times faster than 'untrackedcache only'
    2.52 ± 0.03 times faster than 'fsmonitor only'
    2.81 ± 0.03 times faster than 'no untrackedcache, no fsmonitor'

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAD2mqc4rcQPoTOzomnRpKNzQt%3DRpqhEbUZM0u9EOAXUkOAYpXg%40mail.gmail.com.

[Panos Astithas]

Full Linux support seems to have stalled.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAjAU%3DaymfP2PeCt%3DucLThbhR_N661qDzudeQroAjJYtZiTVFw%40mail.gmail.com.

[Avi Drissman]

There are two different features in git with the same name.

core.fsmonitor = true enables the built-in fsmonitor on Windows/Mac. It is your best option for those platforms, as it's tuned to git's needs.

core.fsmonitor = <monitor process> has git consult with an external monitor process. It's available on all platforms, though you would want to use the built-in fsmonitor if you can. However, as you found, even this feature is well-worth activating.

Avi

[Ken Russell]

This is such a tremendous improvement that I also had to spam the list and say thank you Avi for highlighting it and providing clear instructions for how to enable it!

-Ken

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACWgwAZ5z%3DSiUtyBDo76WE5KfMq2kdMNUGAQjSfu4SKHHvQ6dg%40mail.gmail.com.

[Christian Biesinger]

Thanks for the detailed instructions!

It looks like the git package comes with /usr/share/git-core/templates/hooks/fsmonitor-watchman.sample, so you can probably use that instead of curl+chmod

Also I assume you need to `sudo apt-get install watchman` if not already installed.

Thanks,

Christian

On Thu, Jun 1, 2023 at 2:07 AM Philipp Wollermann <phi...@google.com> wrote:

[K. Moon]

Given so many people are finding it useful, would it be worth documenting in the checkout instructions somewhere? Like, "Hey, Chromium is really big, you'd probably want to turn this Git feature on. Here are some instructions." (The instructions themselves might be on a separate page, or point to an external resource.)

[dan...@chromium.org]

Avi already has done this :)

[K. Moon]

Great!

[Caleb Raitto]

Per cbiesinger@'s mail, I was able to get Linux working with sudo apt install watchman and git config --global core.fsmonitor /usr/share/git-core/templates/hooks/fsmonitor-watchman.sample:

(The first run of git status is slow, but then it's fast after that):

$ time git status
HEAD detached at origin/main

nothing to commit, working tree clean

real    0m0.395s
user    0m0.357s
sys     0m0.050s

-Caleb

[Avi Drissman]

Since it's a one-liner for Mac/Windows, it was easy for me to add it to their build instructions. Could one of you Linux folks add the more detailed instructions to use the fsmonitor hooks to the Linux build instructions?

Avi

[Philipp Wollermann]

I sent a CL to add this to the docs, please free to comment on how to improve it: https://chromium-review.git.corp.google.com/c/chromium/src/+/4583709

Philipp

[Bruce Dawson]

Some follow-up information:

1) I was wondering what was triggering the accumulation of so many fsmonitor processes since they should only spawn when I do git operations. Then I realized that "gclient sync" does some git operations in all of the sub-repos, and that is indeed what triggers so many fsmonitor--daemon processes to appear.

2) On Windows the fsmonitor--daemon processes consume an average of 86 MB of commit. This means they allocate 86 MB of memory but mostly don't use it, however Windows promises that the memory will be available (no overcommit). 86 MB is not much, but 86 MB times 179 processes works out to over 15 GB of commit. This doesn't consume any physical RAM, but most OOM failures on Windows are due to running out of commit, not RAM, so this still matters. You can easily run out of commit and OOM with tens of GB free.

3) If you care about this (on a workstation you probably don't need to) you could increase the size of your pagefile (thus increasing the commit limit) - even though that extra space will never be used. Or, more simply, you can disable fsmonitor globally and just enable it for large repos such as chromium.

So, on Windows my advice would be to set "git config --global core.fsmonitor false" and then on specific large repos (Chromium, v8, ???) set "git config core.fsmonitor true". This gives you the benefits (faster Chromium git operations) without the cost (excessive processes and wasteful commits for repos you rarely use). I will update the Windows documentation.

Similar issues may occur on Linux/OSX, although the ability to overcommit may make them less important. I like having fewer processes so I think I'll follow this policy on my Linux box as well, and I think that this advice is probably appropriate for all platforms.

Other than Chromium and v8 are there any repos that are large enough to need fsmonitor enabled?

P.S. This Task Manager display demonstrates the problem. I have 24 GB of RAM free but if I didn't have a page file I'd be critically low on commit, and large allocations could start failing.

--

Bruce Dawson, he/him

[Bruce Dawson]

VsChromium on Windows also uses file system notifications to track changes, and it also receives notifications for every file written in the out directory. This is necessary in that case because I want VsChromium to be able to search generated code in the out directories.

For a while the CPU usage was problematic but by adjusting the buffer sizes we were able to reduce it to a manageable level. Also, if VsChromium detects high enough activity then it stops monitoring for a while and rescans the src directory later.

I wonder if any of these lessons from Windows could be applied on Linux. I guess you'd have to file a bug against watchman and/or git.

On Sun, Jun 18, 2023 at 11:10 PM Philipp Wollermann <phi...@google.com> wrote:

Thanks Bruce, this is good information!

Two more issues I noticed with this feature on Linux after using it for 2 weeks now:

1) I think watchman might also get inotify events for the build output folder, which results in it using quite a lot of CPU during a build. I wonder if we can exclude the "out" directory somehow.

2) I occasionally got error messages from various tools about my system running out of inotify watches, so I added this to my /etc/sysctl.d/philwo.conf:

fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 10485760

Also, on macOS I ran into issues where all "git" commands would just hang after a while, until I killed the fsmonitor--daemon processes. I disabled the feature on macOS for now.

Cheers,

Philipp

--

Bruce Dawson, he/him

[Joe Mason]

As a quick check, the 10 biggest .git dirs in my Windows checkout are:

$ find . -type d -name .git -print0|xargs -0 du -hs|sort -hr|head
31G     ./.git
2.0G    ./third_party/llvm/.git
1.4G    ./third_party/tflite/src/.git
1.2G    ./third_party/icu/.git
1.2G    ./third_party/angle/third_party/VK-GL-CTS/src/.git
1.1G    ./v8/.git
947M    ./tools/perf/data/.git
940M    ./third_party/sqlite/src/.git
813M    ./third_party/swiftshader/.git
785M    ./third_party/skia/.git

I think that's proportional to the size of the history, not the size of the repo being monitored, but it's probably got a correlation.

I'm not sure if there's a point in turning on fsmonitor in a repo that I'm not actively working in - would it make gclient sync faster? 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAE5mQiO9evXq6p_mRppeYju%2Bqa%2B5U2CxadTF3_hbMsjwSWDBnQ%40mail.gmail.com.

[Bruce Dawson]

Thanks for the data!

If you run gclient sync frequently (once to prime the cache, subsequent times to reap the benefits) then you would probably get a modest speedup by having fsmonitor enabled in repos you don't otherwise use, but the time savings are probably modest. Given the costs (on Windows at least) and the tiny savings for non-Chromium repos I'm sticking with just enabling fsmonitor for the few large repos that I ever work on.

--

Bruce Dawson, he/him

[Philipp Wollermann]

Thanks Bruce, this is good information!

Two more issues I noticed with this feature on Linux after using it for 2 weeks now:

1) I think watchman might also get inotify events for the build output folder, which results in it using quite a lot of CPU during a build. I wonder if we can exclude the "out" directory somehow.

2) I occasionally got error messages from various tools about my system running out of inotify watches, so I added this to my /etc/sysctl.d/philwo.conf:

fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 10485760

Also, on macOS I ran into issues where all "git" commands would just hang after a while, until I killed the fsmonitor--daemon processes. I disabled the feature on macOS for now.

Cheers,

Philipp

[Erik Jensen]

I have my build directory next to src instead of under it, which prevents git from scanning it. The downside is I can't run git commands if I'm in the build tree.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CA%2BAhZogPROoENfYobWLHoEv5-GLBUofjrXwhBaPLY7PR0dQ0vg%40mail.gmail.com.

[Jinyoung Hur]

For me, configuring the build directory next to src,  my Sublime Text on Windows gets much faster while building.

However, Chromium build for Windows requires the out directory should be under the src directory.

https://chromium.googlesource.com/chromium/src/+/main/docs/windows_build_instructions.md#setting-up-the-build

So I had to modify here and there to build and run tests.

It would be great if we can configure the out directory freely for Windows.

2023년 6월 23일 금요일 오전 5시 52분 51초 UTC+9에 Erik Jensen님이 작성:

[Dirk Pranke]

We don't intentionally require the build to be under //out/$something . It's just that almost no one uses anything else.

I would think we'd accept patches to make building outside the src directory work.

-- Dirk

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/38d6149f-6c67-41e5-a59e-1b2bdf7aa842n%40chromium.org.

[Sylvain Defresne]

While we don't explicitly require the build to be //out/$something, there is some code in //base that assumes the root of the repository can be accessed as $dir_exe/../.. (at least on macOS, see base/base_paths_mac.mm). This is one of the location this assumption exists, but it may be possible that other such assumptions are made in the code.

-- Sylvain

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAvvoymj2dJPuRJ_0k%3D2U44wRkLD2msRQCum%2BYXUMYwdQ%40mail.gmail.com.

[Christian Biesinger]

Seems the same code exists on posix, but can be overridden with an env var:

https://source.chromium.org/chromium/chromium/src/+/refs/heads/main:base/base_paths_posix.cc;l=80;drc=afec9eaf1d11cc77e8e06f06cb026fadf0dbf758;bpv=0;bpt=1

Christian

[Nico Weber]

While we don't require the name to be out/foo, we do require the build dir to be two levels deep inside src/, for getting deterministic paths from built binaries to the source code (`../..`).

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAvvoymj2dJPuRJ_0k%3D2U44wRkLD2msRQCum%2BYXUMYwdQ%40mail.gmail.com.

[Dirk Pranke]

This seems like something that could be configurable, e.g. a #define set in a GN file or something. It wouldn't be the first value we've had to set that way.

-- Dirk

[Jinyoung Hur]

> I would think we'd accept patches to make building outside the src directory work.

I've filed https://crbug.com/1457758 and CL is https://crrev.com/c/4644048.

2023년 6월 24일 토요일 오전 4시 24분 48초 UTC+9에 Dirk Pranke님이 작성:

[Greg Thompson]

Perhaps the simpler thing is to configure Sublime to not scan out\FOO\obj. https://coderwall.com/p/bk90bw/exclude-a-directory-from-searching-in-sublime-text-2 and many other places on the internet say how to teach Sublime to not search in places. Maybe that also makes it stop monitoring the filesystem in those places, too. Maybe we already have docs explaining this; see https://chromium.googlesource.com/chromium/src/+/main/docs/sublime_ide.md#making-a-new-project.

(Note: I don't use Sublime, so I can't vouch for the info I linked to.)

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/d2a2db7e-58fd-4a4d-ad8d-fa065a830404n%40chromium.org.

[Erik Jensen]

Is it important that local builds match official builds with respect to path-to-source? I moved my build directories outside of my source directory after I accidentally nuked them with git one too many times. (I work on Chromoting rather than the browser, so that's probably why I haven't yet run into an issue with it.) For my layout, being able to override the source path to `../../src/` in GN or similar would do it, and still be consistent between local build configurations, if not with the official builders.

On Fri, Jun 23, 2023 at 12:19 PM Nico Weber <tha...@chromium.org> wrote:

[Jinyoung Hur]

Greg, thanks for the Sublime Text tips.
However, I'm already using folder_exclude_patterns. I don't see the out directory in the sidebar, so it seems to have been applied well.
I've seen sublime_text.exe's CPU usage spike just by copying files(e.g. src/google_apis folder) into the out directory.
I'm not sure if this is the problem of ST or my PC environment.
Anyway, I've filed an issue on the Sublime Text project. https://github.com/sublimehq/sublime_text/issues/6031

2023년 6월 27일 화요일 오전 4시 14분 54초 UTC+9에 Erik Jensen님이 작성:

[Philipp Wollermann]

Having support for putting the build directory outside the source tree would also make sandboxing and using a virtual file system to access the code easier, so it’d help us research and make such improvements to Chromium’s build.

Cheers,

Philipp

[Keigo Oka]

I documented it in an internal doc go/speed-up-git-operations. so far it's working well. I welcome your feedback.

[Keigo Oka]

I've also updated the public documentation to reflect this: https://chromium.googlesource.com/chromium/src/+/main/docs/linux/build_instructions.md#speed-up-git-operations

Unfortunately, the Debian watchman package (https://tracker.debian.org/pkg/watchman) is currently orphaned and unmaintained. In fact, the official Watchman documentation explicitly warns against using the Ubuntu-supplied package, noting it is old and missing critical fixes. For this reason, I recommended following the official installation guidelines. This means that even internally, we cannot easily install the package.

Ideally, someone would step up to maintain the Debian package, but it's unclear if there is any interest. I've filed an internal tracking bug (b/467533724) regarding this.

[Junichi Uekawa (上川純一)]

Hi

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1019497

Seems like the package is fairly involved to update, but happy to review and help if anyone is interested 

Junichi Uekawa
Google

2025年12月11日(木) 8:51 'Keigo Oka' via Chromium-dev <chromi...@chromium.org>:

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/7a238290-76b9-41fd-814e-4c31b4e553ecn%40chromium.org.

[Sidhin Thomas]

FYI - had some trouble when installing watchman manually in ubuntu - See   Liblzma prevents me from installing packages · Issue #1283 · facebook/watchman

[Eldar Pulatov]

Hey all!

The git performance improvements (fsmonitor, untrackedCache, feature.manyFiles, commit-graph, git maintenance, checkout.workers, …) are currently spread across several doc pages.

Applying them today is a manual thing: every new engineer - or anyone starting from a new machine or directory - has to track them all down across the docs, remember them, and apply each one correctly.
On top of that, nothing in-tree ever checks or benchmarks git performance itself (please correct me if I'm wrong).

So I wrote a small Python CLI tool that turns all this advice into a single command: `git perf`. It:
- knows and correctly applies all the needed settings, respecting per-platform and per-git-version constraints (e.g. it won't enable fsmonitor on macOS with git < 2.43, because of crbug.com/1475405);
- validates your git setup and configures other useful options (e.g. merge.conflictStyle);
- warns about counterproductive settings like core.compression=0;
- flags the manual steps it can't perform itself (Spotlight/Defender exclusions, for example) and suggests them;
- has minimal dependencies and integrates flawlessly with gclient/depot_tools (tested mostly on downstream fork workflows);
- and last but not least, it can also benchmark the performance of your current checkout and trace it via git trace2, so you can see what git is actually doing under the hood on a Chromium checkout across different workflows.

Would anyone here be interested in upstreaming this tool? If so, where should it live? depot_tools feels like the natural home, but I'm not sure of the rules for committing to depot_tools as an outside contributor - it may not be a welcome place for this. src/tools/git/ would also work (that's where it currently lives, since I work mostly in Chromium).

P.S. My impression is that Google engineers get much of this functionality via internal provisioning (the internal `.goog` git build?), so the benefit here would land mostly on external contributors. If there's an internal equivalent already on a path to being open-sourced, I'm happy to wait for it and compare mine against it.

P.P.S. Git now ships Scalar by default, but that is incompatible with Chromium infra, given the gclient/depot_tools architecture..

Thanks,
Eldar

[Philipp Wollermann]

Hi Eldar,

That looks very interesting! :) Incidentally, I've also been benchmarking Git's performance on various Chromium use cases across platforms this past week, looking for potential optimizations to Git itself or to the settings we recommend.

I would love to help you review and upstream your tool. We might even want to extend it in the future to include performance checks for build-related issues as well. depot_tools sounds like the best place to me, too, if Gavin agrees.

How should we proceed? Do you want to send a CL to gavinmak@ and me, and we'll see if we can get it merged?

Best,

Philipp

This e-mail may contain CONFIDENTIAL AND PROPRIETARY INFORMATION and/or PRIVILEGED AND CONFIDENTIAL COMMUNICATION intended solely for the recipient and, therefore, may not be retransmitted to any party outside of the recipient’s organization without the prior written consent of the sender. If you have received this e-mail in error please notify the sender immediately by telephone or reply e-mail and destroy the original message without making a copy.

[Eldar Pulatov]

Hi Philipp,

I'm excited to work with you on this!

Before I send the CL, a few questions so I set things up the right way:

1. Contribution guidelines / CQ access: are there specific guidelines for contributing to depot_tools that differ from Chromium's own? I have only infra-dev group in mind
2. Design: Is there a design doc or anything I should read first? and would you prefer I write a short design doc before uploading the CL and send it to you, or is it fine to start the discussion directly on the CL?
3. Git specifics on Google infra: Is there anything git-related in depot_tools (or in the Google infra setup) that I should account for initially? So far I've only tested against a downstream fork, so I'm not familiar with all the git specifics on Google's side, and I'd like to avoid making assumptions that only hold downstream. If it's too private, I guess we can work on CL straightaway.

Thanks a lot!

Duplicating this to the discussion group, since my previous message didn't go to reply-all.

Cheers,
Eldar