PSA: Turn on Git’s fsmonitor!

1,589 views
Skip to first unread message

Avi Drissman

unread,
May 26, 2023, 10:45:42 AM5/26/23
to Chromium-dev
Not too long ago, Git got a feature called "fsmonitor" which significantly helps with large git repos and with platforms like Mac and Win where filesystem performance is an issue. It's pretty simple as long as you have an up-to-date git:

> git config --global core.untrackedCache true
> git config --global core.fsmonitor true

The speed gains are astonishing. I highly recommend that you enable this feature on git.

Avi

dan...@chromium.org

unread,
May 26, 2023, 10:52:19 AM5/26/23
to a...@google.com, Chromium-dev
Thanks Avi

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACWgwAYAfMOaQmO6n1C6Ft9zEaTiPrHoYWu5rYBbOiN9TEi0SA%40mail.gmail.com.

Dale Curtis

unread,
May 26, 2023, 12:28:33 PM5/26/23
to dan...@chromium.org, a...@google.com, Chromium-dev
+1, this is amazing. I didn't realize this was now built into git. On macOS git status went from ~4s to 0.6s.

$ git status
On branch ms
Your branch is ahead of 'main' by 1 commit.
  (use "git push" to publish your local commits)


It took 3.19 seconds to enumerate untracked files,
but the results were cached, and subsequent runs may be faster.
See 'git help status' for information on how to improve this.

nothing to commit, working tree clean

$ time git status
On branch ms
Your branch is ahead of 'main' by 1 commit.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean
git status  0.61s user 0.07s system 84% cpu 0.802 total

- dale

dan...@chromium.org

unread,
May 26, 2023, 12:53:23 PM5/26/23
to Dale Curtis, a...@google.com, Chromium-dev
This should go in

I see the mac one does mention untrackedCache though not on the well lit path, and doesn't have fsmonitor yet.

Any takers?

Avi Drissman

unread,
May 26, 2023, 1:00:36 PM5/26/23
to dan...@chromium.org, Dale Curtis, Chromium-dev
I'll add it. Just wanted to scream from the top of the mountain how good this is.

dan...@chromium.org

unread,
May 26, 2023, 1:03:18 PM5/26/23
to Avi Drissman, Dale Curtis, Chromium-dev
Thanks, without the email only new folks would ever know!

Feras Aldahlawi

unread,
May 26, 2023, 1:18:08 PM5/26/23
to dan...@chromium.org, Avi Drissman, Dale Curtis, Chromium-dev
This cannot be real!!! 

FYI, if you are using Mac-M1 you might need to run `git fsmonitor--daemon start` before you see gains. Check the status using `git fsmonitor--daemon status`.

Peter Kasting

unread,
May 26, 2023, 1:27:52 PM5/26/23
to a...@google.com, Chromium-dev
On Fri, May 26, 2023 at 7:44 AM 'Avi Drissman' via Chromium-dev <chromi...@chromium.org> wrote:
> git config --global core.untrackedCache true

For me this one is the big win -- I've had fsmonitor on for awhile without too much speed gain, but this setting really cuts `git status` times on Windows.

PK 

Steve Kobes

unread,
May 26, 2023, 1:56:18 PM5/26/23
to pkas...@chromium.org, a...@google.com, Chromium-dev
Wow, core.untrackedCache speeds up Linux too.  For me git status went from 2s to 0.8s.

Interestingly it seems slightly slower with both core.untrackedCache and core.fsmonitor, compared to setting only core.untrackedCache.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

Avi Drissman

unread,
May 26, 2023, 2:58:22 PM5/26/23
to Feras Aldahlawi, dan...@chromium.org, Dale Curtis, Chromium-dev
You should not need to explicitly start it.

https://git-scm.com/docs/git-fsmonitor--daemon:

When core.fsmonitor is set to true (see git-config[1]) commands, such as git status, will ask the daemon for changes and automatically start it (if necessary).

$ git fsmonitor--daemon status

for me shows the daemon running even though I did not explicitly start it.

Bruce Dawson

unread,
May 29, 2023, 2:08:25 PM5/29/23
to Chromium-dev, Avi Drissman, dan...@chromium.org, Dale Curtis, Chromium-dev, Feras Aldahlawi
The speedup for me is particularly noticeable on "git cl status". I often try to avoid this command because it is too slow, and now I don't think I'll need to.

Bruce Dawson

unread,
May 29, 2023, 2:59:24 PM5/29/23
to Chromium-dev, Avi Drissman, dan...@chromium.org, Dale Curtis, Feras Aldahlawi
While experimenting with this feature on Windows and monitoring for any possible issues I came across this article which then led me to notice that I now have 178 fsmonitor--daemon processes running on my laptop, measured with this command:

    wmic process where "commandline like 'git fsmon%'" get processid, commandline | find /c "fsmonitor--daemon"

The count doesn't seem to grow over time and it appears to be an implementation detail, with each process monitoring a different directory. You can see the directory being monitored by running this command:
    handle -p %%d | find "RWD"
where %%d is the PID.

I haven't tested on Linux or OSX so I don't know if the behavior is similar. The process count does increase if I run "git status" in another repo, which makes sense. So, FYI.

--
Bruce Dawson, he/him

Avi Drissman

unread,
May 29, 2023, 3:32:30 PM5/29/23
to Bruce Dawson, Chromium-dev, dan...@chromium.org, Dale Curtis, Feras Aldahlawi
On my macOS box, I do have a bunch as well:

~> ps a | grep [f]smonitor | wc -l
     169

I'm ok with it; the daemons are quiescent and are blocked when nothing is happening, so there's no system impact.

fsmonitor does not function on Linux; inotify support has not been added.

Nearly every aspect of git is sped up by fixing status; the article that I linked in the first post of this thread illustrates why. Happy to hear that it's helped you!

Avi

K. Moon

unread,
May 30, 2023, 9:41:57 AM5/30/23
to Avi Drissman, Bruce Dawson, Chromium-dev, dan...@chromium.org, Dale Curtis, Feras Aldahlawi
I suspect the usual caveats apply to anything performing file system monitoring: Since operations are now eager, if you touch a lot of the file system at once, you'll probably incur high resource usage during the updates. Otherwise, I can't see much downside. (That said, I'm also too lazy to change anything, as I don't really spend that much of my day on Git operations.)

Mark Mentovai

unread,
May 30, 2023, 10:43:20 AM5/30/23
to a...@google.com, Bruce Dawson, Chromium-dev, dan...@chromium.org, Dale Curtis, Feras Aldahlawi
It’s one git fsmonitor--daemon process per repository. 169 and 178 are entirely plausible numbers of git repositories in a full gclient-based Chrome checkout.

If you don’t like it, you could set core.fsmonitor = true just for the repository where you need it (git config --local inside Chrome, for example) instead of git config --global.

Since the daemon watching the Chrome checkout can (and does) see changes in other repositories, technically there should be a way to reduce this to just a single daemon and single filesystem watcher for all of Chrome. This is essentially the “it doesn’t understand submodules” limitation from the git-fsmonitor--daemon man page, although the man page focuses on the “outer repository’s watch picks up changes to inner repositories but has no use for them” aspect, and not the “watches on inner repositories duplicate the watch on the outer” aspect. It seems like enhancing this would make a suitable Summer of Code-type project.

Caleb Raitto

unread,
May 30, 2023, 1:01:51 PM5/30/23
to Chromium-dev, Mark Mentovai, Bruce Dawson, Chromium-dev, dan...@chromium.org, Dale Curtis, Feras Aldahlawi, a...@google.com
For some reason, on a gLinux cloud machine, I'm not sure this works?

$ git fsmonitor--daemon status
fatal: fsmonitor--daemon not supported on this platform

$ git --version
git version 2.41.0.rc0.172.g3f132b7071-goog

Seems like the version should be new enough, although the Google internal version may be customized...

I didn't notice any speedup either after running the 2 git config commands.

-Caleb

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

Christian Biesinger

unread,
May 30, 2023, 1:22:12 PM5/30/23
to cara...@chromium.org, Chromium-dev, Mark Mentovai, Bruce Dawson, dan...@chromium.org, Dale Curtis, Feras Aldahlawi, a...@google.com
Yes, this does not work on Linux.

You could try the approach described in https://www.spinics.net/lists/git/msg458447.html

Christian

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

Bruce Dawson

unread,
May 31, 2023, 12:09:56 PM5/31/23
to Chromium-dev, Christian Biesinger, Chromium-dev, Mark Mentovai, Bruce Dawson, dan...@chromium.org, Dale Curtis, Feras Aldahlawi, a...@google.com, cara...@chromium.org
> Since operations are now eager, if you touch a lot of the file system at once, you'll probably incur high resource usage during the updates

I took a brief look at this on Windows by doing a build and monitoring accumulated CPU time across all of the fsmonitor processes. One of them accumulated about eight seconds of CPU time and the others accumulated none. Compared to the cost of a build, and compared to the number of files generated, this is tiny so it looks like (on Windows anyway) the overhead is not a problem.

After a couple of days of use the busiest fsmonitor process has now accumulated 22 seconds of CPU time and the others are still sitting at zero. This seems fine, but I recommend checking the "CPU time" column after a few days if you think your usage patterns might be more problematic.

K. Moon

unread,
May 31, 2023, 12:50:34 PM5/31/23
to bruce...@google.com, Chromium-dev, Christian Biesinger, Mark Mentovai, dan...@chromium.org, Dale Curtis, Feras Aldahlawi, a...@google.com, cara...@chromium.org
I was thinking something more like, "You decided to change every single copyright header in the source tree," rather than a build. But I agree that it's probably negligible, unless you're doing the rewrites in a tight loop for some reason.

Avi Drissman

unread,
May 31, 2023, 2:32:26 PM5/31/23
to K. Moon, bruce...@google.com, Chromium-dev, Christian Biesinger, Mark Mentovai, dan...@chromium.org, Dale Curtis, Feras Aldahlawi, cara...@chromium.org
The amount of CPU and usage used by tracking changes as they occur is far less than the CPU and usage of repeatedly scrubbing through the git directory that it prevents.

No matter how much CPU the fsmonitor would take, it would be worth the tradeoff.

Avi

K. Moon

unread,
May 31, 2023, 2:36:08 PM5/31/23
to Avi Drissman, bruce...@google.com, Chromium-dev, Christian Biesinger, Mark Mentovai, dan...@chromium.org, Dale Curtis, Feras Aldahlawi, cara...@chromium.org
I don't think the overall conclusion changes, but I don't think that's strictly true: ~Infinity times a small number is still bigger than ~1 times a big number. The actual answer is still workload-dependent.

Henrik Andreasson

unread,
May 31, 2023, 3:17:24 PM5/31/23
to km...@chromium.org, Avi Drissman, bruce...@google.com, Chromium-dev, Christian Biesinger, Mark Mentovai, dan...@chromium.org, Dale Curtis, Feras Aldahlawi, cara...@chromium.org

Mason Freed

unread,
Jun 1, 2023, 10:57:32 AM6/1/23
to Chromium-dev, Christian Biesinger, Chromium-dev, Mark Mentovai, Bruce Dawson, dan...@chromium.org, Dale Curtis, Feras Aldahlawi, a...@google.com, cara...@chromium.org
I just wanted to say thanks to Avi for calling this out! It is so much better than before.


On Tuesday, May 30, 2023 at 10:22:12 AM UTC-7 Christian Biesinger wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

Philipp Wollermann

unread,
Jun 1, 2023, 10:59:19 AM6/1/23
to hen...@chromium.org, km...@chromium.org, Avi Drissman, bruce...@google.com, Chromium-dev, Christian Biesinger, Mark Mentovai, dan...@chromium.org, Dale Curtis, Feras Aldahlawi, cara...@chromium.org
I wanted to enable this on Linux and it seems that this made it work.

git config --global core.fsmonitor $HOME/bin/query-watchman
git config --global core.untrackedcache true

Benchmark:

philwo@philwo:chromium/src ‹7ed625e6e8f54›$ hyperfine \
    --shell /bin/bash \
    --warmup 1 \
    --prepare "git config --global core.untrackedcache false; git config --global core.fsmonitor false" \
    --command-name "no untrackedcache, no fsmonitor" \
    "git status" \
    --prepare "git config --global core.untrackedcache true; git config --global core.fsmonitor false" \
    --command-name "untrackedcache only" \
    "git status" \
    --prepare "git config --global core.untrackedcache false; git config --global core.fsmonitor $HOME/bin/query-watchman" \
    --command-name "fsmonitor only" \
    "git status" \
    --prepare "git config --global core.untrackedcache true; git config --global core.fsmonitor $HOME/bin/query-watchman" \
    --command-name "untrackedcache + fsmonitor" \
    "git status"

Benchmark 1: no untrackedcache, no fsmonitor
  Time (mean ± σ):      1.149 s ±  0.008 s    [User: 0.605 s, System: 4.074 s]
  Range (min … max):    1.135 s …  1.160 s    10 runs
 
Benchmark 2: untrackedcache only
  Time (mean ± σ):     573.5 ms ±   2.1 ms    [User: 367.4 ms, System: 3747.5 ms]
  Range (min … max):   569.6 ms … 576.2 ms    10 runs
 
Benchmark 3: fsmonitor only
  Time (mean ± σ):      1.030 s ±  0.006 s    [User: 0.582 s, System: 0.445 s]
  Range (min … max):    1.022 s …  1.040 s    10 runs
 
Benchmark 4: untrackedcache + fsmonitor
  Time (mean ± σ):     408.4 ms ±   4.3 ms    [User: 372.6 ms, System: 37.9 ms]
  Range (min … max):   400.3 ms … 415.0 ms    10 runs
 
Summary
  'untrackedcache + fsmonitor' ran
    1.40 ± 0.02 times faster than 'untrackedcache only'
    2.52 ± 0.03 times faster than 'fsmonitor only'
    2.81 ± 0.03 times faster than 'no untrackedcache, no fsmonitor'


Panos Astithas

unread,
Jun 1, 2023, 11:00:51 AM6/1/23
to sko...@chromium.org, pkas...@chromium.org, a...@google.com, Chromium-dev

Avi Drissman

unread,
Jun 1, 2023, 11:25:15 AM6/1/23
to Philipp Wollermann, hen...@chromium.org, km...@chromium.org, bruce...@google.com, Chromium-dev, Christian Biesinger, Mark Mentovai, dan...@chromium.org, Dale Curtis, Feras Aldahlawi, cara...@chromium.org
There are two different features in git with the same name.

core.fsmonitor = true enables the built-in fsmonitor on Windows/Mac. It is your best option for those platforms, as it's tuned to git's needs.

core.fsmonitor = <monitor process> has git consult with an external monitor process. It's available on all platforms, though you would want to use the built-in fsmonitor if you can. However, as you found, even this feature is well-worth activating.

Avi

Ken Russell

unread,
Jun 1, 2023, 12:00:57 PM6/1/23
to a...@google.com, Philipp Wollermann, hen...@chromium.org, km...@chromium.org, bruce...@google.com, Chromium-dev, Christian Biesinger, Mark Mentovai, dan...@chromium.org, Dale Curtis, Feras Aldahlawi, cara...@chromium.org
This is such a tremendous improvement that I also had to spam the list and say thank you Avi for highlighting it and providing clear instructions for how to enable it!

-Ken



Christian Biesinger

unread,
Jun 1, 2023, 12:34:30 PM6/1/23
to Philipp Wollermann, hen...@chromium.org, km...@chromium.org, Avi Drissman, bruce...@google.com, Chromium-dev, Mark Mentovai, dan...@chromium.org, Dale Curtis, Feras Aldahlawi, cara...@chromium.org
Thanks for the detailed instructions!

It looks like the git package comes with /usr/share/git-core/templates/hooks/fsmonitor-watchman.sample, so you can probably use that instead of curl+chmod

Also I assume you need to `sudo apt-get install watchman` if not already installed.

Thanks,
Christian

On Thu, Jun 1, 2023 at 2:07 AM Philipp Wollermann <phi...@google.com> wrote:

K. Moon

unread,
Jun 1, 2023, 1:33:21 PM6/1/23
to Christian Biesinger, Philipp Wollermann, hen...@chromium.org, Avi Drissman, bruce...@google.com, Chromium-dev, Mark Mentovai, dan...@chromium.org, Dale Curtis, Feras Aldahlawi, cara...@chromium.org
Given so many people are finding it useful, would it be worth documenting in the checkout instructions somewhere? Like, "Hey, Chromium is really big, you'd probably want to turn this Git feature on. Here are some instructions." (The instructions themselves might be on a separate page, or point to an external resource.)

dan...@chromium.org

unread,
Jun 1, 2023, 1:37:42 PM6/1/23
to K. Moon, Christian Biesinger, Philipp Wollermann, hen...@chromium.org, Avi Drissman, bruce...@google.com, Chromium-dev, Mark Mentovai, Dale Curtis, Feras Aldahlawi, cara...@chromium.org
Avi already has done this :)

K. Moon

unread,
Jun 1, 2023, 1:42:59 PM6/1/23
to dan...@chromium.org, Christian Biesinger, Philipp Wollermann, hen...@chromium.org, Avi Drissman, bruce...@google.com, Chromium-dev, Mark Mentovai, Dale Curtis, Feras Aldahlawi, cara...@chromium.org
Great!

Caleb Raitto

unread,
Jun 1, 2023, 2:33:28 PM6/1/23
to K. Moon, dan...@chromium.org, Christian Biesinger, Philipp Wollermann, hen...@chromium.org, Avi Drissman, bruce...@google.com, Chromium-dev, Mark Mentovai, Dale Curtis, Feras Aldahlawi
Per cbiesinger@'s mail, I was able to get Linux working with sudo apt install watchman and git config --global core.fsmonitor /usr/share/git-core/templates/hooks/fsmonitor-watchman.sample:

(The first run of git status is slow, but then it's fast after that):

$ time git status
HEAD detached at origin/main

nothing to commit, working tree clean

real    0m0.395s
user    0m0.357s
sys     0m0.050s

-Caleb

Avi Drissman

unread,
Jun 1, 2023, 2:45:06 PM6/1/23
to Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, Philipp Wollermann, hen...@chromium.org, bruce...@google.com, Chromium-dev, Mark Mentovai, Dale Curtis, Feras Aldahlawi
Since it's a one-liner for Mac/Windows, it was easy for me to add it to their build instructions. Could one of you Linux folks add the more detailed instructions to use the fsmonitor hooks to the Linux build instructions?

Avi

Philipp Wollermann

unread,
Jun 2, 2023, 11:36:49 AM6/2/23
to Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, bruce...@google.com, Chromium-dev, Mark Mentovai, Dale Curtis, Feras Aldahlawi
I sent a CL to add this to the docs, please free to comment on how to improve it: https://chromium-review.git.corp.google.com/c/chromium/src/+/4583709

Philipp

Bruce Dawson

unread,
Jun 16, 2023, 5:59:00 PM6/16/23
to Philipp Wollermann, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Chromium-dev, Mark Mentovai, Dale Curtis, Feras Aldahlawi
Some follow-up information:

1) I was wondering what was triggering the accumulation of so many fsmonitor processes since they should only spawn when I do git operations. Then I realized that "gclient sync" does some git operations in all of the sub-repos, and that is indeed what triggers so many fsmonitor--daemon processes to appear.
2) On Windows the fsmonitor--daemon processes consume an average of 86 MB of commit. This means they allocate 86 MB of memory but mostly don't use it, however Windows promises that the memory will be available (no overcommit). 86 MB is not much, but 86 MB times 179 processes works out to over 15 GB of commit. This doesn't consume any physical RAM, but most OOM failures on Windows are due to running out of commit, not RAM, so this still matters. You can easily run out of commit and OOM with tens of GB free.
3) If you care about this (on a workstation you probably don't need to) you could increase the size of your pagefile (thus increasing the commit limit) - even though that extra space will never be used. Or, more simply, you can disable fsmonitor globally and just enable it for large repos such as chromium.

So, on Windows my advice would be to set "git config --global core.fsmonitor false" and then on specific large repos (Chromium, v8, ???) set "git config core.fsmonitor true". This gives you the benefits (faster Chromium git operations) without the cost (excessive processes and wasteful commits for repos you rarely use). I will update the Windows documentation.

Similar issues may occur on Linux/OSX, although the ability to overcommit may make them less important. I like having fewer processes so I think I'll follow this policy on my Linux box as well, and I think that this advice is probably appropriate for all platforms.

Other than Chromium and v8 are there any repos that are large enough to need fsmonitor enabled?

P.S. This Task Manager display demonstrates the problem. I have 24 GB of RAM free but if I didn't have a page file I'd be critically low on commit, and large allocations could start failing.
image.png
--
Bruce Dawson, he/him

Bruce Dawson

unread,
Jun 19, 2023, 12:21:42 PM6/19/23
to Philipp Wollermann, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Chromium-dev, Mark Mentovai, Dale Curtis, Feras Aldahlawi
VsChromium on Windows also uses file system notifications to track changes, and it also receives notifications for every file written in the out directory. This is necessary in that case because I want VsChromium to be able to search generated code in the out directories.

For a while the CPU usage was problematic but by adjusting the buffer sizes we were able to reduce it to a manageable level. Also, if VsChromium detects high enough activity then it stops monitoring for a while and rescans the src directory later.

I wonder if any of these lessons from Windows could be applied on Linux. I guess you'd have to file a bug against watchman and/or git.


On Sun, Jun 18, 2023 at 11:10 PM Philipp Wollermann <phi...@google.com> wrote:
Thanks Bruce, this is good information!

Two more issues I noticed with this feature on Linux after using it for 2 weeks now:

1) I think watchman might also get inotify events for the build output folder, which results in it using quite a lot of CPU during a build. I wonder if we can exclude the "out" directory somehow.
2) I occasionally got error messages from various tools about my system running out of inotify watches, so I added this to my /etc/sysctl.d/philwo.conf:

fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 10485760

Also, on macOS I ran into issues where all "git" commands would just hang after a while, until I killed the fsmonitor--daemon processes. I disabled the feature on macOS for now.

Cheers,
Philipp



--
Bruce Dawson, he/him

Joe Mason

unread,
Jun 19, 2023, 12:33:39 PM6/19/23
to bruce...@google.com, Philipp Wollermann, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Chromium-dev, Mark Mentovai, Dale Curtis, Feras Aldahlawi
As a quick check, the 10 biggest .git dirs in my Windows checkout are:

$ find . -type d -name .git -print0|xargs -0 du -hs|sort -hr|head
31G     ./.git
2.0G    ./third_party/llvm/.git
1.4G    ./third_party/tflite/src/.git
1.2G    ./third_party/icu/.git
1.2G    ./third_party/angle/third_party/VK-GL-CTS/src/.git
1.1G    ./v8/.git
947M    ./tools/perf/data/.git
940M    ./third_party/sqlite/src/.git
813M    ./third_party/swiftshader/.git
785M    ./third_party/skia/.git

I think that's proportional to the size of the history, not the size of the repo being monitored, but it's probably got a correlation.

I'm not sure if there's a point in turning on fsmonitor in a repo that I'm not actively working in - would it make gclient sync faster? 

Bruce Dawson

unread,
Jun 19, 2023, 1:08:28 PM6/19/23
to Joe Mason, Philipp Wollermann, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Chromium-dev, Mark Mentovai, Dale Curtis, Feras Aldahlawi
Thanks for the data!

If you run gclient sync frequently (once to prime the cache, subsequent times to reap the benefits) then you would probably get a modest speedup by having fsmonitor enabled in repos you don't otherwise use, but the time savings are probably modest. Given the costs (on Windows at least) and the tiny savings for non-Chromium repos I'm sticking with just enabling fsmonitor for the few large repos that I ever work on.
--
Bruce Dawson, he/him

Philipp Wollermann

unread,
Jun 21, 2023, 9:08:48 PM6/21/23
to Bruce Dawson, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Chromium-dev, Mark Mentovai, Dale Curtis, Feras Aldahlawi
Thanks Bruce, this is good information!

Two more issues I noticed with this feature on Linux after using it for 2 weeks now:

1) I think watchman might also get inotify events for the build output folder, which results in it using quite a lot of CPU during a build. I wonder if we can exclude the "out" directory somehow.
2) I occasionally got error messages from various tools about my system running out of inotify watches, so I added this to my /etc/sysctl.d/philwo.conf:

fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 10485760

Also, on macOS I ran into issues where all "git" commands would just hang after a while, until I killed the fsmonitor--daemon processes. I disabled the feature on macOS for now.

Cheers,
Philipp


Erik Jensen

unread,
Jun 22, 2023, 4:52:51 PM6/22/23
to phi...@google.com, Bruce Dawson, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Chromium-dev, Mark Mentovai, Dale Curtis, Feras Aldahlawi
I have my build directory next to src instead of under it, which prevents git from scanning it. The downside is I can't run git commands if I'm in the build tree.

Jinyoung Hur

unread,
Jun 22, 2023, 11:13:45 PM6/22/23
to Chromium-dev, Erik Jensen, Bruce Dawson, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Chromium-dev, Mark Mentovai, Dale Curtis, Feras Aldahlawi, phi...@google.com
For me, configuring the build directory next to src,  my Sublime Text on Windows gets much faster while building.
However, Chromium build for Windows requires the out directory should be under the src directory.
So I had to modify here and there to build and run tests.

It would be great if we can configure the out directory freely for Windows.

2023년 6월 23일 금요일 오전 5시 52분 51초 UTC+9에 Erik Jensen님이 작성:

Dirk Pranke

unread,
Jun 23, 2023, 12:21:01 PM6/23/23
to hur...@gmail.com, Chromium-dev, Erik Jensen, Bruce Dawson, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Mark Mentovai, Dale Curtis, Feras Aldahlawi, phi...@google.com
We don't intentionally require the build to be under //out/$something . It's just that almost no one uses anything else.

I would think we'd accept patches to make building outside the src directory work.

-- Dirk

Sylvain Defresne

unread,
Jun 23, 2023, 12:34:28 PM6/23/23
to dpr...@google.com, hur...@gmail.com, Chromium-dev, Erik Jensen, Bruce Dawson, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Mark Mentovai, Dale Curtis, Feras Aldahlawi, phi...@google.com
While we don't explicitly require the build to be //out/$something, there is some code in //base that assumes the root of the repository can be accessed as $dir_exe/../.. (at least on macOS, see base/base_paths_mac.mm). This is one of the location this assumption exists, but it may be possible that other such assumptions are made in the code.
-- Sylvain

Christian Biesinger

unread,
Jun 23, 2023, 1:22:59 PM6/23/23
to Sylvain Defresne, dpr...@google.com, hur...@gmail.com, Chromium-dev, Erik Jensen, Bruce Dawson, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, hen...@chromium.org, Mark Mentovai, Dale Curtis, Feras Aldahlawi, phi...@google.com

Nico Weber

unread,
Jun 23, 2023, 3:20:34 PM6/23/23
to Dirk Pranke, hur...@gmail.com, Chromium-dev, Erik Jensen, Bruce Dawson, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Mark Mentovai, Dale Curtis, Feras Aldahlawi, phi...@google.com
While we don't require the name to be out/foo, we do require the build dir to be two levels deep inside src/, for getting deterministic paths from built binaries to the source code (`../..`).

Dirk Pranke

unread,
Jun 23, 2023, 3:24:48 PM6/23/23
to Nico Weber, hur...@gmail.com, Chromium-dev, Erik Jensen, Bruce Dawson, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Mark Mentovai, Dale Curtis, Feras Aldahlawi, phi...@google.com
This seems like something that could be configurable, e.g. a #define set in a GN file or something. It wouldn't be the first value we've had to set that way.

-- Dirk

Jinyoung Hur

unread,
Jun 25, 2023, 5:49:53 AM6/25/23
to Chromium-dev, Dirk Pranke, hur...@gmail.com, Chromium-dev, Erik Jensen, Bruce Dawson, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Mark Mentovai, Dale Curtis, Feras Aldahlawi, phi...@google.com, Nico Weber
> I would think we'd accept patches to make building outside the src directory work.

I've filed https://crbug.com/1457758 and CL is https://crrev.com/c/4644048.

2023년 6월 24일 토요일 오전 4시 24분 48초 UTC+9에 Dirk Pranke님이 작성:

Greg Thompson

unread,
Jun 26, 2023, 10:10:28 AM6/26/23
to hur...@gmail.com, Chromium-dev, Dirk Pranke, Erik Jensen, Bruce Dawson, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Mark Mentovai, Dale Curtis, Feras Aldahlawi, phi...@google.com, Nico Weber
Perhaps the simpler thing is to configure Sublime to not scan out\FOO\obj. https://coderwall.com/p/bk90bw/exclude-a-directory-from-searching-in-sublime-text-2 and many other places on the internet say how to teach Sublime to not search in places. Maybe that also makes it stop monitoring the filesystem in those places, too. Maybe we already have docs explaining this; see https://chromium.googlesource.com/chromium/src/+/main/docs/sublime_ide.md#making-a-new-project.

(Note: I don't use Sublime, so I can't vouch for the info I linked to.)

Erik Jensen

unread,
Jun 26, 2023, 3:14:54 PM6/26/23
to Nico Weber, Dirk Pranke, hur...@gmail.com, Chromium-dev, Bruce Dawson, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Mark Mentovai, Dale Curtis, Feras Aldahlawi, phi...@google.com
Is it important that local builds match official builds with respect to path-to-source? I moved my build directories outside of my source directory after I accidentally nuked them with git one too many times. (I work on Chromoting rather than the browser, so that's probably why I haven't yet run into an issue with it.) For my layout, being able to override the source path to `../../src/` in GN or similar would do it, and still be consistent between local build configurations, if not with the official builders.

On Fri, Jun 23, 2023 at 12:19 PM Nico Weber <tha...@chromium.org> wrote:

Jinyoung Hur

unread,
Jun 27, 2023, 11:12:11 AM6/27/23
to Chromium-dev, Erik Jensen, Dirk Pranke, hur...@gmail.com, Chromium-dev, Bruce Dawson, Avi Drissman, Caleb Raitto, K. Moon, dan...@chromium.org, Christian Biesinger, hen...@chromium.org, Mark Mentovai, Dale Curtis, Feras Aldahlawi, phi...@google.com, Nico Weber
Greg, thanks for the Sublime Text tips.
However, I'm already using folder_exclude_patterns. I don't see the out directory in the sidebar, so it seems to have been applied well.
I've seen sublime_text.exe's CPU usage spike just by copying files(e.g. src/google_apis folder) into the out directory.
I'm not sure if this is the problem of ST or my PC environment.
Anyway, I've filed an issue on the Sublime Text project. https://github.com/sublimehq/sublime_text/issues/6031


2023년 6월 27일 화요일 오전 4시 14분 54초 UTC+9에 Erik Jensen님이 작성:

Philipp Wollermann

unread,
Jun 27, 2023, 4:10:43 PM6/27/23
to Erik Jensen, Avi Drissman, Bruce Dawson, Caleb Raitto, Christian Biesinger, Chromium-dev, Dale Curtis, Dirk Pranke, Feras Aldahlawi, K. Moon, Mark Mentovai, Nico Weber, dan...@chromium.org, hen...@chromium.org, hur...@gmail.com
Having support for putting the build directory outside the source tree would also make sandboxing and using a virtual file system to access the code easier, so it’d help us research and make such improvements to Chromium’s build.

Cheers,
Philipp
Reply all
Reply to author
Forward
0 new messages