Reduce debug build size

[William Allen]

I am currently working to update a project I work on to use Bazel 6.0.0 instead of Bazel 5.3.1.  On macOS, our debug builds are currently quite large and Bazel 6.0.0 increases the size just enough that it exceeds our available CI disk space (200 GB).  I am currently trying to reduce the amount of disk space used so we are able to remain under this hard cap of 200 GB.

As a crude measure of disk space usage, I did some basic testing and came up with the following results by calculating the size of the directory `/private/var/tmp/_bazel_william.allen/`.  In each case, I ran `bazel clean --expunge` before running another test.

Size after `bazel clean --expunge`: 9.6GB
Size after `bazel build --compilation_mode=dbg //...` with bazel 6.0.0: 168 GB
Size after `bazel build --compilation_mode=dbg //...` with bazel 5.3.1: 158 GB
Size after `bazel build //...` with bazel 6.0.0: 23 GB
Size after `bazel build //...` with bazel 5.3.1: 23 GB

Simply getting the debug size back down to something close to the original 158 GB would fix my problem for the moment, but I am also curious about other approaches which might reduce the size even more.

When running `bazel test --compilation_mode=dbg //...`, an initial build process occurs and then tests start running.  Throughout the testing process, the size of `/private/var/tmp/_bazel_william.allen/` begins around 9.6 GB and continues to grow until it reaches roughly 168 GB when the tests finish.  Is there a way to remove intermediary products during the testing process so less disk space is used overall?  I found the flag `--experimental_sandbox_async_tree_delete_idle_threads`, but the documentation is a little bit misleading, and it actually increased the amount of space required.