How to tell Bazel how many instances of a particular action it can run in parallel?

[Konstantin]

One of our custom rules runs the action which is very much network I/O bound and not CPU bound at all. 

For that particular action there is no point to limit parallelism to the number of available cores. 

Instead it makes sense to start as many of those actions as there are ready for execution. 

I wonder if there is a way to convey that info to Bazel to let it do parallel actions scheduling more efficiently. 

Thank you!

Konstantin

[Austin Schuh]

https://docs.bazel.build/versions/master/test-encyclopedia.html#other-resources
? It lets you add tags = [ "cpu:5" ] to signal that a test uses more
CPU. That isn't quite what you are looking for, but is very close. I
wonder if tags = [ "cpu:0" ] would work.

Austin

> --
> You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/bcaf1944-18dc-45ab-a67f-dc433533055bo%40googlegroups.com.

[Konstantin]

This is an interesting finding, but it seems to be pertained to test rules only. Probably implemented in test executor. It does not seem to be applicable to the custom non-test rules. 

On Sunday, September 20, 2020 at 1:31:42 PM UTC-7, Austin Schuh wrote:

https://docs.bazel.build/versions/master/test-encyclopedia.html#other-resources
?  It lets you add tags = [ "cpu:5" ] to signal that a test uses more
CPU.  That isn't quite what you are looking for, but is very close.  I
wonder if tags = [ "cpu:0" ] would work.

Austin

On Sun, Sep 20, 2020 at 12:32 PM Konstantin <kon...@ermank.com> wrote:
>
> One of our custom rules runs the action which is very much network I/O bound and not CPU bound at all.
> For that particular action there is no point to limit parallelism to the number of available cores.
> Instead it makes sense to start as many of those actions as there are ready for execution.
> I wonder if there is a way to convey that info to Bazel to let it do parallel actions scheduling more efficiently.
>
> Thank you!
> Konstantin
>
> --
> You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-...@googlegroups.com.

[Andrew Allen]

Forking the thread since my question is related but not identical.

On Sun, Sep 20, 2020 at 2:31 PM Austin Schuh <austin...@gmail.com> wrote:

https://docs.bazel.build/versions/master/test-encyclopedia.html#other-resources
?  It lets you add tags = [ "cpu:5" ] to signal that a test uses more
CPU.  That isn't quite what you are looking for, but is very close.  I
wonder if tags = [ "cpu:0" ] would work.

This is a very interesting feature! I have a set of tests that consumes a significant amount of ram (8gb in some cases). Our CI machine has 32gcu and 32gb of ram and for most of our tests running with huge parallelism is wonderful. --jobs (and the test equivalent) set to 16 or 32 provides great parallelism, but sometimes we get unlucky and schedule 8 jobs that each try to take 8gb (2x the amount of RAM we have on the box). Each test does its memory hungry thing, decides there isn't enough RAM and then crashes which fails the test.

I tried `tags = ["mem:8gb"]` (just for fun) but that didn't seem to do anything. Is there a similar tag to affect memory?

We have already tagged them as enormous hoping that we could limit the parallelism that way, but there doesn't appear to be a way to say "enormous is actually a user defined amount of memory".

Do you, or anyone else on this list, know of a way to describe how much memory a test will consume to limit parallelism?

Austin

On Sun, Sep 20, 2020 at 12:32 PM Konstantin <kon...@ermank.com> wrote:
>
> One of our custom rules runs the action which is very much network I/O bound and not CPU bound at all.
> For that particular action there is no point to limit parallelism to the number of available cores.
> Instead it makes sense to start as many of those actions as there are ready for execution.
> I wonder if there is a way to convey that info to Bazel to let it do parallel actions scheduling more efficiently.
>
> Thank you!
> Konstantin
>
> --
> You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/bcaf1944-18dc-45ab-a67f-dc433533055bo%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CABsbf%3DHew0xMmuSRtjZ6%2BkgD9ofJTEgi5XccxKHKFhOSxjO6KQ%40mail.gmail.com.

[Konstantin]

I have opened the issue to track this.

On Monday, September 21, 2020 at 9:49:22 AM UTC-7, Andrew Allen wrote:

Forking the thread since my question is related but not identical.

On Sun, Sep 20, 2020 at 2:31 PM Austin Schuh <austi...@gmail.com> wrote:

https://docs.bazel.build/versions/master/test-encyclopedia.html#other-resources
?  It lets you add tags = [ "cpu:5" ] to signal that a test uses more
CPU.  That isn't quite what you are looking for, but is very close.  I
wonder if tags = [ "cpu:0" ] would work.

This is a very interesting feature! I have a set of tests that consumes a significant amount of ram (8gb in some cases). Our CI machine has 32gcu and 32gb of ram and for most of our tests running with huge parallelism is wonderful. --jobs (and the test equivalent) set to 16 or 32 provides great parallelism, but sometimes we get unlucky and schedule 8 jobs that each try to take 8gb (2x the amount of RAM we have on the box). Each test does its memory hungry thing, decides there isn't enough RAM and then crashes which fails the test.

I tried `tags = ["mem:8gb"]` (just for fun) but that didn't seem to do anything. Is there a similar tag to affect memory?

We have already tagged them as enormous hoping that we could limit the parallelism that way, but there doesn't appear to be a way to say "enormous is actually a user defined amount of memory".

Do you, or anyone else on this list, know of a way to describe how much memory a test will consume to limit parallelism?

Austin

On Sun, Sep 20, 2020 at 12:32 PM Konstantin <kon...@ermank.com> wrote:
>
> One of our custom rules runs the action which is very much network I/O bound and not CPU bound at all.
> For that particular action there is no point to limit parallelism to the number of available cores.
> Instead it makes sense to start as many of those actions as there are ready for execution.
> I wonder if there is a way to convey that info to Bazel to let it do parallel actions scheduling more efficiently.
>
> Thank you!
> Konstantin
>
> --
> You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/bcaf1944-18dc-45ab-a67f-dc433533055bo%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-...@googlegroups.com.