How to unconditionally execute an action?

[sbag...@grailbio.com]

Hello,

I have an action that captures some system state like version of installed tools, etc. My rules then depend on this state so that I can invalidate the cache for builds when the system state has changed.

How do I make this action run unconditionally on every build without using the local action cache?

I looked into repository_rule and even the `local` attribute there does not give me unconditional execution. I know that people depend on environment variables to capture system state, but I really do want to execute a few commands to get version information, etc.

The other option I know is to include my state in bazel's workspace_status, but that would invalidate cache for targets of all rule types that depend on the volatile or stable status files, and would also slow down every bazel invocation, not just the ones that build targets of my rule type.

Thanks.

Sid

[Jochen Issing]

You mean like a phony target (in Makefile language) similar to what the build status does? We did a little hack here, which only works on linux platforms: we depend on /proc/uptime.

Maybe that helps :)

/jochen

--


This email message, including attachments, may contain private,
proprietary, or privileged information and is the confidential information
and/or property of GRAIL, Inc., and is for the sole use of the intended
recipient(s). Any unauthorized review, use, disclosure or distribution is
strictly prohibited. If you are not the intended recipient, please contact
the sender by reply email and destroy all copies of the original message.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/afdc0c35-6022-4220-8d9f-9f202d2a7ecd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Siddhartha Bagaria]

Oh, that's a neat trick! But it won't work without the proc filesystem. More specifically, it won't work on darwin for us. I tried, and it definitely works on linux.

I found this field on spawn actions, but no way to set it... :-(

[Jochen Issing]

That’s interesting and maybe could lead to a correctly designed solution. Another one is actually in place for the build info, but I haven’t checked the sources. 

Thinking of a multi-platform hack: you coukd have a genrule creating some random content (similar to /dev/random but limited to some bytes, e.g. dd if=/dev/random count=20...) and depend on the outcome of that rule. Something like that could work cross platform. But it will still be a hack ofc.

:)

Cheers, jochen

[Andrew Suffield]

It sounds like what you're asking for is a more generic version of workspace_status, which is controlled via starlark. That's an interesting idea, although it doesn't exist today.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAJXhWqWE42vay3npVimdpdHVOO0iUkOXgYo8opHcQbm8MD7rmw%40mail.gmail.com.

[Siddhartha Bagaria]

The problem with the genrule is also that once built, it will be cached.

I don't think there is a way right now to mark an action to be unconditionally executed. I thought that the 'no-cache' key in execution_requirements will be enough, but it does not apply to the action cache.

[sbag...@grailbio.com]

Bumping up this thread again. Have there been any developments that allow unconditional execution of an action without using hacks like depending on /proc/uptime?

Thanks.

[Jared Neil]

This seem like exactly what workspace status command is for. https://docs.bazel.build/versions/2.0.0/user-manual.html#flag--workspace_status_command

Can you explain why that doesn't work for you?

[sta...@gmail.com]

Using the workspace status command for this purpose, i.e. changing the stable status vars, will unconditionally execute all actions that ingest the stable status file. I want only one specific action to unconditionally execute.

[Jared Neil]

Ah, yes. We got around this by writing a genrule that extracts just the variables we care about for later actions.

workspace_status.sh:

echo "STABLE_TOOLS $(determine-tool-versions)"

BUILD.bazel:

genrule(
    name = "tool-versions",
    outs = ["tool-versions.txt"],
    cmd = "grep '^STABLE_TOOLS' bazel-out/stable-status.txt > $@",
    stamp = 1,
)

Then all later actions depend on the :tool-versions target, and will only run if tool-versions.txt actually changed

[Siddhartha Bagaria]

Aah! Can't do that either. There are many 3p rules like rules_go, rules_docker, etc. that consume the entire file and are not within my control.

[Jingwen Chen]

> I looked into repository_rule and even the `local` attribute there does not give me unconditional execution.  I know that people depend on environment variables to capture system state, but I really do want to execute a few commands to get version information, etc.

You can create a repository_rule and mark it as `configure`. Then, before every build, run `bazel sync --configure`. This will trigger a forced evaluation of the rule. Then, if the rule generates a different BUILD file, your downstream targets will be automatically invalidated.

This email message, including attachments, may contain private, proprietary, or privileged information and is the confidential information and/or property of GRAIL, Inc., and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/abcb6fad-2632-4f74-8bc7-99aa992e7729%40googlegroups.com.

[Siddhartha Bagaria]

Thanks. This may work out. I will have to redesign some of the internals in my rules, because I was using a `rule` and not a `repository_rule`. But it could be OK if it goes with the broader story of how users are expected to tell bazel to re-read the system state, i.e. `bazel sync --configure`.