Quick check in

[Pallavi Kappaguntula]

Hi Folks,

I wanted to send a quick note to provide visibility into our current plan. We are actively working on open sourcing native C++ and Java rule implementations and hope to get this done within a month (there are a lot of Google dependencies to untangle, so this is a nontrivial task). We are also determining the feasibility of releasing the Android and iOS rules in their current form.

We’re taking some time to do due diligence and experiment with the Android development experience with Buck. We will reach out to you with additional questions.

Are all the right people in the bazel-discuss@ mailing list?

Are there other ways in which we can enhance your understanding of Bazel?

We will reach out to you when we have clear understanding of timelines. Meanwhile, keep the questions coming!

Cheers,

Pallavi

[dreiss]

Hi, Bazel team.  I'd like to call your attention to a post we just made about using Buck in an Android project: http://facebook.github.io/buck/article/exopackage.html .  I think this could be useful to you in a few ways.

- It gives an overview of the steps required to port an existing project to Buck, including demonstrations with two projects.  The post links to AntennaPod diffs, but that repository also has a branch with a set of patches that show how to get the iosched app working.

- It gives a little more detail about exopackage, which is one of our most important optimizations.  I didn't describe it before because it wasn't quite ready for external consumption.  I'm quite sure that it will be possible for you to implement the build side of it in Bazel, but it does require a little extra coordination between the build tool and the installation process.

--David

[dreiss]

Here are some questions that I've been thinking about:

- Can we expect an open-source action/spawn cache?  This will be pretty important for us if we're going to try to use Bazel for fbandroid.  We can provide our own storage backend, but we'd need Bazel to provide the key computation.

- We have some new numbers of compile servers that you should be able to reproduce.  The AntennaPod and iosched apps show about a 3x improvement in clean build time when using a warm buckd vs a fresh one.  I don't think our fbandroid developers could tolerate losing this speedup.

- What sort of scripting do you currently have in Blaze?  Since BUCK files are full Python, we frequently use this to define "macros" (Python functions) that expand to multiple rule definitions, which allows us to have one high-level call like "fb_android_binary" produce multiple genrules and android_libraries required to assemble our final binary.

- My impression is that the spawn cache will be able to handle all of the cases that we currently use AbiRule for, so that's good.  We'll have to make a few changes, but I think it will be worth it.

- Is it possible to dynamically create new rules after the analysis phase?  My impression from the Shake paper is that they can construct rules in the middle of the build.  Is this possible with Bazel?

[Han-Wen Nienhuys]

On Wed, Jul 23, 2014 at 8:56 PM, dreiss <dre...@gmail.com> wrote:
> Here are some questions that I've been thinking about:
>
> - Can we expect an open-source action/spawn cache? This will be pretty
> important for us if we're going to try to use Bazel for fbandroid. We can
> provide our own storage backend, but we'd need Bazel to provide the key
> computation.

We're thinking about remote execution, but we don't have anything
concrete and it will take a while.

A spawn cache like Buck has should actually be pretty simple if you
don't do remote execution. The spawn is an immutable structure, so you
can compute a key by hashing what comes out of the accessors together

https://github.com/google/bazel/blob/master/src/main/java/com/google/devtools/build/lib/actions/Spawn.java

execution strategies are already pluggable. You'd have to add
something along the lines of

https://github.com/google/bazel/blob/master/src/main/java/com/google/devtools/build/lib/standalone/LocalSpawnStrategy.java

which does a lookup before spawning locally. If you annotate it as

@ExecutionStrategy(name = { "cassandra" }, contextType =
SpawnActionContext.class)

you can do

bazel build --spawn_strategy=cassandra //my:target

which will route actions through the newly added strategy.

> - What sort of scripting do you currently have in Blaze? Since BUCK files
> are full Python, we frequently use this to define "macros" (Python
> functions) that expand to multiple rule definitions, which allows us to have
> one high-level call like "fb_android_binary" produce multiple genrules and
> android_libraries required to assemble our final binary.

We have a Python preprocessor, internally. People can use it to
preprocess their BUILD files, but also to load extensions from other
parts of the depot ("subincludes").

We're not very happy with it for the following reasons:

* Our infrastructure that understands and edits BUILD automatically
cannot deal with full-blown python.
* It's prone to misuse: people use it to generate large numbers of
targets, which in turn cause production problems.
* It's hard to make completely hermetic/secure.
* It's not good enough to write real rules (the BUILD file is parsed
before the target platform is decided)
* the code is a bit of a mess.

Compared the Buck preprocessor, it's a little more advanced. Build
extensions have labels, and are part of packages, so visibility labels
are enforced for extensions, and all kinds of wonky scenarios where
they move between packages, or are symlinked about are handled
correctly.

We hope that the skylark extension model can cater for most use cases,
but I'm not sure we will escape adding some form of preprocessing back
for bazel at some point in the future. The extension point is here

https://github.com/google/bazel/blob/master/src/main/java/com/google/devtools/build/lib/blaze/BlazeModule.java#L102

with the preprocessor defined here

https://github.com/google/bazel/blob/master/src/main/java/com/google/devtools/build/lib/packages/Preprocessor.java

we have a dummy implementation, which is not yet shipped in Bazel,
unfortunately.

> - Is it possible to dynamically create new rules after the analysis phase?
> My impression from the Shake paper is that they can construct rules in the
> middle of the build. Is this possible with Bazel?

We probably use different terminology: "rule" is the construct in the
input file, from which we generate Targets and Actions in the Analysis
phase. By definition, the analysis phase is after rules have been
processed.

What kind of process were you thinking of?

> --
> You received this message because you are subscribed to the Google Groups
> "bazel-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bazel-discus...@googlegroups.com.
> To post to this group, send email to bazel-...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bazel-discuss/59b4f6f8-f4c9-484a-9195-a18b09901d9a%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.



--
Han-Wen Nienhuys
Google Munich
han...@google.com

[Lukács T. Berki]

Sorry about the delay. This message got lost in the bowels of my inbox.

On Wed, Jul 23, 2014 at 8:56 PM, dreiss <dre...@gmail.com> wrote:

Here are some questions that I've been thinking about:

- Can we expect an open-source action/spawn cache?  This will be pretty important for us if we're going to try to use Bazel for fbandroid.  We can provide our own storage backend, but we'd need Bazel to provide the key computation.

- We have some new numbers of compile servers that you should be able to reproduce.  The AntennaPod and iosched apps show about a 3x improvement in clean build time when using a warm buckd vs a fresh one.  I don't think our fbandroid developers could tolerate losing this speedup.

- What sort of scripting do you currently have in Blaze?  Since BUCK files are full Python, we frequently use this to define "macros" (Python functions) that expand to multiple rule definitions, which allows us to have one high-level call like "fb_android_binary" produce multiple genrules and android_libraries required to assemble our final binary.

- My impression is that the spawn cache will be able to handle all of the cases that we currently use AbiRule for, so that's good.  We'll have to make a few changes, but I think it will be worth it.

 

- Is it possible to dynamically create new rules after the analysis phase?  My impression from the Shake paper is that they can construct rules in the middle of the build.  Is this possible with Bazel?

As Han-Wen said, there is a bit of a terminology issue: for us, "Rule" is what one writes in the BUILD file. Each executed step (e.g. a shell command, a C++ compile) in the build is called an "Action". I believe this is called a "Step" in Buck parlance. And the reason why you ask this question in particular is that the granularity of incrementality in Buck is a Rule, whereas it's an Action in Blaze.

Sadly, we cannot create Actions during the execution phase, or, phrased in another way, the set of Actions has to be known before we execute any of them. This creates a few limitations, mostly in the area of test sharding: we have to determine the number of shards before we build the tests, so we cannot do it adaptively based on how many test methods there are.

re: exopackage -- I didn't quite understand how this works on the device. Is the trick that you install a smaller APK (that does not have the resources, only the code) on each incremental step, then access that other APK when the application starts up? How does this not work with a few devices?