How to incrementally build on circleci

[srobe...@codeit.com]

Hi gang, congrats on reaching beta! loving Bazel.

One thing I'm struggling with is getting Bazel to do incremental builds on circleci.

Circleci storage is ephemeral, your build is moved to different hosts every time. You have to specify directories that are cached/restored. Currently I've told circleci to cache /home/ubuntu/.cache/bazel

Unfortunately Bazel thinks everything needs to be rebuilt every time. Anyone have any luck getting incremental builds to work on circle ci? What else do I need to cache?

What does Bazel look at to determine if something has been built. Is it timestamps, md5 fingerprints, etc... If it's timestamps I'm thinking that when circleci restores files they're set to the time they've been restored from cache.

[Damien Martin-guillerez]

Hi, thanks :)

AFAIU Bazel use md5 over a short description of the file (timestamp, name, filesize, ..., not the whole file content) for the output directory. It wasn't made to be cached over multiple installation. Someone with better understanding of that part would be better answering here.

Another way is to use the script in scripts/ci/ci.sh that does look at what have changed using git diff and query to only build specific target.

In Bazel we do run a hot server for our incremental build on CI (we have instances spawn on GCE).

The incoming work with distributed caching might help with your setup but is probably still a few month away.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/85dafd68-a915-411f-b132-ef67c4dda103%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[srobe...@codeit.com]

>
>
> Another way is to use the script in scripts/ci/ci.sh that does look at what have changed using git diff and query to only build specific target.
> In Bazel we do run a hot server for our incremental build on CI (we have instances spawn on GCE).

Thanks for pointing me in this direction. It doesn't quite work for two reasons for me.

1. Bazel rc2 baloons over circle ci's 4GB when you query
2. I'm using Bazel to compile allot of bulid tooling, the biggest speed improvement I'd get is if it could use the results of the previous builds.

So correct me if if I'm wrong, but it looks like currently there's no way to take a snapshot of the current bazel output and relocate it to another machine/instance/node/server/whatever. So my best option is to have a server up continuously?

[Damien Martin-guillerez]

I am afraid it is until we have distributed caching.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/405da657-f30f-4e91-9590-aef237f9e153%40googlegroups.com.

[Eric Fellheimer]

In principle you should be able to copy the output files and action_cache files in the output_base of the build to the corresponding directory of the other machine. (I have not personally experimented / verified this). This may be a relatively expensive operation, but may save some time compared to a fully clean build.

Eric Fellheimer
fe...@google.com

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/405da657-f30f-4e91-9590-aef237f9e153%40googlegroups.com.

[Jeff Cox]

Is it not possible with CircleCI to request a persistent build server? Copying files around may help incremental build performance, but it will still be slower than taking advantage of Bazel's in-memory caching.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAJMmbx%2BBACjDYWCPzLeBidfQpXbbofWP-RqXLdoLccK8CcXCzw%40mail.gmail.com.

[Scott Robertson]

On Fri, Sep 11, 2015 at 8:59 AM Eric Fellheimer <fe...@google.com> wrote:

In principle you should be able to copy the output files and action_cache files in the output_base of the build to the corresponding directory of the other machine. (I have not personally experimented / verified this). This may be a relatively expensive operation, but may save some time compared to a fully clean build.

Where would the action_cache located? I'm restoring everything under 

/home/ubuntu/.cache/bazel  which I believe houses the output_base.  I see the symlinks in the workspace root point to that directory.

[Scott Robertson]

I don't think it's possible to get a persistent server.

In my particular case copying build tools around that I've built is a huge win and helps us stay as they say hermetically sealed. 

[Damien Martin-guillerez]

Coming back to that discussion as I finally got some time to do some tests.

If you can save the ~/.cache folder and preserve the path to your workspace, you should get a pretty good incrementality. What will be recomputed on an unchanged build is loading the package and computing the checksums of the files.

It's not as good as a hot a server but it is like you do bazel shutdown; bazel build.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAEtCvMy6GU55B%2Bcqih1mk6v9ZrYR4Cs62LoO4u-gN_HBFrU7nA%40mail.gmail.com.

[srobe...@codeit.com]

> If you can save the ~/.cache folder and preserve the path to your workspace, you should get a pretty good incrementality. What will be recomputed on an unchanged build is loading the package and computing the checksums of the files.
>
>

Yes!! actually after some further tweaking I discovered I get incremental builds if I run bazel tests during CircleCI's dependency phase rather than test phase. It's the only phase that actually caches when it's done running. Here's a snippet from my circle.yml for the curious.

======= circle.yml =======

dependencies:
cache_directories:

- "/home/ubuntu/.cache/bazel"

override:
- curl -L -O https://github.com/bazelbuild/bazel/releases/download/0.1.0/bazel-0.1.0-installer-linux-x86_64.sh
- chmod 755 bazel-0.1.0-installer-linux-x86_64.sh && ./bazel-0.1.0-installer-linux-x86_64.sh --user
- cp ~/ntropy/build_tools/bazelrc.circleci ~/.bazelrc
- bazel test --explain=$CIRCLE_ARTIFACTS/bazel-test.log --verbose_explanations :all //scripts/migrations

test:
override:
- echo "best test ever"

====================

However, there's still times when it decides to not build incrementally. And I can't figure out why. I've enabled verbose explanations but I get the following useless output.

==============

Build options: --package_path=%workspace%:/home/ubuntu/.bazel/base_workspace --define='buildenv=circleci' --spawn_strategy=standalone --genrule_strategy=standalone --nodistinct_host_configuration --define='buildenv=circleci' --spawn_strategy=standalone --genrule_strategy=standalone --test_output=errors --nodistinct_host_configuration --explain=/tmp/circle-artifacts.epERZLw/bazel-test.log --verbose_explanations
Executing action 'BazelWorkspaceStatusAction stable-status.txt': unconditional execution is requested.
Executing action 'Executing genrule //scripts:bin-wheels': One of the files has changed.
Executing action 'Creating source manifest for //:java-code': action command has changed.
Executing action 'Creating runfiles tree bazel-out/local_linux-fastbuild/bin/java-code.runfiles': One of the files has changed.
Executing action 'Testing //python:test': One of the files has changed.
Executing action 'Building java projects //:build-java': One of the files has changed.
Executing action 'Gathering python requirements //scripts:nosetest': One of the files has changed.
Executing action 'Testing //scripts/automation:test': One of the files has changed.
Executing action 'Testing //:java-code': One of the files has changed.

====================

Seeing that "One of the giles has change" isn't very helpful, obviously I know that. :) Is there away to get Bazel to tell me which file Bazel thinks has changed?