How do I prevent bazel downloading dependencies and recompile them?

[wangjun...@gmail.com]

I'm working on a project which uses Tensorflow and other libraries as dependencies. I include them all by native.new_http_archive and native.http_archive.

Bazel sometimes redownloads my dependencies and recompile them if I switch to another git branch. Or did this even without changing any code. Is there any idea why this happens and how to prevent it? Thanks!

[Jingwen Chen]

You can use the flag --experimental_repository_cache=/some/path to keep the downloaded artifacts cached across branches and projects, as long as the checksums are provided.

On Thu, Oct 26, 2017, 10:15 PM <wangjun...@gmail.com> wrote:

I'm working on a project which uses Tensorflow and other libraries as dependencies. I include them all by native.new_http_archive and native.http_archive.

Bazel sometimes redownloads my dependencies and recompile them if I switch to another git branch. Or did this even without changing any code. Is there any idea why this happens and how to prevent it? Thanks!

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/6a5a3a7e-ab81-439f-b360-4e746ad9078c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oleg Tsarev]

But why re-download may occur?...

[wangjun...@gmail.com]

I don't really the reason. But if I switch my workspace to among some branches or pull the update from the code base, the re-download will occur even I didn't change any line about my dependencies and build flags.

[Oleg Tsarev]

Probably you have unstable reference to repo like "master"?

In this case on some check bazel can found updates in external repo.

Can you show me part of the WORKSPACE, where you reference to TensorFlow?

You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/_otGfLuHN_Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/6e5af094-b7ce-469f-ba84-56614feacc84%40googlegroups.com.

[wangjun...@gmail.com]

在 2017年10月27日星期五 UTC-7下午7:23:12，Oleg Tsarev写道：

> Probably you have unstable reference to repo like "master"?
> In this case on some check bazel can found updates in external repo.
>
>
> Can you show me part of the WORKSPACE, where you reference to TensorFlow?
>
>
> On Sat, Oct 28, 2017 at 1:33 AM, <wangjun...@gmail.com> wrote:
> I don't really the reason. But if I switch my workspace to among some branches or pull the update from the code base, the re-download will occur even I didn't change any line about my dependencies and build flags.
>
>
>
> On Friday, October 27, 2017 at 3:44:10 PM UTC-7，Oleg Tsarev wrote:
>
> > But why re-download may occur?...
>
> >
>
> > On Friday, October 27, 2017 at 7:59:24 AM UTC+2, Jingwen Chen wrote:
>
> > You can use the flag --experimental_repository_cache=/some/path to keep the downloaded artifacts cached across branches and projects, as long as the checksums are provided.
>
> >
>
> >
>
> >
>
> > On Thu, Oct 26, 2017, 10:15 PM  <wangjun...@gmail.com> wrote:
>
> > I'm working on a project which uses Tensorflow and other libraries as dependencies. I include them all by native.new_http_archive and native.http_archive.
>
> >
>
> >
>
> >
>
> > Bazel sometimes redownloads my dependencies and recompile them if I switch to another git branch. Or did this even without changing any code. Is there any idea why this happens and how to prevent it? Thanks!
>
> >
>
> >
>
> >
>
> > --
>
> >
>
> > You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
>
> >
>
> > To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
>
> >
>
> > To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/6a5a3a7e-ab81-439f-b360-4e746ad9078c%40googlegroups.com.
>
> >
>
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
>
> You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
>
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/_otGfLuHN_Y/unsubscribe.
>

> To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.

>
> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/6e5af094-b7ce-469f-ba84-56614feacc84%40googlegroups.com.
>
>
>
> For more options, visit https://groups.google.com/d/optout.

I point the Tensorflow to https://github.com/tensorflow/tensorflow/archive/v1.4.0-rc0.tar.gz, which is not a 'master' repo for tensorflow. Another wired thing is that bazel also rebuild protobuf as well, I run sha256sum to the output(libprotobuf.a) of protobuf. And I found that the output files are exactly the same for each build. BTW, I invoke all dependencies by http_archive with sha256 checksum, I don't think this is the reason.

Do you have any idea about how bazel compute action hash? It might help me to find out the reason. Thanks!

[Oleg Tsarev]

Sounds like bazel bug. Please fill ticket.

On Sat, Oct 28, 2017 at 4:40 AM, <wangjun...@gmail.com> wrote:

On Sat, Oct 28, 2017 at 7:23:12, Oleg Tsarev wrote:
> Probably you have unstable reference to repo like "master"?
> In this case on some check bazel can found updates in external repo.
>
>
> Can you show me part of the WORKSPACE, where you reference to TensorFlow?
>
>
> On Sat, Oct 28, 2017 at 1:33 AM,  <wangjun...@gmail.com> wrote:
> I don't really the reason. But if I switch my workspace to among some branches or pull the update from the code base, the re-download will occur even I didn't change any line about my dependencies and build flags.
>
>
>
> On Friday, October 27, 2017 at 3:44:10 PM UTC-7，Oleg Tsarev wrote:
>
> > But why re-download may occur?...
>
> >
>
> > On Friday, October 27, 2017 at 7:59:24 AM UTC+2, Jingwen Chen wrote:
>
> > You can use the flag --experimental_repository_cache=/some/path to keep the downloaded artifacts cached across branches and projects, as long as the checksums are provided.
>
> >
>
> >
>
> >
>
> > On Thu, Oct 26, 2017, 10:15 PM  <wangjun...@gmail.com> wrote:
>
> > I'm working on a project which uses Tensorflow and other libraries as dependencies. I include them all by native.new_http_archive and native.http_archive.
>
> >
>
> >
>
> >
>
> > Bazel sometimes redownloads my dependencies and recompile them if I switch to another git branch. Or did this even without changing any code. Is there any idea why this happens and how to prevent it? Thanks!
>
> >
>
> >
>
> >
>
> > --
>
> >
>
> > You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
>
> >
>
> > To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
>
> >
>
> > To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/6a5a3a7e-ab81-439f-b360-4e746ad9078c%40googlegroups.com.
>
> >
>
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
>
> You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
>
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/_otGfLuHN_Y/unsubscribe.
>

> To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.

>
> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/6e5af094-b7ce-469f-ba84-56614feacc84%40googlegroups.com.
>
>
>
> For more options, visit https://groups.google.com/d/optout.

在 2017年10月27日星期五 UTC-7下午7:23:12，Oleg Tsarev写道：

> Probably you have unstable reference to repo like "master"?
> In this case on some check bazel can found updates in external repo.
>
>
> Can you show me part of the WORKSPACE, where you reference to TensorFlow?
>
>
> On Sat, Oct 28, 2017 at 1:33 AM,  <wangjun...@gmail.com> wrote:
> I don't really the reason. But if I switch my workspace to among some branches or pull the update from the code base, the re-download will occur even I didn't change any line about my dependencies and build flags.
>
>
>
> On Friday, October 27, 2017 at 3:44:10 PM UTC-7，Oleg Tsarev wrote:
>
> > But why re-download may occur?...
>
> >
>
> > On Friday, October 27, 2017 at 7:59:24 AM UTC+2, Jingwen Chen wrote:
>
> > You can use the flag --experimental_repository_cache=/some/path to keep the downloaded artifacts cached across branches and projects, as long as the checksums are provided.
>
> >
>
> >
>
> >
>
> > On Thu, Oct 26, 2017, 10:15 PM  <wangjun...@gmail.com> wrote:
>
> > I'm working on a project which uses Tensorflow and other libraries as dependencies. I include them all by native.new_http_archive and native.http_archive.
>
> >
>
> >
>
> >
>
> > Bazel sometimes redownloads my dependencies and recompile them if I switch to another git branch. Or did this even without changing any code. Is there any idea why this happens and how to prevent it? Thanks!
>
> >
>
> >
>
> >
>
> > --
>
> >
>
> > You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
>
> >
>
> > To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
>
> >
>
> > To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/6a5a3a7e-ab81-439f-b360-4e746ad9078c%40googlegroups.com.
>
> >
>
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
>
> You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
>
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/_otGfLuHN_Y/unsubscribe.
>

> To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.

>
> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/6e5af094-b7ce-469f-ba84-56614feacc84%40googlegroups.com.
>
>
>
> For more options, visit https://groups.google.com/d/optout.

I point the Tensorflow to https://github.com/tensorflow/tensorflow/archive/v1.4.0-rc0.tar.gz, which is not a 'master' repo for tensorflow. Another wired thing is that bazel also rebuild protobuf as well, I run sha256sum to the output(libprotobuf.a) of protobuf. And I found that the output files are exactly the same for each build. BTW, I invoke all dependencies by http_archive with sha256 checksum, I don't think this is the reason.

Do you have any idea about how bazel compute action hash? It might help me to find out the reason. Thanks!

--

You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/_otGfLuHN_Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/0e9f3416-eb24-4737-8e25-7dc680d8d3e2%40googlegroups.com.

[Oleg Tsarev]

I do not know. I am not the developer of bazel, just enthusiast. But I suppose you meet some bug, because I never see something like that in my bazel usage.

--

You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/_otGfLuHN_Y/unsubscribe.

To unsubscribe from this group and all its topics, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/06eccfdc-a271-42bd-890b-1a6c098f5084%40googlegroups.com.

[Damien Martin-Guillerez]

The details on how a repository is invalidated was explained in that doc: https://www.bazel.build/designs/2016/10/18/repository-invalidation.html.

If you change branch often, I recommend you use --experimental_repository_cache= together with --experimental_local_disk_cache --experimental_local_disk_cache_path= to have an action cache (I actually always use the former). There are not made default just because we do not have a deletion strategy yet.

To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/06eccfdc-a271-42bd-890b-1a6c098f5084%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAPuPPcs16yrm50WSfy1Jm3w_-OJDLfwjEm8PboLW%2BB1i8L1Wyw%40mail.gmail.com.

[wangjun...@gmail.com]

在 2017年10月30日星期一 UTC-7上午5:23:01，Damien Martin-Guillerez写道：

Thank you! The rebuild and redownload seems to be solved after I enable linux-sandbox strategy. I also add these two flags, the bazel should be working fine now.