repository_cache with remote cache

[Ian O'Connell]

Hi,

I couldn't find it in the docs so apologies if it is somewhere, but is it possible to have bazel store the remotely fetched/repo cache entries in the remote cache?

Context here is things like github go down (its partially down right now), but we mostly fetch from github with https which makes caching the results with a proxy server not really doable. So rather than manually mirroring everything I was hoping we could use the remote cache we use for bazel artifacts to store things from 3rdparty sites?

Does it exist already maybe?

Thanks,

Ian.

[Jakob Buchgraber]

I believe --repository_cache currently only supports a directory as a value. I am not aware of any plans to support a http:// URL.

+Klaus Aehlig and/or +Jingwen Chen would know about plans for the --repository_cache.

Jakob Buchgraber

Software Engineer

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

    

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAP_RQpgvefoRUuAEGdSTYM6czwxj3EdHcifD-FqYVoT%2BhotGbw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Klaus Aehlig]

> I believe --repository_cache currently only supports a directory as a
> value. I am not aware of any plans to support a http:// URL.

> +Klaus Aehlig <aeh...@google.com> and/or +Jingwen Chen
> <jin...@google.com> would

> know about plans for the --repository_cache.

The repository cache currently is distfile cache; it stores files that
are downloaded by bazel, indexed by their hash. So, in order to obtain
a cache hit, it has to be
- a single file downloaded by bazel (specified by some URL, regardless
if http, https, or file), and
- a sha256 hash of the file to be downloaded has to be specified.
If this is the case, and a file with the given hash is in the cache, then
the file is taken from the cache (the URL is not even accessed).
That's all there currently is with respect to repository caching.

However, there are plans to add some form of cache for git repositories
(with some details still to be decided). See the ongoing discussion
on https://github.com/bazelbuild/bazel/pull/5928

Regards,
Klaus

--
Klaus Aehlig
Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschaeftsfuehrer: Paul Terence Manicle, Halimah DeLaine Prado

[Jakob Buchgraber]

Hi all,

I have just talked to aehlig@ and we mutually agreed that it makes sense that if a --remote_cache

is specified that Bazel should be able to read and write remote repository files to it, similar as it does

right now to local disk. We think that this solution makes sense for users who are running a remote

cache on stateless VMs/containers to speed up their repository downloads and to make their downloads

more reliable.

I would love for this to be implemented as part of a community contribution :-) and would be happy

to work closely with anyone willing to take on this task!

Best,

Jakob

Jakob Buchgraber

Software Engineer

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

    

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[ittai zeidman]

Interesting!
Can this work with RBE?
Let’s assume I have 10 remote repositories of http_file with their sha256 specified.
Currently locally I can use —repository_cache and be content.
On CI I use RBE for my remote execution and remote cache. Will that work (conceptually) or is it limited to only using remote_cache without remote execution?

[Jakob Buchgraber]

Yes it can also be made to work with RBE.

Jakob Buchgraber

Software Engineer

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

    

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/e0564db9-cb45-4758-a327-7c389d978e71%40googlegroups.com.

[Austin Schuh]

Hi Jacob,

Do you have a guess on how much work is involved?  I'm realizing that this is becoming the limiting factor in terms of scalability right now for us.  And if I don't have to add another infrastructure dependency to keep up, that would be wonderful.

Austin

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAGQ4vn3Tq0TDzK4UDg3e1X-GkgjfDETTkwT3gBPyNbpdfMXBCA%40mail.gmail.com.