remote caching warnings/errors

164 views
Skip to first unread message

ens...@google.com

unread,
Aug 2, 2018, 5:07:39 AM8/2/18
to bazel-discuss
hi,

We're using
build --remote_http_cache=https://storage.googleapis.com/${GCS_BUCKET_NAME}
to speed up our continous integration build jobs. We're using baszel-15.0.0

It mostly works, but we get a few kinds of warnings/errors
1)
WARNING: Error writing to the remote cache:
502 Bad Gateway
<html reply here>

2)
WARNING: Error writing to the remote cache:
429 Too Many Requests
<?xml version='1.0' encoding='UTF-8'?><Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><Details>Changing object ${GCS_BUCKET_NAME}/cas/fc503dd74d0add4a88a492388ee56f46e71ecd117dbac8177c0846d2d44f5221 too quickly, please slow down</Details></Error>

I think I saw others too. Should I file bugs for them or are we using it wrong?

Stefan

Jakob Buchgraber

unread,
Aug 2, 2018, 5:12:19 AM8/2/18
to Stefan Sauer, bazel-discuss
Hi Stefan,

On Thu, Aug 2, 2018 at 11:07 AM ensonic via bazel-discuss <bazel-...@googlegroups.com> wrote:
hi,

We're using
build --remote_http_cache=https://storage.googleapis.com/${GCS_BUCKET_NAME}
to speed up our continous integration build jobs. We're using baszel-15.0.0

It mostly works, but we get a few kinds of warnings/errors
1)
WARNING: Error writing to the remote cache:
502 Bad Gateway
<html reply here>

Please file a bug for this error with some more details and a reproducer ideally. 

2)
WARNING: Error writing to the remote cache:
429 Too Many Requests
<?xml version='1.0' encoding='UTF-8'?><Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><Details>Changing object ${GCS_BUCKET_NAME}/cas/fc503dd74d0add4a88a492388ee56f46e71ecd117dbac8177c0846d2d44f5221 too quickly, please slow down</Details></Error>


This error is benign and we are well aware of it. Cloud Storage doesn't allow to modify a file more than once per second. So
two actions trying to upload the same file twice will trigger this error. One could argue that we should not be uploading the
same file twice :-), and we'll fix that soonish.
 
I think I saw others too. Should I file bugs for them or are we using it wrong?

Stefan

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/8dafc5d4-8dd1-4210-a413-f5860407f640%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Stefan Sauer

unread,
Aug 2, 2018, 9:15:12 AM8/2/18
to Jakob Buchgraber, bazel-...@googlegroups.com
On Thu, Aug 2, 2018 at 11:12 AM Jakob Buchgraber <buc...@google.com> wrote:
Hi Stefan,

On Thu, Aug 2, 2018 at 11:07 AM ensonic via bazel-discuss <bazel-...@googlegroups.com> wrote:
hi,

We're using
build --remote_http_cache=https://storage.googleapis.com/${GCS_BUCKET_NAME}
to speed up our continous integration build jobs. We're using baszel-15.0.0

It mostly works, but we get a few kinds of warnings/errors
1)
WARNING: Error writing to the remote cache:
502 Bad Gateway
<html reply here>

Please file a bug for this error with some more details and a reproducer ideally. 


2)
WARNING: Error writing to the remote cache:
429 Too Many Requests
<?xml version='1.0' encoding='UTF-8'?><Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><Details>Changing object ${GCS_BUCKET_NAME}/cas/fc503dd74d0add4a88a492388ee56f46e71ecd117dbac8177c0846d2d44f5221 too quickly, please slow down</Details></Error>


This error is benign and we are well aware of it. Cloud Storage doesn't allow to modify a file more than once per second. So
two actions trying to upload the same file twice will trigger this error. One could argue that we should not be uploading the
same file twice :-), and we'll fix that soonish.
 

Awesome. We have some trouble w.r.t. https://github.com/bazelbuild/bazel/issues/4558 and that was why I was eyeing more on the cache.

Are there any cleanup tools to avoid it growing unbounded? This parts of the docs does not really help

 
I think I saw others too. Should I file bugs for them or are we using it wrong?

Stefan

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/8dafc5d4-8dd1-4210-a413-f5860407f640%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Stefan Sauer | Software Engineer | ens...@google.com

Google Germany GmbH | Erika-Mann-Str 33 | 80636 München | Germany
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Wenn Sie nicht der richtige Adressat sind, leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen Sie die E-Mail und alle Anhänge. Vielen Dank.
       
This e-mail is confidential. If you are not the right addressee please do not forward it, please inform the sender, and please erase this e-mail including any attachments. Thanks.

Jakob Buchgraber

unread,
Aug 2, 2018, 9:40:23 AM8/2/18
to Stefan Sauer, bazel-discuss
Hi Stefan,

yes please file bugs for issues. Regarding garbage collection, Google Cloud Storage offers this functionality

Best,
Jakob 

Jakob Buchgraber

Software Engineer


Google Germany GmbH

Erika-Mann-Straße 33

80636 München


Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg


Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

    

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.




Stefan Sauer

unread,
Aug 2, 2018, 10:50:28 AM8/2/18
to Jakob Buchgraber, bazel-...@googlegroups.com
Awesome. Totally missed this line. Thanks.

Keith Smiley

unread,
Oct 29, 2019, 7:31:10 PM10/29/19
to bazel-discuss
I've just pinged this GitHub issue as well but has there been any movement on a solution to this? I just set this up with bazel 1.0.0 and immediately saw this 502 from GCP.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-...@googlegroups.com.


--
Stefan Sauer | Software Engineer | ens...@google.com

Google Germany GmbH | Erika-Mann-Str 33 | 80636 München | Germany
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Wenn Sie nicht der richtige Adressat sind, leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen Sie die E-Mail und alle Anhänge. Vielen Dank.
       
This e-mail is confidential. If you are not the right addressee please do not forward it, please inform the sender, and please erase this e-mail including any attachments. Thanks.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages