Known Issue: codereview.chromium.org is unreachable

[Aaron Gable]

This is, unfortunately, the same as yesterday's problems with bugs.chromium.org. Today the affected sites (and available workarounds) are:

codereview.chromium.org -> chromiumcodereview.appspot.com

crashpad.chromium.org -> crashpad-home.appspot.com

This is due to bad interactions with the fact that we serve both SNI and VIP, and the new Google Cloud Console isn't letting us set the VIP correctly.

friedman@ is working on the fix again today. The tracking issue is 657064. Further communication will be in this thread when we have more information.

Aaron

Apologies for the spam if you have received this message multiple times; due to communicating both internally and externally, sometimes multiple messages have to be sent.

[Daniel Jacques]

For code review uploading and interfacing, you can use the (new) flag, "--force-codereview-server", to override your repository's code review server during the outage. You will have to update "depot_tools" to access the new flag and re-authenticate with the new server URL.

The flag to use for "codereview.chromium.org" is: --force-codereview-server=https://chromiumcodereview.appspot.com

Note that this will not work with "git cl try".

--
You received this message because you are subscribed to the Google Groups "infra-announce" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-announc...@chromium.org.
To post to this group, send email to infra-a...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-announce/CAH58R2coFY2ktn0gdQEsCUq%3DA-5V-jwL%3Dc52fPR-Q3r%2BQ7_J_A%40mail.gmail.com.

[Aaron Gable]

30 minute update:

The issue is due to the deprecation of VIP, which our forwarding of various .chromium.org subdomains relied upon. friedman@ is working with the Cloud teams to either manually re-enable VIP for us, or to do an ahead-of-schedule DNS push to remove VIP from our configuration and only use SNI instead. The former will end the outage sooner but will require more work in the future; the latter is future-proof but slower.

Another update forthcoming in half an hour.

Aaron

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-announce/CAD1RUU9BjEvfWbQcUMOs8mAnfnHvwxV1qO_s%3DxK5B_m6GfJLZg%40mail.gmail.com.

[Prathmesh Prabhu Chromium]

+Chromium OS dev 

I think we should subscribe chromium-os-dev to infra-a...@chromium.org (just like chromium-dev) so that these announcements automatically get there too.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-announce/CAH58R2dSfZAHYQ2rPU9BVnLKvJ-7Q%3D0ef%2BSW9dM4zrbDvOYodQ%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Aaron Gable]

Final update: a new VIP has been set (the former option mentioned before), and the immediate issue has been resolved. Everyone should have access to codereview.chromium.org again.

A postmortem will be written in the next couple days. Hopefully it won't contain any internal-only information, and we'll be able to share it publicly as well.

Thanks for your patience,

Aaron

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-announce/CANG1%2Bn7SOj8mPBnJ%2BCq2y9wckhL7pBk4dAmeL6_qWa4%3DsqCMuA%40mail.gmail.com.

[Aaron Gable]

Followup: As part of the cleanup after the outage, another piece of configuration was just changed. The codereview.chromium.org site is still up, but 'git cl upload' (and potentially other 'git cl' commands) is temporarily broken. We're tracking this aftershock in this bug: https://bugs.chromium.org/p/chromium/issues/detail?id=657216

Aaron

[Thiago Farina]

On Tuesday, October 18, 2016, Aaron Gable <aga...@chromium.org> wrote:

Followup: As part of the cleanup after the outage, another piece of configuration was just changed. The codereview.chromium.org site is still up, but 'git cl upload' (and potentially other 'git cl' commands) is temporarily broken. We're tracking this aftershock in this bug: https://bugs.chromium.org/p/chromium/issues/detail?id=657216

Yeah, I'm unable to upload right now. Thanks for the notice.

--

Thiago Farina

[Aaron Gable]

Investigation shows that this is due to a DNS change which removed the VIP from our CNAME, causing clients to use SNI directly. Theoretically this would have prevented us from ever running into this morning's problems again. However, it turns out that the vast majority of our developer machines -- and our buildbot fleet -- don't actually support SNI (everyone uses Python 2.7.6; Python only added native SNI support in 2.7.9). This is resulting in massive breakage.

The DNS change is being rolled back, but doing so may take a very long time (the change which caused this outage took over 6 hours to propagate). Further updates will be forthcoming.

Aaron

[Paul Jensen]

FWIW I successfully uploaded by rolling back my depot_tools prior to the revert of the "--force-codereview-server" option (with "git checkout a8fb3a28") and using "git cl upload --force-codereview-server=https://chromiumcodereview.appspot.com"

--
You received this message because you are subscribed to the Google Groups "infra-announce" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-announce+unsubscribe@chromium.org.

To post to this group, send email to infra-a...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-announce/CAFenBsa62awA-KB_hTQuzJCXninJj6rqsyRuOfZ6cjyeGMY95g%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

[Anton Vayvod]

I wonder why the change was reverted so far (or at all), it seems pretty small and harmless.

On Tue, Oct 18, 2016 at 8:56 PM Paul Jensen <paulj...@chromium.org> wrote:

FWIW I successfully uploaded by rolling back my depot_tools prior to the revert of the "--force-codereview-server" option (with "git checkout a8fb3a28") and using "git cl upload --force-codereview-server=https://chromiumcodereview.appspot.com"

On Tue, Oct 18, 2016 at 8:48 PM, Thiago Farina <tfa...@chromium.org> wrote:

On Tuesday, October 18, 2016, Aaron Gable <aga...@chromium.org> wrote:

Followup: As part of the cleanup after the outage, another piece of configuration was just changed. The codereview.chromium.org site is still up, but 'git cl upload' (and potentially other 'git cl' commands) is temporarily broken. We're tracking this aftershock in this bug: https://bugs.chromium.org/p/chromium/issues/detail?id=657216

Yeah, I'm unable to upload right now. Thanks for the notice.

--

Thiago Farina

--

You received this message because you are subscribed to the Google Groups "infra-announce" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-announc...@chromium.org.

To post to this group, send email to infra-a...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-announce/CAFenBsa62awA-KB_hTQuzJCXninJj6rqsyRuOfZ6cjyeGMY95g%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org

View archives, change email options, or unsubscribe:

http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "infra-announce" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-announc...@chromium.org.

To post to this group, send email to infra-a...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-announce/CABQTWrm89fBKf64kkaK%3DOW%2BQpUbzgALfiWgZkxcAD%2B9feEsV%3DA%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Daniel Jacques]

It was reverted because it was a temporary hack that was not thoroughly integrated into the tools. A new change has landed that supersedes that one and automatically does the mapping.

[Aaron Gable]

A change has been landed in depot_tools which should un-break developers (primarily 'git cl upload') and un-break the trybots (which use 'apply_issue'). Please update your depot_tools to pick up the change.

If you're still seeing errors locally after getting the change, please update the tracking bug.

Aaron

[Aaron Gable]

We believe the situation to be resolved, and you shouldn't expect any further updates on this thread. In case updates are needed, I am handing off communication on this to chrishall@, part of our wonderful Sydney crew.

Thanks again for your patience,

Aaron