Google BigQuery is currently experiencing an elevated rate of job failures.

12 views
Skip to first unread message

Google Cloud Platform Status

unread,
Jul 23, 2020, 7:34:37 PM7/23/20
to bigquery-dow...@googlegroups.com
Description: Google BigQuery is currently experiencing an elevated rate of job failures in the US multi-region.

Symptoms: Google BigQuery job failures returning an "Internal Error" message.

Mitigation is under way and our engineering team is working on it.

We will provide an update by Thursday, 2020-07-23 18:00 US/Pacific with current details.

Diagnosis: You may be affected if you notice Google BigQuery jobs failing with an "Internal Error" message.

Workaround: None at this time.

Google Cloud Platform Status

unread,
Jul 23, 2020, 7:39:21 PM7/23/20
to bigquery-dow...@googlegroups.com
Description: Google BigQuery is currently experiencing an elevated rate of job failures in the US multi-region.

Symptoms: Google BigQuery job failures returning an "Internal Error" message.

Mitigation is under way and our engineering team is working on it.

We will provide an update by Thursday, 2020-07-23 18:05 US/Pacific with current details.

Google Cloud Platform Status

unread,
Jul 23, 2020, 8:52:41 PM7/23/20
to bigquery-dow...@googlegroups.com
Description: Mitigation work is still underway by our engineering team. Affected projects are being mitigated in batches, so retrying the failed job periodically may succeed.

We will provide an update by Thursday, 2020-07-23 20:00 US/Pacific with current details.

Diagnosis: You may be affected if you notice Google BigQuery jobs failing with an "Internal Error" message.

Workaround: Retry the failed jobs periodically, as they will start succeeding once your project is mitigated.

Google Cloud Platform Status

unread,
Jul 23, 2020, 10:39:56 PM7/23/20
to bigquery-dow...@googlegroups.com
Description: Mitigation work is still underway by our engineering team.

The mitigation is expected to complete by Thursday, 2020-07-23 20:30 US/Pacific.

We will provide more information by Thursday, 2020-07-23 20:45 US/Pacific.

Google Cloud Platform Status

unread,
Jul 23, 2020, 11:21:15 PM7/23/20
to bigquery-dow...@googlegroups.com
Description: Mitigation work is still underway by our engineering team.

We are making progress in the mitigation, but it is taking longer than our initially expected timeline.

We will provide more information by Thursday, 2020-07-23 21:30 US/Pacific.

Google Cloud Platform Status

unread,
Jul 24, 2020, 12:15:43 AM7/24/20
to bigquery-dow...@googlegroups.com
Description: Mitigation work is still underway by our engineering team.

We will provide more information by Thursday, 2020-07-23 22:30 US/Pacific.

Google Cloud Platform Status

unread,
Jul 24, 2020, 1:20:10 AM7/24/20
to bigquery-dow...@googlegroups.com
Description: As the full mitigation is taking longer than we have expected, we are trying to resolve individual projects manually at the same time. If you already have a support case open with an affected project, we will let you know individually once your project is mitigated.

We will provide more information by Thursday, 2020-07-23 23:30 US/Pacific.

Google Cloud Platform Status

unread,
Jul 24, 2020, 2:28:33 AM7/24/20
to bigquery-dow...@googlegroups.com
Description: Mitigation work is making progress, and we are seeing error rates down from the peak at 1% to under 0.01%.

We are mitigating the affected projects in batch and individually. If you already have a support case open with an affected project, we will let you know individually once your project is mitigated.

We will provide more information by Friday, 2020-07-24 01:00 US/Pacific.

Google Cloud Platform Status

unread,
Jul 24, 2020, 4:02:56 AM7/24/20
to bigquery-dow...@googlegroups.com
The issue with Google BigQuery experiencing an elevated rate of job failures is believed to be affecting a very small number of projects and our Engineering Team is working on it.

If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

We thank you for your patience while we're working on resolving the issue.
Reply all
Reply to author
Forward
0 new messages