Gerrit's "submit to cq" button replaced with "submit to master" button?

[Nico Weber]

Hi,

https://chromium-review.googlesource.com/c/chromium/src/+/1169302 at least has a "Submit to master" button where "submit to cq" used to be.

I note most the last few commits on build.chromium.org (https://chromium-review.googlesource.com/c/chromium/src/+/1163933, https://chromium-review.googlesource.com/c/chromium/src/+/1169305, https://chromium-review.googlesource.com/c/chromium/src/+/1168084) were landed directly, instead of going through the cq.

Committers: Hit 'a' to bring up the 'mail comments' dialog and hit cq+2 there to commit changes through the cq.

Gerrit folks: This is likely an unintended change -- can we roll back to the regular CQ button asap? Else the tree will likely be pretty rough today if everybody (unintentionally) just lands without using the CQ.

Nico

[Philip Rogers]

I think this may be related to https://crbug.com/872729

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAMGbLiFR172NrMfr%2B9MmeQDAbwDRNXZ0T%3DNSxjjMF9sDCR7H%3DA%40mail.gmail.com.

[Nico Weber]

https://bugs.chromium.org/p/chromium/issues/detail?id=872722 is a bit earlier, so duped 872729 into that. https://bugs.chromium.org/p/gerrit/issues/detail?id=9546 too, but can't dupe into there.

[Nico Weber]

But to repeat, until this is fixed: Committers: Hit 'a' to bring up the 'mail comments' dialog and hit cq+2 there to commit changes through the cq instead of hitting the "submit button that's where the "cq" button usually is.

[Elly Fong-Jones]

On the chromium side, maybe also related to https://bugs.chromium.org/p/chromium/issues/detail?id=872722

This is a chromium infra emergency - I am attempting to get in touch with whoever is oncall for gerrit right now.

-- elly

[Elly Fong-Jones]

As sheriff on duty, I have closed the tree and will keep it closed until I get the good word from the Gerrit folks.

-- elly

[Elly Fong-Jones]

Update: We are working with Infra to figure out what is going on and why. There are knock-on effects on the rest of our infrastructure so expect it to take some time to sort things out.

-- elly

[Wez]

Does closing the tree actually prevent committers from submitting directly?  I thought tree open/closed was a CQ/presubmit check?

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/10b48142-f8c5-4491-9e69-2ae1d0cf1c79%40chromium.org.

[Mike Frysinger]

it doesn't block, but at least it makes it a bit of a red flag that some people will notice

-mike

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CALekkJdQURMwD2mkKHsO23c%2B0AWgwOxLRPnjE4xVhW0xCV88CQ%40mail.gmail.com.

[Elly Fong-Jones]

No, it does not. I haven't yet been able to get in touch with anyone who has the technical ability to prevent direct submits.

-- elly

[Joe Mason]

Is it ok to do cq dry runs while this is going on?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/4462c669-0101-41fa-a06c-6e4fa4211467%40chromium.org.

[Elly Fong-Jones]

Update:

Gerrit folks are working to roll back a gerrit canary rollout that happened during the timeframe in which the incident started.

Chrome Ops folks (tandrii@) are figuring out how we are going to remediate this after the fact - i.e., how to repair the state of chromium/src and v8/src.

Chrome Ops have also globally disabled submits to chromium/src, pending fixing the root cause and repairing the damage to the repo

Chromium Sheriffs are doing our best with the state of the world :)

-- elly

[Andrii Shyshkalov]

CQ dry runs are fine.

But if you see a submit button anywhere, please let me know :)

[Elly Fong-Jones]

Update:

Gerrit is still working on rolling back to the prior version. There is no ETA for this yet.

Submits are now even more globally disabled than previously - some folks were still able to submit CLs. There is no ETA yet for re-enabling submits or reopening the tree.

There is a postmortem draft in progress.

Unfortunately, there are many other infra issues happening as consequences of this one - sheriffs and ops are triaging them as we see them.

-- elly

[Elly Fong-Jones]

Update:

The ETA for gerrit to be fixed is approximately 1 hour.

The ETA for the tree to be reopened and submits to be allowed again is still unknown - it depends heavily on how we repair the tree.

The postmortem draft is available (Google only) here: https://docs.google.com/document/d/11UwPvlhjK5DLKSOBOpF5fRsT8zx1urEIDLavuAcTotY/edit#

The remediation proposal doc (again Google only) is here: https://docs.google.com/document/d/15MR-B50dPzzY483-2jsAEAS8VlNpOm2Xc_RrQeqpBn0/edit

Both docs are still in very early states so please do not comment on them at this time unless you are actively involved in handling this incident.

-- elly

[Elly Fong-Jones]

Update:

The fixed version of gerrit is deployed and the serving version of gerrit is now fixed.

We are beginning steps to repair the repository. Once those are complete, we will still need to see what the state of downstream infra and tests are. There is no ETA for this.

-- elly

[Elly Fong-Jones]

Update:

We are ~70% done with the repository repair work. Once that is complete, we will validate that gerrit is working properly on the live repository, then reopen submits and reopen the tree.

Future updates will come from pkasting@, who is taking over as incident commander.

-- elly

[Peter Kasting]

Update: The repository has been repaired and https://chromium.googlesource.com/chromium/src now points to a good commit.  Changes which landed during the breakage have been reverted through the standard revert mechanisms (so if you submitted one, you should have seen email); you'll need to re-land them once the tree opens.

We're in the process of landing a test commit to ensure the CQ and associated machinery is all working correctly and the tree is reasonably green.  Once we assess the resulting state I will reopen.

PK

[Peter Kasting]

Update: A direct commit was stamped with the proper footers.  However we have not yet managed to successfully trigger tryjobs/land a change through the CQ.  Hopefully that's close, at which point we can enable the CQ to operate for everyone (but won't yet open the tree).

The tree is building the direct commit; builders seem largely green, test bots have not yet run.  Once the CQ is up and the tests look reasonable, we should be able to open.  ETA for that is 45 min.

Future updates here may come from dpranke@, who will take over as incident commander some time in the next hour.

PK

[becca...@chromium.org]

Is code review on chromium/src disabled too as I see "You don't have permission to edit this label." next to "Code-Review"?

[Peter Kasting]

Update: This outage should be over.

The CQ has been re-enabled, submit rights have been restored on all branches, and the tree is open.  The CQ seems to be landing patches.

Special thanks to tandrii@ and dborowitz@ for leading the Chromium and Gerrit sides of the response here and to ellyjones@ for being the incident commander through the majority of the time.  If there are further followups we will reply here.  A postmortem will be forthcoming at some point.

PK

[Peter Kasting]

On Thu, Aug 9, 2018 at 2:37 PM <becca...@chromium.org> wrote:

Is code review on chromium/src disabled too as I see "You don't have permission to edit this label." next to "Code-Review"?

I don't see that on any of the CLs I own or am a reviewer for.  Link to affected CL?

PK 

[Lei Zhang]

Try jobs on chromium_presubmit [1] has been going purple for the last
hour. CLs without a recent, green chromium_presubmit run probably
can't get through CQ.

[1] https://ci.chromium.org/p/chromium/builders/luci.chromium.try/chromium_presubmit

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
> ---
> You received this message because you are subscribed to the Google Groups
> "Chromium-dev" group.
> To view this discussion on the web visit

> https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAHOzFApi-niRr1KaisCXW2CQ5FOKU4WVFAMb1Q%2B7oS%2Bk2FAeg%40mail.gmail.com.

[Peter Kasting]

On Thu, Aug 9, 2018 at 2:51 PM Lei Zhang <the...@chromium.org> wrote:

Try jobs on chromium_presubmit [1] has been going purple for the last
hour. CLs without a recent, green chromium_presubmit run probably
can't get through CQ.

[1] https://ci.chromium.org/p/chromium/builders/luci.chromium.try/chromium_presubmit

Yep, this should clear itself out fairly quickly (we hope); tandrii@ just added 20 more bots to the queue to work through the backlog.

PK 

[Dirk Pranke]

I think "fairly quickly" is probably open to some debate :).

I actually expect things to be kinda bumpy for the next few hours, but we'll be keeping an eye on it as per normal sheriff and trooper duties.

-- Dirk

 

PK 

[Stefan Zager]

The repository munging seems to have broken my merge-based flow:

$ git cl upload

ERROR: Your diff contains 20 commits already in c53e53438d29ce4c55c6df6fd1d646ad2f5047f6.

Run "git log --oneline a944b0d7f322400a54def81b313f16b9e125167b..HEAD" to get a list of commits in the diff.  If you are using a custom git flow, you can override the reference used for this check with "git config gitcl.remotebranch <git-ref>".

Git sanity check failure

$

Any guidance?

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAHOzFApi-niRr1KaisCXW2CQ5FOKU4WVFAMb1Q%2B7oS%2Bk2FAeg%40mail.gmail.com.

[Stefan Zager]

On Thu, Aug 9, 2018 at 3:22 PM Stefan Zager <sza...@google.com> wrote:

The repository munging seems to have broken my merge-based flow:

$ git cl upload

ERROR: Your diff contains 20 commits already in c53e53438d29ce4c55c6df6fd1d646ad2f5047f6.

Run "git log --oneline a944b0d7f322400a54def81b313f16b9e125167b..HEAD" to get a list of commits in the diff.  If you are using a custom git flow, you can override the reference used for this check with "git config gitcl.remotebranch <git-ref>".

Git sanity check failure

$

Any guidance?

I did try various values for gitcl.remotebranch, without success...

$ git config gitcl.remotebranch refs/remotes/origin/master    # nope

$ git config gitcl.remotebranch $(git rev-parse HEAD)    # nope

$ git config gitcl.remotebranch $(git merge-base origin/master HEAD)    # nope

[Dirk Pranke]

Andrii can probably help better than I, since I'm almost no help at all :(.

I think if you happened to create a branch off of one of the commits that we overrode, you needed to create a new branch and cherry-pick your commits across.

For a merge workflow, I honestly have no idea what you'd do since I don't know how git-cl upload figures out what branch to diff against (I'm guessing maybe that's what remotebranch does?).

-- Dirk

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

[Alexei Filippov]

Hi folks,

Something strange still happening. I cannot land my patch https://chromium-review.googlesource.com/c/chromium/src/+/1169533

It fails on presubmit, which says that lgtms are missing for files that are not part of the patch. Excerpt from presubmit stdout:

** Presubmit ERRORS **

Missing LGTM from an OWNER for these files:

    ash/public/cpp/OWNERS

    base/BUILD.gn

    cc/trees/layer_tree_host_pixeltest_blending.cc

    chrome/android/java/src/org/chromium/chrome/browser/media/remote/DefaultMediaRouteController.java

    chrome/android/java/src/org/chromium/chrome/browser/photo_picker/PickerCategoryView.java

....

Does anyone know why's that?

Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Lei Zhang]

It looks like the parent commit of your CL is [2], which does not have
a commit position number. So I'm guessing that's one of the CLs that
got erased from the master branch's history?

[2] https://chromium.googlesource.com/chromium/src/+/27378e03

> https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/242c0d3b-9d04-4a6e-a4a6-9e0eddeab619%40chromium.org.

[Alexei Filippov]

Ah, that could be. I cherry-picked it during the outage. I'll try rebasing it.

[Scott Violet]

Is there an easy way to erase the same commits locally? I have a branch with a lot of commits that touches a lot of files and it's a pain to cherry pick.

  -Scott

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACSHbcSnO3bYq0zAt6ocR8-z8G-RxTRdd3BZWC_62RxNcmKo0g%40mail.gmail.com.

[Christian Biesinger]

You can do:

git rebase --onto origin $LAST_GIT_HASH_BEFORE_YOUR_COMMITS

Christian

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAKARY_mJbLxXYLrGaZzhj%2BsVucL63g2LGsFkshL2Q%2BL6tL3NzQ%40mail.gmail.com.

[Scott Violet]

Christian,

Does that git rebase only work if in my branch I've done rebases vs merges? Would it be possible to instead revert the commits that were erased (in my branch)?

  -Scott

[Christian Biesinger]

Oh... I think that command will only work for a rebase workflow. I'm
not sure of a good fix for a merge workflow.

Christian