Hi all,
You've probably noticed that there's been a huge flood of emails from
luci-notify lately with reports of build failures in the
"publish_tarball" and "Build From Tarball" bots. Thankfully, they're
finally over now.
This ended up happening for a few separate, somewhat independent reasons:
* The notification settings were changed recently
(
https://crrev.com/c/4226079) and this list was set to receive emails
when the "Build From Tarball" bot failed.
* The "Build From Tarball" bot was not totally green back then though. I
landed a few changes to fix some of the failures
(
https://crrev.com/c/4231332,
https://crrev.com/c/4239837,
https://crrev.com/c/4236459), but it took a while for some of the
changes to be rolled in due to some other, unrelated issues on the
Chromium infra side.
* On the "publish_tarball" side,
https://crrev.com/c/4228579 landed to
fix
https://crbug.com/1413633, but it ended up uncovering a lot of older
releases (since M74) that had not been processed in the past due to
limitations of the old API the bot was using. This caused several
hundreds of new builds to be queued, including builds for very old
releases which the "publish_tarball" infrastructure no longer supports
and which caused the bot to fail. I kind of fixed this in
https://crrev.com/c/4235127 but by then a lot of ancient builds had
already been queued. thomasanderson@ also landed
https://crrev.com/c/4237587 to process only Linux releases rather than
every non-iOS release.
After about a week and all the changes above, both bots seem to have
processed all queued builds and everything looks calm and green again,
and I can finally change my membership settings to start receiving
emails from this list again :-)