[llvm-dev] Upcoming upgrade of LLVM buildbot

[Galina Kistanova via llvm-dev]

Hello everyone,

The buildbot upgrade is entering the phase when the results to become visible.

No change is required at this time on any of the builders. The bot owners could upgrade the buildbot on build computers later, at their convenience, as this is not on the critical path.

We are going to upgrade the staging bot first. Then, once that is stable and all detected issues are resolved, we will upgrade the production bot.

I will need some help with testing, and will be asking to move some of the builders temporarily to the staging. LLVM buildbot is a piece of critical infrastructure, so more eyes and hands in making sure it works properly the better.

I'll be posting updates and ETA of particular changes in this thread.

Please feel free to ask if you have any questions or concerns.

Thanks

Galina

[Galina Kistanova via llvm-dev]

Hello everyone,

Starting tomorrow we will be upgrading the staging and production LLVM build bot.

To make the transition smooth we would not accept any change to zorg from tomorrow 11:00 AM PDT till the production bot is up and running the new version. Please feel free to talk to me if you will have a special situation and if you would absolutely have to make changes. Thanks for your patience and understanding.

If you are a bot owner, you do not need to do anything at this point, unless I'll ask you to help.

Here is the plan.

* Tomorrow, on September 30th, 2020, at 11:00 AM PDT we will upgrade the staging build bot to buildbot v2.8.4.

If you are an owner of one of the staged builders, you would not need to do anything. The staging will go down for a short period of time, and then a new version will come up and will accept connections from your bots.

We will be watching the staging and will be improving zorg for the next week or so. Meaning that staging will be restarted more often than before, and restarts will be done without further notices.

* Once the staging is reliable you may want to move your bot from production to staging to make sure it will work as expected after the upgrade. I will send a note when this could be done.

* After staging is good and we have about a week of running history, we will upgrade the production bot. I will send a separate announcement for this closer to the date.

Once the production is up and running, I will need your feedback about blame e-mails delivery, IRC reporting issues, and anything you could spot wrong with the new bot. I  hope the transition will go smoothly and we will handle issues quickly if any would come up.

* After production is good and we have about a week of running history, I'll ask the bot owners to upgrade buildbots on their side. Please do not upgrade your buildbots unless I'll ask you to. We are trying to limit a number of moving parts at this stage.

Thanks for your support and help. And please feel free to ask if you have questions.

Galina

[Galina Kistanova via llvm-dev]

Hello everyone,

The staging buildbot was up and running for 6 days now, and looks good.

Tomorrow at 12:00 PM PDT we will switch the production buildbot to the new version.

If you are a bot owner, you do not need to do anything at this point, unless I'll ask you to help.

The buildbot will go down for a short period of time, and then a new version will come up and will accept connections from your bots.

Please note that the new version has a bit different URL structure. You will need to update the bookmarks or scripts if you have stored direct URLs to inside the buldbot.

We will be watching the production and staging bots and will be improving zorg for the next week or so.

I will need your feedback about blame e-mails delivery, IRC reporting issues, and anything you could spot wrong with the new bot. I  hope the transition will go smoothly and we will handle issues quickly if any would come up.

After production is good and we have about a week of running history, I'll ask the bot owners to upgrade buildbots on their side. Please do not upgrade your buildbots unless I'll ask you to. We are trying to limit a number of moving parts at this stage. We will start accepting change to zorg at this point. Please feel free to talk to me if you will have a special situation and if you would absolutely have to make changes.

Thanks for your support and help. And please feel free to ask if you have questions.

Galina

On Thu, Sep 10, 2020 at 9:35 PM Galina Kistanova <gkist...@gmail.com> wrote:

[Vitaly Buka via llvm-dev]

It looks like all sanitizer builder are still offline http://lab.llvm.org:8011/#/builders

_______________________________________________
cfe-commits mailing list
cfe-c...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[Galina Kistanova via llvm-dev]

They are online now - http://lab.llvm.org:8011/#/waterfall?tags=sanitizer

AnnotatedCommand has severe design conflict with the new buildbot.

We have changed it to be safe and still do something useful, but it will need more love and care.

Please let me know if you have some spare time to work on porting AnnotatedCommand.

Thanks

Galina

[Andrzej Warzynski via llvm-dev]

Our Flang-aarch64 buildbots just won't connect to the main Buildbot
master anymore. I switched them to the staging buildbot master instead
and it seems fine for now. Is there anything that we can/should tweak at
our end?

http://lab.llvm.org:8014/#/waterfall?tags=flang

-Andrzej

> cfe-c...@lists.llvm.org <mailto:cfe-c...@lists.llvm.org>
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe...@lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

[Galina Kistanova via llvm-dev]

Hello bot owners,

I see a lot of builders went down and not connecting back to the production buildbot.

Could you check your bots to make sure they are up and running, please?

And report connectivity issues if any with quotes from the logs directly to me.

Thanks

Galina

[Paula Askar via llvm-dev]

Hey Andrzej,

What are you seeing in your buildbot logs? Is it this error?
`twisted.spread.flavors.NoSuchMethod: No such method:
remote_getWorkerInfo`

If so, you might want to try updating your buildbot worker.
I updated llvmlibc's to 2.8.4 and that seemed to solve the connection
problem: https://github.com/llvm/llvm-project/commit/f60686f35cc89504f3411f49cf16a651a74be6eb

Best,
Paula Askar

[Galina Kistanova via llvm-dev]

Hi Paula,

This error is fine. The buildbot has tested the worker version. 0.8.x apparently does not have that method.

The error gets handled gracefully on the server side. At least it seems so so far.

That should not prevent your bot from connecting.

Thanks

Galina

[Andrzej Warzynski via llvm-dev]

I switched one of our workers to the main Buildbot and everything seems
fine #fingers-crossed. I guess that that was a temporary glitch?

We haven't updated our local Buildbot installations - still on 0.8.5.
Should we update?

-Andrzej

> > > <mailto:vital...@google.com <mailto:vital...@google.com>>>

> wrote:
> > >
> > >     It looks like all sanitizer builder are still offline
> > > http://lab.llvm.org:8011/#/builders
> > >
> > >     On Tue, 6 Oct 2020 at 00:34, Galina Kistanova via cfe-commits
> > >     <cfe-c...@lists.llvm.org
> <mailto:cfe-c...@lists.llvm.org>

> <mailto:cfe-c...@lists.llvm.org

> <mailto:cfe-c...@lists.llvm.org <mailto:cfe-c...@lists.llvm.org>>

> > > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
> > >
> > >
> > > _______________________________________________
> > > cfe-dev mailing list

> > > cfe...@lists.llvm.org <mailto:cfe...@lists.llvm.org>

> > > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> > >
> > _______________________________________________
> > LLVM Developers mailing list

> > llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

[Galina Kistanova via llvm-dev]

Hello Andrzej,

Please do not update your bots yet. I will explicitly ask later.

Feel free to move other reliably green bots of yours to the production buildbot.

Thanks

Galina

[Vitaly Buka via llvm-dev]

Thanks, I see them.

On Wed, 7 Oct 2020 at 16:32, Galina Kistanova <gkist...@gmail.com> wrote:

They are online now - http://lab.llvm.org:8011/#/waterfall?tags=sanitizer

AnnotatedCommand has severe design conflict with the new buildbot.

We have changed it to be safe and still do something useful, but it will need more love and care.

Please let me know if you have some spare time to work on porting AnnotatedCommand.

It's unlikely in near future.

What is missing exactly?

[Reid Kleckner via llvm-dev]

On Wed, Oct 7, 2020 at 4:32 PM Galina Kistanova via lldb-commits <lldb-c...@lists.llvm.org> wrote:

They are online now - http://lab.llvm.org:8011/#/waterfall?tags=sanitizer

AnnotatedCommand has severe design conflict with the new buildbot.

We have changed it to be safe and still do something useful, but it will need more love and care.

Please let me know if you have some spare time to work on porting AnnotatedCommand.

That's unfortunate, it would've been good to know that earlier. I and another team member have spent a fair amount of time porting things to use more AnnotatedCommand steps, because it gives us the flexibility to test steps locally and make changes to the steps without restarting the buildbot master. IMO that is the Right Way to define steps: a script that you can run locally on a machine that satisfies the OS and dep requirements of the script.

I am restarting the two bots that I am responsible for, and may need some help debugging further issues soon. I'll let you know. 

[Galina Kistanova via llvm-dev]

We have a better version of AnnotatedCommand on the staging. It should be a functional equivalent of the old one.

We need to stress test it well before moving to the production build bot.

For that we need all sanitizer + other bots which use the AnnotatedCommand directly or indirectly moved temporarily to the staging.

Please let me know when that could be arranged.

Thanks

Galina

[Vitaly Buka via llvm-dev]

Switched all but PPC, I don't have access to them. But they run the same script as sanitizer-x86_64-linux.

http://lab.llvm.org:8014/#/waterfall?tags=sanitizer

[Vitaly Buka via llvm-dev]

Looks like staging AnnotatedCommand fixed step statuses, so we can see which one is green.

Please let me know when to switch bots back from the staging.

Thank you!

[Galina Kistanova via llvm-dev]

Thanks, Vitaly!

Let's have them there for at least 24 hours, shall we?

Could you move sanitizer-buildbot1, sanitizer-buildbot3, sanitizer-buildbot7 as well, please?

AnnotatedCommand on the staging has been tested functionally and is good. My only concern at this point is how it would handle a heavy load, so the more bots we will have on the staging the better.

If somebody else could move their AnnotatedCommand bots to the staging area, that would be much appreciated.

Thanks

Galina

[Vitaly Buka via llvm-dev]

On Mon, 12 Oct 2020 at 22:01, Galina Kistanova <gkist...@gmail.com> wrote:

Thanks, Vitaly!

Let's have them there for at least 24 hours, shall we?

We can do that.

 

Could you move sanitizer-buildbot1, sanitizer-buildbot3, sanitizer-buildbot7 as well, please?

Done. 

[Reid Kleckner via llvm-dev]

FWIW, I don't see any issues with my two bots that use buildbot annotated commands:

http://lab.llvm.org:8011/#/builders/sanitizer-windows

http://lab.llvm.org:8011/#/builders/clang-x64-windows-msvc

The individual steps don't highlight as green or red, but that's OK for now.

[Siva Chandra via llvm-dev]

On Mon, Oct 12, 2020 at 10:01 PM Galina Kistanova via lldb-commits
<lldb-c...@lists.llvm.org> wrote:
> If somebody else could move their AnnotatedCommand bots to the staging area, that would be much appreciated.

I moved the libc bots to staging to now.

Thanks,
Siva Chandra

_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org

https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

[Vitaly Buka via llvm-dev]

They do on staging. 

http://lab.llvm.org:8014/#/builders/sanitizer-windows
http://lab.llvm.org:8014/#/builders/clang-x64-windows-msvc

Galina, when do you plan to push this to the primary server?

If it's a few days, I'd rather keep bots on staging to have colors.

[Vitaly Buka via llvm-dev]

I switched sanitizers and two rnk@ bots back to the primary server.

http://lab.llvm.org:8011/#/waterfall?tags=sanitizer

[Galina Kistanova via llvm-dev]

Thanks everyone!

I would like to keep the bots in the staging till Friday. And if everything is good, then apply the changes to the production and let you move all the green bots back there.

Thanks

Galina

[Vitaly Buka via llvm-dev]

Ok, I can switch them back to staging. However today's staging was frequently offline.

[Galina Kistanova via llvm-dev]

Thanks everyone for keeping your annotated builders in the staging area! Much appreciated.

Please feel free to move all the green builders back to the production. It has a new AnnotatedCommand now.

Thanks

Galina

[Vitaly Buka via llvm-dev]

Thanks!

Done.

[Vitaly Buka via llvm-dev]

There is some issue, at least on staging.

http://lab.llvm.org:8014/#/builders/89/builds/184 has multiple failed stages but marked as SUCCESS

[Galina Kistanova via llvm-dev]

Hello everyone,

We have reached the point when all the bots could be updated to a newer buildbot-worker software.

We should be fine to go with whatever the version is current for the OS on your bots.

If you are a bot owner, please feel free to move your bots to the staging, upgrade the buildbot worker, check to make sure it is still reliably green, and move it back to the production build bot. Please talk to me if for any reason you want to stay with the version you currently have installed.

Feel free to ping me if you will get issues or will need help.

In the meantime, we will lock the llvm-zorg repository for changes for a few days. I'm going to clean up the patches for the updated zorg and will commit them. Once that is done we are back to accepting the changes.

Please let me know if you have any questions.

Thanks

Galina

On Tue, Sep 29, 2020 at 11:24 PM Galina Kistanova <gkist...@gmail.com> wrote:

Hello everyone,

Starting tomorrow we will be upgrading the staging and production LLVM build bot.

To make the transition smooth we would not accept any change to zorg from tomorrow 11:00 AM PDT till the production bot is up and running the new version. Please feel free to talk to me if you will have a special situation and if you would absolutely have to make changes. Thanks for your patience and understanding.

If you are a bot owner, you do not need to do anything at this point, unless I'll ask you to help.

Here is the plan.

* Tomorrow, on September 30th, 2020, at 11:00 AM PDT we will upgrade the staging build bot to buildbot v2.8.4.

If you are an owner of one of the staged builders, you would not need to do anything. The staging will go down for a short period of time, and then a new version will come up and will accept connections from your bots.

We will be watching the staging and will be improving zorg for the next week or so. Meaning that staging will be restarted more often than before, and restarts will be done without further notices.

* Once the staging is reliable you may want to move your bot from production to staging to make sure it will work as expected after the upgrade. I will send a note when this could be done.

* After staging is good and we have about a week of running history, we will upgrade the production bot. I will send a separate announcement for this closer to the date.

Once the production is up and running, I will need your feedback about blame e-mails delivery, IRC reporting issues, and anything you could spot wrong with the new bot. I  hope the transition will go smoothly and we will handle issues quickly if any would come up.

* After production is good and we have about a week of running history, I'll ask the bot owners to upgrade buildbots on their side. Please do not upgrade your buildbots unless I'll ask you to. We are trying to limit a number of moving parts at this stage.

Thanks for your support and help. And please feel free to ask if you have questions.

Galina

[Vitaly Buka via llvm-dev]

it's not just staging, this build must be marked as failed http://lab.llvm.org:8011/#/builders/74/builds/122

[Galina Kistanova via llvm-dev]

Thanks, Vitaly.

Somebody is looking at this.

[Florian Hahn via llvm-dev]

On Oct 20, 2020, at 03:35, Galina Kistanova via llvm-dev <llvm...@lists.llvm.org> wrote:

Hello everyone,

We have reached the point when all the bots could be updated to a newer buildbot-worker software.

We should be fine to go with whatever the version is current for the OS on your bots.

If you are a bot owner, please feel free to move your bots to the staging, upgrade the buildbot worker, check to make sure it is still reliably green, and move it back to the production build bot. Please talk to me if for any reason you want to stay with the version you currently have installed.

Feel free to ping me if you will get issues or will need help.

In the meantime, we will lock the llvm-zorg repository for changes for a few days. I'm going to clean up the patches for the updated zorg and will commit them. Once that is done we are back to accepting the changes.

Please let me know if you have any questions.

Thanks for all the work making the buildbot upgrade possible!

One thing I noticed is that since the upgrade, it appears like the IRC llvm-build bot stopped mentioning people on the blamelist for broken builds. Unfortunately I don’t know much about the bot, is there anything that needs adjusting to bring the blamelist functionality back?

Cheers,

Florian

[Galina Kistanova via llvm-dev]

Thanks, Florian! I will look into this.

Cheers

Galina

[Vitaly Buka via llvm-dev]

Any news regarding false-negatives?

+Mitch Phillips 

[Vitaly Buka via llvm-dev]

It looks fixed. I haven't seen them in recent reports.

[Galina Kistanova via llvm-dev]

Hello Vitaly,

This should be fixed.

Please let me know if you still see the issue.

Thanks

Galina

[Galina Kistanova via llvm-dev]

Hello everyone,

The LLVM buildbot is upgraded.

Thanks for your patience and support!

llvm-zorg source code has been updated, and now is open for new changes. If you have pending patches, please rebase them on top of the llvm-zorg master and update your reviews on the Phabricator. I'll start reviewing them shortly.

If you are a bot owner and your bot is still using buildbot 0.8.x version, please move your bots to the staging, upgrade the buildbot worker software, check to make sure it is still reliably green, and move it back to the production build bot. Please talk to me if for any reason you want to stay with the version you currently have. Feel free to ping me if you will get issues or will need help.

Please let me know if you have any questions or see issues.

Thanks

Galina

[Galina Kistanova via llvm-dev]

Hello Florian,

Thanks for pointing to the problem. Appreciate this.

It has been fixed and now failed builds reported with the blamelist.

Thanks

Galina

On Wed, Oct 21, 2020 at 5:30 AM Florian Hahn <floria...@apple.com> wrote:

[John Paul Adrian Glaubitz via llvm-dev]

Hello Galina!

On 10/28/20 2:53 AM, Galina Kistanova via llvm-dev wrote:
> llvm-zorg source code has been updated, and now is open for new changes. If
> you have pending patches, please rebase them on top of the llvm-zorg master
> and update your reviews on the Phabricator. I'll start reviewing them
> shortly.

Done [1]. The SPARC bot is patiently now for being added as a new worker ;-).

> [1] https://reviews.llvm.org/D90148

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - glau...@debian.org
`. `' Freie Universitaet Berlin - glau...@physik.fu-berlin.de
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913