ONOS Jenkins CI not working

[Jian Li]

Hi, 

This is Jian from SK Telecom.

We recently noticed that the ONOS jenkins CI was working properly, so that all ONOS check-ins are not reviewed properly by CI bot, which in turn prevents all commits from being merged.

It would be great that someone can take a look at this issue.

http://jenkins.onosproject.org/ 

Many thanks!

// Jian

[Alessio Giorgetti]

added Joey in the loop...

alessio

------ Messaggio originale ------

Da: "Jian Li" <pyg...@gmail.com>

A: onos-d...@opennetworking.org

Inviato: 23/05/2024 11:06:58

Oggetto: ONOS Jenkins CI not working

--
You received this message because you are subscribed to the Google Groups "ONOS Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onos-discuss...@opennetworking.org.
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/onos-discuss/CAN2QPWgRs8SnDjKzbXaHL8yek4y6p01gyYYXSZD6sN2MZ0%2BEwA%40mail.gmail.com.
For more options, visit https://groups.google.com/a/opennetworking.org/d/optout.

[Jian Li]

Any update on this?

Thanks,

// Jian

2024년 5월 23일 (목) 오후 6:29, Alessio Giorgetti <alessio....@gmail.com>님이 작성:

[Joey Armstrong]

Hi Alessio & Jian,

I can give some background on the problem but we do not have a resolution available as yet.

There is an ongoing effort to reduce monthly budgetary costs for all of the projects.  During recent AWS cleanup effort someone inadvertently deleted a few AMI images thinking they were unused. Unfortunately later on it was found the AWS AMIs were a dependency and in this case some were a base image for spinning up jenkins nodes to run builds and tests on.  ONOS was not the only project affected, Broadband/VOLTHA has been as well, all jenkins jobs are currently failing due to a lack of nodes to run them on.

At present I am working on trying to reconstruct the Broadband jenkins nodes from a clean distribution decorated with appropriately versioned packages.

Not sure how close the ONOS jenkins nodes will be but a similar effort will likely be needed to get the ONOS jenkins server back online.

Let me know if there are any questions

Joey Armstrong

The Linux Foundation

jarmstrong@linuxfoundation

New York (EST)

On Thu, May 30, 2024 at 3:09 AM Alessio Giorgetti <alessio....@gmail.com> wrote:

Dear Joey,

I guess you are the one that could help on this point... 

Please let us know.

alessio

------ Messaggio originale ------

Da: "Jian Li" <pyg...@gmail.com>

A: "Alessio Giorgetti" <alessio....@gmail.com>

Cc: onos-d...@opennetworking.org; "jarms...@linuxfoundation.org" <jarms...@linuxfoundation.org>

Inviato: 30/05/2024 05:50:05

Oggetto: Re: ONOS Jenkins CI not working

[Joey Armstrong]

Hi,

Looping in David to see if he has cycles to work on the problem.

Joey Armstrong

The Linux Foundation

jarmstrong@linuxfoundation

New York (EST)

[David Ferguson]

Hi, All,

I have been working on this a bit and want to give you an update on progress.   There were actually two issues:

1.  The AMIs (system images) that were used for your build jobs got deleted by my error.

2.  The load balancer in front of the Jenkins server was recommended for updating and was intermittently failing to communicate with the server quickly enough.

With all jobs failing due to #1, I chose to bring up the server on a different server, and eliminate the load balancer as traffic had been insufficiently high enough to justify the overhead/cost of the LB.   That process was started Sunday and the server should be available and running (but not doing anything yet) this evening.   My apologies, something else took precedence Tues and much of Wed that delayed this portion.

Resolving the missing AMI's is of greater complexity.    The jobs within the ONOS project were running on three different OS versions (16.04, 18.04, and 20.04) and I do not have access to the exact configuration of the machines.  A simple rebuild of what was done 3 years ago of a build image was insufficient when attempted.     To guide my efforts in prioritizing which tests/build machines to focus on, is it possible to provide a prioritized list of which Jenkins jobs are of highest priority to your efforts?

An associated question is how much effort should be put into restoring what WAS available, as opposed to getting what is needed running?   I'll admit that trying to get 16.04 running seems semi-pointless given the maturity of that release and it's imminent demise, but if the installed base of users are still primarily there, it will be necessary.    Similar issues for 18.04 exist.

Fundamentally, I'm hoping the ONOS community can provide some guidance concerning whether priority should be getting the existing jobs running on the previously running OS's or whether effort should be concentrated on aiming at jobs running on the more recent OS's and migrating older ones up.

I've been unaware of much interest or community use of the ONOS Jenkins server, and apologize for not correcting/maintaining some of these issues earlier, but do want to take advantage of this time to ensure that the work done in this area does match the community's needs.   Please provide whatever priority guidance is available on which jobs should be prioritized.

-daf

[Alessio Giorgetti]

Dear David, Joey,

thank you very much for the update. 

I guess having a fully operational gerrit is fundamental for the ONOS community since it represents the only possibility to progress in the development.

Since the community is not large, I guess that no relevant resources are needed to deply a reliable and operational system.

As best of my understanding the gerrit is still not able to perform builds... 

Please keep us updated.

alessio 

------ Messaggio originale ------

Da: "David Ferguson" <dfer...@linuxfoundation.org>

A: "Joey Armstrong" <jarms...@linuxfoundation.org>

Cc: "Alessio Giorgetti" <alessio....@gmail.com>; "Jian Li" <pyg...@gmail.com>; onos-d...@opennetworking.org; "Andrew Grimberg" <agri...@linuxfoundation.org>

Inviato: 30/05/2024 22:54:45

Oggetto: Re: Re[2]: ONOS Jenkins CI not working

[Jian Li]

Hi David,

Thanks for sharing the updates.

Currently, the majority of Jenkins CI is built against the onos-2.7 and master branches. The other branches are almost obsolete. I am not quite sure which VM base image (either Ubuntu 18.04 or 20.04) each branch is using.

It would be very helpful if all check-ins for the onos-2.7 and master branches could trigger the CI build, ensuring that the latest changes can be merged promptly.

Thanks,

// Jian 

2024년 6월 10일 (월) 오후 11:11, Alessio Giorgetti <alessio....@gmail.com>님이 작성:

[David Ferguson]

Hi  all,

Apologies for this late response.   I'll be focusing on getting the checkin builds completing as desired today.

-daf