Elastic Sponsorship for OpenTelemetry usage for community Jenkins instances

[Damien Duportal]

[Cross posted on jenkins-dev and jenkins-infra]

Hello dear community,

During the last Jenkins infrastructure weekly meeting (2021-05-25 - Notes -with recording link: https://hackmd.io/sBywc0IfRraU93aNETVwkw), Victor Martinez and Iván Fernández from Elastic did a demonstration of the opentelemetry plugin used with an Elastic APM platform.
The goal was to demonstrate how the build metrics could be used with distributed tracing: please check the notes, video and https://plugins.jenkins.io/opentelemetry/ for more details.

Victor and Ivan's employer, Elastic Inc. is proposing to sponsor the Jenkins project by providing an APM instance that could be used along with opentelemetry plugin, and a bit support to get it configured and maintained.
Their interest is to study what are our (Jenkins community) usages (and obviously mentioning that we are using it).

The proposal from the Infrastructure team is to provide this service for the 2 following use cases:

• For users of ci.jenkins.io (core and plugins contributors) to get build metrics and traces of their projects. It could help a lot for performances tracking, and much more as "Jenkins Pipeline level metrics"
• For infrastructure maintainers of ci.jenkins.io to track the agents allocations and their errors, to improve the QoS of the build farm, as the pure infrastructure metrics is not enough because issues come from different areas (infra, Cloud providers, Jenkins configuration, Pipeline usages, etc.)

It could be a great opportunity for whoever interested in better understanding Jenkins behavior. As we'll try to identify typical use cases that we want to better understand and then build dashboards using metric.
The challenge we face on the Jenkins infrastructure is that it's pretty easy to know if Jenkins is running. But not so much if it's working in an efficient way.

The plan would be the following:

• Start by a PoC on a private Jenkins instance  (e.g. infra.ci.jenkins.io)
• Install the plugin
• Check APM instance access with Elastic
• Configure the plugin to point to the APM
• Bring back the conclusion to the community (by email here, blog post, meetup, whatever)
• Once the configuration is straightforward, deploying to ci.jenkins.io, targeting as much publicly available dashboards and data as possible, as service for the ci.jenkins.io users.

What do you think about this topic, what would be your thoughts or red flags?

Cheers,

For the Infra. team,
Damien DUPORTAL

PS: by default, without any blocker or red flags, we (the infra.team) will proceed with the infra.ci setup described in this email the 1st of June.

[Tim Jacomb]

Sounds great

--
You received this message because you are subscribed to the Google Groups "Jenkins Infrastructure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-infr...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/0A4790E7-DF29-4C36-AAF3-320763D230B9%40gmail.com.

[Oleg Nenashev]

Connecting ci.jenkins.io looks perfectly fine to me. There is no sensitive data on this instance. For other Jenkins instances (infra, release, trusted-ci, cert, ...) a sign-off from the Jenkins Security Officer is required IMHO

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/CAH-3Bie3U4UOxmSLpex8sSw48FQeVGwJjXP4KLQTPWBAon%3DS1Q%40mail.gmail.com.

[Cyrille Le Clerc]

Thank you Damien.

Elastic is very excited to collaborate with the Jenkins community on better observability for Jenkins and for CI/CD at large.

Please note that these observability are vendor neutral, there is no vendor lock-in and what we will accomplish with the Elastic Observability backend will be achievable with other observability solution that also integrate with OpenTelemetry such as Jaeger and Prometheus. 

Victor, Ivan and I will do our best to offer an unprecedented User Experience with Elastic Observability there will be no lock in, it will alway be possible to switch to other observability backends :-) 

Cyrille

[Oleg Nenashev]

Hi all,

I have added this topic to the next Governance Meeting agenda: https://docs.google.com/document/d/11Nr8QpqYgBiZjORplL_3Zkwys2qK1vEvK-NYyYa4rzg/edit#heading=h.1gtco63t6ztr

Casting explicit +1 for proceeding with sponsorship and evaluation

Best regards,

Oleg Nenashev

[Mark Waite]

+1 from me for sponsorship and evaluation

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/b119aa43-d46b-4dce-b819-21f0d6965cdan%40googlegroups.com.

[Damien Duportal]

Good point Oleg, thanks a lot for this reminder!

Any test on infra.ci.jenkins.io must be done on a private APM instance, to ensure no sensitive data is leaked. It means that we should only do it for testing purpose, and the feedback to the community will need a special attention on the data presented.

Damien

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/CAPfivLBCAAJGP2p3oAudZztVQ2yT8sp5Zi9m%3DmMHC3REr%3DYNcA%40mail.gmail.com.

[Ullrich Hafner]

Can you elaborate what information you want to gather? What build metrics and traces? 

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/94FF0A8A-5624-414A-A650-BA068641245A%40gmail.com.

[Cyrille Le Clerc]

Hello Ulrich,

We documented all the data we collect on h the README.md page of the plugin. 

We capture :

* the details collected on pipeline execution as span attributes (see list here)

* metrics on job executions, build queue, agent lifecycle, security events (see list here) 

Note that we don't capture logs for the moment but we are interested in covering this in the future.

Cyrille

[Oleg Nenashev]

Hi all,

Just a quick summary from the yesterday's governance meeting:

• We approve Elastic sponsorship and further collaboration on OpenTelemetry for ci.jenkins.io. The Jenkins infra team is clear to proceed
  
• FTR we do not think the governance meeting approval is needed in this case. Jenkins Infra officer can represent the Jenkins community when it comes to infra matters.
• Taking the security and data privacy considerations, we recommend to not connect other Jenkins instances in our infra until there is an approval from the Jenkins Security Officer
  

And thanks again to all contributors!

Best regards,

Oleg Nenashev