Re: GSoC 2019 - Remoting project ideas

66 views
Skip to first unread message

Oleg Nenashev

unread,
Jan 3, 2019, 6:00:52 AM1/3/19
to Pham Vu Tuan, Supun Wanniarachchi, jtho...@cloudbees.com, Devin Nusbaum, jenkinsci-gsoc-all-public, JenkinsCI Developers
I have also added the Jenkins Developers mailing list to Cc so that we can get more inputs and ideas

On Thu, Jan 3, 2019 at 11:20 AM Oleg Nenashev <o.v.ne...@gmail.com> wrote:
Hi all,

Just to follow-up on the Remoting topic, it would be great to have a GSoC project idea for Remoting this year. You were interested in mentoring a project in the area, so it would be great to get your feedback.

One of the ideas I have is to continue improving Remoting over Kafka to improve Remoting behavior in Docker and K8s environments. Examples:
  • Cloud API implementation for Kafka agents
  • Simplify JAR cache warmup while building agent images
  • Update Remoting over Kafka to improve support of Kubernetes there
    • Descriptions
    • Connector logic using credentials from K8s
    • Automatic provisioning of the Kafka cluster from the plugin
    • etc.
  • Create reference K8s specks and Helm charts for Remoting over Kafka system
  • Integrate Remoting messaging with Prometheus monitoring
  • ...
WDYT? If you are interested in such idea, we could create a project idea draft together. I might not be available to be a mentor this year, but I will be happy to be a technical advisor in this project.

Best regards,
Oleg


Jeff Thompson

unread,
Jan 3, 2019, 6:33:27 PM1/3/19
to Oleg Nenashev, Pham Vu Tuan, Supun Wanniarachchi, Devin Nusbaum, jenkinsci-gsoc-all-public, JenkinsCI Developers
These sound like good projects. It would be nice to get a GSoC project going in Remoting. Last year’s project was a very good, productive one.

At this time I don’t know that I can commit to being a primary mentor but I could help out some.

Jeff Thompson

Pham Vu Tuan

unread,
Jan 3, 2019, 7:44:22 PM1/3/19
to Jeff Thompson, Oleg Nenashev, Supun Wanniarachchi, Devin Nusbaum, jenkinsci-gsoc-all-public, JenkinsCI Developers
Hi, I am happy to help as a mentor for GSoC this year. I will try to check the details and get back to you about my ideas asap. I will join the GSoC weekly meeting from next week so that we can discuss about this topic.
(Sorry for not attending the meeting last month, I was a bit busy + holiday period).

Pham Vu Tuan

unread,
Jan 4, 2019, 8:39:00 AM1/4/19
to Jeff Thompson, Oleg Nenashev, Supun Wanniarachchi, Devin Nusbaum, jenkinsci-gsoc-all-public, JenkinsCI Developers
Hi,

After checking through Oleg's suggestions, I am thinking we can move it forward with 2 projects this year:
- Project 1: Update Remoting over Kafka to support K8s - Current demo setup is using Docker compose, and I think K8s is more popular nowadays, maybe we can somehow integrate this. And if this setup is done, maybe auto provisioning of Kafka cluster can be more convinient?

- Project 2: Integrating Remoting messaging with Prometheus monitoring - By this you mean we will do this for JNLP agents monitoring? To replace the traditional monitoring we are having in Jenkins master? I suggest we can rename it to "Integrating remoting with monitoring tools", because there are many monitoring tools out there (not only Prometheus), and we let the student choose one? We can also make the plugin generic such that user can integrate Jenkins with any monitoring system of their choice (but Im not sure it's doable).

I can be the mentor for Project 1 (because I understand about remoting over Kafka), and can help for project 2 with other mentors (I think we need to have someone with deep understanding of remoting for this project).

Supun Wanniarachchi

unread,
Jan 6, 2019, 12:57:11 PM1/6/19
to Pham Vu Tuan, Jeff Thompson, Oleg Nenashev, Devin Nusbaum, jenkinsci-gsoc-all-public, JenkinsCI Developers
Sounds interesting. Last year we had a successful remoting project and it would be nice to have a remoting project this year as well. I also would try to find some spare time to help the project.
--
Thank You & Best Regards,

Supun Wanniarachchi | Software Engineer

Sysco LABS | 55A Dharmapala Mw, Col. 03
( Mobile: +94716326119 | Blog: blog.supun.me)


Github LinkedIn Twitter


Jeff Pearce

unread,
Jan 6, 2019, 2:56:43 PM1/6/19
to Supun Wanniarachchi, Pham Vu Tuan, Jeff Thompson, Oleg Nenashev, Devin Nusbaum, jenkinsci-gsoc-all-public, JenkinsCI Developers
I’m willing to be a co-mentor on either of the two options, depending on how many others are selected 

Sent from my iPhone
--
You received this message because you are subscribed to the Google Groups "jenkinsci-gsoc-all-public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-gsoc-all...@googlegroups.com.
To post to this group, send email to jenkinsci-gs...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-gsoc-all-public/CAGUi3yW%2Bw62m5dA5szSiqjmxvTvhQQNgk7nFE2OKk%2BPQMV3K4w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Oleg Nenashev

unread,
Jan 7, 2019, 6:57:07 PM1/7/19
to Jeff Pearce, Supun Wanniarachchi, Pham Vu Tuan, Jeff Thompson, Devin Nusbaum, jenkinsci-gsoc-all-public, JenkinsCI Developers
Hi all,

I have created the Project Idea Draft for the idea #1 (Improving Remoting over Kafka Plugin): https://docs.google.com/document/d/1CIgjz8IZxihYC6CKqE0pFuKljCKK3SPhKK87IpHs_38/edit?usp=sharing . Feedback will be appreciated (and thanks to all folks willing to be the mentors!). Regarding integration of Remoting with monitoring tools, it would be great if somebody else becomes a champion so that we balance the load a bit. I think that the idea for general monitoring project is useful though I am not sure how much can be already addressed, e.g. by existing Kafka <=> Prometheus integrations (e.g. this post).

As a side note, I have also reached out to the Jenkins X team. Their feedback is that they would rather prefer to see remoting communications over Kubernetes CRD. We could definitely move some logic to CRD (Classloading, transfer of large files), but CRD is not a good choice for heavy RMI operations and log streaming. For log streaming there is a work for external logging in Cloud Native SIG, but it has neever been completed due to various reasons. I am not sure about embarking on this journey until JEP-207/212 are implemented. But we could create such project idea if anybody is willing to mentor it

BR, Oleg

Jesse Glick

unread,
Jan 7, 2019, 9:07:09 PM1/7/19
to Jenkins Dev
On Mon, Jan 7, 2019 at 6:57 PM Oleg Nenashev <o.v.ne...@gmail.com> wrote:
> I have also reached out to the Jenkins X team. Their feedback is that they would rather prefer to see remoting communications over Kubernetes CRD.

That is not actually what was said. JX does not currently use Remoting
in the default buildpacks and I do not think there is any plan to do
so. The reference to CRs was about supporting “agents” in some general
sense.

Matt Sicker

unread,
Jan 8, 2019, 11:02:58 AM1/8/19
to jenkin...@googlegroups.com
If anything, I could see gRPC being a more useful protocol than CRDs here.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3sh_BtG4Cgw%3Ds0ATn0oF7Q1oTsySizyhER5mAPdvShaQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.


--
Matt Sicker
Software Engineer, CloudBees

Oleg Nenashev

unread,
Jan 12, 2019, 4:52:29 PM1/12/19
to Jenkins Developers
Yes, we could do a Remoting over gRPC project.
It would be a good continuation of the JNLP3 protocol contributed by Google. This deprecated protocol actually represents some codebase for what later became gRPC. Maybe some code could be salvaged from there (#ornot).

Also, here is a Jenkins Remoting Monitoring project idea from Vu Tuan: https://docs.google.com/document/d/1nhb-lfdjxT2fAY3US15HIMvePQCCoWU82VV1Y8Mu3lc/edit?usp=sharing

BR, Oleg
Reply all
Reply to author
Forward
0 new messages