[JENKINS-68652] Migrate from ANTLR2 to ANTLR4

[Basil Crow]

Jenkins has two ANTLR grammars: one to parse (a variation of) crontab
entries and one to parse label expressions for build agents.

As described in JENKINS-68652, compiling Jenkins on Java 18 or newer
currently fails in ANTLR Maven Plugin. Our versions of ANTLR and ANTLR
Maven Plugin, dating back 15 years, use terminally deprecated Java
features. The repository for the latter does not appear to have
survived the MojoHaus migration from Subversion to Git. The status quo
does not seem sustainable.

Given the lack of volunteer interest in JENKINS-68652 or alternatives
(e.g., writing a new parser without ANTLR), I have reached out to a
reputable ANTLR consulting firm with experience performing migrations
from ANTLR2 to ANTLR4. They are interested in the project and have
scoped the work required to migrate from ANTLR2 to ANTLR4, including
updating the grammars and the classes operating directly with the
parse-tree.

I am starting this thread to reach consensus on the technical side of
this proposal; namely, migrating from ANTLR2 to ANTLR4 to decrease
technical debt and facilitate compilation on Java 18 or newer. When we
have reached consensus that we would like to perform this migration, I
will then propose an agenda item for the next Jenkins governance
meeting. In that meeting, I plan to introduce the consulting firm,
walk through their proposal and fee, and request funding approval from
the governance board to begin the project.

If approved, I plan to be the primary point of contact with the
consulting firm as they prepare the pull request to migrate from
ANTLR2 to ANTLR4. The code is covered by automated testing, and I plan
to complete additional manual smoke testing beyond that. I also plan
to review and approve the pull request. If there are technical
questions regarding the scope of work or implementation plan, please
let me know and I will pass them along.

[Mark Waite]

On Thursday, September 22, 2022 at 12:54:56 PM UTC-6 Basil Crow wrote:

I am starting this thread to reach consensus on the technical side of
this proposal; namely, migrating from ANTLR2 to ANTLR4 to decrease
technical debt and facilitate compilation on Java 18 or newer. When we
have reached consensus that we would like to perform this migration, I
will then propose an agenda item for the next Jenkins governance
meeting. In that meeting, I plan to introduce the consulting firm,
walk through their proposal and fee, and request funding approval from
the governance board to begin the project.

If approved, I plan to be the primary point of contact with the
consulting firm as they prepare the pull request to migrate from
ANTLR2 to ANTLR4. The code is covered by automated testing, and I plan
to complete additional manual smoke testing beyond that. I also plan
to review and approve the pull request. If there are technical
questions regarding the scope of work or implementation plan, please
let me know and I will pass them along.

I think that is a good approach.  Thanks for proposing it!

Mark Waite 

[Basil Crow]

On Thu, Sep 22, 2022 at 11:54 AM Basil Crow <m...@basilcrow.com> wrote:
>
> I am starting this thread to reach consensus on the technical side of
> this proposal; namely, migrating from ANTLR2 to ANTLR4 to decrease
> technical debt and facilitate compilation on Java 18 or newer.

To summarize our progress in this area, we have reached technical
consensus on the developer list and budgetary consensus with the
governance board, which voted in favor of this engagement at its
meeting on October 3, 2022. Oleg Nenashev has informed me that he is
unwilling to move forward until the process documented in JEP-15 is
followed; namely:

> Budget requests should be submitted to the developer mailing list and discussed there[;] the Governance meeting will be doing a formal approval/rejection based on the community feedback.

Whereas technical consensus has been reached on the developer list,
and whereas budgetary consensus has been reached with the governance
board; now, therefore, be it known that the preceding message
constitutes a formal budget request, and that this formal budget
request has been made to the developer mailing list pursuant to JEP-15
(at revision 023ed9f0) § 3.3.2 ("Use of donations"), line 175.

[Oleg Nenashev]

Hi Basil et al,

Thanks for the follow-up. As we discussed in the DMs, with all due respect to you and to your valuable contributions I do not appreciate the passive aggressive tone you chose here or in the original DM conversation yesterday. Scoffing about the adopted community process for budget approvals does not help either. I do not such an approach helps you to get this particular issue resolved faster, as well as it does not help in the pull requests and other conversations I see here and there. Everyone in the community tries to achieve the best for the project, and please use it as a default assumption for communications. It would make the community a more pleasant place for everyone. Thanks for understanding.

Back to the matter, indeed the budgeting process in JEP-15 follows the general Jenkins community decision making process where decisions happen in async channels, not during the live meetings. As for governance and budgeting matters, this mailing list is used. Nothing has changed in these regards since ages, so there is no reason to be surprised a 1,900 EUR budget request, which is 20% of the total Jenkins funds available on LFX Mentorship, would not be approved without a formal email thread in this mailing list. Thanks for understanding again.

The request itself

Now I would like to share the request Basil, copying the meeting notes from the October 03 meeting. So the feedback will be needed: 

• Given the lack of volunteer interest in JENKINS-68652 or alternatives (e.g., writing a new parser without ANTLR), Basil Crow has reached out to Federico Tomassetti at Strumenta, a consulting firm with experience performing migrations from ANTLR2 to ANTLR4
  • Strumenta is interested in the project and has scoped the work required to migrate from ANTLR2 to ANTLR4
  • Technical consensus has been reached on the developer mailing list (in this thread)
  • Proposal from Strumenta
• Basil is requesting funding of 1,900 Euro, possibly with additional Value-Added Tax (VAT), and approval from the governance board to engage Strumenta to begin the project
  
  • If approved, Basil plans to be the primary point of contact with the Strumenta as they prepare the pull request to migrate from ANTLR2 to ANTLR4
  • The code is covered by automated testing, and Basil plans to complete additional manual smoke testing beyond that
  • Basil also plans to review and approve the pull request
  • As thanks for their discount, we will publish a blog post with technical content describing their work and our results of their work. Basil will coordinate the blog post with Strumenta

Some extra context:

• At the governing board meeting, the following contributors voted in favor of the proposal: Mark Waite, Gavin, me, Basil himself, and Kevin. Note that 3 of these voters are directly involved with the matter and work for the same vendor that might have a commercial interest in the fix. It does not make difference in the consensus building process, especially since this change is definitely beneficial for the project.
• After the meeting an anonymous contributor has already shared their feedback that the requested sum is too much for the scope of work in this project.

Personally I am rather in favor of the request. We have money from Jenkins supporters floating around, and we are not using them. Whether the use of the budget is super efficient or not, as Basil said we are not moving forward with the ticket. And it may impact vendors and end users requiring clear security scans to have Jenkins adopted. So I keep my +1 vote for the request.

Please share your feedback!

Best regards,

Oleg Nenashev

[Basil Crow]

Dear Oleg,

I would like to take this opportunity to publicly reiterate the
sincere appreciation I expressed to you in DM yesterday for your
helpful (because you helped me discover a JEP that was previously
unknown to me) and timely (because you replied to my DMs right away)
assistance. While you may have interpreted my questions about the
process and my resulting gesture of appreciation in a negative light,
I would like to assure you and everyone else reading this thread that
they were both genuine. My previous message to this list has adhered
to the process established in JEP-15, chapter and verse, out of my
sincere and utmost respect for the process itself and for your
stewardship of it. I stand by what I wrote to you in DM yesterday and
what I wrote in my previous message to this list, both of which were
in good faith. It is important to remember that not everyone has been
involved in the project for as long as you have been, and many
newcomers like myself are either unaware of these older JEPs or
unfamiliar with the context behind them. A lot of project
documentation (e.g., the documentation for CLAs) is incomplete or
out-of-date, so it can sometimes be difficult to determine which
documentation is still relevant and which documentation is no longer
relevant. As people like me learn to navigate these established
processes, your patience is appreciated. Thanks for understanding.

One additional point of clarification I would like to make is that I
confirmed with Strumenta that the Linux Foundation meets Strumenta's
criteria for being a registered business, so VAT does not apply.
Regarding the price of the engagement, I think that comes with the
territory of working with legacy technology: there are relatively few
people with ANTLR2 knowledge/experience who are also willing to work
on legacy ANTLR2 code, so they can command a higher market rate. (I
suspect something similar might be the case for COBOL programmers!) I
can also disclose that I came up with this idea on my own without any
prompting (either explicit or implicit) from my employer. Like you, I
think it is in the long-term interest of the project for us to move
forward with this upgrade and eliminate the liability of one of our
most ancient dependencies. Proceeding with this engagement will pay
down technical debt and eliminate an impediment to future Java
Platform improvements. If there are any additional questions or
concerns, please let me know!

Thanks,
Basil

[Oleg Nenashev]

Thanks for the clarification about Strumenta's status!

Just to make sure, does the approval on Oct 31st suit your timeline? We could theoretically have an out of order meeting on 24th (I can host it), or make a change in the voting process to allow for async budget decision on the mailing list. The latter would be beneficial for the community anyway.

Best regards,

Oleg

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/bGmc9bc0mp4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjp2A17egKp2VxSqT%2BEOQx-42dzt3j-Sk2Hf2_RYFCBwbg%40mail.gmail.com.

[Basil Crow]

On Wed, Oct 19, 2022 at 3:24 AM Oleg Nenashev <o.v.ne...@gmail.com> wrote:
>
> Just to make sure, does the approval on Oct 31st suit your timeline?

Yes

[Alex Earl]

I would like to take a look at this, I don't think the grammars are too hard to convert. I will work on it this week and see how far I get and report back. Sorry for the late jump in.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjqVv8ALyTu_CYzVxUXJJ3oBoiWscZ2EXGfZXQKWh4PDpQ%40mail.gmail.com.

--

Website: http://earl-of-code.com

[Alex Earl]

Please see https://github.com/jenkinsci/jenkins/pull/7293. Looking for some good feedback.

--

Website: http://earl-of-code.com