Windows-arm64 native support

[Pierrick Bouvier]

Dear Jenkins developers, as part of Linaro Windows on Arm team [1], we are helping various project to support natively windows-arm64 platform. Jenkins has been of the project requested by our customers.

After an investigation, I identified two native libraries (dll) contained inside jenkins .war file:

- jansi (used by groovy shell)

- winp

It seems like the absence of native JANSI library is not a problem, but for winp, I'm not sure what we are missing by not being able to load this DLL (less control on processes launched?).

After looking more closely at winp repository, it seems like arm support is not a priority [2]. This opened PR [3] didn't seem to get a lot of traction.

So, if there is someone here familiar with winp component of Jenkins, what could we do to help support windows-arm64?

Thanks, Pierrick

[1] https://www.linaro.org/windows-on-arm/

[2] https://github.com/jenkinsci/winp#platform-support

[3] https://github.com/jenkinsci/winp/pull/77

[Verachten Bruno]

Hello Pierrick,

Nice to hear from you. I'm an Arm ambassador and a long-time Linaro
fan, so your email particularly piqued my interest.

Unfortunately, I'm not familiar with the wimp library. However,
perhaps some of us in this community are.

I don't own a Windows arm64 machine myself, but I believe it may not
be necessary to get things moving with the wimp library.

Let's hope someone more knowledgeable about this library will chime in.

Best regards,

Bruno

[Pierrick Bouvier]

Hello Bruno,

thanks for your answer!

wimp seems to be used to kill/list process trees by Jenkins, so I was wondering what happens if it's not available.

It does not seem a big effort to support this platform for wimp component, so I hope we can get one of their developer to answer here :)

Regards,

Pierrick

[Basil Crow]

I am not aware of anyone who is actively maintaining the winp
component. You could take ownership of the winp component, set up CI
builds and tests, and then do a release with Windows ARM64 support.

[Pierrick Bouvier]

Thanks for your help, I'll take a look at this.
Pierrick

[Basil Crow]

On Thu, Mar 28, 2024 at 11:16 PM Pierrick Bouvier
<pierrick...@linaro.org> wrote:
> Thanks for your help, I'll take a look at this.

And thanks for your interest! Unfortunately I think
https://github.com/jenkinsci/winp/pull/112 is premature, as the
repository is not currently in a state where _any_ outside
contributions to the native C++ code can be accepted, as the Jenkins
CI build uses precompiled binaries rather than building the C++ code
from scratch.

In order for us to be able to accept contributions to the native code
like yours, we first need to set up a proper build environment with no
pre-compiled binaries, a CI build (ideally Jenkins CI) that compiles
both the Java code and the native code and runs the tests, and
(ideally) a CD process on GitHub Actions that compiles both the Java
code and the native code and performs the release. Our existing CI/CD
infrastructure is heavily Maven-centric, so it would be preferred to
use Maven as the entrypoint into the process and then to vector into
the native code compilation from within Maven.

My strong preference is for the above work to be done as a
prerequisite to adding ARM support. The status quo is already
untenable, even without taking into consideration the changes needed
to add ARM support. I am a -1 on using GitHub Actions for CI, and I am
a -1 on adding ARM support using the existing status quo of prebuilt
binaries before solving the existing technical debt of the incomplete
CI/CD process. The reason for this is that I believe there will not be
a strong incentive to work on the long-term CI/CD process once the ARM
support is added. Therefore I am against adding any new features,
including ARM support, before the existing technical debt is
addressed.

Since commit access is required to test Jenkinsfile changes on
ci.jenkins.io, I would propose to first add you as a winp committer
alongside the Jenkins core team (but not a committer on any other
repositories). This would allow you to test Jenkinsfile changes on
ci.jenkins.io.

The next steps after that, in this proposal, would be for you to
develop two PRs, the first PR updating the Jenkinsfile to compile the
native code as part of the Maven build rather than using pre-built
binaries, and the second PR implementing CD with GitHub Actions. The
first of these will require collaboration with the Jenkins
infrastructure team to provide an appropriate stateless build
environment. Once this is working, your commit access can be removed,
and the core maintainers can click the CD button to perform a release
from the sources without any prebuilt binaries. Since this release
will have been compiled entirely on Jenkins infrastructure with no
prebuilt binaries, it can be trusted and adopted by Jenkins core. At
this point, the technical debt would be solved and the repository
would become open to new features again.

Unfortunately there is no existing maintainer to complete this
prerequisite work. But after this is all said and done, you can
proceed with your original PR to add ARM support, which should be easy
for any core maintainer to approve and release (with CD) once the
repository has no more prebuilt binaries and a working CI/CD process.

[Gavin Mogan]

As much as I find the response verbose and a little aggressive, I'm going to have to agree, especially after the big XZ backdoor found in linux this week. No binary in source control, libs need to be built.

As much as I agree, that a CI project shouldn't use a competitors CI project, I think asking a new contributor to find someone who can help them build a build pipeline on a server they don't have access to is pretty huge ask and unfair. Thats essentially saying no.

The options should be either create a build pipeline on your own install of jenkins (what I usually did), or build a ci pipeline in github actions in a fork, and then submitting it as a PR.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjpc_seOUNK9wKtkz7g3wkWcOwzBJWAxsjr62EaVjsobmg%40mail.gmail.com.

[Pierrick Bouvier]

Thanks for taking the time to answer and give a direction Basil, it's definitely more constructive than saying "do not ping me".

I'm fine doing this work, it it's allowed on my side. I'll come back to you once I have an answer.

@Gavin: I agree 100%. The problem for me is not to install or use my own jenkins server (I did a lot of this in the past), but to be able to understand what is inside your runner image exactly, and how to request new stuff (plus additional time to wait for someone to answer). In short, It will take 3 days to find information on something that take 1h for someone from the infra team to do. I hope I can get concrete help on this, instead of vague general direction.

[Basil Crow]

On Thu, Apr 4, 2024 at 12:24 AM Pierrick Bouvier
<pierrick...@linaro.org> wrote:
>
> Thanks for taking the time to answer and give a direction Basil, it's definitely more constructive than saying "do not ping me".

"Do not ping me" definitely still stands from my side regarding
questions about the current code, even if I did last touch some of the
files in question, trying to keep this component on life support. I
didn't write the current code, I didn't check in the pre-built
binaries, I can't explain the reasoning behind past decisions, and I
can't help you understand the current implementation. The expectation
is that the contributor will have to figure all this out, as I did
when working on a different abandoned repository (PCT) several months
back.

Of course we are willing to guide anyone to the desired end state, and
that desired end state includes running CI builds on Jenkins.

> I'm fine doing this work, it it's allowed on my side. I'll come back to you once I have an answer.

Great! We'd love to see this finally fixed. As you saw in the previous
contributor's PR, this has been an issue for a long time.

> The problem for me is not to install or use my own jenkins server (I did a lot of this in the past), but to be able to understand what is inside your runner image exactly, and how to request new stuff (plus additional time to wait for someone to answer).

Aren't these the same problems as with GitHub Actions? In both cases,
the user does not have access to the server while the CI job is
running. In both cases, the images should have a reasonable amount of
build software installed in order to facilitate efficient development.
In both cases, the software installed needs to be documented to
facilitate efficient development (in our case,
https://github.com/jenkins-infra/documentation/blob/main/ci.adoc#container-agents
describes the setup on our agents). And in both cases, there needs to
be a way to install or enable software dynamically (e.g. using GitHub
Actions' setup-java, or Jenkins' tool installation subsystem). I am
aware of deficiencies on the Jenkins side with regard to Docker,
documented in https://github.com/jenkins-infra/helpdesk/issues/3587,
but that doesn't apply here.

In short, this is essentially the same process, certainly not "a
pretty huge ask and unfair" or "essentially saying no" as Gavin Mogan
claimed. And as proof of this, I have done exactly the same thing
myself in the past, resurrecting the Packaging repository from an
abandoned state and developing a CI job there against a VM agent where
I install Puppet and run a custom non-Maven build.

[Jesse Glick]

On Thu, Mar 28, 2024 at 8:02 AM Pierrick Bouvier <pierrick...@linaro.org> wrote:

for winp, I'm not sure what we are missing by not being able to load this DLL (less control on processes launched?)

I think it would be worthwhile to search for usages of winp in @jenkinsci—probably mostly confined to Jenkins core, such as in `ProcessTree`, though plugins are not blocked from using it—and checking whether it could simply be retired. Not loading native libraries is generally desirable anyway. Java 9 introduced richer process control APIs which might suffice for the particular use cases in Jenkins. The main usage of all this code that I am aware of is to kill off whole process subtrees during builds, typically on an agent. Say you had a Pipeline such as

node('something-remote') {

  sh '''

    export DISPLAY=:999

    xvfb &

    make gui-tests

  '''

}

with no join on the background task. By the close of the `node` block, it is expected that all processes started during the user script have been killed off. This is currently done by passing a special random “cookie” as an environment variable to the main shell script, which should be inherited by descendant processes, and finally searching for any remaining processes with that environment set and sending them a termination signal. Maybe this could be replaced with a `ProcessHandle.descendants` call? (`durable-task` runs the user process in the background, however, and this can survive parent JVM sessions.)

Even if there is not a simple solution using the Java Platform, I am guessing winp specifically could be removed in favor of executing `wmic` commands or something like that, or that could be a fallback in case of an unsupported binary platform.

[Basil Crow]

I considered writing a similar post myself a few days ago, and I also
think it is time for winp to be retired, but there are a lot of
challenges associated with removing winp in favor of a simpler
implementation using native Java process handling or simpler Windows
primitives. The existing native implementation is highly tuned and
battle-tested, with such behaviors as attaching to the console and
sending literal Control-C events at the console level. I can't think
of a good reason for this, but it is an example of the stereotypical
two-page function described in
https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
as a "hairy mess." I am sure the high-level objectives of tracking
child processes and terminating them gracefully (for some definition
of "gracefully") could be achieved with a rewrite using simpler
primitives, but I also fear this would introduce a high risk of
regression—if anything, I suspect many consumers implicitly depend on
the existing implementation details and that any change in behavior
(even a justifiable one) would create some pushback for at least a
subset of users.

While we do run basic test automation on Windows, we don't have much
in the way of test automation that tests the edge cases of native
process handling. That, coupled with the general difficulty of
reviewing rewrites, means that we'd be relying on users of weekly
releases to test the rewritten code and report issues. And the failure
mode here is subtle: mistakes in process handling might not cause
outright failures, but might cause leaked processes to accumulate over
time, delaying the discovery of the problem.

While we do encourage first-time contributors, I think it would be
tricky for a first-time contributor to submit a rewrite like this,
because potential problems might not be discovered for several weeks,
at which point the motivation (or funding) to work on the project
might have decreased. While not insurmountable, I think these
challenges have caused me to have a preference for incremental
improvement rather than a rewrite, especially given that the interest
is coming from a first-time contributor. As annoying as it is to delve
into an unfamiliar codebase and write a CI/CD script for an abandoned
codebase, at least the problem is well-defined, and we can be assured
that the result won't break existing use cases. With a rewrite, that
risk increases and I think we would have to be very careful about
review, testing, and (potential) regression management. If a long-time
contributor like yourself proposed to do such a rewrite, I would feel
more comfortable with it, especially if that long-time contributor
promised to be around to deal with any regressions.

[Pierrick Bouvier]

On Thursday 4 April 2024 at 19:03:14 UTC+4 m...@basilcrow.com wrote:

> The problem for me is not to install or use my own jenkins server (I did a lot of this in the past), but to be able to understand what is inside your runner image exactly, and how to request new stuff (plus additional time to wait for someone to answer).

Aren't these the same problems as with GitHub Actions? In both cases,
the user does not have access to the server while the CI job is
running. In both cases, the images should have a reasonable amount of
build software installed in order to facilitate efficient development.
In both cases, the software installed needs to be documented to
facilitate efficient development (in our case,
https://github.com/jenkins-infra/documentation/blob/main/ci.adoc#container-agents
describes the setup on our agents). And in both cases, there needs to
be a way to install or enable software dynamically (e.g. using GitHub
Actions' setup-java, or Jenkins' tool installation subsystem). I am
aware of deficiencies on the Jenkins side with regard to Docker,
documented in https://github.com/jenkins-infra/helpdesk/issues/3587,
but that doesn't apply here.

If I follow the documentation you sent (container-agents), it seems that Visual Studio is not available in those containers.

What should be the steps to add Visual Studio into this:

- Create a new container?

- Install it dynamically (takes one hour easily...)?

It's probably the reason why it ended up with committed binaries in the first place: it was too hard to do this in existing infra.

If you need one last argument, AppVeyor CI was added to this repo a few years ago.

Probably because without having to modify anything on infra side, someone could have a pipeline running.

I'm not personnally attached to GitHub Actions, or any other platform, I just took the one closest from where the code is hosted.

The advantage of GitHub Actions (or GitLab, or any general CI platform) runner is that they cover the need of more developers than simply Jenkins ones.

So, you can expect all the usual software to be already there, ready to use (including all jdk, visual studio, all windows sdk, mvn, ...)

https://github.com/actions/runner-images/blob/main/images/windows/Windows2019-Readme.md

I don't blame Jenkins runner to cover only jenkins need, that's why it's made for after all.

But it's a bit rude to hear "it's not that hard" and having a proof like "I installed a java program, and it worked".

[Pierrick Bouvier]

On Friday 5 April 2024 at 02:01:45 UTC+4 jgl...@cloudbees.com wrote:

On Thu, Mar 28, 2024 at 8:02 AM Pierrick Bouvier <pierrick...@linaro.org> wrote:

for winp, I'm not sure what we are missing by not being able to load this DLL (less control on processes launched?)

I think it would be worthwhile to search for usages of winp in @jenkinsci—probably mostly confined to Jenkins core, such as in `ProcessTree`, though plugins are not blocked from using it—and checking whether it could simply be retired. Not loading native libraries is generally desirable anyway. Java 9 introduced richer process control APIs which might suffice for the particular use cases in Jenkins. The main usage of all this code that I am aware of is to kill off whole process subtrees during builds, typically on an agent. Say you had a Pipeline such as

https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/util/ProcessTree.java

You're right, emulating Ctrl-C to kill a process hierarchy is one of the usage. We can find as well:
- retrieve some env var for a given process

- enableDebugPrivilege (?)

I'm not very familiar with java overall, but probably the java standard now has some way to do this.

 

Even if there is not a simple solution using the Java Platform, I am guessing winp specifically could be removed in favor of executing `wmic` commands or something like that, or that could be a fallback in case of an unsupported binary platform.

Nice idea.

[Pierrick Bouvier]

On Friday 5 April 2024 at 02:23:47 UTC+4 m...@basilcrow.com wrote:

While we do encourage first-time contributors, I think it would be
tricky for a first-time contributor to submit a rewrite like this,
because potential problems might not be discovered for several weeks,
at which point the motivation (or funding) to work on the project
might have decreased. While not insurmountable, I think these
challenges have caused me to have a preference for incremental
improvement rather than a rewrite, especially given that the interest
is coming from a first-time contributor. As annoying as it is to delve
into an unfamiliar codebase and write a CI/CD script for an abandoned
codebase, at least the problem is well-defined, and we can be assured
that the result won't break existing use cases. With a rewrite, that
risk increases and I think we would have to be very careful about
review, testing, and (potential) regression management. If a long-time
contributor like yourself proposed to do such a rewrite, I would feel
more comfortable with it, especially if that long-time contributor
promised to be around to deal with any regressions.

I agree with this.

On Linaro side, I have a green light to work on existing wimp component, if you still find interest in it.

If you prefer a full rewrite, just make it clear, so I can let you work on this.

I understand the implications of technical debt you mention, and the existing issues.

However, this thread was just to add arm64 support.

Is there anyone here that see interest in this?

I reiterate politely my demand: Could we merge the changes required, let someone trusted (you, or any Jenkins core dev, not me) build the binaries, commit them, make a new release, and call it a day?

I can remove all the CI part of this. My goal was simply to prove objectively that it compiles, and not to open another pandora's box.

I don't care about the tool I use, as long as it provides me what I need in a finite amount of time (took me only two hours to get a clean GitHub Actions pipeline).

Finally, I would just note that on all this thread, and the PR associated, we never discussed any detail of current arm64 support proposal, but simply a useless debate about CI A vs B. Maybe it would be more constructive.

[Tim Jacomb]

From my point of view GitHub actions is fine.

I don’t think we want to add something that takes so long to install to our image builds.

This isn’t something we really want to maintain but it needed updating for something and it has no one actively maintaining it.

I’m not sure if any of our maintainers would be able to build this on ARM and what happens on the next release of the person who built the current one isn’t available.

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/e4274c93-124f-49d5-a0c9-fcf50788b46cn%40googlegroups.com.

[Pierrick Bouvier]

On Friday 5 April 2024 at 13:40:11 UTC+4 timja...@gmail.com wrote:

From my point of view GitHub actions is fine.

I don’t think we want to add something that takes so long to install to our image builds.

This isn’t something we really want to maintain but it needed updating for something and it has no one actively maintaining it.

I’m not sure if any of our maintainers would be able to build this on ARM and what happens on the next release of the person who built the current one isn’t available.

So far, arm64 binaries can be cross compiled from x64 without any problem (that's what is done, even if you're on windows-arm64). This is what the PR does.

So anyone with windows-x64 can do it (or GitHub Actions for now). I completed documentation to make sure all dependencies are documented.

Concerning testing, at Linaro, we have a pool of machines, attached to our GitLab (https://gitlab.com/Linaro/windowsonarm).

We run some downstream CI on our side on a nightly basis (https://gitlab.com/Linaro/windowsonarm/nightly), so we can catch regression on projects we contributed to, and which do not have yet dedicated hardware for this.

In more, our partner at Microsoft can (on a best effort basis) ship some hardware to companies open to use it (usually volterra devkit). Sometimes, we advise as well to deploy a windows-arm64 virtual machine if you have compatible hardware (Linux aarch64, or macos aarch64), and we have a public page on how to setup this.

All that said, as long as the WindowsOnArm project will be alive inside Linaro (2+ years, and 2 more guaranteed), we'll continue to support projects we contributed to (https://www.linaro.org/windows-on-arm/), so we can help when next releases will be requested. Finally, we encourage projects to ping us when some help is needed or when there is some questions.

Hope it help to convince that our contribution can add real value to the Jenkins project!

[Basil Crow]

On Fri, Apr 5, 2024 at 12:44 AM Pierrick Bouvier
<pierrick...@linaro.org> wrote:
>
> If I follow the documentation you sent (container-agents), it seems that Visual Studio is not available in those containers.
> What should be the steps to add Visual Studio into this:
> - Create a new container?
> - Install it dynamically (takes one hour easily...)?

This is a question for the Jenkins infrastructure team, who from the
beginning has been (trying to) work with you to achieve this. If the
infrastructure team evaluated this request and declined it, then (and
only then) would I support the use of GitHub Actions. We should be
working _with_ the infrastructure team, not around them.

> If you need one last argument, AppVeyor CI was added to this repo a few years ago.
> Probably because without having to modify anything on infra side, someone could have a pipeline running.

But maybe something _should_ have been modified on the infrastructure side.

> But it's a bit rude to hear "it's not that hard" and having a proof like "I installed a java program, and it worked".

Ignoring the first half of your sentence, reaching out to the
infrastructure team _isn't_ that hard, and my proof was that I
installed a _non-Java_ program and it worked.

[Basil Crow]

Not from my perspective. With two concurrent implementations, you
haven't really solved the problem. You now just have two problems.

[Basil Crow]

On Fri, Apr 5, 2024 at 1:02 AM Pierrick Bouvier
<pierrick...@linaro.org> wrote:
>
> I reiterate politely my demand: Could we merge the changes required, let someone trusted (you, or any Jenkins core dev, not me) build the binaries, commit them, make a new release, and call it a day?

Is "demand" a bit aggressive? I reiterate politely my _request_: that
we eliminate the checked-in binaries and implement CI/CD as a
prerequisite to _any_ change that modifies the binaries (including
adding ARM64 support), for the reasons stated earlier: that this work
needs to be done, and there won't ever be a strong incentive to do
this at any other time. (I do not see any reasoning for your demand.)

[Basil Crow]

On Fri, Apr 5, 2024 at 2:40 AM Tim Jacomb <timja...@gmail.com> wrote:
>
> From my point of view GitHub actions is fine.
> I don’t think we want to add something that takes so long to install to our image builds.

That could be a fair position, if it is the consensus of the Jenkins
infrastructure team. But before we draw that conclusion, I think we
should confirm empirically that this is the case. For example, maybe
there is an existing image we could use after adding the Jenkins agent
binaries. In any case, I leave this to the infrastructure team to form
a consensus.

[Pierrick Bouvier]

My intention was not be aggressive, but it seems to be simply a language issue.

In my native one (french), using "request" is much stronger than "demand", which seems to be the opposite in american english. I'll pay attention to use the correct one in the future.

[Pierrick Bouvier]

Sure, I can make a PoC and depending on setup time, we can decide to use it, or leave it.

What is the best way to contact this team? (mailing list, github issue, private email thread).

I'm sorry to ask such basic questions, but it's pretty hard for someone external to a project to understand which communication channel is the best.

[Stéphane MERLE]

Hi pierrick,

the best way will be to open an issue, as it will be discussed and prioritized during our infra team meeting on Tuesday afternoons.

https://github.com/jenkins-infra/helpdesk/issues

[Pierrick Bouvier]

Thanks Stéphane,

I'll create a ticket there, once I have played a little with existing images.

Pierrick

[James Nord]

> Java 9 introduced richer process control APIs which might suffice for the particular use cases in Jenkins

Alas much of that is unimplemented on windows.  Don't know why as it is available as winp prooved..

(Command line and arguments for one, ability to gracefully kill (aka ctrl-c?)) Given that doesn't work and there is not really a hierarchy in windows processes I doubt (but did not check) the child processes work either.

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3HJnh8XuzRebGexFWmWdn5TFT2JiUzgHwmKSxPzP%2B%2B_Q%40mail.gmail.com.