Moving ATH tests closer to the code under test

[Basil Crow]

Hi all,

It seems to me that a number of problems are caused by ATH tests being
in a separate repository from the code under test. When regressions
occur, they are not discovered until much later. This causes pain for
those who investigate ATH test failures. If the ATH tests were moved
to the same repository as the code under test (i.e., core or plugins),
then these issues would be caught prior to the problematic change
being merged and released. This would be a win for those who
investigate ATH failures.

Related is the fact that ATH provides a more realistic execution
environment than JTH. ATH supports modern JavaScript frameworks, but
JTH does not. Being able to easily write ATH tests in core and plugins
would be a win for those who make frontend changes. New features could
be delivered with associated test automation at the time of feature
delivery.

I think moving the ATH tests to the same repository as the code under
test would be a win on both fronts. ATH and JTH would then both be
test _harnesses_ (as their names imply) but not _collections_ of
tests: those would live in the same repository as the code under test,
and passing them would be a prerequisite for merge and release.

Integrating such a system into the toolchain would be a significant
project. Providing first-class support for ATH tests in core and
plugins in the context of local builds, plugin CI builds, and BOM/PCT
builds requires a significant amount of work and likely carries
infrastructure prerequisites. Nevertheless, I think such a project
would be of large benefit. It would increase our confidence when
making UI changes, and it would increase our confidence that BOM bumps
do not contain regressions. These benefits would apply to multiple
parties across the Jenkins project.

Note that this is a proposal for a project and a request for
discussion, but I am explicitly not volunteering to implement the
solution (at least not by myself).

Basil

[Ullrich Hafner]

Thanks Basil for bringing this topic to attention! 

I am all in for this new approach! I think when we spend time to write UI tests (which are expensive to write and maintain, and slow to run, see test pyramid of Mike Cohen) then we should consider to run them as real system tests using a real browser and a real Jenkins instance and not as part of our integration tests with a fake UI (HTMLUnit) and a fake Jenkins (started via JTH). 

Since I already converted several plugins in the way you are suggesting I can give some more insights about the things that need to be done, the advantages, and disadvantages. I can only speak for plugins but in theory the same process should be possible for core as well. 

First of all, the credits for this approach go to Oliver Gondža! He created the building blocks in the acceptance test harness and his own plugins. If we follow this approach, a plugin will have three different types of tests:

1) Unit tests that test classes in isolation.

2) Integration tests that test one or more classes together in a running Jenkins instance (started by JTH). These tests will solely work on Jenkins' object model and will not use the browser HTML output anymore.

3) UI tests as end-to-end tests for one or more scenarios of the plugin use cases.

These tests will then be started using maven-surefire (1) and maven-failsafe (2) in our normal pipelines (buildPlugin() or GitHub Actions). Those results will visible in the usual way in a pull request. The UI tests (3) need to be started as a separate process since they work on the created artifacts. I’m using GitHub Actions in my plugins to start the individual UI tests, but it should be possible to run on Jenkins as well. Those tests will run in a special docker container with a headless browser (chrome or firefox).

All three types of tests are started for every pull request. So if a committer breaks something in the UI it will be visible as a PR check failure. Even if Jenkins core changes from 1.n to 1.(n+1), a Dependabot PR will be triggered and evaluated. 

If you want to use that approach in your plugin you need the following steps:

1) Change the folder structure of a plugin: we need a maven sub-module for the plugin and one for the corresponding ui-tests. Basically, all existing files need to be moved below the plugin folder.

2) Create a new ui-test project for the plugin.

3) Move existing Selenium page objects and test cases from the ATH to the plugin.  

Then you will have the previous ATH tests as part of your plugin. Here are some working examples:

- a feature PR in the Warnings Plugin (https://github.com/jenkinsci/warnings-ng-plugin/pull/1206)

- Jenkins upgrade PR in the Code Coverage Plugin (https://github.com/jenkinsci/code-coverage-api-plugin/pull/377)

Here are two work-in-progress PRs:

- JUnit Plugin (https://github.com/jenkinsci/junit-plugin/pull/355 or https://github.com/jenkinsci/junit-plugin/pull/342)

- PIT Plugin (https://github.com/jenkinsci/pitmutation-plugin/pull/129)

So I think that it something that each plugin author can do without much effort in each plugin. 

So as a summary the advantages:

- UI tests work with a real browser (and do not break due to HTMLUnit incompatibilities)

- integration tests are fast and work with assertions on the Jenkins object model. 

- when you do not use HTMLUnit anymore you will also find better ways to test you Java side models. This will lead to a better Java model! The UI layer should be as thin as possible and the logic will be part of Java that can be tested with Unit tests in Java.

- both types of tests are evaluated for each pull request

The disadvantages:

- not all plugins have UI tests yet, so there is an initial ramp up time required

- ATH based UI tests are more fragile

- build execution times of PRs will increase since the initialization is slow

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjoGK5XdfP38zE1WPH-pcVefRrOc7ndvWqApFMzEGEE0zg%40mail.gmail.com.

[Basil Crow]

On Tue, Apr 26, 2022 at 7:45 AM Ullrich Hafner <ullrich...@gmail.com> wrote:
> Thanks Basil for bringing this topic to attention!

You are welcome!

> […] I already converted several plugins in the way you are suggesting […]

I am pleased that this approach has been working well in your plugins.
Assuming that the idea itself is desirable (and by all means, let me
know if you think it is not!), let me focus on some practical concerns
around the feasibility of adopting such an approach throughout the
core and plugin ecosystem. One of the strengths of this ecosystem is
that we expose a rich and polished framework to plugin developers that
works out-of-the-box for most use cases. Adding this type of testing
to the toolchain while maintaining existing levels of approachability
could be a challenge, so there are a few points I want to discuss.

> […] we need a maven sub-module for the plugin and one for the corresponding ui-tests.

Ideally, I think we would avoid the use of multi-module Maven projects
if possible. The vast majority of Jenkins plugins are single-module
Maven projects, and in my opinion the simplicity and low maintenance
associated with this is desirable. I wonder if we could design a
system such that for the common case only a single Maven project is
needed, while still supporting multi-module Maven projects for more
complex use cases. I am not sure if your existing multi-module Maven
project is due to technical limitations or for
structural/organizational reasons.

> Those tests will run in a special docker container with a headless browser (chrome or firefox).

Ideally, I think we would want to support both Dockerized and
non-Dockerized builds if possible. While most developers today are
running Docker, it adds some complexity to the process of debugging
local builds and adds some infrastructure requirements for CI builds.
I think decreasing the number of requirements and the amount of
complexity would make it easier for such a system to gain widespread
adoption. Also, we need to think about how such changes would impact
existing BOM/PCT builds. If we retain a single Maven module and avoid
the requirement for a Docker daemon, then I think few if any changes
would be needed here. But if we introduce additional complexity, there
might be further infrastructure changes needed in BOM/PCT.

> […] run them as real system tests using a real browser and a real Jenkins instance and not as part of our integration tests with a fake UI (HTMLUnit) and a fake Jenkins (started via JTH) […]

Regarding the use of a "real Jenkins instance," I wonder how the ATH
implementation compares to RealJenkinsRule, which is at a high level
doing something very similar; namely, starting Jenkins in a separate
Java process and tearing it down at the end of the test. If the
RealJenkinsRule implementation is more efficient while still remaining
compatible with the goal of frontend testing with a headless browser,
it may be worth looking into unifying the two implementations in order
to reduce maintenance and/or increase performance.

> not all plugins have UI tests yet, so there is an initial ramp up time required

I think if we build it they will come.

> […] UI tests are more fragile

Has this ever not been the case in the decades you and I have been
programming? I guess the benefit of this idea depends on _how_
fragile. If the fragility is occasional and a retry chases the problem
away, that could be tolerable. On the other hand, if the fragility is
frequent and requires one or more retries, this could undermine the
idea.

> build execution times of PRs will increase since the initialization is slow

I wonder if some targeted optimization couldn't be done through the
use of techniques such as the ones used in RealJenkinsRule.

Basil

[Jesse Glick]

On Tue, Apr 26, 2022 at 12:28 PM Basil Crow <m...@basilcrow.com> wrote:

I wonder if we could design a system such that for the common case only a single Maven project is needed

Sounds tricky. `acceptance-test-harness` has a bunch of dependencies some of which clash with those in Jenkins core or some plugins, so you would need to either shade them all, or otherwise somehow ensure the ATH dependency trails are given minimum priority.

I wonder how the ATH implementation compares to RealJenkinsRule

You could in principle use `RealJenkinsRule` with some GUI testing library other than HtmlUnit that better supports exotic JavaScript. If you wanted to reuse the rich Selenium page object library in ATH you would need to factor that out into a standalone library.

There are other significant differences:

• Although ATH uses RESTish endpoints for a few purposes, for the most part the test setup (not just the actual assertions) uses the browser. In all cases the whole test run is “black-box”. This has its appeal (stronger coverage) but GUI setup makes tests much slower and either REST or GUI setup can also make it a lot more awkward to write tests than when using `[Real]JenkinsRule`, which are “white-box” and let you quickly set up an environment and run some assertions at the Java level. JTH also lets you write test-only extensions, which ATH does not.
• ATH installs plugins, and selects plugin versions, using user-mode tools. `[Real]JenkinsRule` follow the Maven test classpath. Possibly ATH could be given an option to define a mock UC based on a test classpath.
• ATH uses Guice for configuration. JTH does not.
• ATH uses `docker-fixtures` for containerized tests. JTH can use that also, but Testcontainers is recommended instead.

 

the benefit of this idea depends on _how_ fragile.

Also whether the fragility is in the actual assertions, or test setup (point #1 above).

[Basil Crow]

On Tue, Apr 26, 2022 at 11:12 AM 'Jesse Glick' via Jenkins Developers
<jenkin...@googlegroups.com> wrote:

> `acceptance-test-harness` has a bunch of dependencies some of which clash with those in Jenkins core or some plugins, so you would need to either shade them all, or otherwise somehow ensure the ATH dependency trails are given minimum priority.

I see; thank you. I suppose the same problem exists in JTH, with
shading being the solution, so I see no reason why the same couldn't
be done for ATH.

> Although ATH uses RESTish endpoints for a few purposes, for the most part the test setup (not just the actual assertions) uses the browser. In all cases the whole test run is “black-box”. This has its appeal (stronger coverage) but GUI setup makes tests much slower and either REST or GUI setup can also make it a lot more awkward to write tests than when using `[Real]JenkinsRule`, which are “white-box” and let you quickly set up an environment and run some assertions at the Java level. JTH also lets you write test-only extensions, which ATH does not.

I see; very interesting. I wonder how many parts of the test setup
process are common to many tests/plugins versus unique to each
test/plugin. For example, I could imagine that things like going
through the setup wizard, creating a freestyle project, or adding an
agent are common to many tests (and therefore pointless from the
perspective of test coverage to repeat them multiple times), while
other parts of the test setup process, like configuring a specific
plugin, might be unique to each test/plugin. If true, and if the test
time is dominated by the parts of the test setup process that are
common to many tests/plugins, perhaps we could, for each portion of
the test setup process that is common to many tests/plugins, run that
portion with the GUI/REST API in one "primary" repository (e.g., core,
Workflow: Job, etc) and (without loss of test coverage at a macro
level) run that portion with JCasC/REST or even Java/JTH in the
"secondary" repositories as a way of speeding up and de-flaking the
setup process to get to the unique/interesting part.

In other words, I wonder if a middle ground between test coverage and
performance couldn't be found. Or maybe I am hopelessly naive and this
is an exercise in futility in the long term. I will discuss this with
my friend who has experience in this domain.

> ATH installs plugins, and selects plugin versions, using user-mode tools. `[Real]JenkinsRule` follow the Maven test classpath. Possibly ATH could be given an option to define a mock UC based on a test classpath.

Yes, a great example of the "middle ground" approach from the previous
paragraph!

> Also whether the fragility is in the actual assertions, or test setup (point #1 above).

I think your point is that fragility in the test setup would undermine
the idea at its core, while fragility in the actual assertions could
be tolerated on a case-by-case basis in individual test suites. If so,
I concur, and I think that finding a way to mitigate the fragility of
test setup by factoring out that logic as described in the preceding
paragraphs could be one solution.

[Antonio Muñiz]

I was working/planning with Oliver for some time on this. One of the identified downsides of this approach is that the Jenkins UI does often evolve in a backward incompatible way (ie. table to divs change, plugin manager recent changes, etc), all those potential breaking changes would not be caught by acceptance tests running at plugin build/release time (unless the plugin maintainer updates the `jenkins.version` to the latest very often). The current ATH approach runs all acceptance tests against a specific Jenkins version, so possible regressions are caught at Jenkins core release time, as desired.

The alternative would be to define the plugin build/release pipeline in a way that ATHs are run against a dynamic Jenkins version (could be the latest weekly at the time of the build), but that leads to unreproducible builds, which is IMO quite bad. And it does not address the issue of checking for regressions caused on plugins at core release time (unless some complicated pipeline is put in place to run ATHs from some key plugins as part of the Jenkins core build).

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjpmQLNRwhJSGYc3wT28P32gHvt2aF_CJtkDtX47iwBGzw%40mail.gmail.com.

--

Antonio Muñiz

Human, Engineer
CloudBees, Inc.

[Basil Crow]

On Wed, Apr 27, 2022 at 3:37 AM 'Antonio Muñiz' via Jenkins Developers
<jenkin...@googlegroups.com> wrote:
> unless some complicated pipeline is put in place to run ATHs from some key plugins as part of the Jenkins core build

Yes, implied in this proposal was running the plugin BOM/PCT tests
(which, as of this proposal, would also include ATH tests) against
core PRs that are ready for merge prior to merging and releasing them
as described in https://github.com/jenkinsci/bom/issues/758. The
general point is to shift testing as far leftward as possible given
budgetary constraints.

[Ullrich Hafner]

>> Although ATH uses RESTish endpoints for a few purposes, for the most part the test setup (not just the actual assertions) uses the browser. In all cases the whole test run is “black-box”. This has its appeal (stronger coverage) but GUI setup makes tests much slower and either REST or GUI setup can also make it a lot more awkward to write tests than when using `[Real]JenkinsRule`, which are “white-box” and let you quickly set up an environment and run some assertions at the Java level. JTH also lets you write test-only extensions, which ATH does not.
>
> I see; very interesting. I wonder how many parts of the test setup
> process are common to many tests/plugins versus unique to each
> test/plugin. For example, I could imagine that things like going
> through the setup wizard, creating a freestyle project, or adding an
> agent are common to many tests (and therefore pointless from the
> perspective of test coverage to repeat them multiple times), while
> other parts of the test setup process, like configuring a specific
> plugin, might be unique to each test/plugin. If true, and if the test
> time is dominated by the parts of the test setup process that are
> common to many tests/plugins, perhaps we could, for each portion of
> the test setup process that is common to many tests/plugins, run that
> portion with the GUI/REST API in one "primary" repository (e.g., core,
> Workflow: Job, etc) and (without loss of test coverage at a macro
> level) run that portion with JCasC/REST or even Java/JTH in the
> "secondary" repositories as a way of speeding up and de-flaking the
> setup process to get to the unique/interesting part.

This will depend on the type of the plugins. In my analysis plugins my tests start by creating a freestyle job or pipeline. There is no previous setup required. I don't think that this can be speed up with job-dsl or something different.

>
> In other words, I wonder if a middle ground between test coverage and
> performance couldn't be found. Or maybe I am hopelessly naive and this
> is an exercise in futility in the long term. I will discuss this with
> my friend who has experience in this domain.
>
>> ATH installs plugins, and selects plugin versions, using user-mode tools. `[Real]JenkinsRule` follow the Maven test classpath. Possibly ATH could be given an option to define a mock UC based on a test classpath.
>
> Yes, a great example of the "middle ground" approach from the previous
> paragraph!

This is already possible. In my UI tests I am starting the Jenkins under test with the same set of plugins that I am using in the integration tests (JTH). These plugins are just copied from the target folder of the plugin maven module to the target folder of the ui module and the JUT starts with them without installing them.

>
>> Also whether the fragility is in the actual assertions, or test setup (point #1 above).
>
> I think your point is that fragility in the test setup would undermine
> the idea at its core, while fragility in the actual assertions could
> be tolerated on a case-by-case basis in individual test suites. If so,
> I concur, and I think that finding a way to mitigate the fragility of
> test setup by factoring out that logic as described in the preceding
> paragraphs could be one solution.
>

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjpmQLNRwhJSGYc3wT28P32gHvt2aF_CJtkDtX47iwBGzw%40mail.gmail.com.

[Ullrich Hafner]

> Am 27.04.2022 um 12:37 schrieb 'Antonio Muñiz' via Jenkins Developers <jenkin...@googlegroups.com>:
>
> I was working/planning with Oliver for some time on this. One of the identified downsides of this approach is that the Jenkins UI does often evolve in a backward incompatible way (ie. table to divs change, plugin manager recent changes, etc), all those potential breaking changes would not be caught by acceptance tests running at plugin build/release time (unless the plugin maintainer updates the `jenkins.version` to the latest very often).

When you run the ATH tests in a plugin then you run the tests agains the latest Jenkins version, not the baseline version defined in the plugin. I’m using Dependabot to make sure that this version is up to date. (It is still too late since the Jenkins release is already published, but at least you will see it one day after the release)

[Basil Crow]

On Wed, Apr 27, 2022 at 9:06 AM Ullrich Hafner <ullrich...@gmail.com> wrote:
>
> This will depend on the type of the plugins. In my analysis plugins my tests start by creating a freestyle job or pipeline. There is no previous setup required. I don't think that this can be speed up with job-dsl or something different.

The point is that creating the project isn't the "interesting" part of
your test. It could be done with white-box JTH/RealJenkinsRule code
rather than black-box ATH/Selenium code and you presumably wouldn't
care, as long as the "interesting" part of your test (verifying that
analysis results are present, clicking into graphs in the UI, etc) was
tested by black-box ATH/Selenium code. And as long the parts that are
"uninteresting" in the context of your analysis plugins are deemed
"interesting" by some other plugin (or core) and tested by black-box
ATH/Selenium logic there, then we can maintain the same level of
coverage across the whole Jenkins project at a macro level.

In this example, we could have a black-box ATH/Selenium test that
creates a Freestyle job in core and a black-box ATH/Selenium test that
creates a Pipeline job in Workflow: Job. Functional testing for
creating those job types would then be covered sufficiently, and in
your analysis plugins you could take the shortcut of using white-box
JTH/RealJenkinsRule logic to create the job efficiently (and
reliably!) before proceeding to the "interesting" part of your test
with black-box ATH/Selenium logic.

The point is that efficiency and reliability can be improved by
removing duplicate effort. Expensive and flaky black-box ATH/Selenium
logic to create a Freestyle job doesn't need to run thousands of
times. It only needs to run a handful of times in key repositories,
and the other repositories can assume that it would work and test on
top of that, a form of proof by induction.

[Ullrich Hafner]

> Am 26.04.2022 um 18:28 schrieb Basil Crow <m...@basilcrow.com>:
>
> On Tue, Apr 26, 2022 at 7:45 AM Ullrich Hafner <ullrich...@gmail.com> wrote:
>
>> Those tests will run in a special docker container with a headless browser (chrome or firefox).
>
> Ideally, I think we would want to support both Dockerized and
> non-Dockerized builds if possible.

You can run the build locally without Docker. Docker is just used in CI (GitHub Action or Jenkins). I am locally using a different Maven profile and run the tests without Docker in Firefox or Chrome. You can even run it from IntelliJ and set breakpoints.

>
>> […] run them as real system tests using a real browser and a real Jenkins instance and not as part of our integration tests with a fake UI (HTMLUnit) and a fake Jenkins (started via JTH) […]
>
> Regarding the use of a "real Jenkins instance," I wonder how the ATH
> implementation compares to RealJenkinsRule, which is at a high level
> doing something very similar; namely, starting Jenkins in a separate
> Java process and tearing it down at the end of the test. If the
> RealJenkinsRule implementation is more efficient while still remaining
> compatible with the goal of frontend testing with a headless browser,
> it may be worth looking into unifying the two implementations in order
> to reduce maintenance and/or increase performance.
>
>> not all plugins have UI tests yet, so there is an initial ramp up time required
>
> I think if we build it they will come.
>
>> […] UI tests are more fragile
>
> Has this ever not been the case in the decades you and I have been
> programming? I guess the benefit of this idea depends on _how_
> fragile. If the fragility is occasional and a retry chases the problem
> away, that could be tolerable. On the other hand, if the fragility is
> frequent and requires one or more retries, this could undermine the
> idea.
>
>> build execution times of PRs will increase since the initialization is slow
>
> I wonder if some targeted optimization couldn't be done through the
> use of techniques such as the ones used in RealJenkinsRule.
>
> Basil
>

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjphex3uXLfjsAfCpEaFvMNpXgBocgnX5tHaSuqTV1iZ7w%40mail.gmail.com.

[jn...@cloudbees.com]

Going back to the earlier premise

> It seems to me that a number of problems are caused by ATH tests being

in a separate repository from the code under test

I would say the biggest problem is not so much that - rather that we do not run enough of these tests for the code that is affected, early enough, nor is there probably a good set of well written smoke test candidates.

Moving the code is one possible solution to the first issue, but it also has other problems.  For example If we update the PageObjects you then need to update N repos (or just not bother) to account for the change in API.  (granted changes in API are hopefully minimal, but that is not always the case, esp wrt things like table-2-div).

I discussed with directly with Mark Waite, but to me we need some new tests that better cover what we consider to be "critical functionality", (ie an Acceptance test), what we tend to have today is a bunch of limited tests where things are testing some things in isolation.

I would love to see a test that was run on a core PR that at least created a credentials (in different scopes, stores), an agent a a pipeline.  And check that the pipeline can checkout from a Git Repo  (github would be nice, but attempting to minimize network operations and hence flakes and also brings security issues into play...) and build some PRs and a main branch.
And something similar with a FreeStyleJob.  (include in that Junit reporting).  Currently the smoke tests for core PRs are too fine grained to me.

I think we could make some quick wins with the following form of attack

*  extend buildPlugin so that you can specify a list of ATH tests to run (currently most of the tests are in the form of plugins.<plugin-name>Test

* introduce a few new tests like discussed above that are run on Jenkins PRs.

Whatever we have, if we end up having ATHs on core PRs to detect regressions early then we need to be prepared to maintain the ATH tests/framework or to not accept the changes.  The maintenance has mostly fallen to a few individuals like Tim, Vincent and myself.  We had a passing build last October, but it did not last (as things where not run with any associated changes).

I am all in favour of improving the current situation however that happens and am happy to see this gaining traction.

/James

[Tim Jacomb]

I'm in favour of this although I can't dedicate time to it currently.

> Whatever we have, if we end up having ATHs on core PRs to detect regressions early then we need to be prepared to maintain the ATH tests/framework or to not accept the changes.  

If PRs break the ATH and they are in the same repository of course we would expect them to be fixed as part of the PR.

> We had a passing build last October, but it did not last (as things where not run with any associated changes).

We have fairly consistently passing builds, (apart from a number of agent disconnections requiring re-runs).

Occasionally a core PR will be merged that wasn't run against ATH and breaks it, or a plugin changes and needs corresponding updates in ATH.

Both of the above issues would be solved by moving the tests to corresponding plugins and core and then having an aggregator project that can run tests against all of them like BOM.

If the framework was in core and we replaced existing HtmlUnit tests I think a lot more would be written.

Recently both Jan and myself have hit a number of issues with HtmlUnit not supporting modern JavaScript. (You implement it and it works fine in the browser but then the PR test builder blows up and you need to re-implement it)

Thanks

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/2ec3290f-93cc-40d1-926f-bedffa1c9d2fn%40googlegroups.com.

[Basil Crow]

On Wed, May 11, 2022 at 7:32 AM Tim Jacomb <timja...@gmail.com> wrote:
> Recently both Jan and myself have hit a number of issues with HtmlUnit not
> supporting modern JavaScript. (You implement it and it works fine in the
> browser but then the PR test builder blows up and you need to re-implement
> it)

jenkinsci/jenkins-test-harness#569 (comment) describes an idea I had recently:

Another solution would be to teach Can I use/Browserslist/Babel which features are (or are not) supported by HtmlUnit/Rhino. Then we could transpile our code to something that would run in JTH/HtmlUnit/Rhino with minimal modifications. This would be a modest amount of effort and would require upstream changes, but it seems relatively achievable in the medium term.