New servlet container for Jenkins CI

[Jakub Bartecek]

Hi,
I created new servlet container for Jenkins CI based on Undertow (new
web server for WildFly).
It replaces the old implementation based on Winstone and Jetty.
I aimed to keep backward compatibility with options in the current
container.
All essential options are supported and no problems with Jenkins was
found out.

I named this project Undertow4Jenkins and it is on GitHub (all commits
since February 2014):
https://github.com/jbartece/undertow4jenkins

The integration of this container into Jenkins is implemented in these
forked repositories:
extras-executable-war module:
https://github.com/jbartece/extras-executable-war
Jenkins CI: https://github.com/jbartece/jenkins

Performance tests:
During processing performance tests were compared current version
and version with new container Jenkins CI.
I tested application in 3 scenarios during 8 minutes long tests.
Each scenario was run 10 times for both
versions of Jenkins.
If you are interested in it, I can describe tests with more details.

Scenarios:
1, Loading main page using HTTP GET: New implementation was about
12% faster.
2, Loading freestyle job configuration using HTTP GET: New
implementation was about 4% faster.
3, Creating freestyle job using HTTP POST: New implementation was
about 0.1% slower.

I made it as my master's thesis and provide it as an alternative to
current version, but I do not force anyone to use it ;).
I think that it has potential to be better than current container.

Cheers,
Jakub

[Stephen Connolly]

Do you think your 8 minutes of testing got past the compilation threshold of 10,000 method invocations to ensure that the JVM had an the chance to optimize?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jesse Glick]

On Wed, Jun 4, 2014 at 10:31 AM, Jakub Bartecek <jbar...@redhat.com> wrote:
> https://github.com/jbartece/undertow4jenkins

Interesting work indeed. I guess section 2.6.3 of

https://github.com/jbartece/undertow4jenkins/blob/master/thesis/tex/projekt.pdf?raw=true

about comparison with Jetty is the crucial thing to explore, if we are
to move off of a familiar and battle-tested container.

Runtime responsiveness is definitely of interest; it sounds like the
improvements could be on the order of a few percent, which matters. Is
the measured speedup mostly related to serving static resources, or
better general stream handling, or something else?

Startup time of the servlet container is probably negligible compared
to that of Jenkins core. You suggested that startup time might be
significant in the context of Jenkins’ hour-long integration tests,
but I doubt it is a major contributor, certainly not 50%; would be
happy to be proven wrong. (By the way replacing the bundled container
has no effect on the web container used by integration tests; this is
hardcoded to be Jetty, in JenkinsRule/HudsonTestCase. Acceptance tests
using the default “Winstone” controller run Jenkins as a JAR and so
would use the currently bundled container; these also make actual HTTP
requests heavily, whereas integration tests only occasionally do so,
and so could really be affected by the choice of container.)

2% reduction in application size would be nice, though not a prime
consideration.

[Vojtech Juranek]

Hi,
few more comments:
first, you maybe wonder why another build-in servlet container, as Kohsuke did
the migration to Jetty more than half year ago. Jakub enrolled his thesis on
the university at the beginning of October (IIRC same week when Kohsuke
announced switch to Jetty) and after that it would be quite difficult to
completely change the topics already approved by the university, so he decided
to continue with this topic (btw: I wasn't thesis supervisor, just an adviser)

> Do you think your 8 minutes of testing got past the compilation threshold
> of 10,000 method invocations

not sure what Jakub used for testing, but on my laptop I get about 1,500
iterations per minute, so I would say it's not sufficient for JIT. On the other
hand, do you think JIT would have any significant impact? IMHO server internal
stuff (async. request processing, any kind of caching, thread pooling etc.) has
much bigger impact

> https://github.com/jbartece/undertow4jenkins/blob/master/thesis/tex/projekt.
> pdf?raw=true

FIY: this is not a text of the thesis, I already asked Jakub to write a blog
post and sum up performance results there

> Is
> the measured speedup mostly related to serving static resources, or
> better general stream handling, or something else?

AFAIK it wasn't investigated into such details - main subject of the thesis
was migration to Undertow. To have answers for these questions (and really
reliable comparison), much in-depth analysis and perf. testing needs to be
done IMHO

Vojta

[Kohsuke Kawaguchi]

If it's faster and smaller, it seems to me that this should be something
we should work toward integrating.

How complete is this in terms of the command line option compatibility?

I also notice that Undertow implements Servlet 3.1 but without requiring
Java SE 7. That's great, except I wonder if it's violation of the
servlet spec which would probably demand SE 7 as the minimum?

Anyone else cares to make a pitch on why we should prefer this over the
current Jetty-based one?

--
Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
Try Jenkins Enterprise, our professional version of Jenkins

[Jakub Bartecek]

I implemented all basic options, which are in --help

Not supported options are these:
--httpDoHostnameLookups, httpsDoHostnameLookups,
handlerCountStartup, logThrowingLineNo - these are not supported in
Winstone 2.0
--webappsDir, hostsDir - because only one instance of Jenkins can
run in container
--handlerCountMaxIdle - not using own ExecutorService
--debug, logThrowingLineNo
--spdy - not currently supported in Undertow, but it will be in
next release, which is in version beta
--httpKeepAliveTimeout, httpsKeepAliveTimeout - currently used for
the whole container (cannot be set separately)

As you said, it runs on Java SE 6.

Undertow currently does not support loading of web.xml automatically (in
one function call as Jetty). I implemented
own parser, which support all tags used in Jenkins, but it does not
support all tags in servlet specification. BUT it is designed
to be easily extended.

[Jakub Bartecek]

Yeah. On first server I run Jenkins and on the second I run test script. After the start of Jenkins, the script was running
for 8 minutes in 50 threads. It sent much more than 10 000 requests.
The scenario of creating freestyle job run only for 1 minute, so the amount of requests was smaller, but more than 5000.

It was only basic tests and I am sure, that the testing was not sufficient, but I think that it has potential to be faster than current
implementation.

The performance can be also improved by using XNIO library wrote in C. Stuart Dougles, the main developer of Undertow,
told on one lecture, that it can be about 10% faster (but I didn't test it).

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

[jieryn]

It's interesting, but ultimately seems to me to be not correct.

The goal of the java -jar jenkins.war is just ease of use. It isn't
about tuning it to squeeze out a modest 12% improvement in a single
scenario. And you still would face Stepen's very real JIT argument.
Additionally, these gains are so modest I don't think any changes to
the underlying base case of java -jar jenkins.war should be made.

Here are more objections which should be addressed:

* Jetty works very well already, it has been thoroughly tested for
compatibility with the legacy Winston driver.
* Jetty itself is very well tested and has lots of active developers
and huge amount of community support and knowledge base.
* Jetty is the underlying test server for Jenkins plugin development,
having a match between default plugin testing and default production
should not be understated.
* Jetty supports SPDY (HTTP/2.0), enablement by default would
dramatically speed up the only scenario you list which was noticeably
in favor of Undertow. Undertow does not yet support SPDY.
** Turning on SPDY for sites like a typical Jenkins installation (tons
of individual static resources, all from the same single domain, and
all having very long cache times) has yielded 50-60% speed
improvements even from engineers not paid by Google.
* Undertow, or any other application server, can already be easily
leveraged by simply deploying it to the application server.

I consider each of these arguments to individually be sufficient to
keep the status quo for java -jar jenkins.war, let alone what they
represent in aggregate.

On Wed, Jun 4, 2014 at 10:31 AM, Jakub Bartecek <jbar...@redhat.com> wrote:

[Jakub Bartecek]

You are right that Jetty has longer history, but the SPDY support is
coming soon in Undertow and can be easily added to this container.

[Kohsuke Kawaguchi]

OK, I didn't realize some people feel strongly about this.

IIUC, Undertoe is the web container for JBoss commercial JavaEE server
products, so I just assumed that RedHat has vested interest in keeping
it strong. Because of that I mentally put it in the same battle-tested
level as Jetty (compared to, say, GlassFish, which is sadly "just an RI"
nowadays.) And we have lots of RedHat people in this community, but not
many Eclipse people.

Originally, I started "java -jar jenkins.war" just as a convenience, but
as I discovered in another thread [1], more than 80% of the installation
base is running on this container today. So it's worth some effort to
get a good one in place.

As we contemplate the UI improvements, we'd like to start taking
advantages of newer versions of the servlet. One thing I liked about
Undertoe is that it seems to provide servlet 3.1 without requiring Java7.

But above all, I just hate to see Jakub's work wasted. He clearly spent
a lot of time and effort into doing this, and it is a good work. As an
older guy, I feel like there's a certain moral obligation to make best
of what a junior guy came up with (sorry if you aren't, Jakub.)


[1]
https://docs.google.com/spreadsheets/d/14YzFgKBB6BvbRU_1OjChC3efECWPs77TEGTU09t3KGw/edit#gid=0

[Jesse Glick]

On Thu, Jun 5, 2014 at 2:12 PM, Kohsuke Kawaguchi
<kkawa...@cloudbees.com> wrote:
> I started "java -jar jenkins.war" just as a convenience,

> but […] more than 80% of the installation base is running on this container today.

FYI, many of the large companies CloudBees has a support customers run
Jenkins via native packages, typically RPM, and these use the built-in
container. So making it fast by default is definitely laudable.

As to mismatch with hpi:run, this is definitely a concern—although
there is no reason hpi:run (and hudson-dev:run, used by
jenkinsci/jenkins/war) could not be switched to Undertow as well,
especially if it started a bit faster. Generally speaking we ought to
be running acceptance tests in a variety of containers to catch
container-specific problems: at least Jetty, Undertow, and Tomcat, and
at least during LTS RC testing (may be too expensive to do more often
than that).