Pebble Performance

[Derek van Daal]

Hi Mitchell and team

So was thinking you might been wondering how Pebble stacks up to the rest of the template engines out there in terms of performance.

I have cloned the benchmark test at https://github.com/greenlaw110/template-engine-benchmarks locally and added a test case for Pebble.

Here are the results on my local machine:

header:
     [echo]
     [echo] #################################################################
     [echo]            
     [echo]             JAVA TEMPLATE ENGINE BENCHMARKER v1.0
     [echo]            
     [echo]                Output method:   w
     [echo]                warm up loops:   10
     [echo]                benchmark loops: 1000
     [echo]                buffer mode:     false
     [echo]                rythm new API:   true
     [echo]                
     [echo]             type 'ant help' to see help messsage
     [echo] ################################################################
     [echo]        

init:

compile:

stringbuilder:
     [java] ntimes: 1000, real time: 110(msec)

freemarker:
     [java] ntimes: 1000, real time: 388(msec)

velocity:
     [java] ntimes: 1000, real time: 250(msec)

jamon:
     [java] ntimes: 1000, real time: 103(msec)

httl:
     [java] ntimes: 1000, real time: 138(msec)

pebble:
     [java] ntimes: 1000, real time: 1212(msec)

jmte:
     [java] ntimes: 1000, real time: 393(msec)

jangod:
     [java] ntimes: 1000, real time: 1487(msec)

thymeleaf:
     [java] ntimes: 1000, real time: 2815(msec)

all:

BUILD SUCCESSFUL
Total time: 10 seconds


I really love the django-style template engine and I'm pushing for the use of Pebble for a new system we will be building at work. Perhaps we could look at improving the performance of Pebble. I'm unfortunately a bit swamped at work at the moment, but would be more than willing to join the team once I have a bit more free time.

Regards
Derek

[Mitchell Bösecke]

Hi Derek,

I'm glad you're interested in Pebble. I threw together my own fork of that benchmark library and I noticed similar results to what you posted. After some time trying to understand jprofiler, I was able to make a couple of performance improvements to pebble. I just now released a new version of pebble, version 1.4.1, which includes these enhancements. Now, pebble consistently falls somewhere between velocity and freemarker. Here are results I'm getting:

header:

     [echo] 

     [echo] #################################################################

     [echo]             

     [echo]             JAVA TEMPLATE ENGINE BENCHMARKER v1.0

     [echo]             

     [echo]                Output method:   os

     [echo]                warm up loops:   100

     [echo]                benchmark loops: 10000

     [echo]                buffer mode:     false

     [echo]                rythm new API:   false

     [echo]                

     [echo]             type 'ant help' to see help messsage

     [echo] ################################################################

     [echo]         

init:

compile:

stringbuilder:

     [java] ntimes: 10000, real time: 439(msec)

freemarker:

     [java] ntimes: 10000, real time: 1709(msec)

velocity:

     [java] ntimes: 10000, real time: 1127(msec)

rythm:

     [java] ntimes: 10000, real time: 769(msec)

jamon:

     [java] ntimes: 10000, real time: 957(msec)

httl:

     [java] ntimes: 10000, real time: 515(msec)

jmte:

     [java] ntimes: 10000, real time: 1507(msec)

jangod:

     [java] ntimes: 10000, real time: 6827(msec)

thymeleaf:

     [java] ntimes: 10000, real time: 12621(msec)

mustache:

     [java] ntimes: 10000, real time: 672(msec)

pebble:

     [java] ntimes: 10000, real time: 1320(msec)

all:

BUILD SUCCESSFUL

Total time: 41 seconds

So it's definitely faster although I have no doubt that there is still room for further improvement. 

[Derek van Daal]

Wow! That's a massive improvement already, Mitchell. Thanks! Was actually going to run the code through JProfiler myself.

Well, I know that some of these other engines compile templates into java bytecode at runtime (httl, rythm), so pretty tough competitors if performance was pebble's main focus ;)

[Mitchell Bösecke]

It's interesting because Pebble used to compile everything into bytecode; that was my original intention. However, by compiling to bytecode I encountered many downsides:

• Had to include a full compiler as a dependency which is massive in size
• Increased complexity when developing pebble core
• Increased complexity when other developers want to extend pebble with their own extensions (this was the biggest driving factor for me to avoid bytecode compilation)
• Initial compilation of templates is slow; users would almost certainly need to pre-compile all templates using a build tool 

So ultimately, even though I had bytecode compilation working just fine I scrapped it all in one massive commit about a year ago. Now, the overarching design goal of Pebble is to be the best template engine for 90% of use cases as opposed to being a niche template engine. The focus points would be the following:

• Fast enough to compete with the big three: thymeleaf, velocity, freemarker
• Few enough dependencies and small enough to be used in multiple environments (servlet container, standalone, mobile, etc)
• Competitively clean syntax and rich feature set
• Easily extendable and adaptable to most projects

At this point I think we're doing pretty well on all of those points but there's always room for improvement!

[João Carvalho]

Hi guys.

We have been rocking Pebble on our production environments for a few months now (the initial version went live last May), and apart from a minor customization (a custom tag to allow lazy iteration over Java 8 Streams and Collections), we’ve been pretty satisfied with Pebble’s performance.

We are using Pebble on two main areas:

- Theming for a presentation framework [1] that allows integration between other frameworks, such as Spring, Struts, etc. On this one, Pebble does not generate the full page, only the layout, but its performance is pretty solid.

- Our CMS [2]. Using Pebble, and custom Loader implementations, we are able to support a fully-feature Java CMS (with a bit of Nashorn as well), with live customizability, and once again, Pebble excels in terms of performance.

On a side note; to make those benchmarks more reliable, how about porting them to JMH [3]?

[1] https://confluence.fenixedu.org/display/BENNU/Portal+Themes

[2] https://github.com/FenixEdu/fenixedu-cms

[3] http://openjdk.java.net/projects/code-tools/jmh/

---

João Carvalho

Software developer at FenixEdu (http://fenixedu.org)
joao.pedr...@tecnico.ulisboa.pt

[Mitchell Bösecke]

Hi João. I'm glad to hear that you're happy with the performance in a production environment. Also, thanks again for porting the pebble-benchmark library to JMH; this helped me a LOT by ensuring I was getting much more accurate and repeatable results. The only change I've done since then was change it from measuring "average time" to "throughput" instead; I felt like this is a bit more representative of what people typically look for in a template engine.

And as an update to everyone, over the past couple of days I've managed to squeeze even more performance out of Pebble. According to the original template-engine-benchmark library, Pebble has now surpassed apache velocity. I'm hoping by the end of the week I'll release version 1.4.2 which will include these improvements.

[Mitchell Bösecke]

If anyone is still interested, I have completely revamped the original template-engine-benchmark library into a new JMH based benchmarking library, as per João's suggestion.