do you have any comments about this benchmark

[Wei-Cheng Sun]

Hi all,

I found a new benchmark was posted on May 28th.
The author tested lift 2.3 with scala 2.8.1 as well as other language/
frameworks.
Because, I didn't see his full source code. I want to ask the
following questions:
Do you have any comments about his results? Is this testing credible?
If not, what is the better method for benchmarking.

http://www.jtict.com/blog/rails-wicket-grails-play-lift-jsp/

Thanks,

Wei-Cheng

[Jeppe Nejsum Madsen]

Haven't read it in detail so don't know how he warmed up the JVM etc. but this

"No sessions/cookies nor HTTP request managers were set in JMeter, so
each HTTP request was performed independently of the other requests."

basically means he starts a new session on each request, probably not
what you want to test

/Jeppe

[David Pollak]

On Mon, May 30, 2011 at 6:42 AM, Jeppe Nejsum Madsen <je...@ingolfs.dk> wrote:

On Mon, May 30, 2011 at 12:59 PM, Wei-Cheng Sun <sunwe...@gmail.com> wrote:
> Hi all,
>
> I found a new benchmark was posted on May 28th.
> The author tested lift 2.3 with scala 2.8.1 as well as other language/
> frameworks.
> Because, I didn't see his full source code. I want to ask the
> following questions:
> Do you have any comments about his results? Is this testing credible?
> If not, what is the better method for benchmarking.
>
> http://www.jtict.com/blog/rails-wicket-grails-play-lift-jsp/

Haven't read it in detail so don't know how he warmed up the JVM etc. but this

"No sessions/cookies nor HTTP request managers were set in JMeter, so
each HTTP request was performed independently of the other requests."

Or at least mark the Lift page as stateless (See the Loc.Stateless LocParam)

And, yes, warmup is critical... hitting that page at least 100K times before doing the benchmark and making sure you're using the latest 64 bit JVM from Sun on the benchmark.

There are also ways to speed up a Lift app.  These include:

• Explicitly defining the snippet or view loading
• Using Views rather than static template
• Using binding rather than CSS Selector Transforms (I have not done benchmarks, but the binding code has less work to do)
• Moving to Lift 2.4-SNAPSHOT (in prior versions of Lift, Lift used some of the Scala XML transforms... these turn out to be very slow... moving away from these will likely speed up your app).
  

There is a balance between the amount of code one writes, the amount of markup in the Scala code (e.g., Views).

 "tr" #> products.map{ p =>

    (".productname *" #> <div class="product"><img src={p.name + ".jpg"} /><span class="productname">{p.name}</span>, <span class="price">${p.price}</span></div> &

    ".categories *" #> p.categories.map(_.name).mkString(", "))

  }

Or given that you're already embedding markup in your Scala, why not go whole hog:

class ProductsSnippet {

 

  def showProducts: NodeSeq = {

  val products = Service.products

  products.map(p => <tr><td><div class="product"><img src={p.name + ".jpg"} /><span class="productname">{p.name}</span>, <span class="price">${p.price}</span></div></td><td>{p.categories.map(_.name).mkString(", ")}</td></tr>)
  }
}


The above code is computationally less intense and I'd be surprised if (under Lift 2.4-SNAPSHOT) you don't get performance on par with JSP.
 

basically means he starts a new session on each request, probably not
what you want to test

/Jeppe

--
You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.

--
Lift, the simply functional web framework http://liftweb.net

Simply Lift http://simply.liftweb.net
Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im

[Wei-Cheng Sun]

thanks for reply,

for a newcomer, it will be nice to have a set of performance rules,
best practices and development guide. is there such resource around?

play framework committers are already marketing their results on
twitter. there should be some response to that benchmark.

@mbknor: #playframework 1.2.2RC1 with #scala module 0.9.1 is the
fasted webframework in this test! http://bit.ly/isMMq8

On May 30, 10:16 pm, David Pollak <feeder.of.the.be...@gmail.com>

wrote:
> On Mon, May 30, 2011 at 6:42 AM, Jeppe Nejsum Madsen <je...@ingolfs.dk>wrote:
>
>
>
>
>
>
>
>
>

> > On Mon, May 30, 2011 at 12:59 PM, Wei-Cheng Sun <sunweich...@gmail.com>

> > wrote:
> > > Hi all,
>
> > > I found a new benchmark was posted on May 28th.
> > > The author tested lift 2.3 with scala 2.8.1 as well as other language/
> > > frameworks.
> > > Because, I didn't see his full source code. I want to ask the
> > > following questions:
> > > Do you have any comments about his results? Is this testing credible?
> > > If not, what is the better method for benchmarking.
>
> > >http://www.jtict.com/blog/rails-wicket-grails-play-lift-jsp/
>
> > Haven't read it in detail so don't know how he warmed up the JVM etc. but
> > this
>
> > "No sessions/cookies nor HTTP request managers were set in JMeter, so
> > each HTTP request was performed independently of the other requests."
>
> Or at least mark the Lift page as stateless (See the Loc.Stateless LocParam)
>
> And, yes, warmup is critical... hitting that page at least 100K times before
> doing the benchmark and making sure you're using the latest 64 bit JVM from
> Sun on the benchmark.
>
> There are also ways to speed up a Lift app.  These include:
>

>    - Explicitly defining the snippet or view loading
>    - Using Views rather than static template
>    - Using binding rather than CSS Selector Transforms (I have not done

>    benchmarks, but the binding code has less work to do)

>    - Moving to Lift 2.4-SNAPSHOT (in prior versions of Lift, Lift used some

> Simply Lifthttp://simply.liftweb.net

[David Pollak]

On Mon, May 30, 2011 at 7:17 PM, Wei-Cheng Sun <sunwe...@gmail.com> wrote:

thanks for reply,

for a newcomer, it will be nice to have a set of performance rules,
best practices and development guide. is there such resource around?

The best practices are simple:

• Use a 64 bit JVM
• Allocate enough heap (I like 2-3 GB)
• Make sure the JVM has warmed up

In terms of Lift, real world applications rarely reflect micro-benchmarks.  The biggest performance issues I've seen related to Lift have nothing to do with the HTML generation.  Much of the time folks are seeing is in the creation costs of Record and Mapper instances and the associated 10-20 instances (that need to GCed) per field.

But more broadly, Lift app tuning is just like HotSpot tuning.  The most important thing is readable code so that the site can be maintained.  This means that 90-98% of your code is written in a readable style (e.g., with CSS Selector Transforms) and the "HotSpots"... the performance bottlenecks... are tuned after they are explicitly identified.  The specifics of the tuning has to do with the bottleneck.  If it's on the rendering side, then use Scala's XML literals rather than higher level constructs (e.g., bind() or CSS Selector Transforms).
 

play framework committers are already marketing their results on
twitter. there should be some response to that benchmark.

I think anyone who uses Play deserves what they get.

The Play developers are all anti-security (It's the developers responsibility and we're not going to get in their way).  At GeeCON I did the traditional Lift Chat demo (see http://seventhings.liftweb.net/comet )  Part of the presentation was inserting Cross Site Scripting exploits into the chat stream and seeing how, without any developer intervention, Lift protects against this kind of vulnerability.  So, the Play guy does the Play chat demo and opens it up to the audience.  Someone chatted <script>alert('I ownz you');</script> and there was a JavaScript pop-up on everyone else's screen.

So, rather than focusing on a micro-benchmark, I focus on the actual Lift successes.  Stuff like http://twitter.com/#!/rasmus/status/5929904263 (and for those of you who don't know who Rasmus is... Google him.)  Stuff like the high volume sites that are Lift powered and that continue to be very successful with Lift.  Or real-world enterprise app stats like http://twitter.com/#!/dpp/status/75329297925144576

I do not mean to disparage your efforts with the benchmark, however, the likes of Foursquare, UK Guardian and Novell Vibe are all testaments to Lift's scalability not to mention the ability to roll out complex applications and feature-after-feature while keeping a stable, secure site speak far more to what Lift is about.

Thanks,

David

PS -- I'd be interested to see how much extra code you have to put into each of the benchmarks to guard against Cross Site Scripting vulnerabilities in the benchmark and how that would impact performance.
 

--

Lift, the simply functional web framework http://liftweb.net

Simply Lift http://simply.liftweb.net

[Andreas Joseph Krogh]

On 05/31/2011 07:35 PM, David Pollak wrote:

On Mon, May 30, 2011 at 7:17 PM, Wei-Cheng Sun <sunwe...@gmail.com> wrote:

thanks for reply,

for a newcomer, it will be nice to have a set of performance rules,
best practices and development guide. is there such resource around?

The best practices are simple:

• Use a 64 bit JVM
• Allocate enough heap (I like 2-3 GB)
• Make sure the JVM has warmed up

Just curious:
JVM "warm-up" has several times been mentioned as a key issue when discussing Lift performance and Lift vs. framework-X. Is there any reason to believe Lift is more sensitive to this than other JVM-based frameworks?

-- 
Andreas Joseph Krogh <and...@officenet.no>
Senior Software Developer / CTO
Public key: http://home.officenet.no/~andreak/public_key.asc
------------------------+---------------------------------------------+
OfficeNet AS            | The most difficult thing in the world is to |
Rosenholmveien 25       | know how to do a thing and to watch         |
1414 Trollåsen          | somebody else doing it wrong, without       |
NORWAY                  | comment.                                    |
Org.nr: NO 981 479 076  |                                             |
                        |                                             |
Tlf:    +47 24 15 38 90 |                                             |
Fax:    +47 24 15 38 91 |                                             |
Mobile: +47 909  56 963 |                                             |
------------------------+---------------------------------------------+

[David Pollak]

On Tue, May 31, 2011 at 10:58 AM, Andreas Joseph Krogh <andrea...@gmail.com> wrote:

On 05/31/2011 07:35 PM, David Pollak wrote:

On Mon, May 30, 2011 at 7:17 PM, Wei-Cheng Sun <sunwe...@gmail.com> wrote:

thanks for reply,

for a newcomer, it will be nice to have a set of performance rules,
best practices and development guide. is there such resource around?

The best practices are simple:

• Use a 64 bit JVM
• Allocate enough heap (I like 2-3 GB)
• Make sure the JVM has warmed up

Just curious:
JVM "warm-up" has several times been mentioned as a key issue when discussing Lift performance and Lift vs. framework-X. Is there any reason to believe Lift is more sensitive to this than other JVM-based frameworks?

HotSpot (especially 64 bit HotSpot running in server mode) does an amazing job of inlining code, especially monomorphic code.  It turns out that Scala's small functions (that get turned into anonymous inner classes) slot perfectly into HotSpot's inlining methodology.  But it takes a lot of calls to a given method to let HotSpot gather the statistics it needs to inline the code correctly.

So, it's my opinion based on my experience (no hard facts here), that it takes a little more time for HotSpot to optimize Scala code because of the code that ScalaC generates, but once optimized, Scala code performs very, very well.
 

-- 
Andreas Joseph Krogh <and...@officenet.no>
Senior Software Developer / CTO
Public key: http://home.officenet.no/~andreak/public_key.asc
------------------------+---------------------------------------------+
OfficeNet AS            | The most difficult thing in the world is to |
Rosenholmveien 25       | know how to do a thing and to watch         |
1414 Trollåsen          | somebody else doing it wrong, without       |
NORWAY                  | comment.                                    |
Org.nr: NO 981 479 076  |                                             |
                        |                                             |
Tlf:    +47 24 15 38 90 |                                             |
Fax:    +47 24 15 38 91 |                                             |
Mobile: +47 909  56 963 |                                             |
------------------------+---------------------------------------------+

--

You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.

[Andreas Joseph Krogh]

On 05/31/2011 08:04 PM, David Pollak wrote:

On Tue, May 31, 2011 at 10:58 AM, Andreas Joseph Krogh <andrea...@gmail.com> wrote:

On 05/31/2011 07:35 PM, David Pollak wrote:

On Mon, May 30, 2011 at 7:17 PM, Wei-Cheng Sun <sunwe...@gmail.com> wrote:

thanks for reply,

for a newcomer, it will be nice to have a set of performance rules,
best practices and development guide. is there such resource around?

The best practices are simple:

• Use a 64 bit JVM
• Allocate enough heap (I like 2-3 GB)
• Make sure the JVM has warmed up

Just curious:
JVM "warm-up" has several times been mentioned as a key issue when discussing Lift performance and Lift vs. framework-X. Is there any reason to believe Lift is more sensitive to this than other JVM-based frameworks?

HotSpot (especially 64 bit HotSpot running in server mode) does an amazing job of inlining code, especially monomorphic code.  It turns out that Scala's small functions (that get turned into anonymous inner classes) slot perfectly into HotSpot's inlining methodology.  But it takes a lot of calls to a given method to let HotSpot gather the statistics it needs to inline the code correctly.

So, it's my opinion based on my experience (no hard facts here), that it takes a little more time for HotSpot to optimize Scala code because of the code that ScalaC generates, but once optimized, Scala code performs very, very well.

Thanks for explaining, makes sense.

[Diego Medina]

On Tue, May 31, 2011 at 1:35 PM, David Pollak
<feeder.of...@gmail.com> wrote:
>
>
> On Mon, May 30, 2011 at 7:17 PM, Wei-Cheng Sun <sunwe...@gmail.com>
> wrote:
>>

[cut]

> So, rather than focusing on a micro-benchmark, I focus on the actual Lift
> successes.  Stuff like http://twitter.com/#!/rasmus/status/5929904263 (and
> for those of you who don't know who Rasmus is... Google him.)  Stuff like

I remember the day I got my novell vibe account, the first thing I did
after sign up was try to break it using all kinds of xss. It was very
puzzling how it was using "weird" form field names that kept changing
on page reload (this was before I met lift).

So I second what David said, not having to even think about security
is very refreshing.

my 2 cents

/Diego

--
Diego Medina
Web Developer
http://www.fmpwizard.com

[Wei-Cheng Sun]

thanks for your explanation. it makes sense, and lift is a great
modern framework.

sorry for taking so long to reply.

On Jun 1, 11:16 am, Diego Medina <di...@fmpwizard.com> wrote:
> On Tue, May 31, 2011 at 1:35 PM, David Pollak<feeder.of.the.be...@gmail.com> wrote:
>
> > On Mon, May 30, 2011 at 7:17 PM, Wei-Cheng Sun <sunweich...@gmail.com>

> >> fasted webframework in this test!http://bit.ly/isMMq8

> Diego Medina
> Web Developerhttp://www.fmpwizard.com

[Ben Phelan]

Yeah, that is a pretty nice thing.  Not having to religiously double-check all of your team's code for a hundred subtle security vulnerabilities is nice too.

[arac...@gmail.com]

Hi,

sorry for ressurrecting this, but I just noticed it thanks to a
discussion in Stack Overflow. I just want to add one piece of
information and ask a question. As a disclaimer, I'm a Play developer,
but I also used Lift (although time ago), and my intention is not to
create a "flame" on Play vs Lift but to get more information and
knowledge in the area.

The piece of information is about the security issues. As per
documentation http://www.playframework.org/documentation/1.2.2/security
since 1.0.2 I believe Play provides the major security features you
may need. It may not avoid some security issues caused by the
developer, but I believe that to be true for all frameworks. I would
like to know exactly what extra security features Lfit has
implemented. It would be good to add these features, if any, to all
frameworks as this would make the web more safe.

On the question, it's related to the evaluation done here on the
performance test. I'm leaving aside the fact that this is a micro test
and the impact on a real application may not be as clear as shown
here. As far as I understood (and please correct me if I'm wrong), the
test is not considered valid as it doesn't respect the warming of the
JVM. But I would say that all the frameworks subjected to the test are
in the same condition, that is, the warming issue is present in all of
them. Why then you invalidate the test by this? I acknowledge that the
final results may be somehow different after the warming, but given
that the conditions are identical for all the tests (or at least we
are assuming so) I don't understand the issue.

Please note that I'm aware of the impact of warming up a JVM on the
initial run vs subsequent runs, I'm not arguing against that. I'm
asking why, given that the warming happens in all of them, this is
considered a major issue. All servers will suffer the same issue.

Thanks

[Naftoli Gugenheim]

On Wed, Jul 27, 2011 at 11:27 AM, pere.v...@gmail.com <arac...@gmail.com> wrote:

But I would say that all the frameworks subjected to the test are
in the same condition, that is, the warming issue is present in all of
them. Why then you invalidate the test by this? I acknowledge that the
final results may be somehow different after the warming, but given
that the conditions are identical for all the tests (or at least we
are assuming so) I don't understand the issue.

Please note that I'm aware of the impact of warming up a JVM on the
initial run vs subsequent runs, I'm not arguing against that. I'm
asking why, given that the warming happens in all of them, this is
considered a major issue. All servers will suffer the same issue.

Thanks

Suppose framework A does better than framework B without either warming up first, and if both warm up first then B does better. Which is actually more performant?

The only benchmark that matters is the one that gives an indication of how well the framwork does in real usage. In real usage, most of the time the JVM has warmed up already. So to use a benchmark as a prediction of a framework's real-use performance, it has to be comparable to real use, such as by being warmed up already.

[David Pollak]

On Wed, Jul 27, 2011 at 5:27 PM, pere.v...@gmail.com <arac...@gmail.com> wrote:

Hi,

sorry for ressurrecting this, but I just noticed it thanks to a
discussion in Stack Overflow. I just want to add one piece of
information and ask a question. As a disclaimer, I'm a Play developer,
but I also used Lift (although time ago), and my intention is not to
create a "flame" on Play vs Lift but to get more information and
knowledge in the area.

The piece of information is about the security issues. As per
documentation http://www.playframework.org/documentation/1.2.2/security
since 1.0.2 I believe Play provides the major security features you
may need. It may not avoid some security issues caused by the
developer, but I believe that to be true for all frameworks. I would
like to know exactly what extra security features Lfit has
implemented.

First, Play doesn't protect against XSS vulnerabilities as documented.  I saw a room full of people do an XSS exploit at GeeCON against the Play chat demo.
 

It would be good to add these features, if any, to all
frameworks as this would make the web more safe.

There's plenty of documentation around about Lift's use of GUIDs to protect against XSRF and replay attacks as well as form/parameter tampering.

More specifically, my discussions with sadek drobi at the Functional Programming Exchange this year were very clear on the matter.  Sadek was adamant that Play's philosophy was that security was up to the developer and Play would do nothing to get in the way of raw HTTP requests making it through to the program.  Sadek was strong in his opposition to Lift because Lift puts layers in between the HTTP request and the developer unless the developer actively asks for more raw information.
 

On the question, it's related to the evaluation done here on the
performance test. I'm leaving aside the fact that this is a micro test
and the impact on a real application may not be as clear as shown
here. As far as I understood (and please correct me if I'm wrong), the
test is not considered valid as it doesn't respect the warming of the
JVM. But I would say that all the frameworks subjected to the test are
in the same condition, that is, the warming issue is present in all of
them. Why then you invalidate the test by this?

Lift and Lift-based applications make extensive use of Scala and Scala-generated functions.  There is a lot of loading of classes than implement functions, determining code paths (which classes are monomorphic and which are polymorphic), which classes can be inlined, etc.  This is all stuff that takes a lot of page loading and rendering and also is much better on 64 bit JVMs running in server mode than on 32 bit JVMs which always run either mixed or in client mode.

 

I acknowledge that the
final results may be somehow different after the warming, but given
that the conditions are identical for all the tests (or at least we
are assuming so) I don't understand the issue.

Please note that I'm aware of the impact of warming up a JVM on the
initial run vs subsequent runs, I'm not arguing against that. I'm
asking why, given that the warming happens in all of them, this is
considered a major issue. All servers will suffer the same issue.

It depends on how the application and the framework are coded.  Lift is coded to take advantage of Scala at the cost of some warm-up time (look at the number of classes in the core Lift framework: more than 38,000... that's a lot of classes to load and optimize during program initialization [granted not all of these are necessary to run the benchmark.]).  I'm betting that Play has less than 2,000 classes.  Most of the classes in Lift are small and represent Scala functions... there's a lot to load, but those loaded entities represent just a few instructions and can typically get inlined by HotSpot.

 

 

Thanks

On Jun 12, 12:23 pm, Ben Phelan <bem...@gmail.com> wrote:
> Yeah, that is a pretty nice thing.  Not having to religiously double-check
> all of your team's code for a hundred subtle security vulnerabilities is
> nice too.

--
You received this message because you are subscribed to the Google Groups "Lift" group.

To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.

--

Lift, the simply functional web framework http://liftweb.net

Simply Lift http://simply.liftweb.net