Yesod vs ASP.NET simple test suite up

[Chris Swires]

Hi all,

A simple test suite allowing you to run the basic framework tests
(Pong/staticfile/bigtable) is now up at: http://christopherswires.com/homepage/testmenu,
if this is useful to anyone. It is now up on an EC2 High CPU medium to
avoid the micro's randomness.

I am working on a database test as Felipe suggested but unfortunately
my deadline is only two weeks away now and I have a lot of offline
tests to run (for concurrency etc) and a fair amount of the document
to get down. If I am unable to finish this before the deadline I will
complete it afterwards regardless (as well as a few other tests I have
in mind, and potentially adding further frameworks now that I have
Amazon funding) as it doesn't look like I'll be going straight into a
job thanks to my now fairly niche skill-set.

Anyone care to take a stab at why the Yesod results show so much
variation? I had thought that it was an issue with the EC2 but the
ASP.NET results are far more consistent.

Thanks in advance,

Best,

Chris.

[Chris Swires]

I would like to agree but unfortunately there is no consistent degradation over long periods, simply fairly extreme fluctuations. Though I imagine you're right and it could be an issue with GHC. My Ec2 (from top) is only showing memory usage of around 22% though with CPU spiking at around 15% when a test is run. 

Very odd. 

Chris.

On Monday, 17 September 2012 04:04:37 UTC+1, Smith wrote:

I chalked it up to GHC's runtime or something.

High loads over a long period of time results in degradation in performance in my case.  Yesod would initially start out fairly well but quickly slowed down.

It's quite the opposite with other languages and frameworks I've dealt with in my experience. Sure, the initial perf is great but under heavy load, frameworks like Play and sometimes even ASP.NET MVC outstrips Yesod in most cases and has much more predictable performance.  Of course, I could be doing something wrong and if anyone has the opposite experience, I'd love to hear what they did.

For what it's worth, I'm using the 64-bit version of GHC on Linux with gobs of RAM.

(...put down the pitchforks.  Haskell is one of my favorite languages :-))

[Smith]

Sorry.  I did not elaborate much on my tests.

What I did was deploy to a server to a Linux VM we have on one of our servers.  I tested both rendering a page and returning json.  While using ab/httperf, I monitored Yesod's RAM usage.  What would happen is Yesod would actually eat up a ridiculous amount of RAM all while slowing down.  After a while, the GC kicks in and Yesod would speed up, only to slow down again and the cycle repeats.  So in essence, my results are similar to yours.

I don't know much about the internals of GHC but from my testing, I concluded that every time a thread is forked, GHC does not immediately reclaim memory after a thread has finished executing.  On why that would slow Yesod down, I have no idea.

If you'd like to duplicate the behavior for yourself, just deploy to a server where you can closely monitor processes or even try on your own machine.  Hopefully, it's just my environment but I highly doubt that, having tested on 3-4 different machines already.

[Chris Swires]

Thanks for the reply, I ran the tests again a good 40 times this morning and got largely the same results. Monitoring the ram usage it was fairly constant at around 24%. 

I did however run the same tests using httperf (I usually use ab), and here I encountered the same issues as you so it may possibly be an issue with this rather than Yesod that is causing the effect you described? 

Try running the tests with ab only maybe and see what you come up with? 

Thanks again for the reply, 

Best, 

Chris. 

[Smith]

Did you actually run the tests back to back?

I had a ruby script that was essentially

while true do

  `ab -n 10000 -c 100 [ip-address-here]`

end

It allowed me to see the speed of requests being processes although it's somewhat qualitative. If you did then hmm... I have no idea then.

[Chris Swires]

I did, although a used an sh script and allowed a 10 second wait period between tests so perhaps this is where the difference between our results lie (mine obviously being far less stressful). This was mostly so as not to over tax the ec2 as it is still hosting a live part of my site. 

I will fire up another ec2 later on and run them back to back to see if the results then correspond with yours. 

Thanks again, 

Chris. 

[Chris Swires]

I do in fact get similar results to yours, the memory usage grows fairly steadily until around the 50% mark (using the static test), and gradually slowing from around 8k rps to 2k, it then resets and the cycle completes. 

I presume this is in fact as you previously stated an issue with the garbage collection of GHC though I'm hardly qualified to give a definitive answer. 

This however does not explain my varied results as the runs are infrequent enough to be unaffected by this issue. 

As a side note I ran the same tests using httperf and ab. While ab consumed memory slower (and thus the cycle was extended), the problem was effectively the same, so disregard my previous hypothesis.  

Best, 

Chris. 

[Smith]

Ah.  Thanks for confirming.  Glad to know it's not just me.

Anyway, that's the only explanation I got, which is shaky at best.  Maybe someone else can chime in.

On an unrelated note, while rereading my previous posts, I noticed my writing is absolutely terrible.  I should actually proofread. :-/