Databen.ch - Persistence Benchmark for the JVM

[Flavio W. Brasil]

http://databen.ch

The first result of the databen.ch project is available. It uses a simple financial system model to test persistence frameworks scalability and consistency. The project core is implemented in Scala.

An Amazon EC2 hi1.4xlarge instance is used for the execution and the AMI (Amazon Image) is published with the results.

There are implementations for the Java frameworks: BatooJpa, Db4o, Ebean, EcliseLink, Hibernate and Prevayler. Also for the Scala frameworks: Activate, Slick and Squeryl.

The project is open to receive pull requests and suggestions to enhance the current implementations.

-- 

Flávio W. Brasil

Databen.ch contributor

[Alec Zorab]

No doubt you'll get dozens of people weighing in on this, but glancing at your slick implementation, I'd have thought you could get a significant performance increase simply by changing line 85 from

         for (account <- Accounts if (account.id === id))

to

         for (account <- Accounts if (account.id === id.bind))

Though I'm not in a position to test that.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Stefan Zeiger]

On 2013-04-09 16:02, Alec Zorab wrote:

No doubt you'll get dozens of people weighing in on this, but glancing at your slick implementation, I'd have thought you could get a significant performance increase simply by changing line 85 from

         for (account <- Accounts if (account.id === id))

to

         for (account <- Accounts if (account.id === id.bind))

Though I'm not in a position to test that.

I haven't run the benchmark yet but I looked at some of the implementations and I expect the major source of Slick's bad performance is that it's very heavy on updates and Slick needs to recompile two update queries per transaction. "select" queries can be precompiled but updates still cannot. The ticket for that is here: https://github.com/slick/slick/issues/17

-sz

[Flavio W. Brasil]

Hi Alec.

I tested the modification and in my machine it not modified the results. Maybe it can be reflected on the next official execution. I've just integrated it in the repo.

-- 

Flávio W. Brasil

[Flavio W. Brasil]

Hi Stefan,

I did some profiling during the Slick implementation. Looks like the absence of an optimistic offline locking mechanism and caches are the main scalability limitations.

Would be nice if Slick provided an out-of-the-box connection pool mechanism too. Probably people are getting in trouble using Slick without a connection pool.

Please feel free to send suggestions of enhancement or pull requests.

Regards,

-- 

Flávio W. Brasil

--

[Fernando Racca]

Great work!

Looking forward for Slick connection pool features too.

[Alois Cochard]

I'm working on a connection pool implementation that use Akka :)

That benchmark will be a great way to check performance of my stuff!

On Tuesday, 9 April 2013 17:35:15 UTC+1, fwbr...@gmail.com wrote:

[Alois Cochard]

Just to put things in perspective, it should be noted that the author of the benchmark is the author of the Activate Framework.

This seriously question the quality of the current implementation.

Just saying but a disclosure on the databen.ch website would be appreciated.

[anli]

On Thursday, April 11, 2013 4:55:40 PM UTC+4, Alois Cochard wrote:

This seriously question the quality of the current implementation.

I'm sure such words must not exist without technical prove. Please, supply one and retry to doubt about quality. And, by the way, bench author asks all interested developers to supply pull requests. Have you missed this suggestion?

[Alois Cochard]

No I haven't missed that suggestion and I don't understand why you wanted to bring it here, as I said I wanted to put things into perspective because I'm very surprised the author of the benchmark didn't realize it would be good to be clear about his role into one of the solution tested.

If I'm question the qualitiy of the implementation is not because of a technical fact, but simply because history showed me that benchmark can be easily biased in favor of one implementation, specially when the author of the benchmark is actually the author of one of the solution. I'm sure you can understand this skepticism.

Of course, I'll hope you'll get PR from different specialist to improve other implementations, in the mean time I think it would be fair to say that the initial version was created by Activate author, and I really wonder why it wasn't done in first place.

[Flavio W. Brasil]

Hi Alois,

Let me know if you have any technical argument about the benchmark. 

I will not discuss this type of tentative to discredit the project by non-technical arguments.

-- 

Flávio W. Brasil

[Andrew Gaydenko]

On Thursday, April 11, 2013 5:20:04 PM UTC+4, Alois Cochard wrote:

...and I really wonder why it wasn't done in first place.

I guess the intention is to get neutral trait of the started bench project. OTOH we all know here who is who ;) Moreover, the project is a continuation of this interesting discussion: https://groups.google.com/forum/#!folder/last/scala-user/Xc2NlYgNP7w Unfortunately both TypeSafe and Akka teams' memebers are not too open and active in discussions related to performance. Probably I'm wrong here, but moderateness of the core teams doesn't help in decision making from our, Scala users/lovers side.

[√iktor Ҡlang]

Wdym?

The only reason I haven't replied to the latest email in there is because I'll have to find a time slot where I can go analyze things in full.

I'm pretty sure it's OK for me to do due diligence.

Cheers,

√

 

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Viktor Klang
Director of Engineering

Typesafe

Twitter: @viktorklang

[Alois Cochard]

On Thursday, 11 April 2013 14:55:17 UTC+1, Andrew Gaydenko wrote:

OTOH we all know here who is who ;)

I'm afraid I don't, and I wouldn't expect all reader of this mailling list to know that Flavio is Activate author. 

On Thursday, 11 April 2013 14:45:21 UTC+1, fwbr...@gmail.com wrote:

I will not discuss this type of tentative to discredit the project by non-technical arguments.

Thanks you, I appreciate your open-minded approach to debate and the fact you have carefully evaluated my request of adding a disclosure. 

I feel even more motivated into spending time helping you making databench the most objective benchmark out there!

I wish you best of luck with your projects and looking forward contributing to databench.

[Andrew Gaydenko]

On Thursday, April 11, 2013 6:01:43 PM UTC+4, √iktor Klang wrote:

The only reason I haven't replied to the latest email in there is because I'll have to find a time slot where I can go analyze things in full.

Viktor, great, thanks, I'm sure not to be the only waiting for your (and other team members') comments.
I just remember you to be extremely fast and sufficiently elaborative in discussions. I see, just not this time :)

Saying about openness I mean something like was at those times when every scala developer cried because of scalac "speed". Now scalac (more strictly - official development environment in a whole) is much closer to be acceptable. Yes, it is indeed very big and important Scala's step. Look around now.. We all see signs of developers'  discomposure related to (base on) Scala code runtime performance. Probably I'm too naive (and too old to want to look smart), but I'm waiting from Scala/TypeSafe team side something like "yes, we have more rapid compiler now, lets make our products - play, slick, ... - really rapid also".

It would be the last step to dominate all over the world :)

[Stefan Zeiger]

To add my two cents to this discussion: I appreciate the efforts of keeping this benchmark open and welcoming improvements from others. I also had a quick look at some of the implementations and they seem to be generally well done. But there's still the question of the design of the benchmark: This is not a generic benchmarking suite but a test for a rather specific use case (highly contended atomic updates to a simple data model). There's nothing wrong with that per se and maybe it wasn't even a conscious decision, but I think people should know on whose turf they are competing (even if the rules of the competition are designed fairly).

(Disclaimer: I am the tech lead for Slick, one of the frameworks being benchmarked)

[√iktor Ҡlang]

On Thu, Apr 11, 2013 at 4:54 PM, Andrew Gaydenko <andrew....@gmail.com> wrote:

On Thursday, April 11, 2013 6:01:43 PM UTC+4, √iktor Klang wrote:

The only reason I haven't replied to the latest email in there is because I'll have to find a time slot where I can go analyze things in full.

Viktor, great, thanks, I'm sure not to be the only waiting for your (and other team members') comments.
I just remember you to be extremely fast and sufficiently elaborative in discussions. I see, just not this time :)

Just bad timing atm :)

 

Saying about openness I mean something like was at those times when every scala developer cried because of scalac "speed". Now scalac (more strictly - official development environment in a whole) is much closer to be acceptable. Yes, it is indeed very big and important Scala's step. Look around now.. We all see signs of developers'  discomposure related to (base on) Scala code runtime performance. Probably I'm too naive (and too old to want to look smart), but I'm waiting from Scala/TypeSafe team side something like "yes, we have more rapid compiler now, lets make our products - play, slick, ... - really rapid also".  

It would be the last step to dominate all over the world :)

Is that a Promise[true.type]? :-)

Cheers,

√

 

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Tim Pigden]

From a punter's perspective, the more and more diverse benchmarks the better. So if you were to concoct a benchmark that favoured Slick it would also be very useful. What we need to do is know when a particular flavour of persistence is going to be strong so we can extrapolate onto our own use cases and start making useful decisions. Knowing who favours what benchmark is actually pretty useful (so Flavio, a declaration of your interest would be useful, not just allay Alois suspicions, but as a message as to where you think your stuff is targetted).

Let's face it - none of the Scala ORM/persistence mechanisms are big-budget developments so they are all going to be biased in favour of some type of operation. 

Naturally I'd expect people to produce benchmarks that reflect their own domain of interest - and that's hopefully going to show their solutions to best advantage.

Alois - I think you're being a bit unfair on Flavio here. You may not realise he's the author of Activate - but I'm pretty sure most interested parties did - and simply clicking on his email for github on the last page of the benchmark does indeed take you right there.

Tim 

--
Tim Pigden
Optrak Distribution Software Limited
+44 (0)1992 517100
http://www.linkedin.com/in/timpigden
http://optrak.com
Optrak Distribution Software Ltd is a limited company registered in England and Wales.
Company Registration No. 2327613 Registered Offices: Orland House, Mead Lane, Hertford, SG13 7AT England 
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Optrak Distribution Software Ltd. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.

[Alois Cochard]

On Thursday, 11 April 2013 16:34:38 UTC+1, Tim P wrote:

From a punter's perspective, the more and more diverse benchmarks the better. So if you were to concoct a benchmark that favoured Slick it would also be very useful. What we need to do is know when a particular flavour of persistence is going to be strong so we can extrapolate onto our own use cases and start making useful decisions. Knowing who favours what benchmark is actually pretty useful (so Flavio, a declaration of your interest would be useful, not just allay Alois suspicions, but as a message as to where you think your stuff is targetted).

Let's face it - none of the Scala ORM/persistence mechanisms are big-budget developments so they are all going to be biased in favour of some type of operation. 

Naturally I'd expect people to produce benchmarks that reflect their own domain of interest - and that's hopefully going to show their solutions to best advantage.

Alois - I think you're being a bit unfair on Flavio here. You may not realise he's the author of Activate - but I'm pretty sure most interested parties did - and simply clicking on his email for github on the last page of the benchmark does indeed take you right there.

I agree with you except for the last point obviously ;)

I realized Flavio was the Activate author after seeing him promoting Activate on twitter by responding to someone complaining about Slick performance shown in this benchmark...

I don't see what's unfair here about excepting such information to be shown on the 'fairness' page:

http://databen.ch/fairness/

Or in the email sent here...

I don't think one should have to search for this information, this is the kind of things someone doing a benchmark *must* mention... but you know, that's just my opinion ;)

I think I make it pretty clear and I don't want to argue about it because it just so obvious to me.

BTW, I clearly agree with you on other points and such benchmarks are always useful

[Andrew Gaydenko]

Flavio, by the way, TypeSafe repo "Typesafe Repo" at "http://repo.typesafe.com/typesafe/repo/" has got last postgresql driver "postgresql" % "postgresql" % "9.2-1002.jdbc4"

[Flavio W. Brasil]

Alois,

I am really starting to doubt about your real intention with this kind of nasty information. The person was complaining about the Slick features, not about the benchmark result:

Declan Conlon @possiblywrong

Scala Slick, possibly the worst library I have ever used. Can't do anything beyond simple selects and deletes. Useless.

And I suggested to try Activate.

-- 

Flávio W. Brasil

[Alois Cochard]

On Thursday, 11 April 2013 17:45:55 UTC+1, fwbr...@gmail.com wrote:

Alois,

I am really starting to doubt about your real intention with this kind of nasty information. The person was complaining about the Slick features, not about the benchmark result:

Declan Conlon @possiblywrong

Scala Slick, possibly the worst library I have ever used. Can't do anything beyond simple selects and deletes. Useless.

 

My bad, sorry for that. I miss-understood the original tweet complain, but still it's how I realized you was doing promotion for Activate on tweeter, and what made me question about the objectivity of the benchmark.

My intention are obviously to find the best possible object mapping solution on the JVM platform! and thanks to databench project, I'll be able to do it :-)

But you are definitely right about questioning my intention, as I clearly question yours.

[Flavio W. Brasil]

Thanks Andrew. I will update.

-- 

Flávio W. Brasil

On Thursday, 11 de April de 2013 at 13:05, Andrew Gaydenko wrote:

Flavio, by the way, TypeSafe repo "Typesafe Repo" at "http://repo.typesafe.com/typesafe/repo/" has got last postgresql driver "postgresql" % "postgresql" % "9.2-1002.jdbc4"

[Naftoli Gugenheim]

Flavio, I'm not sure why you think Alois was trying to discredit anything. It's a normal thing to include a "full disclosure" note. It doesn't mean you are saying you are biased. It just means that people don't get the impression that you're hiding anything. Even if that impression is wrong, people will feel that way. It's called transparency.

[Flavio W. Brasil]

Hi Naftoli.

Alois was trying to discredit the benchmark. He said:

This seriously question the quality of the current implementation.

I couldn't continue the discussion with him due the nature of his argument.

I understand the transparency question. First, we thought that the github link on the "Community" page was enough, but, after some suggestions, the authors information was added to the "Fairness" page.

Best regards,

-- 

Flávio W. Brasil

[Alois Cochard]

On Friday, 12 April 2013 01:39:10 UTC+1, fwbr...@gmail.com wrote:

Hi Naftoli.

Alois was trying to discredit the benchmark. He said:

This seriously question the quality of the current implementation.

I couldn't continue the discussion with him due the nature of his argument.

I understand the transparency question. First, we thought that the github link on the "Community" page was enough, but, after some suggestions, the authors information was added to the "Fairness" page.

I'm glad you took into account my suggestion even if you can't discuss with me anymore, sorry for that, I didn't expected you'll react such negatively on me sharing my opinion.

Cool stuff, the update of the 'fairness' page! Thanks to that I now discovered that the other contributor of databench is the author of Prevayler, that's kind of funny :-)

You see this is definitely useful, specially for people like me non-familiar with those projects, I really appreciate the effort you did to make databench more transparent. If only the pharmaceutical industry was as concern about transparency as you are!

Thanks

 

Best regards,

-- 

Flávio W. Brasil

On Thursday, 11 de April de 2013 at 21:06, Naftoli Gugenheim wrote:

Flavio, I'm not sure why you think Alois was trying to discredit anything. It's a normal thing to include a "full disclosure" note. It doesn't mean you are saying you are biased. It just means that people don't get the impression that you're hiding anything. Even if that impression is wrong, people will feel that way. It's called transparency.

On Thursday, April 11, 2013, Flavio W. Brasil wrote:

Hi Alois,

Let me know if you have any technical argument about the benchmark. 

I will not discuss this type of tentative to discredit the project by non-technical arguments.

-- 

Flávio W. Brasil

On Thursday, 11 de April de 2013 at 10:20, Alois Cochard wrote:

On Thursday, 11 April 2013 14:01:59 UTC+1, Andrew Gaydenko wrote:

On Thursday, April 11, 2013 4:55:40 PM UTC+4, Alois Cochard wrote:

This seriously question the quality of the current implementation.

I'm sure such words must not exist without technical prove. Please, supply one and retry to doubt about quality. And, by the way, bench author asks all interested developers to supply pull requests. Have you missed this suggestion?

No I haven't missed that suggestion and I don't understand why you wanted to bring it here, as I said I wanted to put things into perspective because I'm very surprised the author of the benchmark didn't realize it would be good to be clear about his role into one of the solution tested.

If I'm question the qualitiy of the implementation is not because of a technical fact, but simply because history showed me that benchmark can be easily biased in favor of one implementation, specially when the author of the benchmark is actually the author of one of the solution. I'm sure you can understand this skepticism.

Of course, I'll hope you'll get PR from different specialist to improve other implementations, in the mean time I think it would be fair to say that the initial version was created by Activate author, and I really wonder why it wasn't done in first place.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scala-user+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "scala-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scala-user+unsubscribe@googlegroups.com.