How about a performance test suite for all Haxe frameworks

[Robert Konrad]

So Dmitry Hryppa is currently torturing lots of innocent bunnies to talk us all into improving the performance of our frameworks: http://themozokteam.com/playground/frameworkstest/ The end justifies the means they say and I tend to agree in this case. Drawing the same bunny over and over again is an interesting performance characteristic (because it resembles for example particle systems and tilemaps). And believe it or not, the biggest change that was made to draw more bunnies in Kha was a fix in the Haxe compiler for constructor inlining by Simon and everybody can now benefit from that.

But there are many many more performance characteristics of a media framework than just bunny drawing that shouldn't go untested. Also there are many more systems than a Windows PC with a beefy graphics card, that shouldn't go untested either.

So I discussed that a little with Lars Doucet already and I would like to suggest that we all work together to set up a big performance test suite for all Haxe frameworks. We should include Non-Haxe frameworks, too, just like Dmitry also does - I think that is really great, gives everybody a much better perspective about where we are performance wise.

The "test on many systems" part of that is kinda hard, but building the test suite itself - not so much. If Dmitry agrees we could just take his stuff, setup a Github project (maybe even let Dmitry manage that, if he would like to do that?)

 and take pull requests. We could basically implement a process where whenever somebody wants to test a specific performance characteristic of a framework (can very well be the framework developers themselves) sends in a pull request for that - and when the developers of some other framework want to participate in that specific test, they port it and send a pull request themselves. When all those tests just output a performance number when run, statistics could be created automatically for all tests and be updated regularly on a webpage.

So, how about that?

[Lars Doucet]

It sounds good to me; and honestly doesn't take a lot to get started. 

This method also seems to address the usual issues with isolated benchmarks (not providing source code for your tests, only running on one machine, only running one narrow benchmark that doesn't have much real-world relevance).

[Robert Konrad]

And thinking about the "many systems" aspect - if we can make it really easy to run the tests, we can maybe crowd source the results?

[Khaled Garbaya]

I really like the idea but it would be nice to mention the in which platform the test was done and for OpenFL benchmark was it normal display list or gpu accelerated display list

[Dmitry Hryppa]

Cool :)

No problem, I will publish all sources and binaries of my bunnymark on GitHub soon.

Will let you know guys when this mission will be complete.

[Dmitry Hryppa]

Okay, here is first part of my sources:

https://github.com/dmitryhryppa/XNA_Monogame_bunnies

https://github.com/dmitryhryppa/kha_bunnies

https://github.com/dmitryhryppa/openfl_bunnies

[Dmitry Hryppa]

And NME: 

https://github.com/dmitryhryppa/nme_bunnies

[Lars Doucet]

Great Dmitry!

Could you start a new repo that we can organize an entire benchmark suite in?

I figure it would look something like this:

(root repository)

--> bunnymark

-----> kha

-----> nme

-----> openfl
-----> heaps

-----> luxe

-----> (etc)

--> someOtherTest1

-----> kha

-----> nme

-----> openfl
-----> heaps

-----> luxe

-----> (etc)

--> someOtherTest2

-----> kha

-----> nme

-----> openfl
-----> heaps

-----> luxe

-----> (etc)

On Wednesday, August 19, 2015 at 3:54:30 PM UTC-5, Dmitry Hryppa wrote:

And NME: 

https://github.com/dmitryhryppa/nme_bunnies

[Dmitry Hryppa]

Sure :)

https://github.com/dmitryhryppa/Frameworks_test

четверг, 20 августа 2015 г., 0:02:47 UTC+3 пользователь Lars Doucet написал:

[Robert Konrad]

Wonderful. And off we go.

Next I think we should try to automate test runs as much as possible. For rendering tests we should measure time, so that would change the bunnymark slightly to something ala "rendering one million bunnies takes xx milliseconds". And beware - we want to also test stuff on JIT-Compilers, so warm up phases are important. Can prepare a pull request for the Kha bunnies.

[Justin L Mills]

It would be nice to see Starling AIR - Haxe Openfl, or AS3 versions within the tests, it's important that Adobe realize that Haxe is a good target for thier tools, such support would open many opportunities for developers still using AIR to allow them to use Haxe instead. FlashIDE is a brilliant tool which has always lacked export options.  It is also very important to smoothing integration of Haxe with photoshop and illustrator.   It's not that Haxe is not amazing despite corporate support, but, ...  it is often a limiting factor for developers, and limits coding choice for work projects, and more graphics support from Adobe, could make a real difference to the number of corporate and design agencies interested.

--
To post to this group haxe...@googlegroups.com
http://groups.google.com/group/haxelang?hl=en
---
You received this message because you are subscribed to the Google Groups "Haxe" group.
For more options, visit https://groups.google.com/d/optout.

[Lars Doucet]

Are you volunteering? :D

[Hugh]

I would also like to see stuff less related to the framework, and more about the target, eg a physical simulation.

So we can compare JS vs hxcpp vs java vs c# for this aspect, orthogonally to the rendering/framework.

This would make performance improvements to hxcpp easier and more likely to happen.

Hugh