speed.rubyspec.org has launched

24 views
Skip to first unread message

Chuck Remes

unread,
Jun 20, 2011, 11:08:16 PM6/20/11
to code...@googlegroups.com
Many thanks to the folks on this list (and Miquel) for help in getting acquainted with codespeed. It is a really neat little tool.

Please check out the site at http://speed.rubyspec.org and look around. There isn't much data posted yet. The EC2 instance is churning through a few commits I cherry-picked from early May through the present day. It is working on today's commit(s) first and will eventually chug through the rest of the backlog.

When clicking on the Timeline tab, it is going to load up a *lot* of benchmarks. The Ruby world has a lot of low-level benchmarks so we can track the perf of each method in the core (and eventually the standard library). We even have separate benchmarks for the same method but with different arguments (e.g. "a sample string".split(' ') and "a sample string".split('|')) so we can test average and worst case scenarios.

As you can see from that page, it really stresses your browser's javascript engine. :)

Again, thanks!

cr

Stefan Marr

unread,
Jun 21, 2011, 5:13:29 AM6/21/11
to codespeed
Hi:

On Jun 21, 5:08 am, Chuck Remes <cremes.devl...@mac.com> wrote:
> Please check out the site at http://speed.rubyspec.organd look around. There isn't much data posted yet. The EC2 instance is churning through a few commits I cherry-picked from early May through the present day. It is working on today's commit(s) first and will eventually chug through the rest of the backlog.
>
Interesting approach with EC2. What are the guarantees you get there?
Are you guaranteed to have your VM image to be scheduled always on the
same machine? And always access to the same resources i.e. memory and
CPU time?




> When clicking on the Timeline tab, it is going to load up a *lot* of benchmarks.
Indeed, Chrome asked my twice and then I canceled. Safari actually
showed the webpage.
Think, there is some performance tuning necessary, otherwise many of
your users will just not be patient enough.

In case you are doing something in that direction, I would be
interested, since I also have a lot benchmarks and performance
troubles.


Best regards
Stefan

Chuck Remes

unread,
Jun 21, 2011, 7:31:07 AM6/21/11
to code...@googlegroups.com
On Jun 21, 2011, at 4:13 AM, Stefan Marr wrote:

> Hi:
>
> On Jun 21, 5:08 am, Chuck Remes <cremes.devl...@mac.com> wrote:
>> Please check out the site at http://speed.rubyspec.organd look around. There isn't much data posted yet. The EC2 instance is churning through a few commits I cherry-picked from early May through the present day. It is working on today's commit(s) first and will eventually chug through the rest of the backlog.
>>
> Interesting approach with EC2. What are the guarantees you get there?
> Are you guaranteed to have your VM image to be scheduled always on the
> same machine? And always access to the same resources i.e. memory and
> CPU time?

The EC2 instance was donated by a member of the community. I don't know what kinds of guarantees it offers, if any. I've heard EC2 performance can fluctuate as much as 20% over time, but for the purposes of tracking perf trends across runtimes I still think it's sufficient. If any major change shows up, interested developers can zero in on it and see if it's a real problem or just an artifact of EC2 usage.


>> When clicking on the Timeline tab, it is going to load up a *lot* of benchmarks.
> Indeed, Chrome asked my twice and then I canceled. Safari actually
> showed the webpage.
> Think, there is some performance tuning necessary, otherwise many of
> your users will just not be patient enough.

I don't even know where to begin. When I click on that tab, I see that python goes to 100% for a period of time. I don't know how to improve that. It looks to me like a *browser* performance issue.

We are using the built-in web server for sending out the results. I have run out of time to figure out how to do a deployment with apache/nginx/lighttpd/etc so it will have to do for now.

If people who are interested in the data cancel the operation before it completes, then perhaps they aren't *really interested* in the data. I don't imagine it will be a high traffic site except for an initial flurry of activity now that it has launched.

> In case you are doing something in that direction, I would be
> interested, since I also have a lot benchmarks and performance
> troubles.

In a few weeks I hope to have time to fix up some of the things where I had to take a short-cut. I am also hoping that our experiences and results will turn into a few issues that can be tackled by members of the codespeed community who are familiar with the code base. In particular, I think it would be neat to have "collapsable" views for the Timeline tab where we can define a hierarchy for the benchmarks and by default they are hidden unless someone clicks on them to expand (think of the List view in MacOSX Finder).

e.g.
core/array/bench_append
core/array/bench_dup
core/array/bench_pack
core/array/bench_map
core/array/bench_sort
core/hash/bench_dup
core/hash/bench_sort

these would collapse to

> core

then expand to

>core->array
>core->hash

and again to

>core->array
bench_append
bench_dup
etc
>core->hash
bench_dup
bench_sort


Like I said, I'll start to file issues on these kinds of things in a few weeks. I want to get a little distance from the problem so that I can see the forest for the trees before I start suggesting a bunch of enhancements/changes to the UI.


cr

Frank Becker

unread,
Jun 21, 2011, 8:13:39 AM6/21/11
to code...@googlegroups.com
Chuck Remes [2011-06-21, 13:31 +0200]:

> On Jun 21, 2011, at 4:13 AM, Stefan Marr wrote:
>

Hi,

> > On Jun 21, 5:08 am, Chuck Remes <cremes.devl...@mac.com> wrote:
> >> Please check out the site at http://speed.rubyspec.organd look around.
> >> There isn't much data posted yet. The EC2 instance is churning through
> >> a few commits I cherry-picked from early May through the present day.
> >> It is working on today's commit(s) first and will eventually chug
> >> through the rest of the backlog.
> >>

[...]


> >> When clicking on the Timeline tab, it is going to load up a *lot* of
> >> benchmarks.
> > Indeed, Chrome asked my twice and then I canceled. Safari actually
> > showed the webpage. Think, there is some performance tuning necessary,
> > otherwise many of your users will just not be patient enough.
>
> I don't even know where to begin. When I click on that tab, I see that
> python goes to 100% for a period of time. I don't know how to improve
> that. It looks to me like a *browser* performance issue.

>
> We are using the built-in web server for sending out the results. I have

That is a built-in development server. It is single threaded and *only* for
development! IOW it is not able to serve more than one client at a time.

> run out of time to figure out how to do a deployment with
> apache/nginx/lighttpd/etc so it will have to do for now.

I'm happy to help w/ Apache or nginx. Tell me what you prefer I explain how
to configure it using WSGI.

> > In case you are doing something in that direction, I would be
> > interested, since I also have a lot benchmarks and performance
> > troubles.

Can you please provide a little more details? Thanks.

Bye,

Frank

--
Frank Becker <f...@alien8.de> (jabber|mail) | http://twitter.com/41i3n8
GnuPG: 0xADC29ECD | F01B 5E9C 1D09 981B 5B40 50D3 C80F 7459 ADC2 9ECD
SILC-Net: a8 | Home: http://www.alien8.de | <<</>> http://www.c3d2.de
"> Freedom is just chaos, with better lighting. <" Alan Dean Foster

signature.asc

Chuck Remes

unread,
Jun 21, 2011, 9:00:25 AM6/21/11
to code...@googlegroups.com
On Jun 21, 2011, at 7:13 AM, Frank Becker wrote:

> Chuck Remes [2011-06-21, 13:31 +0200]:
>> On Jun 21, 2011, at 4:13 AM, Stefan Marr wrote:
>>
>
> Hi,
>
>>> On Jun 21, 5:08 am, Chuck Remes <cremes.devl...@mac.com> wrote:
>>>> Please check out the site at http://speed.rubyspec.organd look around.
>>>> There isn't much data posted yet. The EC2 instance is churning through
>>>> a few commits I cherry-picked from early May through the present day.
>>>> It is working on today's commit(s) first and will eventually chug
>>>> through the rest of the backlog.
>>>>
> [...]
>>>> When clicking on the Timeline tab, it is going to load up a *lot* of
>>>> benchmarks.
>>> Indeed, Chrome asked my twice and then I canceled. Safari actually
>>> showed the webpage. Think, there is some performance tuning necessary,
>>> otherwise many of your users will just not be patient enough.
>>
>> I don't even know where to begin. When I click on that tab, I see that
>> python goes to 100% for a period of time. I don't know how to improve
>> that. It looks to me like a *browser* performance issue.
>
>>
>> We are using the built-in web server for sending out the results. I have
> That is a built-in development server. It is single threaded and *only* for
> development! IOW it is not able to serve more than one client at a time.

It hasn't been a problem yet! :)

>> run out of time to figure out how to do a deployment with
>> apache/nginx/lighttpd/etc so it will have to do for now.
> I'm happy to help w/ Apache or nginx. Tell me what you prefer I explain how
> to configure it using WSGI.

I would prefer nginx. Telling me all of the right components to 'apt-get' would be great, plus which files I should touch for codespeed/django to use it.

>>> In case you are doing something in that direction, I would be
>>> interested, since I also have a lot benchmarks and performance
>>> troubles.
> Can you please provide a little more details? Thanks.

That isn't my quote; it was from Stefan. Hopefully he will elaborate.

cr

Stefan Marr

unread,
Jun 21, 2011, 10:33:49 AM6/21/11
to codespeed
Hi:

On Jun 21, 2:13 pm, Frank Becker <f...@alien8.de> wrote:
> > On Jun 21, 2011, at 4:13 AM, Stefan Marr wrote:
> > > In case you are doing something in that direction, I would be
> > > interested, since I also have a lot benchmarks and performance
> > > troubles.
>
> Can you please provide a little more details? Thanks.

Well, I am benchmarking our RoarVM and have scaling benchmarks from 1
to 59 cores. I don't do exactly 59 benchmarks, but a few core
configurations and a few different benchmark types. That multiplies...

So, the idea of grouping would be a nice one, perhaps picking one
representative benchmark of a group as an entry point.

Another idea would be to try to change the rendering process.

I already forgot how it works currently, but I think I remember a few
for-loops going and setting up the data for rendering.
Perhaps we could change that to have it done asynchronously, step-wise
somehow. So, that the display is updated after a new plot is rendered.
Impatience is a subtle problem, especially if the user does not know
what to expect and just thinks that the damn page is crashing his
browser.

Miquel, any ideas?

I think I should be able to spend some time on codespeed this week,
since we are setting up a 'production' deployment here.
Will also take the chance to update to the latest code base and
convert my modifications to feature branches.

Best regards
Stefan

--
Stefan Marr
Software Languages Lab
Vrije Universiteit Brussel
Pleinlaan 2 / B-1050 Brussels / Belgium
http://soft.vub.ac.be/~smarr
Phone: +32 2 629 2974
Fax: +32 2 629 3525

Miquel Torres

unread,
Jun 21, 2011, 3:09:22 PM6/21/11
to code...@googlegroups.com
Hi all,

great to see you got that Speed center up and running. Congrats!

First of all, I think it is terrific that Chuck is bringing up all
this issues (instead of just giving up and not providing feedback),
and that you all are helping out in finding solutions.

It is clear that Codespeed wasn't designed to do well beyond a number
of parameters (exes*envs*benchs < 50 maybe), neither UI-wise, nor
performance-wise. With enough users and contributors now, I think we
have the chance to rethink how we could improve on that so that it
covers most use-cases. Chuck's "benchmark grouping" idea and Stefan
progressive rendering are going in the right direction, IMHO.

I'll release 0.8 shortly and use the chance to start the 0.9 release
cycle discussion.

Cheers!

PS:@Frank: and if you have any questions while helping out Chuck, just ask!

2011/6/21 Stefan Marr <goo...@stefan-marr.de>:

Steffen Schwigon

unread,
Jun 22, 2011, 12:26:29 AM6/22/11
to code...@googlegroups.com
Miquel Torres <tob...@googlemail.com> writes:
> I'll release 0.8 shortly and use the chance to start the 0.9 release
> cycle discussion.

Can you please have a look whether a8's Django template stuff
refactoring discussed in

https://github.com/tobami/codespeed/issues/25

can be merged before 0.8?

I haven't seen a pull request for that, or maybe it already is merged,
not sure, anyway, at least I'm interested in those for my setup.

Thanks.

Kind regards,
Steffen
--
Steffen Schwigon <s...@renormalist.net>

Steffen Schwigon

unread,
Jun 22, 2011, 12:31:21 AM6/22/11
to code...@googlegroups.com
Steffen Schwigon <s...@renormalist.net> writes:
> Can you please have a look whether a8's Django template stuff
> refactoring discussed in
>
> https://github.com/tobami/codespeed/issues/25
>
> can be merged before 0.8?

Ha, I should read all emails before answer. :-)
I just read it's not yet in 0.8 but planned for 0.9.

Frank Becker

unread,
Jun 22, 2011, 4:04:40 AM6/22/11
to code...@googlegroups.com
On 6/22/11 6:31 AM, Steffen Schwigon wrote:
> Steffen Schwigon <s...@renormalist.net> writes:

Hi,

>> Can you please have a look whether a8's Django template stuff
>> refactoring discussed in
>>
>> https://github.com/tobami/codespeed/issues/25
>>
>> can be merged before 0.8?
>
> Ha, I should read all emails before answer. :-)
> I just read it's not yet in 0.8 but planned for 0.9.

And, as the ticket reads I keep that branch
https://github.com/a8/codespeed-a8/tree/separation
rebased on top of tobami's master.

Thanks,

signature.asc

Frank Becker

unread,
Jun 22, 2011, 9:00:01 AM6/22/11
to code...@googlegroups.com
On 6/21/11 3:00 PM, Chuck Remes wrote:

Hi,

>>> run out of time to figure out how to do a deployment with
>>> apache/nginx/lighttpd/etc so it will have to do for now.
>> I'm happy to help w/ Apache or nginx. Tell me what you prefer I explain how
>> to configure it using WSGI.
>
> I would prefer nginx. Telling me all of the right components to 'apt-get' would be great, plus which files I should touch for codespeed/django to use it.

Please have a look at:
https://github.com/a8/codespeed-a8/blob/separation/example/README.md
section: Installing for production
and tell me at which step you might fail or what has to be documented
differently/better.

Thanks,

signature.asc

Chuck Remes

unread,
Jun 22, 2011, 9:58:04 AM6/22/11
to code...@googlegroups.com
On Jun 22, 2011, at 8:00 AM, Frank Becker wrote:

> On 6/21/11 3:00 PM, Chuck Remes wrote:
>
> Hi,
>
>>>> run out of time to figure out how to do a deployment with
>>>> apache/nginx/lighttpd/etc so it will have to do for now.
>>> I'm happy to help w/ Apache or nginx. Tell me what you prefer I explain how
>>> to configure it using WSGI.
>>
>> I would prefer nginx. Telling me all of the right components to 'apt-get' would be great, plus which files I should touch for codespeed/django to use it.
> Please have a look at:
> https://github.com/a8/codespeed-a8/blob/separation/example/README.md
> section: Installing for production
> and tell me at which step you might fail or what has to be documented
> differently/better.

Frank,

I've gotten stuck right away.

I can't accomplish step #3 because I cannot find any directory 'deploy' with the nginx.default-site.conf file. Also, my codespeed directory doesn't have a directory called sitestatic (I can't find that anywhere).

Step #5 doesn't work. There is no command 'collectstatic' that is accepted by manage.py.

Step #6 doesn't work unless 'gunicorn' is added to the INSTALLED_APPS array in settings.py.


Perhaps your fork has diverged a bit more from the codespeed master than you recalled?

cr

Chuck Remes

unread,
Jun 22, 2011, 10:11:10 AM6/22/11
to code...@googlegroups.com

I see that a number of these things only exist on the 'separation' branch. Going through it now.

cr

Frank Becker

unread,
Jun 22, 2011, 10:29:46 AM6/22/11
to code...@googlegroups.com
On 6/22/11 3:58 PM, Chuck Remes wrote:
> On Jun 22, 2011, at 8:00 AM, Frank Becker wrote:
>
>> On 6/21/11 3:00 PM, Chuck Remes wrote:

Hi,

>>>>> run out of time to figure out how to do a deployment with
>>>>> apache/nginx/lighttpd/etc so it will have to do for now.
>>>> I'm happy to help w/ Apache or nginx. Tell me what you prefer I explain how
>>>> to configure it using WSGI.
>>>
>>> I would prefer nginx. Telling me all of the right components to 'apt-get' would be great, plus which files I should touch for codespeed/django to use it.
>> Please have a look at:
>> https://github.com/a8/codespeed-a8/blob/separation/example/README.md
>> section: Installing for production
>> and tell me at which step you might fail or what has to be documented
>> differently/better.
>
> Frank,
>
> I've gotten stuck right away.

Sorry, right. My fault. All those files are in that branch [0] on
github. The purpose of that branch is to "decouple" the codespeed app
(Django wording for [extension|plugin|module|...]) That makes it way
more simple to integrate codespeed into other Django projects - among
other advantages. History of that is in ticket #25 [1]. One concern was
that it may be more difficult to set up a new speedcenter site from
scratch. Hence, I tried to write up some install docu.

The part that describes how to configure Nginx or Apache should work on
the current v0.8 tobmai/master.

To get the files please check out my tree:

git clone g...@github.com:a8/codespeed-a8.git
cd codespeed-a8
git checkout -b separation origin/separation
cd example/deploy

OR

find the files on github via the Web:
https://github.com/a8/codespeed-a8/tree/separation/example/deploy


[0] https://github.com/a8/codespeed-a8/tree/separation
[1] https://github.com/tobami/codespeed/issues/25


Good luck,

signature.asc

Frank Becker

unread,
Jun 22, 2011, 10:44:55 AM6/22/11
to code...@googlegroups.com
On 6/22/11 4:29 PM, Frank Becker wrote:
> On 6/22/11 3:58 PM, Chuck Remes wrote:
>> On Jun 22, 2011, at 8:00 AM, Frank Becker wrote:
>>
>>> On 6/21/11 3:00 PM, Chuck Remes wrote:

Hi,

> The part that describes how to configure Nginx or Apache should work on
> the current v0.8 tobmai/master.
After a 2nd thought that is not right. Django v1.3 has a former Django
app called staticfiles [0] integrated. That deals w/ serving static
files (done by e. g. ngnix and not Django). One part of my branch is to
switch to use that.

You will need an alias for admin_media e. g. in Apache speak:

Alias /admin_media/ /usr/share/pyshared/django/contrib/admin/media/

and please ignore step:
5 Prepare static files cd /path/to/speedcenter/ python ./manage.py \
collectstatic


[0] https://docs.djangoproject.com/en/1.3/howto/static-files/

Good luck again,

signature.asc

Chuck Remes

unread,
Jun 22, 2011, 11:25:46 AM6/22/11
to code...@googlegroups.com
On Jun 22, 2011, at 9:44 AM, Frank Becker wrote:

> On 6/22/11 4:29 PM, Frank Becker wrote:
>> On 6/22/11 3:58 PM, Chuck Remes wrote:
>>> On Jun 22, 2011, at 8:00 AM, Frank Becker wrote:
>>>
>>>> On 6/21/11 3:00 PM, Chuck Remes wrote:
>
> Hi,
>
>> The part that describes how to configure Nginx or Apache should work on
>> the current v0.8 tobmai/master.
> After a 2nd thought that is not right. Django v1.3 has a former Django
> app called staticfiles [0] integrated. That deals w/ serving static
> files (done by e. g. ngnix and not Django). One part of my branch is to
> switch to use that.
>
> You will need an alias for admin_media e. g. in Apache speak:
>
> Alias /admin_media/ /usr/share/pyshared/django/contrib/admin/media/
>
> and please ignore step:
> 5 Prepare static files cd /path/to/speedcenter/ python ./manage.py \
> collectstatic
>
>
> [0] https://docs.djangoproject.com/en/1.3/howto/static-files/

Frank, I got allof it working with nginx. The hardest part of the instructions to understand was the static file stuff.

In my /etc/nginx/site-enabled/default file I made the path to my static files equal to /path/to/speedcenter/media/$1 and it's working okay. The site is a *lot* faster with this option enabled.

I ran into a bug with some python library (python_meld?) and supervisor, so I switched over to using a shell script and upstart to start/stop it. It's working great. I now have 3 gunicorn workers to help spread the load.

Also, I found this blog post very helpful in understanding what I was doing:

http://davidpoblador.com/run-django-apps-using-gunicorn-and-nginx/

cr

Frank Becker

unread,
Jun 22, 2011, 11:41:13 AM6/22/11
to code...@googlegroups.com
On 6/22/11 5:25 PM, Chuck Remes wrote:

Hi,

> Frank, I got allof it working with nginx. The hardest part of the instructions to understand was the static file stuff.
Congratulations. OK, the static file stuff will come back.
I'll try to explain that a little more.

> In my /etc/nginx/site-enabled/default file I made the path to my static files equal to /path/to/speedcenter/media/$1 and it's working okay. The site is a *lot* faster with this option enabled.
>
> I ran into a bug with some python library (python_meld?) and supervisor, so I switched over to using a shell script and upstart to start/stop it. It's working great. I now have 3 gunicorn workers to help spread the load.

Debian or Ubuntu? Oh I see, Ubuntu 11.04.
The problem is in the xml module. I will see if there is a Ubuntu bug
open already tomorrow.

> Also, I found this blog post very helpful in understanding what I was doing:
>
> http://davidpoblador.com/run-django-apps-using-gunicorn-and-nginx/

:-) I've learned about supervisor from that page. Upstart is not
available everywhere.
The root directive in the ngnix config did not work for me though.


Bye,

signature.asc

Frank Becker

unread,
Jun 27, 2011, 11:26:58 AM6/27/11
to code...@googlegroups.com
On 6/22/11 5:25 PM, Chuck Remes wrote:
> On Jun 22, 2011, at 9:44 AM, Frank Becker wrote:
>
>> On 6/22/11 4:29 PM, Frank Becker wrote:
>>> On 6/22/11 3:58 PM, Chuck Remes wrote:
>>>> On Jun 22, 2011, at 8:00 AM, Frank Becker wrote:
>>>>
>>>>> On 6/21/11 3:00 PM, Chuck Remes wrote:

Hi,

> I ran into a bug with some python library (python_meld?) and supervisor, so I switched over to using a shell script and upstart to start/stop it. It's working great. I now have 3 gunicorn workers to help spread the load.
Just for completeness here is the Ubuntu bug for supervisor incl. some
work arounds:

https://bugs.launchpad.net/ubuntu/+source/supervisor/+bug/777862

Bye,

signature.asc
Reply all
Reply to author
Forward
0 new messages