python/ruby benchmark.
"</script>
http://shootout.alioth.debian.org/benchmark.php?test=all&lang=ruby&lang2=python&sort=fullcpu
and found the tests on ruby disapointing :(.
So what I'm asking for is a link to some ruby-specific optimisation tips.
--
Regards, Groleo!
# touch universe
# chmod +x universe
# ./universe
gabriele renzi
> I took a look at
> http://shootout.alioth.debian.org/benchmark.php?test=all&lang=ruby&lang2=python&sort=fullcpu
> and found the tests on ruby disapointing :(.
> So what I'm asking for is a link to some ruby-specific optimisation tips.
a simple thing: do the most normal thing instead of writing code in ruby
that is innatural, i.e. instead of rewritine an heapsort by hand, use
the builtin sort/sort!/sort_by routines. Instead of writing a random
function by hand, use the builtin rand().
If you really feel ruby (or python) is slow you can act later optimizing
it, not doing it before you ever need to be faster. "premature
optimization is the root of all evil", people say.
Glenn Parker
Avoid creating new objects, e.g. re-use strings and arrays, or modify
them in place when possible, especially large ones.
Use built-in iterators as much as possible.
Pray for YARV! :)
--
Glenn Parker | glenn.parker-AT-comcast.net | <http://www.tetrafoil.com/>
Ryan Leavengood
>
> If you really feel ruby (or python) is slow you can act later optimizing
> it, not doing it before you ever need to be faster. "premature
> optimization is the root of all evil", people say.
I heartily agree with this, and it doesn't just apply to Ruby and Python,
but every language. I used to find myself wasting a lot of time trying to
come up with some clever "fast" solution, when the reality is in almost
every conceivable case the "dumb" solution does the job very fast on
modern machines.
But Python does seem to be faster in most cases, and this is because the
Python interpreter uses bytecode while the current Ruby interpreter just
interprets the AST tree after parsing. But there is currently a lot of
impressive work going into a Ruby bytecode interpreter, called YARV:
http://www.atdot.net/yarv/
You can expect the Ruby interpreter that uses YARV to be 3 or 4 times
faster (I've compiled it and ran the benchmarks.)
Ryan
Vincent Foley
in Ruby. Also, the tests must use the exact same algorithm, so this
disadvantages Ruby is some situations. I say the test should compare
same algorithms, but also the one best suited to the language.
Lothar Scholz
gr> "</script> ha scritto:
>> I took a look at
>> http://shootout.alioth.debian.org/benchmark.php?test=all&lang=ruby&lang2=python&sort=fullcpu
>> and found the tests on ruby disapointing :(.
>> So what I'm asking for is a link to some ruby-specific optimisation tips.
gr> a simple thing: do the most normal thing instead of writing code in ruby
gr> that is innatural, i.e. instead of rewritine an heapsort by hand, use
gr> the builtin sort/sort!/sort_by routines. Instead of writing a random
gr> function by hand, use the builtin rand().
Sorry you don't understand this benchmark. It is there to do this
algorithms to compare the basic data operations.
The benchmark is not useable to compare compiled languages with
interpretered, but comparing python with ruby is a valid comparision.
And that ruby is so worse then python is a real problem. If you still
try to deny this, then you have a real mental problem.
If ruby is 9 times slower then python in for example function calling intensive
code that means that ruby will slow down your development time a lot
as it forces you to use optimized code or even C code much more often
then python.
--
Best regards, emailto: scholz at scriptolutions dot com
Lothar Scholz http://www.ruby-ide.com
CTO Scriptolutions Ruby, PHP, Python IDE 's
Kent Sibilev
web app using Rails is an order of magnitude faster than the
development of a similar web app using Struts/Hibernate frameworks.
There are other factors that can slow down your development time
besides the speed of your language of choice.
Cheers,
Kent
Just another guy with huge mental problems.
Dibya Prakash
I definitely agree.Only feature of a particular programming language
cannot determine the speed of software.Other factors are also
crucially important.
I didnot know about YARV,but gr8 that I came to know it thru u
people.I'll surely check it out.
Bye
Dibya Prakash
Happy coding
Lothar Scholz
KS> Java is an order of magnitude faster than Ruby. The development of a
KS> web app using Rails is an order of magnitude faster than the
KS> development of a similar web app using Struts/Hibernate frameworks.
KS> There are other factors that can slow down your development time
KS> besides the speed of your language of choice.
If i can't use the language because it is to slow to solve the problem
then the development time doesn't matter. There are just too many
problem domains left where you can't ignore it.
But we all hope and pray for YARV.
Ryan Leavengood
>
> If i can't use the language because it is to slow to solve the problem
> then the development time doesn't matter. There are just too many
> problem domains left where you can't ignore it.
I would like to hear some more specifics on these problem domains. I'm not
trying to start an argument, I'm just wondering if the problems you are
thinking of are just not appropriate for any interpreted language.
Ruby or Python or Java will never be as fast as C, and C in most cases is
not as fast as good hand-coded assembly. Any problem domain that requires
major number crunching will need to be coded in the most efficient
language, sometimes even requiring special hardware (like 3D graphics.)
Of course with modern computers and compilers, I doubt anyone would code
an entire application in assembly, but rather entirely in C or with just
the hot spots coded in assembly. The same philosophy applies when moving
to the higher level of Ruby, with C for the hot spots. I certainly
wouldn't try to write a web application with C these days.
In fact I would say that in most cases, for many problem domains, the huge
increase in productivity that you get with higher level languages totally
outweighs performance issues.
Ryan
Gavri Fernandez
> I would like to hear some more specifics on these problem domains. I'm not
> trying to start an argument, I'm just wondering if the problems you are
> thinking of are just not appropriate for any interpreted language.
So you're saying that when the performance requirements crosses a
certain threshold, interpreted languages should not be used.
The idea here is that if Ruby is faster, that threshold where should a
switch needs to be made is shifted.
--
Gavri
http://livejournal.com/users/ga_woo
Ryan Leavengood
>
> So you're saying that when the performance requirements crosses a
> certain threshold, interpreted languages should not be used.
> The idea here is that if Ruby is faster, that threshold where should a
> switch needs to be made is shifted.
Of course, and I'm all for that. I want Ruby to be as fast as possible. In
fact I have some application domains in mind I'd like to use Ruby for that
may run into this exact problem (sound processing.)
But still, at this point in the state of computing, I would not use Ruby
in certain applications:
- operating system level code.
- heavy duty 3D rendering.
- device drivers.
- any major number crunching (math, video processing, low-level image
manipulation.)
But hey, maybe with some special hardware (a Ruby Chip?), all the above
would be possible and fast with Ruby. That may be the next level of
computing: hardware accelerated high level languages.
Ryan
Ara.T.Howard
> Gavri Fernandez said:
>>
>> So you're saying that when the performance requirements crosses a
>> certain threshold, interpreted languages should not be used.
>> The idea here is that if Ruby is faster, that threshold where should a
>> switch needs to be made is shifted.
>
> Of course, and I'm all for that. I want Ruby to be as fast as possible. In
> fact I have some application domains in mind I'd like to use Ruby for that
> may run into this exact problem (sound processing.)
>
> But still, at this point in the state of computing, I would not use Ruby
> in certain applications:
>
> - operating system level code.
> - heavy duty 3D rendering.
> - device drivers.
> - any major number crunching (math, video processing, low-level image
> manipulation.)
check out some of the things people are doing
http://sciruby.codeforpeople.com/wiki.cgi/InterestingProjects
this is for realtime video processing
cheers.
-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| My religion is very simple. My religion is kindness.
| --Tenzin Gyatso
===============================================================================
Gyoung-Yoon Noh
I've rewritten that code using 'set' as following:
require 'set'
d = Set.new
File.open('/usr/share/dict/words') do |f|
d << $_.chop! while f.gets
end
STDIN.each do |l|
puts l unless d.include? l.chop!
end
Here are timed results in my machine.
$ time echo zuul | ruby18 spellcheck-hash.rb
zuul
real 0m1.688s
user 0m1.635s
sys 0m0.045s
$ time echo zuul | ruby18 spellcheck-set.rb
zuul
real 0m1.796s
user 0m1.748s
sys 0m0.048s
$ time echo zuul | ruby19 spellcheck-hash.rb
zuul
real 0m1.694s
user 0m1.624s
sys 0m0.070s
$ time echo zuul | ruby19 spellcheck-set.rb
zuul
real 0m1.465s
user 0m1.411s
sys 0m0.053s
$ time echo zuul | ~/usr/bin/ruby -rite spellcheck-hash.rb
zuul
real 0m1.541s
user 0m1.485s
sys 0m0.054s
$ time echo zuul | ~/usr/bin/ruby -rite spellcheck-set.rb
zuul
real 0m1.178s
user 0m1.108s
sys 0m0.062s
YARV is not that faster in this case. I guess ruby's collection
types (Array, Hash, Set...) are not sufficiently optimized than python.
I've heard that python 2.4 boosted 'list' performance upto 30% than
previous version from a python developer. I got a strong impression
on much lower memory usage through 'generator' too.
Anyway, ruby is very slower than many other languages.
But we can prove not so slower than currently presented in that site.
How about discussing about 'howto raise ruby ranking in the shootout
site'? ;-)
Phil Tomson
Ryan Leavengood <mrc...@netrox.net> wrote:
>Gavri Fernandez said:
>>
>> So you're saying that when the performance requirements crosses a
>> certain threshold, interpreted languages should not be used.
>> The idea here is that if Ruby is faster, that threshold where should a
>> switch needs to be made is shifted.
>
>Of course, and I'm all for that. I want Ruby to be as fast as possible. In
>fact I have some application domains in mind I'd like to use Ruby for that
>may run into this exact problem (sound processing.)
>
>But still, at this point in the state of computing, I would not use Ruby
>in certain applications:
>
>- operating system level code.
>- heavy duty 3D rendering.
>- device drivers.
>- any major number crunching (math, video processing, low-level image
>manipulation.)
>
>But hey, maybe with some special hardware (a Ruby Chip?),
RubyChip: Introduced in 2010. Apple adopts it in 2012, moves away from
Intel. ;-)
>all the above
>would be possible and fast with Ruby. That may be the next level of
>computing: hardware accelerated high level languages.
Well, maybe not so new. There was a Lisp machine back in the 80's as I
recall. There was also a Forth chip made by Chuck Moore back then too.
Of course there are also HW implementations of the JVM (PicoJava).
Now that we've got fairly inexpensive FPGAs (like the Spartan 3 family
from Xilinx) it's possible that you can do this sort of thing in FPGAs.
It's certainly an intriguing idea.
Phil
Lothar Scholz
RL> But still, at this point in the state of computing, I would not use Ruby
RL> in certain applications:
RL> - operating system level code.
RL> - heavy duty 3D rendering.
RL> - device drivers.
RL> - any major number crunching (math, video processing, low-level image
RL> manipulation.)
Don't forget
- code that must do a lot of parsing
- code that works on large datasets (a GC problem)
- any algorithmic problem
Ara.T.Howard
> Using 'set' is somewhat faster than Hash in 1.9.
> I've rewritten that code using 'set' as following:
>
> require 'set'
>
> d = Set.new
>
> File.open('/usr/share/dict/words') do |f|
> d << $_.chop! while f.gets
> end
>
> STDIN.each do |l|
> puts l unless d.include? l.chop!
> end
>
> Here are timed results in my machine.
hmmm. since set uses hash yet has at least another level of method calls
that's hard to believe... this is what i'm getting:
harp:~ > ruby a.rb
========< set >========
3.818156
========< hash >========
2.572087
harp:~ > ruby a.rb
========< set >========
3.824544
========< hash >========
2.600603
harp:~ > cat a.rb
require 'set'
def bench label
fork {
GC.disable
STDOUT.sync = true
puts "========< #{ label } >========"
a = Time::now.to_f
yield
b = Time::now.to_f
printf "%f\n", b - a
puts
}
Process::wait
end
words = IO::readlines('/usr/share/dict/words').map{|word| word.strip}
set = Set::new
hash = Hash::new
words.each{|word| set << hash[word] = word}
bench('set'){ 42.times{ words.each{|word| set.include? word}}}
bench('hash'){ 42.times{ words.each{|word| hash.has_key? word}}}
note that times are the time it takes to look up every single word in the list
42 times.
James Edward Gray II
> Hello Ryan,
>
> RL> But still, at this point in the state of computing, I would not
> use Ruby
> RL> in certain applications:
>
> RL> - operating system level code.
> RL> - heavy duty 3D rendering.
> RL> - device drivers.
> RL> - any major number crunching (math, video processing, low-level
> image
> RL> manipulation.)
>
> Don't forget
From a person that maintains a Ruby IDE, you're comments always
surprise me...
> - code that must do a lot of parsing
I've actually used Ruby for a pretty hefty parsing application in my
work. It was my experience that this is an area where Ruby really
shines. Funny to see you bring it up as the opposite of that now.
> - code that works on large datasets (a GC problem)
I don't have much experience here, so I'll take your word for it.
> - any algorithmic problem
You serious?! I have to say that I think that's absurd. Browsing
RubyForge and RAA makes me think I'm not alone either. Have you
looked at the Ruby Quiz solutions?
So basically what you're saying is that Ruby is unusable for most
applications? I think it's safe to say you're in the minority with
this opinion.
I hear you, that Ruby is slow. I agree. I work with Java and it's
definitely faster. I've benchmarked similar Perl scripts and they're
faster. Ruby may even be the slowest language I use and the amount
of time this actually affects me is incredibly low. I want YARV as
much as you do, I'm sure, but your comments seem ridiculous to me.
James Edward Gray II
Dion Almaer
Actually, at a macro level there is no reason that Ruby or Python or Java
can not be as fast as C.
An example here is IronPython. Some of the python benchmarks run FASTER
under IronPython compared to the C Python.
Virtual Machine's are pretty darn smart now, and with the abstraction can do
smart things, like changing out the native code on the fly to end up with
smarter versions (vs. a one pass compilation step).
As an industry we often move up a chain, especially for app dev work.
I have never run into a bottleneck in a web application that occurred due to
the programming language. As soon as you are dealing with networks, disks,
and the like, architecture is always the key.
Just my 2p.
Cheers,
Dion
Steven Jenkins
>> On Jun 9, 2005, at 4:15 PM, Lothar Scholz wrote:
>> - code that must do a lot of parsing
>
> I've actually used Ruby for a pretty hefty parsing application in my
> work. It was my experience that this is an area where Ruby really
> shines. Funny to see you bring it up as the opposite of that now.
Here's a little benchmark. I wrote a parser in Racc to read the export
format from a system engineering modeling tool. The grammar is
non-trivial: 173 rules, 371 states. Running it on a fairly large file
(65k lines, 3.6 MB) takes about 6 seconds on a 3.2 GHz P4 running Linux.
Note that this is a lot more than parsing. The application is
constructing, in a correct but not very efficient way, the entire data
model in memory. I suspect that parsing itself is a small fraction of
the total runtime.
Steve
Matt Lawrence
> Well, maybe not so new. There was a Lisp machine back in the 80's as I
> recall. There was also a Forth chip made by Chuck Moore back then too.
> Of course there are also HW implementations of the JVM (PicoJava).
Don't forget the SOAR (Smalltalk On A RISC) project. There was even a
_very_ interesting book published about it.
Also, given the distance (in clock cycles) between main memory and the
CPU, I'm expecting to see a programmable microcode design come out. Such
things were built in the 70's & 80's with some success. And, being a fan
of Forth, using it as the internal microcode could be really interesting.
> Now that we've got fairly inexpensive FPGAs (like the Spartan 3 family
> from Xilinx) it's possible that you can do this sort of thing in FPGAs.
> It's certainly an intriguing idea.
I would enjoy learning how to design with these. And, I would love to
write a bunch of tools in Ruby for developing with them.
-- Matt
Nothing great was ever accomplished without _passion_
Gyoung-Yoon Noh
Oops, I posted absolute stupidity without any context information.
Actually, I've been looking into some cases in which ruby is
irrationally slow, so 'spellcheck' code come from at that time.
See http://shootout.alioth.debian.org/benchmark.php?test=spellcheck&lang=ruby&id=0&sort=fullcpu
Anyway, Ara, I got a similar result as you'd presented. Have you
tested my code? I doubt 'set' prevails in constructing large data
set compared to Hash. In case of 'spellcheck', it seems that the
bench would be performed iterative execution of the code.
If then, I guess 'set' version probably wins.
Sorry for posting without any mention about specific context.
Regards,
Austin Ziegler
> James Edward Gray II wrote:
> >> On Jun 9, 2005, at 4:15 PM, Lothar Scholz wrote:
> >> - code that must do a lot of parsing
> > I've actually used Ruby for a pretty hefty parsing application in my
> > work. It was my experience that this is an area where Ruby really
> > shines. Funny to see you bring it up as the opposite of that now.
> Here's a little benchmark. I wrote a parser in Racc to read the export
> format from a system engineering modeling tool. The grammar is
> non-trivial: 173 rules, 371 states. Running it on a fairly large file
> (65k lines, 3.6 MB) takes about 6 seconds on a 3.2 GHz P4 running Linux.
Similarly, my ~75 page manual for PDF::Writer is generated from a
parsed source in 1m 6s on a similar P4 running Windows XP. (It takes
significantly longer -- about 6 - 8m -- on my 1Ghz Crusoe laptop.)
This may not seem fast, but it's incredible when you look at the
complexity of the manual involved. The majority of the time isn't in
calculation or in parsing, either -- it's in the transaction
capabilities required by a good formatting engine.
-austin
--
Austin Ziegler * halos...@gmail.com
* Alternate: aus...@halostatue.ca
gabriele renzi
> Hello gabriele,
>
> gr> "</script> ha scritto:
>
>>>I took a look at
>>>http://shootout.alioth.debian.org/benchmark.php?test=all&lang=ruby&lang2=python&sort=fullcpu
>>>and found the tests on ruby disapointing :(.
>>>So what I'm asking for is a link to some ruby-specific optimisation tips.
>
>
> gr> a simple thing: do the most normal thing instead of writing code in ruby
> gr> that is innatural, i.e. instead of rewritine an heapsort by hand, use
> gr> the builtin sort/sort!/sort_by routines. Instead of writing a random
> gr> function by hand, use the builtin rand().
>
> Sorry you don't understand this benchmark. It is there to do this
> algorithms to compare the basic data operations.
I understand the benchmark.
I'm just saying that it shows "you should not implement X in ruby" not
that "X takes a lot of time in ruby" since you may not need to implement
X (i.e. because it is already there or because you would not actually
ever need a recursive floating point function).
> The benchmark is not useable to compare compiled languages with
> interpretered, but comparing python with ruby is a valid comparision.
>
> And that ruby is so worse then python is a real problem. If you still
> try to deny this, then you have a real mental problem.
I don't think I did
> If ruby is 9 times slower then python in for example function calling intensive
> code that means that ruby will slow down your development time a lot
> as it forces you to use optimized code or even C code much more often
> then python.
I agree, It would be nice if ruby was faster. Luckily,
people are working on this.
Phil Tomson
You can get the Xilinx Spartan 3 starter kit for $99. Includes a board
with a reasonably good sized FPGA on it. The software is even free
(Webkit they call it). It's pretty easy to get into FPGA design these
days. Also check out the newsgroup: comp.arch.fpga
As far as writing a bunch of Ruby tools for this sort of thing: I'd like
to do that too.
Phil
Jim Freeze
> I agree, It would be nice if ruby was faster. Luckily,
> people are working on this.
Don't go away thinking that X is faster than Ruby. Do
your own tests. We tested Perl against Ruby in a real
world problem (netlist parsing) and Ruby was faster
than Perl by 30%. When we did a C implementation
(which could now be done automatically with optimize)
it was 100% faster (ie, took half the time.)
These statements that X language is faster than
Ruby is like saying that Einstein is smarter than
Mozart. In what?
--
Jim Freeze
Theory and practice are the same, in theory. -- Ryan Davis
Phil Tomson
Jim Freeze <j...@freeze.org> wrote:
>* gabriele renzi <surren...@remove-yahoo.it> [2005-06-10 09:35:28 +0900]:
>
>> I agree, It would be nice if ruby was faster. Luckily,
>> people are working on this.
>
>Don't go away thinking that X is faster than Ruby. Do
>your own tests. We tested Perl against Ruby in a real
>world problem (netlist parsing) and Ruby was faster
>than Perl by 30%. When we did a C implementation
>(which could now be done automatically with optimize)
>it was 100% faster (ie, took half the time.)
Are you saying that the Ruby netlist parser was 100% faster than the C
implementation or the other way around?
BTW: I recall someone at a PDX.rb meeting who said he had benchmarked
Ruby vs. OO-style Perl and Ruby was quite a bit faster (don't recall the
percentage). As I recall the explanation was that doing OO in perl
requires more levels of indirection and thus more function calls.
Phil
Jim Freeze
> In article <20050610024...@freeze.org>,
> Jim Freeze <j...@freeze.org> wrote:
> >* gabriele renzi <surren...@remove-yahoo.it> [2005-06-10 09:35:28 +0900]:
> >
> >> I agree, It would be nice if ruby was faster. Luckily,
> >> people are working on this.
> >
> >Don't go away thinking that X is faster than Ruby. Do
> >your own tests. We tested Perl against Ruby in a real
> >world problem (netlist parsing) and Ruby was faster
> >than Perl by 30%. When we did a C implementation
> >(which could now be done automatically with optimize)
> >it was 100% faster (ie, took half the time.)
>
> Are you saying that the Ruby netlist parser was 100% faster than the C
> implementation or the other way around?
The C implementation was Ruby C (my bad). So, the Ruby C
implementation ended up twice as fast as the Perl
implementation. Bad comparison you say. Well, in this
environment, we found over four netlist parsers written
in Perl. These parsers were intermingled with the netlist
munging code, so it was difficult to do abstraction and
pull out the netlist parser in Perl and optimize it.
But you say, well all that could and should have been done in
Perl. My response is, well, you had 10 years, what were you
waiting for?
The pragmatic nature of Ruby lent itself to writing better code,
more modular code, and thus the ability to optimize a particular
module muuuchh more than Perl. ;)
Isaac Gouy
-snip-
> How about discussing about 'howto raise ruby ranking in the shootout site'? ;-)
Simplest way: contribute missing programs, find ways to fix broken
programs.
Find them on
http://shootout.alioth.debian.org/great/benchmark.php?test=all&lang=ruby&lang2=ruby&sort=fullcpu
and
http://shootout.alioth.debian.org/old/benchmark.php?test=all&lang=ruby&lang2=ruby&sort=fullcpu
Follow the FAQ instructions for contributing programs:
http://shootout.alioth.debian.org/great/faq.php?sort=fullcpu#help
Isaac Gouy
> As Gabriele mentionned, they implement a lot of stuff that is done in C
> in Ruby.
Specifically?
> Also, the tests must use the exact same algorithm, so this
> disadvantages Ruby is some situations. I say the test should compare
> same algorithms, but also the one best suited to the language.
Do this disadvantage Ruby more than PHP?
http://shootout.alioth.debian.org/great/benchmark.php?test=all&lang=ruby&lang2=php&sort=fullcpu
Or more than Tcl, or more than GNU Smalltalk?
http://shootout.alioth.debian.org/great/benchmark.php?test=all&lang=ruby&lang2=gst&sort=fullcpu
Austin Ziegler
> > Also, the tests must use the exact same algorithm, so this
> > disadvantages Ruby is some situations. I say the test should compare
> > same algorithms, but also the one best suited to the language.
> Do this disadvantage Ruby more than PHP?
When the algorithm, as entered, won't complete, yes. When compared
against a language that has tail recursion optimisation, or a possible
situation where optimisation of the compiler deals with recursion
intelligently? Yes.
The definition of Ackermann numbers is, I presume, well known. Because
some platforms can't modify their stack level (Windows), and because
the stack level isn't modified in the runs for the benchmark, there's
going to be an issue with the deep recursion. In this case, it would
be appropriate to reduce the level of recursion. But given the rules
mentioned, this isn't acceptable.
What's the *purpose* of something like the Ackermann test? Is it to
see how quickly the environment winds and unwinds the stack? How much
state it carries around?
I personally think that most of the shootout items are bogus in just
about every way. Don't get me wrong -- I'm not at all suggesting that
Ruby couldn't be faster. I'm just saying that benchmarks, like
statistics, are lies.
Isaac Gouy
> On 6/10/05, Isaac Gouy <ig...@yahoo.com> wrote:
> > > Also, the tests must use the exact same algorithm, so this
> > > disadvantages Ruby is some situations. I say the test should compare
> > > same algorithms, but also the one best suited to the language.
> > Do this disadvantage Ruby more than PHP?
> > Or more than Tcl, or more than GNU Smalltalk?
>
> When the algorithm, as entered, won't complete, yes. When compared
> against a language that has tail recursion optimisation, or a possible
> situation where optimisation of the compiler deals with recursion
> intelligently? Yes.
Ruby completes ackermann for N=7
http://shootout.alioth.debian.org/great/fulldata.php?test=ackermann&p1=ocaml-0&p2=clean-0&p3=gcc-3&p4=gcc-0&sort=fullcpu#cputable
Which is how there can be a comparison between Ruby and Tcl on
ackermann
http://shootout.alioth.debian.org/great/benchmark.php?test=all&lang=ruby&lang2=tcl&sort=fullcpu#ratio
> The definition of Ackermann numbers is, I presume, well known. Because
> some platforms can't modify their stack level (Windows), and because
> the stack level isn't modified in the runs for the benchmark, there's
> going to be an issue with the deep recursion.
We use Debian Linux, so tell us how to modify the stack level for Ruby.
-snip-
> I personally think that most of the shootout items are bogus in just
> about every way. Don't get me wrong -- I'm not at all suggesting that
> Ruby couldn't be faster. I'm just saying that benchmarks, like
> statistics, are lies.
Sounds like me in editorial mode.
http://shootout.alioth.debian.org/great/miscfile.php?sort=fullcpu&file=benchmarking&title=Flawed%20Benchmarks
Trey Campbell
>
> I personally think that most of the shootout items are bogus in just
> about every way. Don't get me wrong -- I'm not at all suggesting that
> Ruby couldn't be faster. I'm just saying that benchmarks, like
> statistics, are lies.
>
> -austin
> --
> Austin Ziegler * halos...@gmail.com
> * Alternate: aus...@halostatue.ca
>
>
The most succinct way that I've heard it put is that there are "lies,
damn lies, and benchmarks".
Trey Campbell
treyca...@mac.com
Austin Ziegler
> > The definition of Ackermann numbers is, I presume, well known. Because
> > some platforms can't modify their stack level (Windows), and because
> > the stack level isn't modified in the runs for the benchmark, there's
> > going to be an issue with the deep recursion.
> We use Debian Linux, so tell us how to modify the stack level for Ruby.
I'm sorry you do. However, yours should be one of the ones that should
be able to modify the depth of the stack with ulimit.
If I cared enough to get more than annoyed at this damned shootout
with its inane implementation rules, I'd do more. But as it is, the
best I can tell people is that the Alioth shootout is bullshit.
Gyoung-Yoon Noh
It seems like gforge bug. Whenever I request resending
confirmation mail, I got blank mail contains no body text.
Isaac Gouy
There isn't with benchmarks per-se; there can be a problem with how one
chooses to interpret benchmarks.
Isaac Gouy
> I've registered, but cannot get proper comfirmation mail yet.
> It seems like gforge bug. Whenever I request resending
> confirmation mail, I got blank mail contains no body text.
If you're talking about this page
https://alioth.debian.org/account/register.php
and continue to have problems, then perhaps you need to contact an
alioth admin:
http://www.debian.org/intro/organization
Austin Ziegler
>> On Jun 10, 2005, at 1:02 PM, Austin Ziegler wrote:
>>> I personally think that most of the shootout items are bogus in
>>> just about every way. Don't get me wrong -- I'm not at all
>>> suggesting that Ruby couldn't be faster. I'm just saying that
>>> benchmarks, like statistics, are lies.
>> "lies, damn lies, and benchmarks".
> how one chooses to interpret benchmarks.
No, it's the benchmarks themselves that are the problem. Remember
that NVidia got caught detecting a benchmark in their drivers and
optimizing output for the benchmark.
Similarly, this thread was started because someone came across your
language shootout benchmark page and assumed that Ruby is
dramatically slower in Real Life Examples because it's slower in the
benchmarks. Which, I can assure you, it most assuredly NOT.
You get it correct when you indicate that "the best benchmark is
your program." That's the only one that matters. It also lets you
measure benchmarks that can't be categorised in CPU seconds or
memory use or FLOPS or other nonsense like that.
It's hype, but the idea that Rails makes one 10x more productive is
part of the value of Ruby. Unless your program is of a special case
-- and a lot of these special cases are handled already -- then Ruby
programs will be developed faster and more easily maintained than
their non-Ruby counterparts. Enough so that execution speed is
something of a wash.
At any rate, I consider the Alioth shootout to be harmful to all
languages involved. There is no useful value provided by it,
especially as it does not permit language-appropriate modifications
to the algorithms in use.
Isaac Gouy
Austin Ziegler wrote:
> On 6/10/05, Isaac Gouy <ig...@yahoo.com> wrote:
> > There isn't with benchmarks per-se; there can be a problem with
> > how one chooses to interpret benchmarks.
>
> No, it's the benchmarks themselves that are the problem. Remember
> that NVidia got caught detecting a benchmark in their drivers and
> optimizing output for the benchmark.
>
> Similarly, this thread was started because someone came across your
> language shootout benchmark page and assumed that Ruby is
> dramatically slower in Real Life Examples because it's slower in the
> benchmarks.
As I said - There isn't a problem with benchmarks per-se; there can be
a problem with how one chooses to interpret benchmarks.
> Which, I can assure you, it most assuredly NOT.
Not "dramatically slower" than what other language, for what specific
"Real Life Examples"?
> It's hype, but the idea that Rails makes one 10x more productive is
> part of the value of Ruby. Unless your program is of a special case
> -- and a lot of these special cases are handled already -- then Ruby
> programs will be developed faster and more easily maintained than
> their non-Ruby counterparts. Enough so that execution speed is
> something of a wash.
Faster to develop-in than what other languages - Smalltalk? Lisp?
Mathematica?
> At any rate, I consider the Alioth shootout to be harmful to all
> languages involved. There is no useful value provided by it,
> especially as it does not permit language-appropriate modifications
> to the algorithms in use.
What's a language-appropriate modification?
Have you ever asked for one to be accepted in the Alioth shootout?
Glenn Parker
>
> As I said - There isn't a problem with benchmarks per-se; there can be
> a problem with how one chooses to interpret benchmarks.
That's a meaningless distinction. You are suggesting that people who
read your benchmarks will somehow interpret them and find information
that is simply not there. The benchmarks paint an overly simplified
picture, nothing more. The only way to interpret the numbers correctly
is to ignore them and look for more pragmatic examples.
I do think there is some small value in the comparitive coding samples,
but the rigidity of the rules means that there is very little ingenuity
allowed in the code, and thus all the code ends up looking extremely
similar, regardless of the language employed. If you enjoy probing
variations in whitespace and the placement of semi-colons, you will have
lots of fun.
> Faster to develop-in than what other languages - Smalltalk? Lisp?
> Mathematica?
Faster to develop in than C++, Perl, Java, Forth, Assembler, and indeed,
Lisp. God help you if you decide to implement a web framework or a
compiler in Mathematica.
The real problem with the alioth benchmarks is that they are run by
amateurs that allow themselves to be bound by rigid pseudo-academic
dogma, but they can't research and fix trivial problems on their own,
and then they whine when nobody does it for them. If you guys aren't
highly motivated to fix things, why do you think the rest of us will be?
I gave up on them a while ago.
--
Glenn Parker | glenn.parker-AT-comcast.net | <http://www.tetrafoil.com/>
Isaac Gouy
Glenn Parker wrote:
> Isaac Gouy wrote:
> >
> > As I said - There isn't a problem with benchmarks per-se; there can be
> > a problem with how one chooses to interpret benchmarks.
>
> That's a meaningless distinction. You are suggesting that people who
> read your benchmarks will somehow interpret them and find information
> that is simply not there. The benchmarks paint an overly simplified
> picture, nothing more. The only way to interpret the numbers correctly
> is to ignore them and look for more pragmatic examples.
>
> I do think there is some small value in the comparitive coding samples,
> but the rigidity of the rules means that there is very little ingenuity
> allowed in the code, and thus all the code ends up looking extremely
> similar, regardless of the language employed. If you enjoy probing
> variations in whitespace and the placement of semi-colons, you will have
> lots of fun.
>
> > Faster to develop-in than what other languages - Smalltalk? Lisp?
> > Mathematica?
>
> Faster to develop in than C++, Perl, Java, Forth, Assembler, and indeed,
> Lisp. God help you if you decide to implement a web framework or a
> compiler in Mathematica.
My mistake, I didn't realize Ruby was fixed to Rails, somewhere I got
the impression it was more general than that.
> The real problem with the alioth benchmarks is that they are run by
> amateurs that allow themselves to be bound by rigid pseudo-academic
> dogma, but they can't research and fix trivial problems on their own,
> and then they whine when nobody does it for them. If you guys aren't
> highly motivated to fix things, why do you think the rest of us will be?
>
> I gave up on them a while ago.
Who's whining? Those who wish to contribute will, those who don't won't.
Alexandru Popescu
Afaik Python uses bytecodes while Ruby is completely interpretative, so imo there is not possible to
drive conclusions from this benchmark.
A lot of _comparisons_ are made in this list (I see everywhere Ruby code has fewer lines than X,
Ruby dev is better than Y, etc); finally you may expect these other kinds of comparisons to take
place. As far as the 1st comparison was made, I see no benefit from hidding behind the finger and
disregarding from the start the possibility to look at other types of comparisons.
I see here (n.b. on the ML) lots of ideas and open minds while speaking of ruby (_and just ruby_)
and I appreciate this. But everytime, somebody/something is bringing into attention another
tool/language something bad/sad happens and all this openminded is vanishing away.
While looking at Python and Ruby as both being oo scripting languages, I agree I would like to
hear/see/read comparisons: from all possible points of view. And imo one of the most important is
the runtime performance. Maybe the development is easier and cheaper, but if the performance is bad
i will not do it. Also, even if I have to write fewer lines in X, but the tools for Y are offering
much more I will certanly go for Y (disclaimer: I am not speaking from the pov of writting a command
line parser utility or some other little toy that parse the log of my server, but from the pov of
developing a medium/large application).
A few months ago - not very many - I have started looking at Ruby. And I am enjoying its ideas a
lot. But, after some time, I am still not sure I will start working with it. I know, you may say:
`maybe Ruby is not for you´ (and maybe, I can agree with this). I must sadly confess that I haven't
got the same impression while looking at Python (and from the sizes of communities I can deduce many
others haven't felt the same).
Finally, please consider the above not a violent complain, but a sad objection I'm having.
:alex |.::the_mindstorm::.|
Austin Ziegler
> I've run through this thread and i am finally writting this
> because i am in a way astonished.
>
> Afaik Python uses bytecodes while Ruby is completely
> interpretative, so imo there is not possible to drive conclusions
> from this benchmark.
Which is one of the many reasons that I have a problem with the
alioth shootout. Ultimately, it tries to treat compiled, semi-
compiled, and interpreted languages equally. They're not. Compiled
languages will be faster than semi-compiled (bytecode), which will
be faster than interpreted. The level of speed difference is a
matter of scale, and that scale also matters on development time and
clarity.
> A lot of _comparisons_ are made in this list (I see everywhere
> Ruby code has fewer lines than X, Ruby dev is better than Y, etc);
> finally you may expect these other kinds of comparisons to take
> place. As far as the 1st comparison was made, I see no benefit
> from hidding behind the finger and disregarding from the start the
> possibility to look at other types of comparisons. I see here
> (n.b. on the ML) lots of ideas and open minds while speaking of
> ruby (_and just ruby_) and I appreciate this. But everytime,
> somebody/something is bringing into attention another
> tool/language something bad/sad happens and all this openminded is
> vanishing away.
Sorry, but that's just not true. People *are* open-minded on this
list with respect to new ideas. On the other hand, many of us don't
want to import those ideas which we feel will change the fundamental
nature of Ruby. Most of us don't *quite* agree on what that
fundamental nature is, but hey, that's the nature of individuals ;)
No one has said that Ruby can't afford to be faster, or that other
languages ARE faster. What we've said is that, in most cases, Ruby
is Fast Enough...and that it gives us pleasure to work with it and
we feel that we have a better grasp on our problem domain and that
we have a more powerful solution for less development time. All of
these items work together. You know what? Python developers
generally feel the same about their programs.
> While looking at Python and Ruby as both being oo scripting
> languages, I agree I would like to hear/see/read comparisons: from
> all possible points of view. And imo one of the most important is
> the runtime performance. Maybe the development is easier and
> cheaper, but if the performance is bad i will not do it.
See, the problem is that you can't really know what your performance
will be like until you've solved at least part of your problem
domain. The use of benchmarks -- especially brainless ones like
you'll find at the alioth shootout -- won't help you. Unless, of
course, your problem domain is writing benchmarks.
Look, there are people doing OpenGL work with Ruby. It's obviously
fast enough for *them*. ARPA uses Ruby to drive a large-scale Java
application. Rich Kilmer is working on making Ruby drive a Flash
application (what's the status on that, Rich?). Ara Howard uses Ruby
to do NOAA work on large weather pictures and to drive a cluster.
These are all things that require significant computing power. These
people have chosen to use Ruby. Why? Because it gives them pleasure,
and Ruby is Fast Enough in real world tests.
Will they complain if Ruby gets faster? Not at all. They'll be
ecstatic. But they're not complaining about Ruby's speed right now.
> Also, even if I have to write fewer lines in X, but the tools for
> Y are offering much more I will certanly go for Y (disclaimer: I
> am not speaking from the pov of writting a command line parser
> utility or some other little toy that parse the log of my server,
> but from the pov of developing a medium/large application).
IMO, this is a little short-sighted way to look at it. Sure,
VisualStudio gives you great tools to develop .NET applications.
*Damn* but does it ever give you great tools. (I've seen a preview
of what's coming in November. Damn, I say, Damn, that's nice.) But
you have to do a lot more with the languages behind VisualStudio to
get things done. In most cases, the Ruby program will be done or
deployable long before the .NET program (and *way* before a J2EE
program). Real world results will be measured. Real optimisations
can be performed.
> A few months ago - not very many - I have started looking at Ruby.
> And I am enjoying its ideas a lot. But, after some time, I am
> still not sure I will start working with it. I know, you may say:
> `maybe Ruby is not for you´ (and maybe, I can agree with this). I
> must sadly confess that I haven't got the same impression while
> looking at Python (and from the sizes of communities I can deduce
> many others haven't felt the same).
You know, I'm not sure whether Ruby is for you or not. Have you done
development in it? Has it shown itself to be Too Slow for your
problem domain(s)? Does it matter if Ruby finished the job one
second slower than Python? Five seconds? Five minutes? Ultimately,
it depends on the job as to how that can be answered.
Just as another note, I think that the RAA is pretty active. It's
run on WEBrick through an Apache proxy. Wow, huh? WEBrick is a
pretty performant web server written in Ruby.
Isaac Gouy
Austin Ziegler wrote:
> On 6/11/05, Alexandru Popescu <the_mi...@evolva.ro> wrote:
> > I've run through this thread and i am finally writting this
> > because i am in a way astonished.
> >
> > Afaik Python uses bytecodes while Ruby is completely
> > interpretative, so imo there is not possible to drive conclusions
> > from this benchmark.
>
> Which is one of the many reasons that I have a problem with the
> alioth shootout. Ultimately, it tries to treat compiled, semi-
> compiled, and interpreted languages equally. They're not. Compiled
> languages will be faster than semi-compiled (bytecode), which will
> be faster than interpreted.
Saying "Compiled languages will be faster than..." treats them all
equally!
There are compiled, bytecode, interpreter, ... /implementations/ of
programming languages - there are interpreters for C as-well-as
compilers.
Phil Tomson
Austin Ziegler <halos...@gmail.com> wrote:
>On 6/11/05, Alexandru Popescu <the_mi...@evolva.ro> wrote:
>> I've run through this thread and i am finally writting this
>> because i am in a way astonished.
>> Afaik Python uses bytecodes while Ruby is completely
>> interpretative, so imo there is not possible to drive conclusions
>> from this benchmark.
>
>Which is one of the many reasons that I have a problem with the
>alioth shootout. Ultimately, it tries to treat compiled, semi-
>compiled, and interpreted languages equally. They're not. Compiled
>languages will be faster than semi-compiled (bytecode), which will
>be faster than interpreted. The level of speed difference is a
>matter of scale, and that scale also matters on development time and
>clarity.
I have to say that I tend to agree with the astonishment.
Here's what I mean:
Someone comes along and asks about python vs. Ruby performance and
references the alioth benchmarks. Several folks jump on the alioth
benchmarks and call them bogus. I see this as blaming the messanger.
The fact is that these benchmarks exist and they cover lots of different
types of algorithms. They're on the web and people will look at them -
no way to stop that from happening.
Someone mentioned the Ackermann benchmark so I had a look and found that
gawk did much better than Ruby - that's pathetic folks. Now, I know that
if you could do this in gawk it should be possible to do it in Ruby as well.
There aren't even mumbers for N=8 or N=9 for the Ruby version. Maybe it's
because the Ruby version was written poorly. Maybe it's because of some
issue within the Ruby interpreter itself - I don't know, I didn't really
take time to look into it just now. Either way, wouldn't it be valuable
to try to improve our score on that benchmark? If it's because it's
poorly written, then it's gravy. If it's an issue with the interpreter
then maybe it would be good to know about the issue for the YARV design.
Now sure you can say that the benchmarks are bogus (and who knows, maybe
they are) and then take all your marbles and go home, but the benchmarks
remain and other languages communities are trying to hone their numbers.
If we totally sit it out, well, that's not going to look too good.
Sure these benchmarks don't tell the whole story about how nice it is to
develop in Ruby vs. gawk, but performance _is_ an issue for some people
and they will use this set of benchmarks (or another) as a factor in
deciding which langauge to use. If Ruby consistently shows up down
towards the bottom of the list... well, it can give a false impression.
If we tell them that they really shouldn't worry about performance (and
who knows, maybe we would be right in doing so) it just sounds like so
much excuse-making.
I think that as a community we should be trying to compete in these
benchmarks just like other langauge communities are doing. Perhaps the
results can help us as we do move to a VM - if nothing else to show
that we're making improvements (look at them as a set of unit tests for
performance). We know that Ruby needs help in the
performance area. If we keep telling ourselves that Ruby is 'fast
enough' for our application (and it may well be) are we going to be
sitting still while other languages improve performance? Have we been
the hare for the last few years sitting around while the tortise (TCL,
for example, once considered a lot slower than Ruby) is passing us up
(by creating or improving their VMs, for example).
Of course we know that we can do all sorts of things to speed things up
by writing C extension code, for example - that's a given, and the fact
that it's so easy to do in Ruby makes it possible to get into a lot of
areas where you might normally need to consider a compiled langauge,
however, a lot of people out there want to see how Ruby performs when
you just write Ruby code (C is a compiled language, of course). In some
sense marrying Ruby and C might seem like cheating to them, I suppose (try
to see it from their perspective).
If it looks like some of the benchmarks are misleading or stacked against
us, maybe we can propose other benchmarks where Ruby shines (BTW: why
does it say that the Object Methods benchmark is being deprecated?),
after all, it appears to be an open source process.
Bottom line: Ruby needs help in the performance area. Let's admit that
and work on improving in that area instead of shooting the messenger -
it'll make us look like whiners if we keep doing the latter. I'd really
rather not have the Ruby community perceived that way from the outside.
In the meantime, perhaps we could have some Ruby quizzes that focus on
choosing a benchmark and optimizing it. I think we would be
better off as a community if we participate in the process rather than
boycott it.
...just my 2cents.
Phil
Alexandru Popescu
>
> Phil
>
>
Lots of thanks Phil for better expressing my thoughts and feelings. That is exactly what I wanted to
say - unfortunately the feelings just cut off my vocabulary.
And just as a confirmation of my last paragraph:
[stupid-quoting-myself]
I know, you may say:
`maybe Ruby is not for you´ (and maybe, I can agree with this). I must sadly confess that I haven't
got the same impression while looking at Python (and from the sizes of communities I can deduce many
others haven't felt the same).
[/stupid-quoting-myself]
I have received offline messages suggesting to go with other solutions and some solid arguments why
I should do it.
I have to agree that it is no pleasure to find your invention `hammered´ by a (possible wrong)
benchmark; but I think when this happens I would go out shouting my benchmarks where i am kicking ass.
I remember that Java community did this a lot while the first c++ vs java benchmarks have been
published.
I remember that a few years ago when such a benchmark (driven by `paid´ money by MS), was proving
that .NET was surclassing J2EE, finally even IBM joined the war and proved it completely wrong.
:alex |.::the_mindstorm::.|
Stefan Lang
[...]
> Bottom line: Ruby needs help in the performance area. Let's admit that
> and work on improving in that area instead of shooting the messenger -
> it'll make us look like whiners if we keep doing the latter. I'd really
> rather not have the Ruby community perceived that way from the outside.
Im slowly getting fed up with this whining, "Ruby is so slow".
Especially if it's compared to Python.
Ever directly compared Ruby vs. Python solutions with regards to
performance?
Python is damn slow at startup, despite loading byte- instead of
sourcecode. On my machine, there is (approximately) a linear
ratio between LOPC and startup time: About 0.1 second per 200
LOPC. I've a Ruby script and a Python script doing the same thing
with a roughly equal model: The Ruby version takes ~0.06s,
the Python script ~0.26s for startup. Just to emphasize that a
bit: that's a factor of 4.3. (And the Ruby script is even
longer because it has some additional features.)
For longer program runs, the Python version comes closer
(but never surpasses) the Ruby one.
In conclusion, IMO Ruby will perform better for batch processing
and I don't know if the opposite is true for bigger/longer running
applications. I don't believe that Python is faster for serious
apps (not just 5 line benchmarks) unless someone shows me a program
written in Ruby and Python were the Python one is significantly faster.
Stefan
Steven Jenkins
> Austin Ziegler <halos...@gmail.com> wrote:
>>Which is one of the many reasons that I have a problem with the
>>alioth shootout. Ultimately, it tries to treat compiled, semi-
>>compiled, and interpreted languages equally. They're not. Compiled
>>languages will be faster than semi-compiled (bytecode), which will
>>be faster than interpreted. The level of speed difference is a
>>matter of scale, and that scale also matters on development time and
>>clarity.
>
> I have to say that I tend to agree with the astonishment.
>
> Here's what I mean:
> Someone comes along and asks about python vs. Ruby performance and
> references the alioth benchmarks. Several folks jump on the alioth
> benchmarks and call them bogus. I see this as blaming the messanger.
That's putting it politely. The argument against, which has merit, is
stated with so much hyperbole that it damages its own credibility.
First, there are the ad hominem attacks. Words like "brainless", "inane"
and "bullshit". Don't tell me these aren't personal; they are.
Then there's the rhetorical exaggeration: If this benchmark has flaws,
then it has no value. If this benchmark has no value, then no benchmarks
have value. Culminating, of course, in "Benchmarks, like statistics, are
lies." I suppose I could be charitable and interpret that as a
hyperbolic way to say "Benchmarks, like statistics, tell only part of
the story. Use caution in their construction and interpretation." If
that's what was meant, then the original statement is itself,
ironically, a "lie" in precisely the same way. If it was meant
literally, then it betrays a deep ignorance of the practice of science
and engineering, among other things.
In my line of work, we use benchmarks and statistics extensively. They
may be useful for lying; so what? They are far more powerful for telling
the truth.
Steve
Austin Ziegler
Ackermann function, why did you say anything? Sorry, but the alioth
benchmarks are crap -- and always will be crap. Trying to improve
Ruby's score in them is a loser's game.
-austin
Austin Ziegler
> In my line of work, we use benchmarks and statistics extensively. They
> may be useful for lying; so what? They are far more powerful for telling
> the truth.
And there's the real tragedy. People who buy into the crap that is benchmarks.
There is only one benchmark that truly matters. Are your users
complaining about the speed of your program? If so, then you need to
optimise it. If not, then it's good enough.
Everything else is lies and marketing. Things like the alioth
benchmarks don't actually shed any light on anything; they are great
at obscuring. Nothing more.
Alexandru Popescu
> On 6/12/05, Steven Jenkins <steven....@ieee.org> wrote:
>> In my line of work, we use benchmarks and statistics extensively. They
>> may be useful for lying; so what? They are far more powerful for telling
>> the truth.
>
> And there's the real tragedy. People who buy into the crap that is benchmarks.
>
> There is only one benchmark that truly matters. Are your users
> complaining about the speed of your program? If so, then you need to
> optimise it. If not, then it's good enough.
>
> Everything else is lies and marketing. Things like the alioth
> benchmarks don't actually shed any light on anything; they are great
> at obscuring. Nothing more.
>
> -austin
Austin, I am still wondering: you develop f.e. 3 months to find out if your client will accept the
software you have invested into? All the companies I have been employed into were not working like
this. We have always built prototypes and decided from that point. And a prototype is a benchmark in
fact.
Also, I agree with the fact the optimization comes later in the dev cycles. But we need to have the
doors open to optimize. Moreover, I read that in Ruby you can always go and code the hot spots in C.
What if you don't have these resources? Which are the costs to bring such a resource and make him
productive?
Maybe I am completely wrong, but this is the way the companies I have worked with are proceeding.
:alex |.::the_mindstorm::.|
Michael Campbell
> Also, I agree with the fact the optimization comes later in the dev cycles. But we need to have the
> doors open to optimize. Moreover, I read that in Ruby you can always go and code the hot spots in C.
> What if you don't have these resources? Which are the costs to bring such a resource and make him
> productive?
How does this issue relate to *ruby*? The same charge could be levied
at python, even J2EE. "What happens if you don't have the resources
to speed up something slow?"
I tend to agree with Austin on this; theoretical, academic arguments
don't mean much when a customer wants his software.
It surprised me coming up through the application development
(financial applications) world, but some customers even *WANT* early
releases to be slow, so they can feel good that they tell the
developers to make it faster, and they do.
Alexandru Popescu
I want to understand how you ruby working guys are managing this. Usually only the success stories
go public, but I know behind these there are lots of failures.
:alex |.::the_mindstorm::.|
Steven Jenkins
> On 6/12/05, Steven Jenkins <steven....@ieee.org> wrote:
>
>>In my line of work, we use benchmarks and statistics extensively. They
>>may be useful for lying; so what? They are far more powerful for telling
>>the truth.
>
> And there's the real tragedy. People who buy into the crap that is benchmarks.
What's more amazing is that these fools why buy into this "crap" have
made fundamental contributions to the theory and practice of digital
communications, image processing, and numerical analysis, practically
invented systems engineering, found volcanoes on Io and landslides on
Venus, communicated with spacecraft beyond the edge of the solar system,
put three rovers on the surface of Mars, and a whole bunch of other
stuff that will go into history books.
But of course, you know better. You have A Blog and have written Some
Software.
Steve
Isaac Gouy
> (BTW: why does it say that the Object Methods benchmark
> is being deprecated?)
The quarrels about what was and wasn't OO became really boring.
object-methods will still exist on The Doug Bagley
http://shootout.alioth.debian.org/old/index.php?sort=fullcpu
And maybe we'll just replace it with dispatch
http://shootout.alioth.debian.org/sandbox/benchmark.php?test=dispatch&lang=all&sort=fullcpu
Austin Ziegler
> What's more amazing is that these fools why buy into this "crap" have
> made fundamental contributions to the theory and practice of digital
> communications, image processing, and numerical analysis, practically
> invented systems engineering, found volcanoes on Io and landslides on
> Venus, communicated with spacecraft beyond the edge of the solar system,
> put three rovers on the surface of Mars, and a whole bunch of other
> stuff that will go into history books.
Huh. They didn't do it with benchmarks. (And I could very easily point
out that those same people have screwed up massively -- forgetting to
convert between English units and Metric units?) Look closely at the
people who espouse benchmarks. They're mostly marketers or fools who
can't tell the difference.
There are *real* measures to deal with; they're not benchmarks. They
aren't and never have been.
For a first person shooter, the real measure is "is the game fun?" The
answer will be different for everyone, but there are some objective
things that will break the "fun" factor for just about everyone.
Frames Per Second. Load Time. These things should be as fast as they
possibly can. Gigaflops never enters the question here. Nor does
specmark or anything else like that.
For image manipulations, they need to be quick. But not once does the
Ackermann function ever enter the question.
> But of course, you know better. You have A Blog and have written Some
> Software.
I do know better. It's not because I have a blog (I do, but I haven't
updated it in months, because I've been busy writing Real Software for
Real People). And I've written a LOT of software. No, I've never
worked for NASA. But I have real world experience, and I know what I
speak of.
Never *once* have I needed to implement an Ackermann function. Not
once. In my entire career. I look at the crap that is on alioth and
there's very little that represents common use. There's some neat
things -- the new DNA transformation ones -- but exactly how many
people will actually be using that in their work?
I won't. Wouldn't have my entire career. Measures that mattered to me
at my last job were "how many bills can I generate in an hour?" At my
current job "what's the average backup speed throughput for this?"
If we're not getting the performance we need, we fix the damn problem.
We don't rely on "benchmarks" -- we rely on real world measurements of
our real problems. Not on pseudo-crap like gigaflops or specmark or
the speed of an airborne swallow. Actually, strike that. The last is
useful.
Curt Hibbs
>
> If we're not getting the performance we need, we fix the damn problem.
> We don't rely on "benchmarks" -- we rely on real world measurements of
> our real problems. Not on pseudo-crap like gigaflops or specmark or
> the speed of an airborne swallow. Actually, strike that. The last is
> useful.
Yeah, I rely on the speed of airborne swallows at least three times as
often as Ackermann functions. ;-)
Curt
Ara.T.Howard
it's 42 right?
-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| My religion is very simple. My religion is kindness.
| --Tenzin Gyatso
===============================================================================
Isaac Gouy
Austin Ziegler wrote:
-snip-
> Never *once* have I needed to implement an Ackermann function. Not
> once. In my entire career. I look at the crap that is on alioth and
> there's very little that represents common use. There's some neat
> things -- the new DNA transformation ones -- but exactly how many
> people will actually be using that in their work?
Never once needed to implement a recursive function?
Some details are always specific to a problem domain and application,
but the same representations and approaches are used across problem
domains and across applications - yes the details are from DNA sequence
analysis, but the programs process ascii strings and hashtables.
Austin Ziegler
> Austin Ziegler wrote:
> -snip-
> > Never *once* have I needed to implement an Ackermann function. Not
> > once. In my entire career. I look at the crap that is on alioth and
> > there's very little that represents common use. There's some neat
> > things -- the new DNA transformation ones -- but exactly how many
> > people will actually be using that in their work?
> Never once needed to implement a recursive function?
Not an Ackermann recursive. Only simple recursive. Most of the time, I
haven't even needed that. There's an important point there. Indeed, I
have sometimes gone in and changed recursive into iterative because it
was too expensive to implement as recursion.
But what are you really testing with this? If you want to test stack
winding and unwinding speeds, then you're probably testing the wrong
thing here. If you're testing something else, it's not clear. And
that's the damnable thing about the whole alioth shootout -- you
refuse to take an editorial stance anywhere (well, sort of) on the
interpretation of the numbers, leaving them to stand on their own --
which leads to people making stupid assumptions about them. You state
that you refuse to take an editorial stance, but you end up doing so
-- both by what tests you accept for the shootout and by what
implementations you'll accept.
I stand by what I said, though: the alioth shootout is a waste of
everyone's time. Frankly, I think it would be better if it were just
taken down.
> Some details are always specific to a problem domain and application,
> but the same representations and approaches are used across problem
> domains and across applications - yes the details are from DNA sequence
> analysis, but the programs process ascii strings and hashtables.
Perhaps. But more often than not, the approaches aren't perfectly portable.
Steven Jenkins
> Huh. They didn't do it with benchmarks.
They did. I was there; you weren't. But don't let facts get in the way.
Steve
Austin Ziegler
Bully for you. Were you there when they mixed unit systems, too? Did
they do *that* with benchmarks?
Sorry, but I don't buy the argument that they did this with
benchmarks. Measurements, perhaps, of the performance of their own
programs (or even prototypes), but *not* with benchmarks. In other
words, I think you're full of it and I'm calling you on it. Provide
some evidence that the decisions were made with benchmarks -- "A
standard by which something can be measured or judged."
Tell me that they did prototyping and I'll believe you. Tell me that
they measured *performance* and I'll believe you. Tell me that
benchmarks were involved in the process and I won't believe you unless
you provide some evidence of this claim.
Isaac Gouy
Austin Ziegler wrote:
> On 6/12/05, Isaac Gouy <ig...@yahoo.com> wrote:
> > Austin Ziegler wrote:
> > -snip-
> > > Never *once* have I needed to implement an Ackermann function. Not
> > > once. In my entire career. I look at the crap that is on alioth and
> > > there's very little that represents common use. There's some neat
> > > things -- the new DNA transformation ones -- but exactly how many
> > > people will actually be using that in their work?
> > Never once needed to implement a recursive function?
>
> Not an Ackermann recursive. Only simple recursive. Most of the time, I
> haven't even needed that. There's an important point there. Indeed, I
> have sometimes gone in and changed recursive into iterative because it
> was too expensive to implement as recursion.
And in some other language implementations that change is unecessary -
perhaps that's the point.
-snip-
> And that's the damnable thing about the whole alioth shootout -- you
> refuse to take an editorial stance anywhere (well, sort of) on the
> interpretation of the numbers, leaving them to stand on their own --
> which leads to people making stupid assumptions about them.
-snip-
Alioth Shootout - a Rorschach test for programmers.
Steven Jenkins
> On 6/12/05, Steven Jenkins <steven....@ieee.org> wrote:
>>They did. I was there; you weren't. But don't let facts get in the way.
>
> Bully for you. Were you there when they mixed unit systems, too? Did
> they do *that* with benchmarks?
Yes, I was there. It's irrelevant to the discussion, but I was there.
> [badgering elided]
>
> Tell me that they did prototyping and I'll believe you. Tell me that
> they measured *performance* and I'll believe you. Tell me that
> benchmarks were involved in the process and I won't believe you unless
> you provide some evidence of this claim.
I've said what I have to say. Believe it or don't, as you see fit. We're
done.
Steve
Lothar Scholz
MC> On 6/12/05, Alexandru Popescu <the_mi...@evolva.ro> wrote:
>> Also, I agree with the fact the optimization comes later in the
>> dev cycles. But we need to have the
>> doors open to optimize. Moreover, I read that in Ruby you can
>> always go and code the hot spots in C.
>> What if you don't have these resources? Which are the costs to
>> bring such a resource and make him
>> productive?
MC> How does this issue relate to *ruby*? The same charge could be levied
MC> at python, even J2EE. "What happens if you don't have the resources
MC> to speed up something slow?"
If a language is a few times faster then the risk factor that this
happens is just lower. This is a management descision.
There are many things to keep in balance when you start a project.
Development time is just one of them and often not that important.
I can just speak for the application domain where i worked for years:
This is products for the mass market. Here you don't have something
like "your customer". In this market you must look a lot about "your
competitor" and this makes it different. Spending a few month more on
development is just okay if the (peak !!) speed is well and
competitive.
This is a domain where Ruby is unfortunately not useable at the
moment (missing a good GUI toolkit is the second reason). And i would
like to see it possible to use it there too.
--
Best regards, emailto: scholz at scriptolutions dot com
Lothar Scholz http://www.ruby-ide.com
CTO Scriptolutions Ruby, PHP, Python IDE 's
Phil Tomson
No, but as you've said, you need quick response from a game in order for
it to be engaging. It comes down to frames per second (as you note).
>
>Never *once* have I needed to implement an Ackermann function. Not
>once. In my entire career.
No one is saying that you would need to. The Ackermann benchmark can be
a measure of how well your language handles recursion (of course it can
be written iteratively as well). Sure it may not be something that you'd
ever use, but if someone is perusing the benchmark results and sees that
Ruby falls way behind gawk in this particular benchmark, they're likely
to draw some conclusions. Now you and I might think that the conclusions
they draw are ot fair or accurate, but that doesn't matter. If I've
never encountered Ruby before and all I know of it is from the alioth
benchmarks, the conclusions I come to will not be positive.
>I look at the crap that is on alioth and
>there's very little that represents common use. There's some neat
>things -- the new DNA transformation ones -- but exactly how many
>people will actually be using that in their work?
>I won't. Wouldn't have my entire career.
Not everyone is doing web development. Some people work in interesting
areas like Bioinformatics. It's great that Ruby is finding a niche in web
development with Rails, but I really hope that doesn't translate into
Ruby only aspiring to be another PHP (ie. being perceived as a language
that is only useful for web development - I'm afraid that's already
starting to happen).
The fact is that it's in various scientific areas where you'll find the
most compute-intensive applications (the other area would be gaming), so
including DNA transformation benchmarks can be very informative.
>Measures that mattered to me
>at my last job were "how many bills can I generate in an hour?" At my
>current job "what's the average backup speed throughput for this?"
>
>If we're not getting the performance we need, we fix the damn problem.
Good; we can all agree on this ;-)
>We don't rely on "benchmarks" -- we rely on real world measurements of
>our real problems. Not on pseudo-crap like gigaflops or specmark or
>the speed of an airborne swallow. Actually, strike that. The last is
>useful.
>
Again, benchmarks don't tell the whole story, but unfortuneately many
will judge Ruby based on these benchmarks - that's just reality. What
are you going to do, buy banner ads on the alioth site which read "Don't
trust these benchmarks!". I don't think that'll work.
Since benchmarks tend to be a uniform measure (we can't all trot out
our 'real world' code to be translated into all the popular languages and
then tested and timed in each one - and if we did that would just become
the next benchmark, wouldn't it?). A good set of benchmarks needs to be
easily implemented in all of the languages being compared and they also
need to exercise a lot of different features that would show up in real
world programs. I'm going to give the alioth folks the benefit of the
doubt here and assume that they want to develop a 'good' set of
benchmarks that can be used to measure relative performance of various
languages. We can either dismiss their efforts with various
pejoratives (which will make people outside of our community wonder
what we're trying to hide) or help out.
Phil
Nura...@aol.com
well, I would not spend money to buy banners on Alioth saying,
"Don't trust these benchmarks!", either.
But I think ,too, that benchmarks do not tell the whole story and in
many cases are misleading.
I am working in theoretical modelling for biology and I was using
mainly Matlab and C because the first one is such a convenient scripting
language
and the latter is so quick before I came to know (and like) Ruby.
I think that benchmarks are significant if you do a very limited amout of
things
and do them very very many times in your work.
So, in PDE models for biological cells, you can use numerical integration,
treat
a zillion different parameter values to search for parameters that might suit
the experimental behaviour you are looking for. I bet that C and maybe many
other languages are faster on that than Ruby.
However, you can treat the same problem using symbolic algebra.
Then you may learn at 8.00 am on some day that something called a Groebner
basis (doesn't matter what that is, just tell somebody and they'll think
you've just dropped from Mars) could be very useful to calculate a solution in
much less steps.
But you have to implement that in C, because you don't find an implementation
that suits your needs quite exactly (it's nice to have fractions in the
parameters
and a non-commutative variant of Groebner bases in my case, and
Maple/Mathematica
didn't quite work for some reason I don't recall at the moment - but nobody
I talked to could
fix in the amount of time they were willing to think about it).
Now, most of my enthusiasm for Ruby comes from the fact that you find
implementations of things that you need (and you can be confident
to do so even for problems you didn't know you would have) which you can
modify
or combine easily in such a way that the speed of C from the benchmark list
looks quite faded, because you need to add all the development time to it
(which is rather huge in my case).
It has even happened that I finished a programming task earlier than
scheduled,
but so far, only when using Ruby.
So, my personal benchmark would be:
How long does it take for me to implement problem X in code and execute it
in language
Y versus language Z?
In reality, for me, this will often mean: I can use C and a stupid problem
finding strategy
(which is fast, but has to check a zillion things, so it's slow after all)
against a
more intelligent strategy, which I could implement faster in Ruby (also, it
doesn't seem
such a daunting task to start implementing it anymore).
Since I use Ruby, I still have a lot of problems, but they keep changing
much faster
than earlier and I do less routine work.
That's what I'd write on a banner if I had money to buy one.
So I agree with Austin's benchmark about customer happiness of customer as a
benchmark.
Best regards,
Axel
Austin Ziegler
> Hello Michael,
>> On 6/12/05, Alexandru Popescu <the_mi...@evolva.ro> wrote:
>>> Also, I agree with the fact the optimization comes later in the
>>> dev cycles. But we need to have the doors open to optimize.
>>> Moreover, I read that in Ruby you can always go and code the hot
>>> spots in C. What if you don't have these resources? Which are
>>> the costs to bring such a resource and make him productive?
>> resources to speed up something slow?"
> If a language is a few times faster then the risk factor that this
> happens is just lower. This is a management descision.
Sometimes yes, sometimes no. More often than I would like, but less
often, I imagine, than you think.
> There are many things to keep in balance when you start a project.
> Development time is just one of them and often not that important.
Actually, in my experience -- both with cusomisable largeware (a
billing system for ISPs) and with consumer-oriented software --
development time is one of the top items of concern. Not *the* top,
but definitely a component in the top (which is usually support
costs, a combination of tech support time, QA time, and developer
time, the cost of each rising).
> This is a domain where Ruby is unfortunately not useable at the
> moment (missing a good GUI toolkit is the second reason). And i
> would like to see it possible to use it there too.
I'd argue that Ruby's speed is secondary to the lack of a good
cross-platform GUI kit. I, too, would like to see Ruby perform
faster than it does. But I don't think that satisfying the alioth
shootout is going to make that happen. The problem *is* known, and
the solutions are at hand. If moving slowly.
Isaac Gouy
-snip-
> So, my personal benchmark would be:
> How long does it take for me to implement problem X in code and execute it
> in language Y versus language Z?
> In reality, for me, this will often mean: I can use C and a stupid problem
> finding strategy (which is fast, but has to check a zillion things, so it's
> slow after all) against a more intelligent strategy, which I could implement
> faster in Ruby (also, it doesn't seem such a daunting task to start
> implementing it anymore).
Or with Language XXX(TM) we could develop an intelligent strategy and
then, after high-level experimentation, compile the code to give a
program with performance 2x-3x of the gnarly C program.
Tanner Burson
providing, or furthering much intelligent discussion. But what on
earth are you attempting to say by this quote? Or are you just
throwing out random things for the sake of saying that somewhere, for
a given problem domain, there is a language more suited than Ruby? If
so thanks, point taken. If not would you mind clarifying by saying
something a bit more descriptive? Is the above quote actually in
reference to a given language you're attempting to introduce everyone
to? Or is it just a generic example with no meaning whatsoever?
>
>
--
===Tanner Burson===
tanner...@gmail.com
http://tannerburson.com <---Might even work one day...
Isaac Gouy
> I've tried to stay out of this thread, because I don't feel it's
> providing, or furthering much intelligent discussion. But what on
> earth are you attempting to say by this quote? Or are you just
> throwing out random things for the sake of saying that somewhere, for
> a given problem domain, there is a language more suited than Ruby? If
> so thanks, point taken. If not would you mind clarifying by saying
> something a bit more descriptive? Is the above quote actually in
> reference to a given language you're attempting to introduce everyone
> to? Or is it just a generic example with no meaning whatsoever?
Sorry.
Only 2 options were presented - slow-to-write fast-to-run C and
fast-to-write slow-to-run Ruby - there are other possibilities. That's
all.
Nura...@aol.com
Sorry.
Yes, of course. I was stating my personal views without claiming
generality.
Mmh, benchmarks seem to be good essentially for one thing, to judge
by this thread - propagate bad mood. I tried to change that a bit , without
success apparently.
If people should really be running away from Ruby or see
it as a language good for web-development only because of bad
benchmarks, my proposal would be to try to identify a small set
of features which are slowing down the relative performance of Ruby in
applications many people feel important and enhance them in a future version.
Maybe that's what this not-so providing thread could eventually do.
So far, I myself am quite happy with Ruby's performance, so I couldn't
open a list, but nevertheless, I think that if it is was possible to identify
some consensus about say the five biggest downslowers in important
applications
and fix them in a later version, this could achieve more than trying to
put much more work into fixing many more problems.
Best regards,
Axel
Ryan Leavengood
tweaked and designed to satisfy the benchmarks, whereas real world
performance may not be that good. It was mentioned how NVidia did this
with video cards. I imagine this is part of the perspective that Austin is
coming from.
These are very specific, pedantic benchmarks that don't seem to allow any
real creativity in implementation. The power or elegance of many of the
languages is lost in implementing an identical algorithm to solve a
certain problem. Surely varied algorithms that give the same answer should
be allowed to show the power and semantics of the given languages?
I'm not as passionate about this as Austin, but I certainly see where he
is coming from.
Ryan
Isaac Gouy
> Surely varied algorithms that give the same answer should
> be allowed to show the power and semantics of the given languages?
Surely that would show the power of algorithms. (Yes, memoising
Ackermann is much much faster.)
lypanov
> If we keep telling ourselves that Ruby is 'fast
> enough' for our application (and it may well be) are we going to be
> sitting still while other languages improve performance?
luckily there is work on this.
what i don't get it why everyone
talks about it rather than actually working on it :)
Alex
David Zhao
applications, performance problems can be solved with adding more
hardware.
When you have to make the trade off between more developer time, ease
of maintainence vs. more hardware, most companies go with the more
hardware approach as it usually is cheaper. People worry about
performance often too much when choosing the language, rather than
focusing on good design and the product itself.
Steven Jenkins
that it is possible to have a useful benchmark. I'm going to do
so by telling about a real benchmark that we used to solve a real
problem in the Deep Space Network. But first, some clarifications:
1. I have not expressed, and do not hold, an opinion regarding the
Alioth Shootout.
2. I have not expressed, and do not hold, an opinion regarding the
Ackermann function as a benchmark.
3. I originally entered this discussion because I objected to
the assertion "Benchmarks, like statistics, are lies." My main
objection was to the inclusion of statistics. Statistics was, for all
practical purposes, invented by a scientist (Gauss) to solve physics
problems. It is utterly indispensable to the practice of science and
engineering. Nowadays, virtually every digital device (MP3 player,
cell phone, network interface) is designed using theoretical concepts
pioneered by Claude Shannon in 1948. Shannon's information theory is
heavily statistical in nature; he called his measure of information
"entropy" because its formulation is so similar to the concept of
the same name in statistical mechanics. People outside the field of
engineering are sometimes surprised to discover just how essential
the deep theories of mathematics and statistics are to ordinary
things like cell phones and airplanes. Here's an interesting
interview with Andrew Viterbi, a first-rate theoretician who made
a huge impact on the practical world, for those who are interested:
http://www.ieee.org/organizations/history_center/oral_histories/transcripts/viterbi.html.
It's an interesting twist that he worked on the Deep Space Network,
which features in the story below.
4. As the discussion was more about benchmarks than statistics,
I'll give what I believe is a counterexample to the claim. I have
several others, some perhaps better than this one, but this is one
that I worked on directly.
5. I've marked this OT because it's not about Ruby and LONG because
it's, well, long. I don't expect to have more to say.
First, a little background. In the early 90s I worked on the Deep
Space Network, which is managed by JPL for NASA. The purpose of the
DSN, among other things, is to communicate with spacecraft beyond
earth orbit. This is a tremendous technical challenge. Voyager 1,
for example, has a transmitter that produces somewhere around 5
watts of power, about the same as a walkie-talkie. It's currently
about 14 billion km from earth. It's hard to fathom just how weak a
signal that is. To hear it and actually get data from it requires
huge antennas, cryogenically-cooled high-power amplifiers, and
exotic error correcting codes based on information theory.
The DSN has antenna complexes in California, Spain, and
Australia. Since long before the Internet, these complexes have been
connected by digital links carried over geosynchronous satellite
circuits.
The people who built this system were world-class telecom engineers,
not computer guys. The DSN ground system had a lot of custom
hardware and software in it, as well as "COTS" products made by
smaller manufacturers you've probably never heard of. By 1990 or
so people began to wonder whether the DSN ground infrastructure
should be re-implemented using Unix systems and TCP/IP. I was in
the group that thought we should.
People on the other side had advanced a theoretical argument against
TCP/IP, based on the quite true fact that the TCP specification then
allowed only 64 kB of sliding window. (Since then the protocol has
been extended to work better on "fat pipes".) It was true that you
could not completely load a T1 circuit, for example, with a single
TCP stream unless the network round-trip time was less than about
330 ms. Unfortunately, the speed of light requires about 230 ms
just to go 35,000 km up to the comsat and back down, which meant
a round trip from Australia to JPL would take at least 460 ms. Not
good enough.
It occurred to me that the flaw in the argument was that the
limitation applied only to a single stream, and that one way around
it would be to open multiple streams and "inverse multiplex" the
data across them. (Other people not at JPL, we discovered later,
had the same idea and built multi-stream FTP servers and clients,
for example.) So we wanted to test this idea out.
Remember where we are now. We have no application to run on these
Unix machines. We have no prototype of an application. We just have
an idea, and we want to see if it looks promising. If we find that
we can work around the single-stream limitation of TCP, then the
Unix/TCP approach is still in the running. If not, it's probably
dead. (It might still be a good idea, just not work because we're
not smart enough to pull it off. But in any case, this idea is
going nowhere unless we can move data.)
So Russ Byrne, Vance Heron (also a Rubyist and reader of this list),
and I built a test configuration and wrote some test code. The
test configuration used a hardware satellite simulator to impose
a variable delay between two routers. We connected a transmitter
machine to one router and a receiver to the other. The code was
about 600 lines of C, and used BSD sockets and select().
I claim that this code is a benchmark. It
satisfies my definition and the one on Wikipedia:
http://en.wikipedia.org/wiki/Benchmark_%28computing%29. It did
not remotely approximate our real ground applications, which do
a lot of formatting, routing, controlling, accounting, etc. It
tested only whether it was possible to achieve something near the
theoretical maximum throughput across the satellite simulator using
a particular technique.
To make a long story shorter, it was. The ability to control
precisely all relevant factors allowed us to answer some important
questions. First of all, we showed that it was possible to fully
load a T1 in the face of worse-than-realistic delays and bit-error
rates. We compared different Unix implementations, and discovered
(surprise) that Sun's TCP stack significantly outperformed some
of their competitors. We compared framing protocols on the serial
circuit and found a bug in Cisco's implementation of HDLC. (Cisco
already knew about it, but we confirmed it independently.) Later,
when we got satellite time, we verified the performance between two
machines linked by a 140,000 km path. And finally, the benchmark
allowed me to answer with confidence the most important question
I've ever been asked at work.
In 1993, the Galileo spacecraft's high-gain antenna failed to open.
This appeared at the time to be a devastating blow. The expected
data rate at Jupiter without the HGA was a factor of some 10,000
below what JPL had planned. (10 bits per second.) Not surprisingly,
the project's highest priority became figuring out how to achieve a
reasonable fraction of the science objectives before the spacecraft
arrived at Jupiter in 1995. The team ended up recommending some
hardware modifications to the ground antennas, development of even
more exotic data compression and error-correction codes, changes
in operations procedures, and a near-real-time arraying technique
that called for combining digital signals from large antennas in
Australia and California. That required moving lots of bits between
ground complexes and not losing any. The manager of the mission
rescue team asked me at one point "If we put in all these routers
and Unix boxes, will they support arraying for Galileo?" He needed
to know right then. I said "Yes, they will." He did, and they did.
This is not a personal moment of glory. Lots of people worked to
get the DSN to the point where it could make a major architectural
change. And the team that developed the engineering changes deserves
the credit. But one part of succeeding was being able to say "we
can do what Galileo needs", and we could say that because we'd used
our benchmark to hammer the problem into submission. We understood
it well enough to convince management to trust our conclusions.
Galileo went on to tremendous success:
http://www.newyorker.com/fact/content/?030908fa_fact.
Steve
Ilmari Heikkinen
On 15.6.2005, at 07:55, Steven Jenkins wrote:
> In this essay I'm going to attempt, one final time, to demonstrate
> that it is possible to have a useful benchmark. I'm going to do
> so by telling about a real benchmark that we used to solve a real
> problem in the Deep Space Network.
Thank you for the interesting tale, it was a pleasure to read.
Ralph "PJPizza" Siegler
> In this essay I'm going to attempt, one final time, to demonstrate
> that it is possible to have a useful benchmark. I'm going to do
> so by telling about a real benchmark that we used to solve a real
> problem in the Deep Space Network. But first, some clarifications:
>
One observation I would make would be that you set up benchmark/test/simulation that was very relevant to your problem domain, you didn't use some industry standard MIPS or TPH or such. That is the real problem with the type of benchmarks such as spawned the debate here, such things are interesting and I like to look at them, but that's as far as their usefulness go.
One of the things I do for a living is to migrate municipal databases from various Unix(tm) to Linux on x86 with that proprietary dbms that postgres will hopefully someday kill. Anyway, the usual transaction per second ratings for various machines are done using a benchmark that uses a standard set of tables from cached memory, so they are completely useless for sizing a cluster of Intel or AMD boxes to replace Unix big iron. We make our own tests with the client's dbms and software. So we could call that a "benchmark", though it's more of a benchmark/simulation/test type of thing.
Ralph PJPizza
Austin Ziegler
> On Wed, Jun 15, 2005 at 01:55:58PM +0900, Steven Jenkins wrote:
>> In this essay I'm going to attempt, one final time, to
>> demonstrate that it is possible to have a useful benchmark. I'm
>> going to do so by telling about a real benchmark that we used to
>> solve a real problem in the Deep Space Network. But first, some
>> clarifications:
> Steve, that was a wonderful and fascinating account, thanks very
> much.
Yes, it was. And, given the broadness of the definition that Steven
used, I agree: it was done with benchmarks. However, I also agree
with you (Ralph) that this is what I would consider a performance
simulation -- a prototype, if you will -- and not a benchmark *as
such*. At least, not in *common* parlance, although the proper
definition does include this.
> One observation I would make would be that you set up
> benchmark/test/simulation that was very relevant to your problem
> domain, you didn't use some industry standard MIPS or TPH or such.
> That is the real problem with the type of benchmarks such as
> spawned the debate here, such things are interesting and I like to
> look at them, but that's as far as their usefulness go.
Right. I *personally* wouldn't consider what Steven has described in
his fascinating story as a benchmark. He may have *benchmarked* some
performance with his simulation, and used that as a baseline for
further performance tests, but the marketers have taken over the
word by and large. The alioth "benchmarks" don't do what Steven did;
things like MIPS, TPS, SpecMARK, FLOPS, and other benchmark values
are pure marketing speak -- just like the alioth shootout.
Thank you, Steven.
James Britt
> ...
> Right. I *personally* wouldn't consider what Steven has described in
> his fascinating story as a benchmark. He may have *benchmarked* some
> performance with his simulation, and used that as a baseline for
> further performance tests, but the marketers have taken over the
> word by and large.
> The alioth "benchmarks" don't do what Steven did;
> things like MIPS, TPS, SpecMARK, FLOPS, and other benchmark values
> are pure marketing speak -- just like the alioth shootout.
So, do we call that "benchmarketing"?
James
--
http://www.ruby-doc.org - The Ruby Documentation Site
http://www.rubyxml.com - News, Articles, and Listings for Ruby & XML
http://www.rubystuff.com - The Ruby Store for Ruby Stuff
http://www.jamesbritt.com - Playing with Better Toys
Austin Ziegler
> Austin Ziegler wrote:
> > Right. I *personally* wouldn't consider what Steven has described in
> > his fascinating story as a benchmark. He may have *benchmarked* some
> > performance with his simulation, and used that as a baseline for
> > further performance tests, but the marketers have taken over the
> > word by and large.
> > The alioth "benchmarks" don't do what Steven did;
> > things like MIPS, TPS, SpecMARK, FLOPS, and other benchmark values
> > are pure marketing speak -- just like the alioth shootout.
> So, do we call that "benchmarketing"?
You know, it's a good thing that I didn't have a drink in my hands
when I read this.
Yeah. Benchmarketing. I like that.
Steven Jenkins
> One observation I would make would be that you set up
> benchmark/test/simulation that was very relevant to your problem domain,
> you didn't use some industry standard MIPS or TPH or such. That is the
> real problem with the type of benchmarks such as spawned the debate
> here, such things are interesting and I like to look at them, but that's
> as far as their usefulness go.
I said I had other examples :-).
It's been a long time since I was involved in one, but I'm reasonably
confident that we use "standard" benchmarks for large procurements. When
you spend US Government money, you have to jump through a lot of hoops
to ensure a level competitive playing field. A protest from a losing
bidder can tie you up for a long time, so you try to avoid that. Using
your own benchmarks for procurement qualification invites protest.
Nobody wins just because their TPC-A or whatever is highest. A Request
for Proposal may give a particular performance threshold, and the
proposing vendors use that to decide which of their products to propose.
They don't want to propose anything more expensive than they have to,
because they're in a cost competition. It's a rough and imperfect but
vendor-neutral way to talk about classes of performance. The real key is
that, if the vendors buy into it, they can't protest on that point.
Obviously, if one vendor claims dramatically better performance than
another in the same price class, that might be worth looking to. For the
most part, however, the benchmarks just establish who's in the game, and
most of the competition is on cost.
Steve
Stephen Kellett
<steven....@ieee.org> writes
>It's been a long time since I was involved in one, but I'm reasonably
>confident that we use "standard" benchmarks for large procurements.
I think some people have lost sight of what "benchmark" means. For
computer apps some people have been claiming its TPS, MIPS or whatever
form of throughput they are proposing. However, take a step back and
think about "benchmark" in more general terms and you get a better idea
for what a benchmark is. This is what Steven Jenkins was identifying
with his satellite TCP/IP benchmark.
A benchmark is something, anything by which you can compare. Typically
it is the best of breed at some point or other. Here is an example:
I play various musical instruments, one of them being the Border Bagpipe
made by Jon Swayne. Jon Swayne is a legend in his own lifetime to many
dancers and many musicians in the UK. For dancers it is because he is
part of Blowzabella, a major musical force in social dancing throughout
the last 25 years. For musicians, and particularly bagpipers, it is
because he took the bagpipe, an instrument known for not typically being
in tune, and if it was, not necessarily in tune with another bagpipe of
the same type (or even by the same maker!) and creating a new standard,
a new benchmark, if you will, by which other bagpipes are judged. Its
not just Jon Swayne, there are some other makers, but they changed
everyones perception and his pipes are the benchmark by which others are
judged (yes, they really are that good). When you talk to pipers in the
UK and mention his name there is a respect that is accorded. You don't
get that without good reason. Anyway I digress.
The benchmark for Steven's satellite test was did it match the
round-trip criteria. I think absolutely Steven's example is a benchmark.
Its much looser than other benchmarks, but thats not the point. The
point is did it serve a purpose?
For other people the benchmark will be does it perform the test within a
given tolerance? For other people it may be how much disk space does it
use? or is the latency between packets between X and Y? For other people
it will be is it faster than X?
Where Austin's point comes in is that he points out the latter test is
meaningless because you are comparing apples with oranges, when you
should really be comparing GMO engineered (optimized) apples with GMO
(optimized) oranges to be even getting close to a meaningful test. Even
so you are still comparing cores to segments and it gets a bit messy
after that, although they both have pips.
Even so, I once worked for a GIS company (A) that wrote their software
in C with an in-house scripting language. We won the benchmarks when in
competition with other GIS companies. The competition won because of
clever marketing. Their customers lost (*) though because the
competitors software was too hard to configure and our marketing people
were not smart enough to identify this and inform the customer of the
problem.
What sort of benchmarks were being tested?
o Time to compute catchment area of potential customer base within X
minutes drive given a drive time to location.
o Time to compute catchment area of potential customer base within X
minutes drive given a drive time from location.
o Time to compute drive time to location of potential customer base
within X minutes drive given a particular post code area.
o Time to compute drive time from location of potential customer base
within X minutes drive given a particular post code area.
o Think up any other bizarre thing you want.
Times to and from are/location may not be the same because of highway
on/off ramps, traffic light network delay bias and one-way systems.
Superstores often don't care much about drive time from, but care a lot
about drive-time to. For example drive time from may be 15mins, but
drive-time to may be only 5mins.
As you can see the customer requirements are highly subjective, but the
raw input data is hard data - maps and fixed road networks. The
computing time etc, thats also a fixed reality given the hardware.
Its all about perception and need.
I think the benchmarketing term is quite apt for most benchmarks.
...and Steven, your story was great. I could really relate to a lot of
that.
Stephen
(*) Its a matter of debate, they also used an in-house language and
finding non-competitor engineers that used the language was nigh on
impossible and thus they were very expensive to hire to do the
configuration. Our (A) stuff was not so configurable, but didn't need to
be.
When were we doing this stuff? 90..94 for me. X11 and Motif was the cool
stuff back then.
--
Stephen Kellett
Object Media Limited http://www.objmedia.demon.co.uk/software.html
Computer Consultancy, Software Development
Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
Ralph "PJPizza" Siegler
>
> I said I had other examples :-).
>
> It's been a long time since I was involved in one, but I'm reasonably
> confident that we use "standard" benchmarks for large procurements. When
> you spend US Government money, you have to jump through a lot of hoops
> to ensure a level competitive playing field. A protest from a losing
> bidder can tie you up for a long time, so you try to avoid that. Using
> your own benchmarks for procurement qualification invites protest.
>
> Nobody wins just because their TPC-A or whatever is highest. A Request
> for Proposal may give a particular performance threshold, and the
I used to do some spending of U.S. D.O.E. money at Fermilab for servers/workstations/networks for CADD/CAE , as you say the standard benchmarks were a starting point to see what vendors might be considered, but for justifications the capabilities for in-house needs were the main thing. My projects were in $100-$200K range, surely a few orders of magnitude smaller than your NASA ones, with the procurement requirements not as burdensome.
Our group made civil engineering packages (all those tunnels and collision halls) for outside bid, and of course there the spec book that accompanied the drawings was what ruled. That could be called a set of benchmarks, I suppose; they were a mix of construction industry standards and what our engineers had calculated.
Ralph "PJPizza" Siegler