Benchmark programming languages

[ISockGo]

Hi,

       Every second day, there is a new programming language or new framework. I am working on Go from last 4-5 months, really liked it easy to write and clear to understand and works fine with end products. Before go, So, I thought, lets benchmark languages: So I started with Go and java speed test. And go is way slower than Java :(. I run a for loop for 30000000 times. 

I am running these test on IMac, Processor  2.7 GHz Intel Core i5, Memory  8 GB 1333 MHz DDR3, OS X 10.8.1 

Go: 

Total time taken: 63.806ms, 30000000, st: 2012-09-10 11:58:08.341411 -0400 EDT, et: 2012-09-10 11:58:08.405217 -0400 EDT

Java:

Total time taken: 4, 30000000, st: 1347291802315,  et: 1347291802320

Code might make difference: So here is the code for both: (improve it if you want to benchmark it better)

Go:---------------------------------------------------------

t1 := time.Now();

j :=0;

for i := 0; i < 30000000; i++ {

j += 1;

}

t2 :=time.Now();

t3 := t2.Sub(t1);

fmt.Printf("Total time taken: %s, %d, st: %s, et: %s\n", t3, j,t1, t2);

Java:-------------------------------------------------------

   public static void  main(String[] args){

        int j = 0;

        long startTime = new Date().getTime();

        for(int i=0; i<30000000; i++){

                j +=1;

        }

        long endTime = new Date().getTime();

        long mSeconds = (endTime - startTime);

        System.out.println("Total time taken: "+mSeconds+", "+j+",st: "+ startTime+", et: "+endTime );

    }

Can some one benchmark it for C, C++, DotNet, Python, (may be php, groovy, perl) and more... 

Also suggest the other methods to benchmark it. 

Keep enjoying coding ..

[Jan Mercl]

On Mon, Sep 10, 2012 at 6:07 PM, ISockGo <amitab...@gmail.com> wrote:

a) Java really insists that 2320 - 2315 equals 4 ???

b) 4e-3 [s] / 3e7 [1] is 133,333333333×10⁻¹² [s] (133 picoseconds) per
cycle, i.e. your CPU, if it spends only one clock cycle per the whole
loop body including the repeat/jump instruction, must run @ 7.5 GHz
minimum. Want! ;-)

-j

[Ethan Burns]

Hi,

Java seems to be optimizing something here that Go is not.  I am guessing that the Java compiler changes the entire loop into a single assignment.  If change the constant 30000000 to a variable, Java takes more time.  Try this one: http://itsapad.appspot.com/295002.

Ethan

[ISockGo]

Sorry.. Java is not 4 ms it was 5, typo..

Java:

Total time taken: 5 30000000, st: 1347291802315,  et: 1347291802320

On Monday, September 10, 2012 12:07:55 PM UTC-4, ISockGo wrote:

[ISockGo]

Yes, its taking longer if I will assign 30000000 to a variable, but still golang taking way longer to execute that. We are not assigning any variable in for loop in golang though.

Java (with defining 30000000 to a variable and then use it)

Total time taken: 21, 30000000,st: 1347296721846, et: 1347296721867



/--------------------------------------------------

[Larry Clapp]

Are you familiar with the Computer Language Benchmark Game?  http://shootout.alioth.debian.org/u64q/benchmark.php?test=all&lang=java&lang2=go

On these benchmarks, Java can be 2-8 times faster, though on the other hand it can use 2-24 times as much memory.

-- Larry

On Monday, September 10, 2012 12:07:55 PM UTC-4, ISockGo wrote:

[Ian Lance Taylor]

On Mon, Sep 10, 2012 at 9:07 AM, ISockGo <amitab...@gmail.com> wrote:
>
> Every second day, there is a new programming language or new
> framework. I am working on Go from last 4-5 months, really liked it easy to
> write and clear to understand and works fine with end products. Before go,
> So, I thought, lets benchmark languages: So I started with Go and java speed
> test. And go is way slower than Java :(. I run a for loop for 30000000
> times.

Good benchmarking is really hard. Honest.

> Go:---------------------------------------------------------
>
> t1 := time.Now();
> j :=0;
>
>
> for i := 0; i < 30000000; i++ {
> j += 1;
> }
>
> t2 :=time.Now();
> t3 := t2.Sub(t1);
> fmt.Printf("Total time taken: %s, %d, st: %s, et: %s\n", t3, j,t1, t2);

If I build this code with gccgo using -O2, it takes 102us. How can it
be that fast? Because the entire loop is optimized away and replaced
with a single assigment to j.

Ian

[Paul]

I can't  see what version of Go you are using. Please note that there have been substancial performance increases recently to the Go-language. 

I also don't see the measurement of application startup time in your benchmark, wouldn't that count on a raw speed test? 

I would find it interesting to see how java compares against  a concurrent tcp benchmark on a parallel machine. 

[ISockGo]

I am using current version of GO lang: go version go1.0.2.

[JussiJ]

> Can some one benchmark it for C, C++, DotNet, Python, (may be php, groovy, perl) and more... 

FWIW here is the C# version:

    // Zeus Editor C# Template

    using System;

    

    namespace SampleApplication

    {

        static class Program

        {

            static void Main()

            {

                int j = 0;

                DateTime startTime = DateTime.Now;

                for(int i=0; i<30000000; i++){

                        j +=1;

                }

                DateTime endTime = DateTime.Now;

                TimeSpan mSeconds = (endTime - startTime);

                Console.WriteLine("Total time taken: "+mSeconds.Milliseconds.ToString()+", "+j+",st: "+ startTime.ToString()+", et: "+endTime.ToString() );

            }

        }

    }

When I compare it to your Go version:

    package main

    

    import "fmt"

    import "time"

    

    func main() {

        t1 := time.Now()

        j := 0

    

        for i := 0; i < 30000000; i++ {

            j += 1

        }

    

        t2 := time.Now()

        t3 := t2.Sub(t1)

        fmt.Printf("Total time taken: %s, %d, st: %s, et: %s\n", t3, j, t1, t2)

    }

I consistently get these results:

    C:\temp>go_loop_test.exe

    Total time taken: 78.125ms, 30000000, st: 2012-09-11 13:01:12.953125 +1000 AUSEST, et: 201 2-09-11 13:01:13.03125 +1000 AUSEST

    

    C:\temp>cs_loop_test.exe

    Total time taken: 187, 30000000,st: 11/09/2012 1:01:18 PM, et: 11/09/2012 1:01:19 PM

So the go version is twice the speed.

NOTE: Why are both nunber so high you might be asking. Because the machine I am running these test on is ancient.

Also the C# numbers are a bit unfair since the version of C# used is very old:

    C:\temp>csc

    Microsoft (R) Visual C# 2008 Compiler version 3.5.30729.1

    for Microsoft (R) .NET Framework version 3.5

    Copyright (C) Microsoft Corporation. All rights reserved.

Cheers Jussi

[Kevin Gillette]

The version of C# that you used may be rather old, but it could also be said that C# (and .NET) 3.5 was at a greater relative level of maturity than Go is right now, with 3.5 coming after nearly 6 years of platform existence, and Go 1.0.2 coming out 3-4 years after the language was announced.

Additionally, Microsoft considers C# .NET their flagship commercial/enterprise language/platform, probably giving it considerably more salaried attention and resources than Go is getting from Google, since Google heavily uses a variety of other languages, including Python and C++, and has more of a utilitarian tool culture, rather than trying to make "one language fit all tasks."

That said, I don't think it's bad at all that gc produces a binary that may be doing 90 million reads and 60 million writes in 78ms, since:

1) no "real" program does intentional "do nothing" loops, and thus it's a very low-value candidate for optimization (if it can be optimized away, it at best indicates poorly written code, and at worst, indicates a bug).

2) considering for me, using JussiJ's Go snippet, and the following C code, compiled -O0 (no optimizations):

#include <stdlib.h>

#include <stdio.h>

#include <sys/time.h>

int main(int argc, char *argv[])

{

int i, j = 0;

double ms;

struct timeval tv1, tv2;

gettimeofday(&tv1, NULL);

for(i = 0; i < 30000000; i++)

j += 1;

gettimeofday(&tv2, NULL);

ms = (tv2.tv_sec-tv1.tv_sec)*1000 + ((double)(tv2.tv_usec-tv1.tv_usec))/1000;

printf("Total time taken: %fms\n", ms);

return 0;

}

Or this python:

from time import time

i = j = 0

start = time()

while i < 30000000:

    j += 1

    i += 1

end = time()

print "Total time taken: %fms" % ((end - start) * 1000)

I get the following results:

Go: Total time taken: 70.884ms ...

C -O0: Total time taken: 62.040000ms

CPython 2.7.3: Total time taken: 22365.461826ms

That tells me it takes at least 62 ms to do that much work on my system. Period. Here, go is only 14% slower for an utterly meaningless cpu-bound task. That's hardly "bad". Python's is 360x (times, not percent) slower, which is to be expected.

For those of you who think this was a good benchmark, read the entirety of the site: <http://shootout.alioth.debian.org/>. They actually have benchmarks there that aren't so brittle, yet they rightly admit that their benchmarks are not representative of the quality of the languages they are benchmarking, and further, all of these benchmarks only deal with cpu-bound tasks.