Re: [go-nuts] Not happy with the numbers we have for Go database tests

[David Symonds]

Have you tried profiling your program and seeing where it's spending its time?

http://blog.golang.org/2011/06/profiling-go-programs.html

[brainman]

On Friday, 19 April 2013 14:48:10 UTC+10, teona...@gmail.com wrote:

Just an idea. You have global "query" variable that stores *sql.Stmt. You prepared query at the start of your program, and, I suspect, assume it is used for all your queries. I could be wrong, but, I believe, your query statement could be re-prepared under the covers, if database/sql package discovers there is a danger of access contention from multiple goroutines. You could easily discover that by putting debug print statements inside of your database driver somewhere where it prepares sql statements.

Alex

[Brad Fitzpatrick]

The code in that link seems fine.  The code is how I would write it.  I don't know how that driver is implemented, though, and I haven't profiled the database code much (and have never profiled any other drivers).

There's probably a lot of low-hanging fruit.  Things in the tree (JSON, HTTP) have seen lots of benchmarking and performance work.  Out-of-tree drivers--- not sure.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[brainman]

Database "prepare" is normally very expensive operation. And it is something that Go has no control over. The only thing you could do with it, is use as little as possible of it.

Alex

[Sanjay]

From a casual reading of the source, it looks to me like the driver is allocating excessively.

In packets.go[1], implementation of readColumns calls readPacket in a loop, once for every column. Each read packet allocates two byte arrays internally. At a guess, I'd say that its the allocation overhead.

[1] https://github.com/go-sql-driver/mysql/blob/master/packets.go

[Sanjay]

Also, I'm curious what the custom implemented buffer[2] offers over bufio.Reader...

[2] https://github.com/go-sql-driver/mysql/blob/master/buffer.go

[Sanjay]

(pull requests incoming)

[Alexey Borzenkov]

Could it be that in the Go case prepared statements are the problem? Simple statements are usually much faster when you are not using prepared statements, so does it change if you use db.QueryRow(fmt.Sprintf("SELECT id, randomNumber FROM World where id = %d", rand.Intn(DB_ROWS)+1))?

On Fri, Apr 19, 2013 at 8:48 AM, <teona...@gmail.com> wrote:

Hi all,

We recently completed Round 3 of our web application framework tests.  This included Go 1.1 and a community-contributed implementation of our single- and multi-query database tests.

http://www.techempower.com/benchmarks/#section=data-r3

Go 1.1 nearly doubled the simple JSON serialization numbers, putting Go in the 200,000+ club.  But the database numbers are alarmingly bad.  There must be something crucially wrong with the database test implementation.

May I ask one or more of the Go experts here to please investigate the following test code?

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/go/src/hello/hello.go

The gist of the test is quite simple: accept a URL query parameter named "queries" which represents the number of distinct queries the server must run to fulfill the request.  For each query, fetch one random row from a table known to have 10,000 total rows.  Put the set of fetched rows together as a list, serialize that to JSON, and send it off as a result.

I didn't write the code, and I am a Go noob.  To my untrained eye the code looks fine.

We'd like to get this repaired and properly represent Go's performance in Round 4.

Thanks so much!  Please let me know if you have any questions, suggestions, or complaints.  :)

[Alexey Borzenkov]

I tried running your go server code (modified to actually check for errors from a Scan call!) locally and here's what I found:

When using prepared statements and any concurrency (e.g. ab -c 3) github.com/go-sql-driver/mysql starts making mysql connections like crazy for some reason and mysql replies with "Error 1040: Too many connections". When not using prepared statements it doesn't happen.

[Julien Schmidt]

https://code.google.com/p/go/issues/detail?id=4805 might be related.

The database/sql package doesn't allow to set a max. number of connections. As a result more connections are opened than the server can handle.

If you have a suggestion how to prevent this on driver side... you got my attention.

[Julien Schmidt]

Untested and I honestly don't like this "solution", but does something like this http://play.golang.org/p/SIE-JmxuMP work? (simple semaphore)

[Alexey Borzenkov]

No, the problem is a little different. By default database/sql has a limit of 2 idle connections, and if connection is returned that is over that limit it gets closed. The problem is that with concurrency=3 there are 3 connections needed, so 1 additional connection over that limit is always created, but it's never closed because connection is in Stmt.css and all connections that ever happened to be used by Stmt are set as dependencies and that dependency is only removed on close.

I think this is incorrect. Dependency should only be added in Stmt.connStmt and removed in releaseConn.

[Alexey Borzenkov]

Submitted a bug report:

https://code.google.com/p/go/issues/detail?id=5323