Improved packet reading

[Phil Bayfield]

I've committed the "prototype" for new packet reader, after much
consideration I think this method will provide the greatest efficiency
for the library.

Firstly there are 3 new packet interfaces:

packetReadable - a readable packet which must implement the read()
method.
packetWritable - writable packet which must implement the write()
method.
packet - implements packetReadable/packetWritable.

The Reader:

See: https://github.com/Philio/GoMySQL/blob/dev/reader.go

The reader primarily consists of the readPacket method, which reads
the packet length, sequence number and the entire packet data into a
slice.
The idea is that readPacket is called with a bitwise value
representing the possible packet types expected, e.g.

readPacket(PACKET_OK|PACKET_ERROR|PACKET_RESULT)

We can then determine which packet type we've just received from the
server and load it into the appropriate packet struct (which must
implement packetReadable) by calling the read method, e.g.

p = &packetOk {
header: hdr,
}
p.read(data)

This will massively simplify all the packet handling, can simply
extract data values right out of the data slice, should in theory be
faster too.

[Evan Shaw]

Hey, I just joined this list because I was messing around with the
MySQL protocols in Go, but realized that there's already a package in
development.

I think you'd be able to simplify things even further if you used a
bytes.Buffer instead of using a byte slice directly. What I'd do:

In reader.go, instead of calling io.ReadFull, create a bytes.Buffer
and call io.Copyn(buf, r.conn, pktLen). (The error checking below that
is also unnecessary; Copyn will return an error if it copies less than
pktLen bytes.) Pass the buffer on when creating the packet.

In packet.go, you'd be able to get rid of packetBase.readSlice and all
the index incrementing and appends. You could probably use encoding/
binary to do the integer packing/unpacking also.

I can code this up later to illustrate. I have some other suggestions
I might try to put into a pull request.

- Evan

[Phil Bayfield]

Hi Evan,

Thanks for your feedback!

I'm quite interested in finding the fastest and most efficient way to handle packets.

The thinking behind the current dev code is that using a slice would be quicker than calling additional libraries and as it's a reference type I can manipulate it a fair bit without any additional memory usage cost.

Other priorities are adding features, implementing the protocol better, correct previous mistakes, avoid duplicated code like current release branch etc.

Certainly be interested to see a simple usage example based on latest dev code if you have time. I'll have to work on some benchmarks to compare processing time etc.

Cheers

Phil.

[Phil Bayfield]

Quick update to this...

The latest dev branch now contains complete connect logic + error
handling.

This could be a good starting point for experimenting and benchmarking.

[Evan Shaw]

On Thu, Jan 27, 2011 at 11:49 AM, Phil Bayfield <ph...@bayfmail.com> wrote:
> I'm quite interested in finding the fastest and most efficient way to handle packets.
> The thinking behind the current dev code is that using a slice would be quicker than calling additional libraries and as it's a reference type I can manipulate it a fair bit without any additional memory usage cost.

I wouldn't expect much of a performance difference using bytes.Buffer.
Underneath, it basically is just a byte slice and an offset like what
you have now. There'd be a small increase in runtime due to function
call overhead and some additional memory costs, but I personally think
the simpler code is worth it. encoding/binary's Read and Write
functions would likely be slower due to the reflection code, so it's
understandable if you wouldn't want that.

> Other priorities are adding features, implementing the protocol better, correct previous mistakes, avoid duplicated code like current release branch etc.

Got it. Any interest in having compression implemented? It seems like
that would be an easy one to tackle, with zlib already part of the
core Go library.

> Certainly be interested to see a simple usage example based on latest dev code if you have time. I'll have to work on some benchmarks to compare processing time etc.

Sure. The easiest thing to do might be to set up a Client that
communicates with a dummy server. That should eliminate variations due
to different servers and mean that the benchmarker doesn't have to
enter things like username and password. (This won't work if you're
actually interested in benchmarking the server, of course, but I'm not
sure why you would be.)

Then it's just a matter of figuring out what commands/packets you want
to pass through.

- Evan

[Phil Bayfield]

On 27 January 2011 00:50, Evan Shaw <eds...@gmail.com> wrote:

On Thu, Jan 27, 2011 at 11:49 AM, Phil Bayfield <ph...@bayfmail.com> wrote:
> I'm quite interested in finding the fastest and most efficient way to handle packets.
> The thinking behind the current dev code is that using a slice would be quicker than calling additional libraries and as it's a reference type I can manipulate it a fair bit without any additional memory usage cost.

I wouldn't expect much of a performance difference using bytes.Buffer.
Underneath, it basically is just a byte slice and an offset like what
you have now. There'd be a small increase in runtime due to function
call overhead and some additional memory costs, but I personally think
the simpler code is worth it. encoding/binary's Read and Write
functions would likely be slower due to the reflection code, so it's
understandable if you wouldn't want that.

I'm not sure what the performance of reflection is in Go, but I've found it to be slow in other languages, whether this is consistent to Go, I guess a benchmark would reveal! I did actually look at the bytes package before writing the current implementation, but figured I would only need a stripped down sub-set of the functionality offered by it.

> Other priorities are adding features, implementing the protocol better, correct previous mistakes, avoid duplicated code like current release branch etc.

Got it. Any interest in having compression implemented? It seems like
that would be an easy one to tackle, with zlib already part of the
core Go library.

Would be a nice feature to add too, and relatively simple to implement.

> Certainly be interested to see a simple usage example based on latest dev code if you have time. I'll have to work on some benchmarks to compare processing time etc.

Sure. The easiest thing to do might be to set up a Client that
communicates with a dummy server. That should eliminate variations due
to different servers and mean that the benchmarker doesn't have to
enter things like username and password. (This won't work if you're
actually interested in benchmarking the server, of course, but I'm not
sure why you would be.)

Sounds like an interesting idea, it would be quite time consuming though to implement something like this. Perhaps just adding timers around the networking code might be quicker and just as effective. I generally just test on localhost so there probably wouldn't be too much overhead this way anyway.

Then it's just a matter of figuring out what commands/packets you want
to pass through.

- Evan

--
You received this message because you are subscribed to the Google Groups "GoMySQL" group.
To post to this group, send email to gom...@googlegroups.com.
To unsubscribe from this group, send email to gomysql+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gomysql?hl=en.