Ip and Port - tracker compact mode

[Guillermo Estrada]

Hello fellow gophers. I'm writing a Bittorrent tracker in Go, with db backend on MongoDB using mgo. (to be released on github soon). I need some insight/ideas in how to create the compact form of string + port.

the compact form is a 6 byte for Ipv4 (4byte + 2 byte port) and 18 byte for ipv6.

The compact extension tells the tracker to send the peers key as a single string that represents all address and ports of peers. For example, a client at the IP 10.10.10.5 listening on port 128 would be coded as a string containing the following bytes 0A 0A 0A 05 00 80

The IP I either get it from request.remoteAddr of the call or it can be supplied in ip= format on the URL, the port must always come from the URL. I was thinking of storing the compact form in the DB along with the string of both IP and Port, in order to save some CPU conversions, or should I just save one of the forms? maybe the compact one?

data := strings.Split(request.RemoteAddr, ":") //includes port
ip := net.ParseIP(data[0])

and then I check len(ip) to either call To4() or To16() on the IP, does this support IPv6?. Afterwards I need to append the port in a 2 byte format. That means:

port, err := strconv.Atoi(request.FormValue("port"))
// check err

Here how do I append the port as a 2 byte []byte slice? does conversion from into to []byte does that? do I have to convert first to uint16?

And then i have to create a string/[]byte to append every IP+port of every peer downloading some torrent. That is why I thought it would be more efficient to store the compact form and in every case the client does not support compact I just expand the IP port with:

compact := //Get value form data store and convert to []byte if it isn't already
//check len(compact)
ip := net.IPv4(compact[0],compact[1],compact[2],compact[3]) // no IPv6 support? How is a 16 byte representation of Ipv4?
// how do I export the port as a []byte slice of 2 bytes into an integer? conversion?

Well, compact mode just creates a string/[]byte of all the address and ports in IPv4 (for IPv6 there is another list) but appending each to a []byte seems not well performant.
For the non compact form a Bencoded dictionary of the fields must be created (more cpu) so I thought of storing both forms, though I dunno how much impact will have. I can work around
most of this perks, but I do ask to have the correct/more efficient way of doing it before hand.
Anyway, ideas, insight? answers? thnx a lot!



[Kyle Lemons]

If I understand you correctly, you're looking for an efficient way to store an unordered list of six-byte tuples?  I would have a monolithic backing store (two, if you need the bencoded and the bytes) with some clever optimizations for addition/removal.

1) A simple equation can be used to figure out where the Nth dataum is.

 - For the IPs, it's 6*N for the IP, and 6*n+4 for the port, and you can pretty easily slice the right four bytes and cast it to a net.IP for your own use.

 - For the bencoded values, a similar X+S*N equation should work, where X is the length of the bencoded header and S is the stride of the bencoded IP/port map.

2) When inserting a new value, append onto the backing store slice.  This will probably require an rwmutex, tough you can write your own append to reduce the critical section and get a more scalable append.

3) When deleting a value, you can shorten the slice and copy the last value over the one you're deleting.  This is less scalable (global rwmutex) but allows you to not have to iterate through a list on send, since it's already what you need.

4) If you can tolerate the two lists not being identical (you probably can), update them separately, with the more canonical source being updated first.  This makes things more scalable because you're not locking down both lists for every delete/append.

[Guillermo Estrada]

Sorry if I didn't made that part clear. I can't store them as an array or an unordered list cause I have more information to store, the peer_id and the info_hash of the torrent it is tracking. When a peer asks for an announce, I just give him a subset of that list based on the info_hash of the torrent provided and a MAX peers value arbitrarily determined, so I actually need to search for those peers (IP+port), they also provide me additional information I store in the DB. Something like...

type Peer struct {
  Id      bson.ObjectId "_id,omitempty"
  PeerId  string        "peer_id"
  Ip      string        "ip"
  Port    string        "port"
  Compact string        "compact"
}

type Torrent struct {
  Id        bson.ObjectId "_id,omitempty"
  InfoHash  string        "info_hash"
  Completed int64         "completed"
}

type Tracker struct {
  Id         bson.ObjectId "_id,omitempty"
  InfoHash   string        "info_hash"
  PeerId     string        "peer_id"
  LastUpdate int64         "last_update"
}

Kindish... (there are more fields on Tracker and such but are not important in this example) but I always find code more easily to understand. i do not know what is more optimal, to store 'ip', 'port' and 'compact' and just calculate the compact form on the first announce of a peer, or store either 'ip' and 'port' or just compact, and give the cpu (and memory) the conversion process on each announce.

Either way i need help converting the port from int to []byte and back, any help there? :P

[David Anderson]

encoding/binary in the stdlib does exactly that, and more.

[Guillermo Estrada]

encoding/binary in the stdlib does exactly that, and more.

Thnx David I'll check it out.