Integrating Go with C: the ZooKeeper binding

[Gustavo Niemeyer]

Greetings,

Some interesting news about the Go and ZooKeeper integration which
we've developed at Canonical:

Integrating Go with C: the ZooKeeper binding experience
http://j.mp/gozkxp

--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/blog
http://niemeyer.net/twitter

[Kai Backman]

Nice work, both on the actual integration and the blog post about it.
I'm just started writing a lockserver in Go and it's interesting to
look at the Go API you designed for Zookeeper.

Kai

--
Kai Backman, programmer
http://tinkercad.com - The unprofessional solid CAD

[k...@xph.us]

Also worth watching is Doozer, a lock service written in Go. It's
designed to avoid much of Zookeeper's unnecessary complexity.

https://github.com/bmizerany/doozer

It's not ready for production, but we're working hard. In the future
we'll have blog posts and more documentation.

kr

[Gustavo Niemeyer]

Oh, that sounds very interesting indeed.

Can you give some background about the project? What are you guys
trying to achieve, and what's the motivation behind it?

Also, can you provide a very ad-hoc roadmap of what's ahead?

[Michael Hoisie]

Yeah, that project looks very interesting. You should fill out the
readme :)

[k...@xph.us]

Thanks for the encouraging words. We're working on a blog post that
will give a lot more context, including our motivation and roadmap.
I'll be sure to send a link this way when it's published. For now, I
will be terse: we want highly available locks, a name service with
consistent caching, and distributed process monitoring.

kr

[Russ Cox]

>    Integrating Go with C: the ZooKeeper binding experience
>    http://j.mp/gozkxp

Thanks for sharing. Nice writeup about cgo.

Russ

[Jeff R. Allen]

Gustav, thanks for your article. It helped me understand slices well
enough to solve a problem I was having getting efficient code to embed
a large amount of data in a Go program so it is available at runtime
in the form of a []byte (i.e. a bit like how a ramdisk is embedded in
the Linux kernel sometimes).

I wrote up what I learned while investigating this here: http://blog.nella.org/?p=793

-jeff

PS: Why isn't there a type based on []byte that satisfies io.Reader or
ioReaderAt (ok, this one is less commonly used, I admit). In order to
decode a Zipfile in a []byte, I made a type based on []byte that
satisfies io.ReaderAt, if anyone is interested. It is also on my blog.

[Eric Clark]

On Mon, Dec 13, 2010 at 9:52 AM, Jeff R. Allen <j...@nella.org> wrote:
> PS: Why isn't there a type based on []byte that satisfies io.Reader or
> ioReaderAt (ok, this one is less commonly used, I admit). In order to
> decode a Zipfile in a []byte, I made a type based on []byte that
> satisfies io.ReaderAt, if anyone is interested. It is also on my blog.

http://golang.org/pkg/bytes/#Buffer

ReadAt shouldn't be too hard to implement for bytes.Buffer.

[Rob 'Commander' Pike]

It's quite hard and a bad idea. The design of bytes.Buffer is to
implement a stream; changing to a random access object would not only
make the code significantly more complex and slower - and speed is
very important to this code - but would change the meaning of "read"
from "consume" to "scan".

It would be much wiser to make a different type, probably one each for
bytes and strings, that implemented seeking and scanning rather than
consuming as the model for reading.

-rob

[Gustavo Niemeyer]

Hi Jeff,

> Gustav, thanks for your article. It helped me understand slices well
> enough to solve a problem I was having getting efficient code to embed
> a large amount of data in a Go program so it is available at runtime
> in the form of a []byte (i.e. a bit like how a ramdisk is embedded in
> the Linux kernel sometimes).

I'm glad you found it useful.

> I wrote up what I learned while investigating this here: http://blog.nella.org/?p=793

Thanks for the credits Jeff.

There's one point worth mentioning about this:

"""
Mucking around with pointers like this can get you in trouble: if the
garbage collector decides that the only reference it has to the thing
at slice.Data, and it is ready to garbage collect your slice, then
it’s going to try to garbage collect your underlying array as well. If
that array came from someplace other than the Go heap, you’re going to
corrupt something. But it is relatively easy to prevent a slice from
ever getting GC’d; all you have to do is hold it in a map at global
scope.
"""

There are indeed many reasons why this may create trouble, but the
issue described isn't one of them. The Go runtime will not attempt to
collect garbage in arbitrary memory locations. Problems are observed
in the other direction: if the underlying memory is deallocated or
reused for something else, the program will silently see garbage or
crash when using the slice.

Also, to be honest, it's not entirely clear to me that Cgo and some of
the described magic is really necessary for what is being done there.
Have you considered using a string for your PNG data? Strings are
immutable, so they'll likely offer something closer to what you want.

[peterGo]

Rob,

> > ReadAt shouldn't be too hard to implement for bytes.Buffer.
>
> It's quite hard and a bad idea. The design of bytes.Buffer is to
> implement a stream; changing to a random access object would not only
> make the code significantly more complex and slower - and speed is
> very important to this code - but would change the meaning of "read"
> from "consume" to "scan".
>
> It would be much wiser to make a different type, probably one each for
> bytes and strings, that implemented seeking and scanning rather than
> consuming as the model for reading.

You had said something similar in an earlier thread. Therefore, I'm
writing and testing a bytes.File type, which is distinct from the
bytes.Buffer type. The bytes.File type adopts the semantics of the io
package interfaces, and the implementation emulates the os package.
The bytes.File ReadAt method satisfies the io.ReaderAt interface and
produces the same results as the os.File ReadAt method. A bytes.File
is a RAM file emulating a disk file.
Peter

[Rob 'Commander' Pike]

On Dec 13, 2010, at 10:06 AM, peterGo wrote:

> Rob,
>
>>> ReadAt shouldn't be too hard to implement for bytes.Buffer.
>>
>> It's quite hard and a bad idea. The design of bytes.Buffer is to
>> implement a stream; changing to a random access object would not only
>> make the code significantly more complex and slower - and speed is
>> very important to this code - but would change the meaning of "read"
>> from "consume" to "scan".
>>
>> It would be much wiser to make a different type, probably one each for
>> bytes and strings, that implemented seeking and scanning rather than
>> consuming as the model for reading.
>
> You had said something similar in an earlier thread. Therefore, I'm
> writing and testing a bytes.File type, which is distinct from the
> bytes.Buffer type. The bytes.File type adopts the semantics of the io
> package interfaces, and the implementation emulates the os package.
> The bytes.File ReadAt method satisfies the io.ReaderAt interface and
> produces the same results as the os.File ReadAt method. A bytes.File
> is a RAM file emulating a disk file.

That sounds like the right approach.

-rob