Avro genration & RPC client in GO

[Ajay Bodhe]

Hello,

I am looking for a library to generate avro based on scehma provided.
Also I need binary encoding & snappy compression.

This avro datum then should be sent using an RPC client to kafka cluster

Is there any avro encoder/decoder & RPC client in Golang.

[Tin Le]

If you can wait a bit, there will be an avro lib that is fast, spec compliant, etc (IMHO) that will be open sourced soon.   I can't say much more until legal give their ok.

Tin

[kar...@gmail.com]

I'm waiting on my company's legal team to give me the okay for the library name.

Benchmarks put it in the same performance playing field as the reference C implementation. API is straightforward.

One creates a codec instance from a JSON schema string, and then uses that codec instance to Encode directly to a io.Writer, or decode directly from an io.Reader.

The same codec is meant to be stateless and long lived, and was designed so I can use it to process data streams from Kafka in concurrently. The constructor works by compiling a state machine with function pointers to encode and decoders. 

// uses codec created above, and an io.Reader, definition not shown
    datum, err := codec.Decode(r)
    if err != nil {
        return nil, err
    }

// uses codec created above, an io.Writer, definition not shown,
    // and some data
    err := codec.Encode(w, datum)
    if err != nil {
        return nil, err
    }

Another example, this time leveraging bufio.Writer:

    // Encoding data using bufio.Writer to buffer the writes
    // during data encoding:
 
    func encodeWithBufferedWriter(c Codec, w io.Writer, datum interface{}) error {
        bw := bufio.NewWriter(w)
        err := c.Encode(bw, datum)
        if err != nil {
            return err
        }
        return bw.Flush()
    }
 
    err := encodeWithBufferedWriter(codec, w, datum)
    if err != nil {
        return nil, err
    }

[Ajay Bodhe]

My basic requirements are:
1. avro schema given, convert json or fields of GO structs into avro, also does binary encoding & snappy compression
2. avro rpc client or http client support to push packets to kafka cluster
3. can we have more than one datum into single avro record/packet to be sent to kafka cluster?
4. schema sharing between client & server, have persistent connection between client & server, dynamic changes in schema should be handled.

@Tin & Karrick,
Are guys working on same lib?
There are few go-avro libs already there on github
https://github.com/search?utf8=%E2%9C%93&q=avro+language%3AGo&type=Repositories&ref=advsearch&l=Go
Have you had evaluated any of them?

Also are you sure about the release of lib that you guys are working on & how many of the above listed capabilities will be available ?

[Damian Gryski]

Hi,

Any update from the appropriate legal departments?

Damian

On Wednesday, February 11, 2015 at 6:30:10 PM UTC+1, kar...@gmail.com wrote:

I'm waiting on my company's legal team to give me the okay for the library name.

[Tin Le]

Yes, the library has been released.

https://github.com/linkedin/goavro

Tin

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/dlKpaHma5bE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[wur...@gmail.com]

looks like rpc is not implemented.

[wur...@gmail.com]

I forkd linkedin/goavro, writting a quick dirty rpc implemention, and fix some bugs. https://github.com/wuranbo/goavro 。 already used in some project, not so solid, but at least can be used for small project. I still rushed my own work. hope to be haved time to finished it.