using capnp with go

[Vasiliy Tolstov]

Hi all. I have troubles to get working capnproto with go.
What i do:
I'm create file message.capnp with:
using Go = import "github.com/jmckaskill/go-capnproto/go.capnp";

# needed to know what package should be used for generated code
$Go.package("main");

# Needed to know how to import types in the capnp file and whether two
# capnp files are in the same package
$Go.import("bitbucket.org/vtolstov/virt-mon");

@0xa04b8a78e1ae5cc0;

struct Header {
version @0 :UInt8;
type @1 :UInt8;
length @2 :UInt16;
id @3 :UInt32;
}

struct Message {
header @0 :Header;
length @1 :UInt16;
data @2 :Data;
}

And tries to do:
capnp compile -ogo *.capnp

but i get errors like:
message.capnp:1:19-64: error: Import failed:
github.com/jmckaskill/go-capnproto/go.capnp
message.capnp:4:2-4: error: Not defined: Go
message.capnp:8:2-4: error: Not defined: Go

Whats wrong in my setup?
Thanks!

--
Vasiliy Tolstov,
e-mail: v.to...@selfip.ru
jabber: va...@selfip.ru

[Kenton Varda]

cc'ing James who did the Go port.

Looking at your code, though, I suspect the file simply doesn't exist.  This line:

  using Go = import "github.com/jmckaskill/go-capnproto/go.capnp";

Says to look for the file "github.com/jmckaskill/go-capnproto/go.capnp" in the same directory as the current file.  Do you actually have a directory called "github.com" in the same directory as message.capnp, in turn containing "jmckaskill/go-capnproto/go.capnp"?

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at http://groups.google.com/group/capnproto.

[Vasiliy Tolstov]

2014-02-03 Kenton Varda <temp...@gmail.com>:

> cc'ing James who did the Go port.
>
> Looking at your code, though, I suspect the file simply doesn't exist. This
> line:
>
> using Go = import "github.com/jmckaskill/go-capnproto/go.capnp";
>
> Says to look for the file "github.com/jmckaskill/go-capnproto/go.capnp" in
> the same directory as the current file. Do you actually have a directory
> called "github.com" in the same directory as message.capnp, in turn
> containing "jmckaskill/go-capnproto/go.capnp"?

No it does not exists. I'm already solve this, but i think that go
port download file =). Sorry.
Now i have nother problem: http://play.golang.org/p/9vs4f8bXnq
I'm try to get data from sended message in server code, but i get empty []byte.

[Kenton Varda]

James will probably have to answer this as I don't know Go.

[James McKaskill]

s := capn.NewBuffer(nil)

h := NewRootHeader(s)
m := NewRootMessage(s)
m.SetHeader(h)

Should be

s := capn.NewBuffer(nil)
m := NewRootMessage(s)
h := NewHeader(s)
m.SetHeader(h)

Only the root structure in a message should use the NewRoot* function
and should be called before creating any other structures.

Also your length field is superfluous as the data field has an embedded length.

If that doesn't fix things, I'll have a further look tonight.

-- James

[Vasiliy Tolstov]

2014-02-03 James McKaskill <ja...@foobar.co.nz>:

> Only the root structure in a message should use the NewRoot* function
> and should be called before creating any other structures.
>
> Also your length field is superfluous as the data field has an embedded length.
>
> If that doesn't fix things, I'll have a further look tonight.

I'm change my code, but nothing changed.

s := capn.NewBuffer(nil)
m := NewRootMessage(s)
h := NewHeader(s)
m.SetHeader(h)

m.SetLength(uint16(len([]byte("hello"))))
m.SetData([]byte("hello"))

But in server side i get empty byte slice.

[Vasiliy Tolstov]

And why we need to go.capnp file to using and not automatic append it
contents while compile?

2014-02-04 Vasiliy Tolstov <v.to...@selfip.ru>:

[Vasiliy Tolstov]

Hi! Any news? As i see nothing setted via set methods. Only default
values specified in capnp file sended.

2014-02-03 James McKaskill <ja...@foobar.co.nz>:

[Vasiliy Tolstov]

2014-02-05 13:29 GMT+04:00 Vasiliy Tolstov <v.to...@selfip.ru>:
> Hi! Any news? As i see nothing setted via set methods. Only default
> values specified in capnp file sended.

=( what advantages at capnproto before msgpack?

[Kenton Varda]

On Sun, Feb 9, 2014 at 11:27 AM, Vasiliy Tolstov <v.to...@selfip.ru> wrote:

=( what advantages at capnproto before msgpack?

Cap'n Proto is statically typed (which has advantages and disadvantages), is much faster than msgpack, and defines an object-capability RPC protocol.

(Sorry that I can't answer the go-specific questions.)

-Kenton

[Albert Strasheim]

Hey Vasiliy

I can take a look if you want. Would it be possible to post a minimal piece of compilable code somewhere on GitHub for me to take a look?

We've been having great success with James's library here.

Cheers

Albert

[Vasiliy Tolstov]

2014-02-11 4:00 GMT+04:00 Albert Strasheim <ful...@gmail.com>:
> Hey Vasiliy
>
> I can take a look if you want. Would it be possible to post a minimal piece
> of compilable code somewhere on GitHub for me to take a look?
>
> We've been having great success with James's library here.
>
> Cheers
>
> Albert

Thanks for help!
https://bitbucket.org/vtolstov/virt-mon/src/abb24932f79ab7ea1738c9f4e38c8fdbf03fd763/main.go?at=master
https://bitbucket.org/vtolstov/virt-mon/src/abb24932f79ab7ea1738c9f4e38c8fdbf03fd763/message.capnp?at=master

[Albert Strasheim]

Hey Vasiliy

On Tue, Feb 11, 2014 at 1:54 AM, Vasiliy Tolstov <v.to...@selfip.ru> wrote:
> 2014-02-11 4:00 GMT+04:00 Albert Strasheim <ful...@gmail.com>:
>> Hey Vasiliy
>> I can take a look if you want. Would it be possible to post a minimal piece
>> of compilable code somewhere on GitHub for me to take a look?
>> We've been having great success with James's library here.
>> Cheers
>> Albert
> Thanks for help!
> https://bitbucket.org/vtolstov/virt-mon/src/abb24932f79ab7ea1738c9f4e38c8fdbf03fd763/main.go?at=master
> https://bitbucket.org/vtolstov/virt-mon/src/abb24932f79ab7ea1738c9f4e38c8fdbf03fd763/message.capnp?at=master

There seems to be two issues here.

One is that using packed encoding doesn't work for this message for
some reason (maybe James has some ideas why).

The second issue is that there seems to be something going on with
reading even an unpacked stream from the UDP socket. I'm taking a
closer look.

Aside: When using UDP you probably want to serialise to a buffer first
so that you are sure that you are sending a message per packet.
WriteTo makes multiple Write calls to the underlying Writer (in your
case, the socket), so it isn't guaranteed that you will get one
message per packet, as the "Stream" part of reading functions imply.
strace -f -e read,write is your friend here.

I'll let you know as I find out more.

Regards

Albert

[Albert Strasheim]

Hello

Figured out the second part. Your code writes:

write(7, "\0\0\0\0\5\0\0\0", 8) = 8
write(7, "\0\0\0\0\1\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0*\0\0\0"...,
40) = 40

Then ReadFromStream reads with CopyN, which does:

read(5, "\0\0\0\0", 4) = 4
# at this point, the next 4 bytes of the first packet go missing
read(5, "\0\0\0\0", 4) = 4

Now you read a bunch of zeros from the second packet, which just
happens to make ReadFromStream think that your message is empty.

The other problem with UDP is that your messages could arrive out of
order, which will also break your stream.

So for UDP, definitely serialise to a buffer first and send that to
the socket with a single Write call.

I'll look into the packed issue a bit later.

Regards

Albert

[Vasiliy Tolstov]

2014-02-11 22:45 GMT+04:00 Albert Strasheim <ful...@gmail.com>:
> Figured out the second part. Your code writes:
>
> write(7, "\0\0\0\0\5\0\0\0", 8) = 8
> write(7, "\0\0\0\0\1\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0*\0\0\0"...,
> 40) = 40
>
> Then ReadFromStream reads with CopyN, which does:

Thanks! I'm try to fix that things. May be in my case i need not
capnproto or some system that can process out of order messages and
work with it? Or how can i deal with buffering when i don't know
message size? What is the best practice?

[Vasiliy Tolstov]

2014-02-11 22:45 GMT+04:00 Albert Strasheim <ful...@gmail.com>:
> Then ReadFromStream reads with CopyN, which does:
>
> read(5, "\0\0\0\0", 4) = 4
> # at this point, the next 4 bytes of the first packet go missing
> read(5, "\0\0\0\0", 4) = 4
>
> Now you read a bunch of zeros from the second packet, which just
> happens to make ReadFromStream think that your message is empty.
>
> The other problem with UDP is that your messages could arrive out of
> order, which will also break your stream.
>
> So for UDP, definitely serialise to a buffer first and send that to
> the socket with a single Write call.
>
> I'll look into the packed issue a bit later.
>
> Regards

I'm remove packed stream usage and use bytes.Buffer. but have no luck.
Can you look again ?
https://bitbucket.org/vtolstov/virt-mon/src/52a4aaaced20cb7debba5610d2a498615e469da9/main.go?at=master

[Vasiliy Tolstov]

2014-02-11 23:32 GMT+04:00 Vasiliy Tolstov <v.to...@selfip.ru>:
> I'm remove packed stream usage and use bytes.Buffer. but have no luck.
> Can you look again ?
> https://bitbucket.org/vtolstov/virt-mon/src/52a4aaaced20cb7debba5610d2a498615e469da9/main.go?at=master

Ok, i'm fix my issues
https://bitbucket.org/vtolstov/virt-mon/src/master/main.go?at=master
But how can i determine message size and allocate needed buffer? I
don't known what message i get 1500 max bytes or 9000... ?
Is that possible to partially read message and determine message len
and allocated needed buffer after that?

[Albert Strasheim]

Hello

On Tue, Feb 11, 2014 at 12:32 PM, Vasiliy Tolstov <v.to...@selfip.ru> wrote:
> 2014-02-11 23:32 GMT+04:00 Vasiliy Tolstov <v.to...@selfip.ru>:
>> I'm remove packed stream usage and use bytes.Buffer. but have no luck.
>> Can you look again ?
>> https://bitbucket.org/vtolstov/virt-mon/src/52a4aaaced20cb7debba5610d2a498615e469da9/main.go?at=master
> Ok, i'm fix my issues
> https://bitbucket.org/vtolstov/virt-mon/src/master/main.go?at=master
> But how can i determine message size and allocate needed buffer? I
> don't known what message i get 1500 max bytes or 9000... ?
> Is that possible to partially read message and determine message len
> and allocated needed buffer after that?

IIRC, for UDP you could simply allocate a 64k buffer which would be
big enough to receive any UDP packet that could be sent. If possible,
reuse the buffer so that you don't make too much garbage.

Then Read into the buffer from the the socket. This Read with return
an integer n or an error.

Then wrap a bytes.Buffer around buf[:n] and use with ReadFromStream.

Depending on how you organise things, you could also use
ReadFromMemoryZeroCopy, if you didn't use packed encoding.

I usually use perf top and go tool pprof to profile this kind of
thing. This will become important if you need to deal with high packet
rates.

Regards

Albert

[Kenton Varda]

On Tue, Feb 11, 2014 at 12:32 PM, Vasiliy Tolstov <v.to...@selfip.ru> wrote:

But how can i determine message size and allocate needed buffer? I
don't known what message i get 1500 max bytes or 9000... ?
Is that possible to partially read message and determine message len
and allocated needed buffer after that?

If your message sizes are not bounded, you probably should not use UDP.  You should use TCP, or some other higher-level protocol that can reliably transmit arbitrary-sized messages.

[Vasiliy Tolstov]

2014-02-12 2:06 GMT+04:00 Albert Strasheim <ful...@gmail.com>:
> IIRC, for UDP you could simply allocate a 64k buffer which would be
> big enough to receive any UDP packet that could be sent. If possible,
> reuse the buffer so that you don't make too much garbage.
>
> Then Read into the buffer from the the socket. This Read with return
> an integer n or an error.
>
> Then wrap a bytes.Buffer around buf[:n] and use with ReadFromStream.
>
> Depending on how you organise things, you could also use
> ReadFromMemoryZeroCopy, if you didn't use packed encoding.
>
> I usually use perf top and go tool pprof to profile this kind of
> thing. This will become important if you need to deal with high packet
> rates.

Thanks. Thats very good. I can use packed data inside message and
don't need packedstream.
How can i store message with zerocopy to file? And later - how can i
read it from file without additional buffering?

[Vasiliy Tolstov]

2014-02-12 2:13 GMT+04:00 Kenton Varda <temp...@gmail.com>:
> If your message sizes are not bounded, you probably should not use UDP. You
> should use TCP, or some other higher-level protocol that can reliably
> transmit arbitrary-sized messages.

is that possible to read small piece of data, determine len and read
needed len from udp?
In case of using plain udp i understand how can i do that, but in case
of capnproto is that possible to read only 4 byte from message?

[Albert Strasheim]

Hello

On Wed, Feb 12, 2014 at 3:48 AM, Vasiliy Tolstov <v.to...@selfip.ru> wrote:
> 2014-02-12 2:06 GMT+04:00 Albert Strasheim <ful...@gmail.com>:
>> IIRC, for UDP you could simply allocate a 64k buffer which would be
>> big enough to receive any UDP packet that could be sent. If possible,
>> reuse the buffer so that you don't make too much garbage.
>> Then Read into the buffer from the the socket. This Read with return
>> an integer n or an error.
>> Then wrap a bytes.Buffer around buf[:n] and use with ReadFromStream.
>> Depending on how you organise things, you could also use
>> ReadFromMemoryZeroCopy, if you didn't use packed encoding.
>> I usually use perf top and go tool pprof to profile this kind of
>> thing. This will become important if you need to deal with high packet
>> rates.
> Thanks. Thats very good. I can use packed data inside message and
> don't need packedstream.

Yes, I still need to debug that.

> How can i store message with zerocopy to file? And later - how can i
> read it from file without additional buffering?

syscall.Mmap is your friend here. It will return a []byte you can use
with ReadFromMemoryZeroCopy. man mmap is the best documentation.

Regards

Albert

[Vasiliy Tolstov]

Thanks! What about partially reading udp that contains capnproto message?
For example in plain udp and custom serializer i can write fixed
header - and read from udp ony it , after that determine buffer for
payload.
How can i deal with it in capnproto?

[Albert Strasheim]

Hello

On Wed, Feb 12, 2014 at 9:57 AM, Vasiliy Tolstov <v.to...@selfip.ru> wrote:
> Thanks! What about partially reading udp that contains capnproto message?
> For example in plain udp and custom serializer i can write fixed
> header - and read from udp ony it , after that determine buffer for
> payload.
> How can i deal with it in capnproto?

Between the packet you send and the capnproto encoding, all the length
information is already there.

If you must use UDP, make a 64k []byte buffer (preferably reusing it
between reads) and Read into it from the UDP socket.

Read will return how many bytes were actually read so that you can
slice your buffer with buf[:n] to the actual size it should be for
following operation.

Then use the sliced buffer directly with ReadFromMemoryZeroCopy (if
you don't have packed encoding) or by wrapping it in a bytes.Buffer
and using ReadFromStream or ReadFromPackedStream.

There are probably some tweaks that could be made to the go-capnproto
library so that it allocates less for this use-case, but fix that when
it comes up in a profile and once you decide how you want to work with
the messages you receive.

Regards

Albert

[Kenton Varda]

On Wed, Feb 12, 2014 at 3:50 AM, Vasiliy Tolstov <v.to...@selfip.ru> wrote:

is that possible to read small piece of data, determine len and read
needed len from udp?
In case of using plain udp i understand how can i do that, but in case
of capnproto is that possible to read only 4 byte from message?

I'm not sure I understand the question.  But, Cap'n Proto (unlike Protobuf) is a self-delimiting format.  There's no need to write the length separately when writing multiple messages to a stream.

[Albert Strasheim]

Hello again

On Tue, Feb 11, 2014 at 10:36 AM, Albert Strasheim <ful...@gmail.com> wrote:

> On Tue, Feb 11, 2014 at 1:54 AM, Vasiliy Tolstov <v.to...@selfip.ru> wrote:
>> 2014-02-11 4:00 GMT+04:00 Albert Strasheim <ful...@gmail.com>:
>>> Hey Vasiliy
>>> I can take a look if you want. Would it be possible to post a minimal piece
>>> of compilable code somewhere on GitHub for me to take a look?
>>> We've been having great success with James's library here.
>>> Cheers
>>> Albert
>> Thanks for help!
>> https://bitbucket.org/vtolstov/virt-mon/src/abb24932f79ab7ea1738c9f4e38c8fdbf03fd763/main.go?at=master
>> https://bitbucket.org/vtolstov/virt-mon/src/abb24932f79ab7ea1738c9f4e38c8fdbf03fd763/message.capnp?at=master
> There seems to be two issues here.
> One is that using packed encoding doesn't work for this message for
> some reason (maybe James has some ideas why).

Daniel Morsing jumped in and fixed this one.

https://github.com/jmckaskill/go-capnproto/pull/10

Regards

Albert

[Vasiliy Tolstov]

2014-02-17 20:41 GMT+04:00 Albert Strasheim <ful...@gmail.com>:
> Daniel Morsing jumped in and fixed this one.
>
> https://github.com/jmckaskill/go-capnproto/pull/10
>
> Regards

Thanks! Does it mean, that i can use packed stream with udp transport
and read message to buffer?