Command-line tool for decoding to text

[Kenton Varda]

I just pushed a change which finally makes Makefile.am build the new compiler!  However, there is no C++ code generator plugin yet, so the Haskell compiler is still needed, but the code generator shouldn't take more than a day to write (right?).  The capnpc-capnp "loopback" plugin is installed, though, so you can at least play with that.

And actually, the new compiler is more than just a compiler.  The binary is now called "capnp" (rather than "capnpc") and features multiple commands (in the style of, say, git):

capnp compile:  The old capnpc functionality.

capnp id:  Generate a new unique ID.

capnp decode:  Given a schema file and a type, read binary instances of that type on stdin and write them as text to stdout.

All commands are fully-documented through their own help text.

I have not yet added a public API to the parser (that could be used by, say, Jason's Python bindings), but that should be easy and I'll do it soon.  On a related note, I'm currently linking the entire schema parser into libcapnp.so, and it has become rather large (1.3MB).  I wonder if it is worthwhile to split up the library.  I'm not sure it matters:  anyone who really cares about binary size is probably bundling the library with some application, and in that case they are probably statically linking the library, and hopefully the linker will only pull in the objects they actually use, so as long as they don't use the schema parser, they're fine...  right?  Also note that there are no global constructors anywhere, so even when dynamically linking against the library, only the code pages that you actually use should be loaded into memory.

-Kenton

[Andreas Stenius]

Cool!

You say there's no C++ plugin yet, but is the code that's calling the plugin in place? (in other words, is it possible to start play with for building plugins for other languages?)

2013/8/3 Kenton Varda <temp...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at http://groups.google.com/group/capnproto.
 
 

[Kenton Varda]

On Sat, Aug 3, 2013 at 5:38 AM, Andreas Stenius <g...@astekk.se> wrote:

Cool!

You say there's no C++ plugin yet, but is the code that's calling the plugin in place? (in other words, is it possible to start play with for building plugins for other languages?)

Correct.  "capnpc compile" invokes plugins in exactly the same way that the Haskell capnpc does.  In fact, if you rename/symlink capnp as capnpc, it will behave exactly like the Haskell capnpc, and should function as a drop-in replacement except for the lack of a built-in C++ code generator.

There is a bug still in that it won't always provide all the schema nodes for imported files that weren't explicitly listed on the command line.  You can work around that for the moment by explicitly listing all dependencies, or just testing against files that don't import anything.  I'll probably fix this Monday.

-Kenton

[Andreas Stenius]

2013/8/3 Kenton Varda <temp...@gmail.com>

On Sat, Aug 3, 2013 at 5:38 AM, Andreas Stenius <g...@astekk.se> wrote:

Cool!

You say there's no C++ plugin yet, but is the code that's calling the plugin in place? (in other words, is it possible to start play with for building plugins for other languages?)

Correct.  "capnpc compile" invokes plugins in exactly the same way that the Haskell capnpc does.  In fact, if you rename/symlink capnp as capnpc, it will behave exactly like the Haskell capnpc, and should function as a drop-in replacement except for the lack of a built-in C++ code generator.

Excellent.

 

There is a bug still in that it won't always provide all the schema nodes for imported files that weren't explicitly listed on the command line.  You can work around that for the moment by explicitly listing all dependencies, or just testing against files that don't import anything.  I'll probably fix this Monday.

Yeah, I don't I think I'll need support for included schemas this early on.. so there's plenty for me to fiddle with, thank you :)

I'll keep y'all posted how it goes ;)

[Andreas Stenius]

> I'll keep y'all posted how it goes ;)

Configure runs fine (using cygwin), but make barfs on missing capnpc..!

Doh!, have I hit a bootstrapping issue here..?

$ make

make: capnpc: Command not found

Makefile:2237: recipe for target `src/capnp/test.capnp.h' failed

make: *** [src/capnp/test.capnp.h] Error 127

## attempt at building just the executable, was hoping that I could bypass the test.capnp.h file.. :p

$ make capnp.exe

In file included from src/capnp/compiler/capnp.c++:24:0:

src/capnp/compiler/lexer.h:27:40: fatal error: capnp/compiler/lexer.capnp.h: No such file or directory

compilation terminated.

Makefile:1027: recipe for target `src/capnp/compiler/capnp.o' failed

make: *** [src/capnp/compiler/capnp.o] Error 1

But, no.. alas, the capnp also needed some schema file itself..

Maybe I could have the needed schema files sent to me.. to overcome the bootstrap phase..? ;-)

Just wondering though, why does capnp need compiled schemas as input at build time?

How's this supposed to go for first time users?

Cheers

[Kenton Varda]

Hah, sorry, yes, I forgot about that.  I've attached the missing files -- unpack the tarball under src.  Be careful not to run "make clean" since it will delete them.  Once the C++ code generator plugin is written I'll change things so that these files are actually checked into the repository rather than built on-demand, with a script to regenerate them when needed.

There are a couple reasons the compiler itself uses Cap'n Proto code:

1) schema.capnp defines the protocol that the compiler uses to talk to code generator plugins.

2) It was really convenient to use Cap'n Proto for some of the compiler's internal representations, like the AST, because it makes it much easier to use unions.

-Kenton

[Andreas Stenius]

Ok, thanks (after sending that I realized it wouldn't be so hard to produce the needed files myself on a linux box.. any how)..

Got a few weird errors:

src/capnp/compiler/capnp.c++: In member function ‘kj::MainBuilder::Validity capnp::compiler::CompilerMain::generateOutput()’:

src/capnp/compiler/capnp.c++:311:87: error: ‘strsignal’ was not declared in this scope

Makefile:1027: recipe for target `src/capnp/compiler/capnp.o' failed

make: *** [src/capnp/compiler/capnp.o] Error 1

and:

src/kj/string.c++: In function 'char* kj::_::{anonymous}::DoubleToBuffer(double, char*)':

src/kj/string.c++:232:65: error: 'snprintf' was not declared in this scope

src/kj/string.c++: In function 'char* kj::_::{anonymous}::FloatToBuffer(float, char*)':

src/kj/string.c++:288:64: error: 'snprintf' was not declared in this scope

Makefile:1043: recipe for target `src/kj/string.lo' failed

make: *** [src/kj/string.lo] Error 1

which I hacked by simply adding the missing declarations for them in the files where they were being used, just to get passed it (as including the headers where they should be defined didn't seem to help!).

Got a little further (hint for others, pass --touch to tar when unpacking, or make will think that the sources for the bootstrap files are newer and try to remake them..)

Now I'm at:

libtool: compile:  g++ -std=c++11 -DHAVE_CONFIG_H -I. -I./src -I./src -O2 -DNDEBUG -MT src/kj/exception.lo -MD -MP -MF src/kj/.deps/exception.Tpo -c src/kj/exception.c++  -DDLL_EXPORT -DPIC -o src/kj/.libs/exception.o

src/kj/exception.c++:28:22: fatal error: execinfo.h: No such file or directory

compilation terminated.

Makefile:1043: recipe for target `src/kj/exception.lo' failed

make: *** [src/kj/exception.lo] Error 1

And are heading for bed, so I'll pick this up again later..

Any hints welcome (I've not looked for the missing header yet ;)

[Kenton Varda]

On Sun, Aug 4, 2013 at 2:38 PM, Andreas Stenius <g...@astekk.se> wrote:

Ok, thanks (after sending that I realized it wouldn't be so hard to produce the needed files myself on a linux box.. any how)..

Got a few weird errors:

src/capnp/compiler/capnp.c++: In member function ‘kj::MainBuilder::Validity capnp::compiler::CompilerMain::generateOutput()’:

src/capnp/compiler/capnp.c++:311:87: error: ‘strsignal’ was not declared in this scope

Makefile:1027: recipe for target `src/capnp/compiler/capnp.o' failed

make: *** [src/capnp/compiler/capnp.o] Error 1

You can fall back to displaying the signal number.

#ifdef CYGWIN

        context.error(kj::str(exeName, ": plugin failed: killed by signal ", WTERMSIG(status)));

#else

        context.error(kj::str(exeName, ": plugin failed: ", strsignal(WTERMSIG(status))));

#endif

 

and:

src/kj/string.c++: In function 'char* kj::_::{anonymous}::DoubleToBuffer(double, char*)':

src/kj/string.c++:232:65: error: 'snprintf' was not declared in this scope

src/kj/string.c++: In function 'char* kj::_::{anonymous}::FloatToBuffer(float, char*)':

src/kj/string.c++:288:64: error: 'snprintf' was not declared in this scope

Makefile:1043: recipe for target `src/kj/string.lo' failed

make: *** [src/kj/string.lo] Error 1

This is weird.  snprintf() is part of the C99 standard and should be present on Cygwin.  This code is actually copied from protobufs, which compiles fine on Cygwin.  OTOH, Win32 doesn't necessarily have snprintf, only _snprintf, hence the #ifdef near the top of the file.  But that shouldn't apply to Cygwin.

 

Now I'm at:

libtool: compile:  g++ -std=c++11 -DHAVE_CONFIG_H -I. -I./src -I./src -O2 -DNDEBUG -MT src/kj/exception.lo -MD -MP -MF src/kj/.deps/exception.Tpo -c src/kj/exception.c++  -DDLL_EXPORT -DPIC -o src/kj/.libs/exception.o

src/kj/exception.c++:28:22: fatal error: execinfo.h: No such file or directory

compilation terminated.

Makefile:1043: recipe for target `src/kj/exception.lo' failed

make: *** [src/kj/exception.lo] Error 1

execinfo.h defines backtrace().  I guess Cygwin doesn't have it.  You can safely avoid backtrace() by pretending it always returns zero:

#ifdef CYGWIN

  traceCount = 0;

#else

  traceCount = backtrace(trace, 16);

#endif

You may also want to update the definition of KJ_STRINGIFY(const Exception& e) so that it doesn't print "stack:" if there is no stack trace.

  return str(strArray(contextText, ""),

             e.getFile(), ":", e.getLine(), ": ", e.getNature(),

             e.getDurability() == Exception::Durability::TEMPORARY ? " (temporary)" : "",

             e.getDescription() == nullptr ? "" : ": ", e.getDescription(),

             e.getStackTrace().size() > 0 ? "\nstack: " : "",

             strArray(e.getStackTrace(), " "), getStackSymbols(e.getStackTrace()));

(Changed part in bold.)

Let me know if you get stuck on anything else.  I'll also take a crack at it myself after I've written the new C++ code generator.

-Kenton

[Andreas Stenius]

Thanks for the pointers. I've been away for a few days, but will get back to this within a day or two..

[Andreas Stenius]

OK, finally :)

I got fed up with cygwin and it's weirdness.. so I took a back to basics debugging technique by tossing in a --save-temps to the compile flags and inspected the pre-compiled version of the file and figured out that __STRICT_ANSI__ was defined.. looked at the compile flags again and noticed the --std=c++11, tried with --std=gnu++11 instead and voilá, presto, profit, etc.. :) (I changed the macro arg to the AX_CXX_.. check in configure.ac from [noext] to [ext])

So, backtrace is solved with if'deffing against __CYGWIN__, but apart from that it compiled just fine. So, now I can proceed with trying it out! :D

A quick sanity check turned out fine:

$ ./capnp.exe id

@0x86f6688c6f7cfb58

Until next update/question..

[Andreas Stenius]

2013/8/8 Andreas Stenius <g...@astekk.se>

Until next update/question..

Just a quick feedback.. would be nice to know what it tried to run here:

$ make

*** Uncaught exception ***

src/capnp/compiler/capnp.c++:297: error from OS: execlp(exeName.cStr(), exeName.cStr(), nullptr): No such file or directory

stack:

capnpc-c++: plugin failed: exit code 1

Makefile:2237: recipe for target `src/capnp/test.capnp.h' failed

make: *** [src/capnp/test.capnp.h] Error 1

Guess that shows when you indeed have a stack trace.. :p

[Andreas Stenius]

2013/8/8 Andreas Stenius <g...@astekk.se>

Doh.. sorry. My bad, for not reading the source to figure out all parts of the error messages.

I missed that the plugin name was on the line with the exit code.

[Andreas Stenius]

OK, so the capnpc-capnp plugin works like a charm :)

But when I try to compile the schema.capnp file, it complains:

$ capnpc -ocapnp src/capnp/schema.capnp

terminate called after throwing an instance of 'kj::ExceptionImpl'

  what():  src/capnp/schema-loader.c++:1193: requirement not met: no schema node loaded for id; id = 13386661402618388268

stack:

capnpc-capnp: plugin failed: Aborted

not sure where look for import issues, yet..

Was trying to see if I could spot the id in one of the .capnp files (schema, c++ resp.) after converting to hex (b9c6f99ebf806000), but it turned up blank.

Not sure if I really need to compile the schema file, but I guess it would be nice with the additional little nuggets of information that the capnp plugin adds..

[Kenton Varda]

On Wed, Aug 7, 2013 at 10:48 PM, Andreas Stenius <g...@astekk.se> wrote:

OK, finally :)

I got fed up with cygwin and it's weirdness.. so I took a back to basics debugging technique by tossing in a --save-temps to the compile flags and inspected the pre-compiled version of the file and figured out that __STRICT_ANSI__ was defined.. looked at the compile flags again and noticed the --std=c++11, tried with --std=gnu++11 instead and voilá, presto, profit, etc.. :) (I changed the macro arg to the AX_CXX_.. check in configure.ac from [noext] to [ext])

Arg.  This is frustrating because -std=gnu++11 breaks the Clang-Linux build because it enables GCC-specific extensions in the stdc++ headers which Clang doesn't yet support.  But I suppose -std=gnu++11 is more correct so I guess I'll look for a different work-around.

On Thu, Aug 8, 2013 at 12:46 AM, Andreas Stenius <g...@astekk.se> wrote:

2013/8/8 Andreas Stenius <g...@astekk.se>

Just a quick feedback.. would be nice to know what it tried to run here:

$ make

*** Uncaught exception ***

src/capnp/compiler/capnp.c++:297: error from OS: execlp(exeName.cStr(), exeName.cStr(), nullptr): No such file or directory

stack:

capnpc-c++: plugin failed: exit code 1

Makefile:2237: recipe for target `src/capnp/test.capnp.h' failed

make: *** [src/capnp/test.capnp.h] Error 1

Guess that shows when you indeed have a stack trace.. :p

Doh.. sorry. My bad, for not reading the source to figure out all parts of the error messages.

I missed that the plugin name was on the line with the exit code.

It's definitely a crappy error message.  The problem is that the "Uncaught exception" message is being printed in the subprocess, due to exec() failing.  I guess I should improve that.

On Thu, Aug 8, 2013 at 1:37 AM, Andreas Stenius <g...@astekk.se> wrote:

OK, so the capnpc-capnp plugin works like a charm :)

But when I try to compile the schema.capnp file, it complains:

$ capnpc -ocapnp src/capnp/schema.capnp

terminate called after throwing an instance of 'kj::ExceptionImpl'

  what():  src/capnp/schema-loader.c++:1193: requirement not met: no schema node loaded for id; id = 13386661402618388268

stack:

capnpc-capnp: plugin failed: Aborted

This is probably the bug I mentioned earlier where you need to list all imported files explicitly.  schema.capnp imports c++.capnp.  This should be fixed head, though -- do a git pull.

Glad to hear this is all it takes to get things working on Cygwin!  I'll definitely make this an officially-supported platform for the next release.

[Andreas Stenius]

2013/8/8 Kenton Varda <temp...@gmail.com>

On Wed, Aug 7, 2013 at 10:48 PM, Andreas Stenius <g...@astekk.se> wrote:

OK, finally :)

I got fed up with cygwin and it's weirdness.. so I took a back to basics debugging technique by tossing in a --save-temps to the compile flags and inspected the pre-compiled version of the file and figured out that __STRICT_ANSI__ was defined.. looked at the compile flags again and noticed the --std=c++11, tried with --std=gnu++11 instead and voilá, presto, profit, etc.. :) (I changed the macro arg to the AX_CXX_.. check in configure.ac from [noext] to [ext])

Arg.  This is frustrating because -std=gnu++11 breaks the Clang-Linux build because it enables GCC-specific extensions in the stdc++ headers which Clang doesn't yet support.  But I suppose -std=gnu++11 is more correct so I guess I'll look for a different work-around.

I'd like to use Clang, but never came around to take the plunge..

What about detecting which compiler is going to be used first, and then run the AX_CXX_.. with arg depending on if it's gcc, clang, or what-not.. ?

 

On Thu, Aug 8, 2013 at 12:46 AM, Andreas Stenius <g...@astekk.se> wrote:

2013/8/8 Andreas Stenius <g...@astekk.se>

Just a quick feedback.. would be nice to know what it tried to run here:

$ make

*** Uncaught exception ***

src/capnp/compiler/capnp.c++:297: error from OS: execlp(exeName.cStr(), exeName.cStr(), nullptr): No such file or directory

stack:

capnpc-c++: plugin failed: exit code 1

Makefile:2237: recipe for target `src/capnp/test.capnp.h' failed

make: *** [src/capnp/test.capnp.h] Error 1

Guess that shows when you indeed have a stack trace.. :p

Doh.. sorry. My bad, for not reading the source to figure out all parts of the error messages.

I missed that the plugin name was on the line with the exit code.

It's definitely a crappy error message.  The problem is that the "Uncaught exception" message is being printed in the subprocess, due to exec() failing.  I guess I should improve that.

Definitely room for improvement, but not a show stopper.. imho :p

 

On Thu, Aug 8, 2013 at 1:37 AM, Andreas Stenius <g...@astekk.se> wrote:

OK, so the capnpc-capnp plugin works like a charm :)

But when I try to compile the schema.capnp file, it complains:

$ capnpc -ocapnp src/capnp/schema.capnp

terminate called after throwing an instance of 'kj::ExceptionImpl'

  what():  src/capnp/schema-loader.c++:1193: requirement not met: no schema node loaded for id; id = 13386661402618388268

stack:

capnpc-capnp: plugin failed: Aborted

This is probably the bug I mentioned earlier where you need to list all imported files explicitly.  schema.capnp imports c++.capnp.  This should be fixed head, though -- do a git pull.

Yeah, I realized you'd mentioned there was an issue regarding file inclusion once I got into bed.. :p

 

Glad to hear this is all it takes to get things working on Cygwin!  I'll definitely make this an officially-supported platform for the next release.

Indeed, it seems to be going fine now :)

I've not yet managed to run any of the tests though..

[Andreas Stenius]

[...]

But when I try to compile the schema.capnp file, it complains:

$ capnpc -ocapnp src/capnp/schema.capnp

terminate called after throwing an instance of 'kj::ExceptionImpl'

  what():  src/capnp/schema-loader.c++:1193: requirement not met: no schema node loaded for id; id = 13386661402618388268

stack:

capnpc-capnp: plugin failed: Aborted

This is probably the bug I mentioned earlier where you need to list all imported files explicitly.  schema.capnp imports c++.capnp.  This should be fixed head, though -- do a git pull.

Yeah, I realized you'd mentioned there was an issue regarding file inclusion once I got into bed.. :p

 

Got a bunch of compilation errors now from head (3b0e4f1e7f35613455ce59217ef5318f53e2627f) (see attached build log).

Any changes to the "bootstrap" files that I might need?

It seems like an easy bug that generates a ton of build errors.. but it'll be far easier for you to spot what it is (I hope, as I'm not at all familiar with the code base yet ;)

Btw,

here's my changes thus far for building on cygwin: https://github.com/kaos/capnproto/tree/cygwin

[Kenton Varda]

Yes, sorry, those errors are because I changed the code generator.

Good news, though:  I just pushed changes which make the C++ code no longer depend on the Haskell code at all.  The bootstrap code is checked in, and the test code is generated using the C++ "capnp compile" built in-tree.

I plan to spend the next couple hours cleaning up and running more tests, including making sure it works on Cygwin out-of-the-box.  Maybe I'll post a release candidate later today.

[Kenton Varda]

Here's the release candidate, fully tested on Cygwin:  http://capnproto.org/capnproto-c++-0.2.0-rc1.tar.gz

[Andreas Stenius]

Awesome! :)

2013/8/10 Kenton Varda <temp...@gmail.com>