Problems with import

[Michi Henning]

I have the following in a file:

using QueryProxy = import "QueryProxy.capnp";
using ReplyProxy = import "ReplyProxy.capnp";
using ValueDict = import "ValueDict.capnp";

One thing that is awkward here is that I have to invent a different namespace name for each import. If I do something like:

using NS = import "foo.capnp";
using NS = import "bar.capnp";

I get an error. That's a bit inconvenient because the definitions are all meant to be used with each other, and there won't be any clashes if I place them all into a single namespace. The split into separate files is mainly for maintainability and to reduce excessive compile times and recompilation. (I don't like have everything in a single file and then watching the universe recompile because of some trivial change.)

But, more seriously, I'm getting incorrect code generated for the above (for the first example with three different namespace). For the above, the generated code contains:

#if CAPNP_VERSION != 3000
#error "Version mismatch between generated code and library headers.  You must use the same version of the Cap'n Proto compiler and library."
#endif

#include "ValueDict.capnp.h"

// ...

Note that there is only a single include directive, for ValueDict.capnp.h. The include directives for QueryProxy.capnp.h and ReplyProxy.capnp.h are not generated, so compilation fails due to missing definitions. When I edit the generated code to add the two missing #includes, the code compiles fine.

Cheers,

Michi.

[Kenton Varda]

Hi Michi,

On Sun, Sep 1, 2013 at 9:23 PM, Michi Henning <michij....@gmail.com> wrote:

I have the following in a file:

using QueryProxy = import "QueryProxy.capnp";
using ReplyProxy = import "ReplyProxy.capnp";
using ValueDict = import "ValueDict.capnp";

One thing that is awkward here is that I have to invent a different namespace name for each import. If I do something like:

using NS = import "foo.capnp";
using NS = import "bar.capnp";

I get an error. That's a bit inconvenient because the definitions are all meant to be used with each other, and there won't be any clashes if I place them all into a single namespace. The split into separate files is mainly for maintainability and to reduce excessive compile times and recompilation. (I don't like have everything in a single file and then watching the universe recompile because of some trivial change.)

The idea here is that it should be possible to tell which file a definition comes from without having to search all imports.  It sounds like you want an "import *" kind of directive, like Java and Python have.  But, I hesitate to implement this because lots of Java and Python experts say you should never use their respective "import *" directives because they make it hard to tell what is being imported.

One thing you can do is explicitly alias each exact type:

using import "QueryProxy.capnp".FooType;

using import "QueryProxy.capnp".BarType;

using import "QueryProxy.capnp".BazType;

And then FooType, BarType, and BazType will all be usable without any qualification.  I have been thinking of simplifying this to something like:

using import "QueryProxy.capnp".(FooType, BarType, BazType);

Do you think that would be enough to make you happy?

 

But, more seriously, I'm getting incorrect code generated for the above (for the first example with three different namespace). For the above, the generated code contains:

#if CAPNP_VERSION != 3000
#error "Version mismatch between generated code and library headers.  You must use the same version of the Cap'n Proto compiler and library."
#endif

#include "ValueDict.capnp.h"

// ...

Note that there is only a single include directive, for ValueDict.capnp.h. The include directives for QueryProxy.capnp.h and ReplyProxy.capnp.h are not generated, so compilation fails due to missing definitions. When I edit the generated code to add the two missing #includes, the code compiles fine.

Does the importing file actually use any of the types defined in QueryProxy or ReplyProxy?  The compiler only outputs includes that are needed.  This is so that things like "/capnp/c++.capnp" which are only used for annotations don't force a useless #include that slows down the C++ compile.

-Kenton

[Michi Henning]

Hi Kenton,

maybe the problem is in part my inappropriate expectation level. I expected import to work along the lines of C++ namespaces, where I can say "using namespace X", and anything in X will become visible.

> The idea here is that it should be possible to tell which file a definition comes from without having to search all imports. It sounds like you want an "import *" kind of directive, like Java and Python have. But, I hesitate to implement this because lots of Java and Python experts say you should never use their respective "import *" directives because they make it hard to tell what is being imported.
>
> One thing you can do is explicitly alias each exact type:
>
> using import "QueryProxy.capnp".FooType;
> using import "QueryProxy.capnp".BarType;
> using import "QueryProxy.capnp".BazType;
>
> And then FooType, BarType, and BazType will all be usable without any qualification. I have been thinking of simplifying this to something like:
>
> using import "QueryProxy.capnp".(FooType, BarType, BazType);
>
> Do you think that would be enough to make you happy?

Realistically, I don't think so. Basically, what I'm doing is using Zero MQ with Cap'n Proto to cobble together a middleware infrastructure. I have interfaces, operations with in and out params, the usual stuff. (If you've ever seen ZeroC Ice and its Slice definition language, you'll know what I mean.)

Ideally, I would like to have something like re-openable namespaces or modules, so I can group interfaces into namespaces, and have interfaces act as a namespace for its operations. Each operation then ends up describing its parameters as a Cap'n Proto struct with appropriate members.

For some namespaces, there are lots of definitions that I would have to import separately with what you suggest, so I'm doubtful this work all that well.

>
> Does the importing file actually use any of the types defined in QueryProxy or ReplyProxy? The compiler only outputs includes that are needed. This is so that things like "/capnp/c++.capnp" which are only used for annotations don't force a useless #include that slows down the C++ compile.

Yes, the types are used. Without the import statements, the specification would not compile, which is why I added the import statements :-)

Basically, what I have is a case of where the .capnp files compile without complaint, but the generated code does not compile. This should never happen, as far as I can see. All legal .capnp specifications should generated code that compiles without error. (And without the import statements, the specification won't compile.)

Cheers,

Michi.

[Kenton Varda]

On Mon, Sep 2, 2013 at 1:03 AM, Michi Henning <mi...@triodia.com> wrote:

maybe the problem is in part my inappropriate expectation level. I expected import to work along the lines of C++ namespaces, where I can say "using namespace X", and anything in X will become visible.

Yeah, that's one way to design a language...  but it's not the way Cap'n Proto is designed.  In Cap'n Proto, the only namespaces are per-file.  An import statement is actually an expression that effectively evaluates to the file's top-level namespace.  The advantage of this approach is that it's unambiguous where any particular declaration comes from and it's impossible for declarations from two different files to conflict.  Certainly, there are also disadvantages that could be debated, but personally I prefer this approach.

I still don't feel like I really understand how this makes your use case difficult.  It would help if you could give me a more complete example.

> Does the importing file actually use any of the types defined in QueryProxy or ReplyProxy?  The compiler only outputs includes that are needed.  This is so that things like "/capnp/c++.capnp" which are only used for annotations don't force a useless #include that slows down the C++ compile.

Yes, the types are used. Without the import statements, the specification would not compile, which is why I added the import statements :-)

Basically, what I have is a case of where the .capnp files compile without complaint, but the generated code does not compile. This should never happen, as far as I can see. All legal .capnp specifications should generated code that compiles without error. (And without the import statements, the specification won't compile.)

That does sound like a bug.  Could you send me an example demonstrating this problem?  I haven't seen this happen in my tests.

[Michi Henning]

The easy work-around for me today was to simply combine all the definitions into a single file. That causes a lot of unnecessary coupling between source files via the generated mongo-header file, but it got me off the hook :-)

Courages as I am, I promptly deleted all the separated files after combining them into a single one :-|

I'll dig out the previous files from my hourly backups tomorrow and try to come up with stand-alone tests case that illustrates the problem.

Thanks heaps for your support, Kenton!

Cheers,

Michi.

[Kenton Varda]

I've reproduced the bug.  This is definitely a release-blocker, but hopefully I'll have it fixed shortly...

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at http://groups.google.com/group/capnproto.

[Michi Henning]

On 03/09/2013, at 6:21 , Kenton Varda <temp...@gmail.com> wrote:

> I've reproduced the bug. This is definitely a release-blocker, but hopefully I'll have it fixed shortly...

Awesome, thanks for that! :-)

Michi.

[Kenton Varda]

OK, let's hope this is the last damned RC...

  http://capnproto.org/capnproto-c++-0.3.0-rc7.tar.gz

Thanks very much for reporting this problem -- it would have been pretty embarrassing to have shipped without the ability to import more than one file.  (I think this problem was new in 0.3 since the code in question has been rewritten since 0.2.)

Michi.

[Michi Henning]

On Tuesday, September 3, 2013 7:37:57 AM UTC+10, Kenton Varda wrote:

OK, let's hope this is the last damned RC...

  http://capnproto.org/capnproto-c++-0.3.0-rc7.tar.gz

Thanks very much for reporting this problem -- it would have been pretty embarrassing to have shipped without the ability to import more than one file.  (I think this problem was new in 0.3 since the code in question has been rewritten since 0.2.)

Hi Kenton,

thanks heaps for the super-fast response! I'll be in the office in a hour or so and try it out then. Will report back! :-)

Cheers,

Michi. 

[Michi Henning]

On Tuesday, September 3, 2013 7:37:57 AM UTC+10, Kenton Varda wrote:

OK, let's hope this is the last damned RC...

  http://capnproto.org/capnproto-c++-0.3.0-rc7.tar.gz

 Just tried it. Works like a charm! :-)

Thanks heaps for all your efforts!

Cheers,

Michi.