Somehow protoc --proto_path does not understand any relative paths

[Artem]

Hi there,

We're a long-time user of Java protobuffs. Just upgraded to 2.2.0 to
experience the power of LITE_RUNTIME. Everything is rocking and
working much faster, except for one annoying problem.

If I use anything but a processed full path for the --proto_path
argument, it does not work. In other words, if I use for example $
{basedir}../.. it does not work, meaning it does not understand to
convert the ../.. to lower directories.

Is this expected behavior? I was able to work around it by having ANT
process the paths (I am no ANT expert so apologies for the below):

<property name="WSL_ROOT" value="../.." />
<path id="WSL_ROOT_PATH" path="${basedir}/${WSL_ROOT}"/>
<property name="WSL_ROOT_ABSOLUTE" value="$
{toString:WSL_ROOT_PATH}" />

<target name="protoc-trackproto">
<echo>Execute: protoc --proto_path=./src --java_out=./src src/com/
wsl/CardioTrainer/track/*.proto</echo>
<apply executable="${full-protoc-command}" failonerror="true" >
<arg value="--proto_path=${WSL_ROOT_ABSOLUTE}/src/com/wsl/
CardioTrainer/track" />
<arg value="--java_out=${WSL_ROOT}/src" />

<fileset dir="${WSL_ROOT}/src/com/wsl/CardioTrainer/track">
<include name="*.proto"/>
</fileset>
</apply>
</target>

Thanks,
Artem

[Artem]

I should clarify that by "not understand" I mean that it thinks the
proto file is not contained within the proto_path, and asks for an
enclosing path.

[Kenton Varda]

What is your exact protoc command line?

I see this in your XML:

  protoc --proto_path=./src --java_out=./src src/com/wsl/CardioTrainer/track/*.proto

This may unfortunately confuse protoc, as it doesn't know that "src/" and "./src/" mean the same thing.  Canonicalizing paths is surprisingly hard (especially when trying to be portable), so protoc looks for exact matches.  This would work better:

  protoc --proto_path=./src --java_out=./src ./src/com/wsl/CardioTrainer/track/*.proto

Or this:

  protoc --proto_path=src --java_out=./src src/com/wsl/CardioTrainer/track/*.proto

[Artem Petakov]

Hi Kenton,

Thanks for the reply and for your amazing & tireless work on this library -- it's invaluable.

What you point out  is exactly the problem -- the ANT Fileset tag generates a fully qualified path for the protocol buffers themselves, and so I have to also fully qualify my "proto_path" as well to make it match. I can certainly appreciate that it's hard to make the canonical paths work, but it does strike me as non-intuitive -- I'm so used to being able to give any path to a tool. If it's too hard, perhaps at least a warning to this effect in that proto_path error message would help save future users time?

Right now the error it gives is:

"D:\Artem\wsl-trunk\wsl_root\src\com\wsl\CardioTrainer\weightloss\weightloss.proto: File does not reside within any path specified using --proto_path (or -I).  You must specify a --proto_path which encompasses this file."

Perhaps add: "Make sure the proto_path is an exact prefix of the file path to the protocol buffer."

Hmm, I see your TODOs as well (in importer.cc), although the objection to realpath strikes me as somewhat less serious than the common error I am facing now. Your call of course. SVN has something for canonicalizing, but it's not super-pretty. http://www.google.com/codesearch/p?hl=en&sa=N&cd=1&ct=rc#yJqH5U7qc7c/subversion-1.4.0/subversion/libsvn_subr/path.c&q=svn_path_canonicalize&l=63

// TODO(kenton):  It would be nice to handle "..", e.g. so that we can figure
//   out that "foo/bar.proto" is inside "baz/../foo".  However, if baz is a
//   symlink or doesn't exist, then things get complicated, and we can't
//   actually determine this without investigating the filesystem, probably
//   in non-portable ways.  So, we punt.
//
// TODO(kenton):  It would be nice to use realpath() here except that it
//   resolves symbolic links.  This could cause problems if people place
//   symbolic links in their source tree.  For example, if you executed:
//     protoc --proto_path=foo foo/bar/baz.proto
//   then if foo/bar is a symbolic link, foo/bar/baz.proto will canonicalize
//   to a path which does not appear to be under foo, and thus the compiler
//   will complain that baz.proto is not inside the --proto_path.

Thanks for the help,

Artem

[Kenton Varda]

I've updated the error message in rev 233.

The symlink issue is a serious one.  If we simply used realpath() to canonicalize paths then it would essentially prevent people from using any symlinks at all in their source tree, which I think in an unacceptable limitation.  Lots of unix users take advantage of symlinks in their build systems -- google included.

[Artem Petakov]

Thanks so much for fixing the message. And your point about symlinks is also very well taken. Amazing how hard it is to just canonicalize a path! Keep up the amazing work. The Java mobile-optimized version is soo much faster to initialize - thanks!

Artem