Protoc and line endings
1,155 views
Skip to first unread message
Marc Gravell
Jul 2, 2009, 2:48:35 AM7/2/09
to Protocol Buffers
I'm using protoc as the raw .proto parser for protobuf-net (I then
process the compiled binary for code-generation); at the moment, it is
very sensitive about line endings - if it isn't LF, it won't work.
This creates a bit of a nag for Windows users, as you have to go out
of your way to get the right line endings. I could patch the input
files, but that gets very tricky if the user is importing multiple
other files (with the wrong line endings) into their proto - and it
means that their CRLF files wouldn't work fully interoperably.
I have no idea how complex it would be, but is there any chance that
protoc could be less choosy about line endings?
For now, I've added some code to protobuf-net's "protogen", so that if
protoc reports failure it checks the *known* files for CR (and advises
the user) - but this is still tricky for the import scenario.
Thoughts?
Marc
process the compiled binary for code-generation); at the moment, it is
very sensitive about line endings - if it isn't LF, it won't work.
This creates a bit of a nag for Windows users, as you have to go out
of your way to get the right line endings. I could patch the input
files, but that gets very tricky if the user is importing multiple
other files (with the wrong line endings) into their proto - and it
means that their CRLF files wouldn't work fully interoperably.
I have no idea how complex it would be, but is there any chance that
protoc could be less choosy about line endings?
For now, I've added some code to protobuf-net's "protogen", so that if
protoc reports failure it checks the *known* files for CR (and advises
the user) - but this is still tricky for the import scenario.
Thoughts?
Marc
Kenton Varda
Jul 2, 2009, 1:03:16 PM7/2/09
to Marc Gravell, Protocol Buffers
Protoc treats \r as plain whitespace, so it should have no problem with Windows line endings. I just tested this and sure enough, protoc works fine with .proto files that use Windows-style line endings.
Mac pre-OSX line endings (\r with no \n) won't work if the file contains any comments.
What kind of errors are you seeing?
Marc Gravell
Jul 2, 2009, 3:44:05 PM7/2/09
to Protocol Buffers
My bad... it isn't the line endings - it is the UTF8 BOM; when I
switched one it switched the other.
(which is annoying; encoding is much trickier than just cr/lf!)
Marc
switched one it switched the other.
(which is annoying; encoding is much trickier than just cr/lf!)
Marc
Kenton Varda
Jul 2, 2009, 4:23:00 PM7/2/09
to Marc Gravell, Protocol Buffers
protoc actually expects its input to be UTF-8 (though non-ASCII characters are only allowed in default values for string fields). It just doesn't like the BOM.
Marc Gravell
Jul 2, 2009, 4:31:50 PM7/2/09
to Protocol Buffers
OK... is there any way it /could/ silently ignore the BOM? ;-p
I can try to advise the caller to use files without BOMs, but protoc
reads UTF8 anyway it seems reasonable to accept a BOM?
Marc
I can try to advise the caller to use files without BOMs, but protoc
reads UTF8 anyway it seems reasonable to accept a BOM?
Marc
Kenton Varda
Jul 2, 2009, 4:37:21 PM7/2/09
to Marc Gravell, Protocol Buffers
If you want to write up a patch to recognize a UTF8 BOM and ignore it, go ahead. You can just modify the Tokenizer class to recognize and discard a BOM appearing at the beginning of the input.
Reply all
Reply to author
Forward
0 new messages