sbt-protoc migration failing

[Dustin Chaloupka]

Hello!

I tried migrating to the version using sbt-protoc but am having some problems.

The first problem is that compiling external *.proto files does not seem to work how the docs suggest which is something like:

libraryDependencies += "com.trueaccord.scalapb" %% "scalapb-runtime" % "0.5.43" % "protobuf"

which this does add the *.proto files to "target/protobuf_external" but when running "sbt compile" it does not try to compile them. I do have the following in my "build.sbt" definition:

PB.targets in Compile := Seq(

  scalapb.gen() -> (sourceManaged in Compile).value

)

Since I did not have any actual defined *.proto files I was able to just add the following to fix this:

PB.protoSources in Compile := Seq(baseDirectory.value / "target" / "protobuf_external")

I also tried using PB.externalIncludePath but couldn't get that to work.

My second problem is that compiling the *.proto files is failing. Doing an "sbt compile" gives something like the following:

[info] Compiling 45 protobuf files to /path/to/project/target/src_managed/main

...

protoc-jar: protoc version: 300, detected platform: mac os x/x86_64

protoc-jar: executing: [/var/folders/6t/4qsyd4m97z95v6j1wk_2_hwc0000gn/T/protoc3121781921990358678.exe, --plugin=protoc-gen-scala=/var/folders/6t/4qsyd4m97z95v6j1wk_2_hwc0000gn/T/protocbridge2232639964928880563, --scala_out=..............]

google/api/service.proto: warning: Import google/protobuf/any.proto but not used.

google/api/service.proto: warning: Import google/api/label.proto but not used.

google/api/distribution.proto: warning: Import google/protobuf/any.proto but not used.

google/api/distribution.proto: warning: Import google/protobuf/timestamp.proto but not used.

--scala_out: protoc-gen-scala: Plugin failed with status code 141.

And doing "sbt last compile:protocGenerate" doesn't help much either:

[info] Compiling 45 protobuf files to /path/to/project/target/src_managed/main

[debug] protoc options:

[info] Compiling schema /path/to/project/target/protobuf_external/google/logging/type/http_request.proto

java.lang.RuntimeException: protoc returned exit code: 1

at scala.sys.package$.error(package.scala:27)

at sbtprotoc.ProtocPlugin$.sbtprotoc$ProtocPlugin$$compile(ProtocPlugin.scala:114)

at sbtprotoc.ProtocPlugin$$anonfun$sourceGeneratorTask$1$$anonfun$4.apply(ProtocPlugin.scala:141)

at sbtprotoc.ProtocPlugin$$anonfun$sourceGeneratorTask$1$$anonfun$4.apply(ProtocPlugin.scala:140)

at sbt.FileFunction$$anonfun$cached$1.apply(Tracked.scala:253)

at sbt.FileFunction$$anonfun$cached$1.apply(Tracked.scala:253)

at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3$$anonfun$apply$4.apply(Tracked.scala:267)

at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3$$anonfun$apply$4.apply(Tracked.scala:263)

at sbt.Difference.apply(Tracked.scala:224)

at sbt.Difference.apply(Tracked.scala:206)

at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3.apply(Tracked.scala:263)

at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3.apply(Tracked.scala:262)

at sbt.Difference.apply(Tracked.scala:224)

at sbt.Difference.apply(Tracked.scala:200)

at sbt.FileFunction$$anonfun$cached$2.apply(Tracked.scala:262)

at sbt.FileFunction$$anonfun$cached$2.apply(Tracked.scala:260)

at sbtprotoc.ProtocPlugin$$anonfun$sourceGeneratorTask$1.apply(ProtocPlugin.scala:150)

at sbtprotoc.ProtocPlugin$$anonfun$sourceGeneratorTask$1.apply(ProtocPlugin.scala:136)

at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)

at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)

at sbt.std.Transform$$anon$4.work(System.scala:63)

at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)

at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)

at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)

at sbt.Execute.work(Execute.scala:237)

at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)

at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)

at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)

at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

I tried digging around the sbt-protoc source code and other places to see if I could figure out what was going on, but the only other information I found was that when trying to use the scalapbc standalone binary the same thing happened as above (exit status 141 without any other information), however, I was able to use it to generate various other language's files with --java_out, --ruby_out etc. Which when looking through the options of the standalone, I don't see scala_out?

protoc-jar: protoc version: 300, detected platform: mac os x/x86_64

protoc-jar: executing: [/var/folders/6t/4qsyd4m97z95v6j1wk_2_hwc0000gn/T/protoc5298092248929576365.exe, --plugin=protoc-gen-scala=/var/folders/6t/4qsyd4m97z95v6j1wk_2_hwc0000gn/T/protocbridge6418930641522036657, --help]

Usage: /var/folders/6t/4qsyd4m97z95v6j1wk_2_hwc0000gn/T/protoc5298092248929576365.exe [OPTION] PROTO_FILES

Parse PROTO_FILES and generate output based on the options given:

  -IPATH, --proto_path=PATH   Specify the directory in which to search for

                              imports.  May be specified multiple times;

                              directories will be searched in order.  If not

                              given, the current working directory is used.

  --version                   Show version info and exit.

  -h, --help                  Show this text and exit.

  --encode=MESSAGE_TYPE       Read a text-format message of the given type

                              from standard input and write it in binary

                              to standard output.  The message type must

                              be defined in PROTO_FILES or their imports.

  --decode=MESSAGE_TYPE       Read a binary message of the given type from

                              standard input and write it in text format

                              to standard output.  The message type must

                              be defined in PROTO_FILES or their imports.

  --decode_raw                Read an arbitrary protocol message from

                              standard input and write the raw tag/value

                              pairs in text format to standard output.  No

                              PROTO_FILES should be given when using this

                              flag.

  -oFILE,                     Writes a FileDescriptorSet (a protocol buffer,

    --descriptor_set_out=FILE defined in descriptor.proto) containing all of

                              the input files to FILE.

  --include_imports           When using --descriptor_set_out, also include

                              all dependencies of the input files in the

                              set, so that the set is self-contained.

  --include_source_info       When using --descriptor_set_out, do not strip

                              SourceCodeInfo from the FileDescriptorProto.

                              This results in vastly larger descriptors that

                              include information about the original

                              location of each decl in the source file as

                              well as surrounding comments.

  --dependency_out=FILE       Write a dependency output file in the format

                              expected by make. This writes the transitive

                              set of input file paths to FILE

  --error_format=FORMAT       Set the format in which to print errors.

                              FORMAT may be 'gcc' (the default) or 'msvs'

                              (Microsoft Visual Studio format).

  --print_free_field_numbers  Print the free field numbers of the messages

                              defined in the given proto files. Groups share

                              the same field number space with the parent

                              message. Extension ranges are counted as

                              occupied fields numbers.

  --plugin=EXECUTABLE         Specifies a plugin executable to use.

                              Normally, protoc searches the PATH for

                              plugins, but you may specify additional

                              executables not in the path using this flag.

                              Additionally, EXECUTABLE may be of the form

                              NAME=PATH, in which case the given plugin name

                              is mapped to the given executable even if

                              the executable's own name differs.

  --cpp_out=OUT_DIR           Generate C++ header and source.

  --csharp_out=OUT_DIR        Generate C# source file.

  --java_out=OUT_DIR          Generate Java source file.

  --javanano_out=OUT_DIR      Generate Java Nano source file.

  --js_out=OUT_DIR            Generate JavaScript source.

  --objc_out=OUT_DIR          Generate Objective C header and source.

  --python_out=OUT_DIR        Generate Python source file.

  --ruby_out=OUT_DIR          Generate Ruby source file.

No idea if that is relevant or not though.

After some more digging around by compiling projects that looked like they should work (scalapbjs-test for example), with just what the examples had, compiling worked fine. However, adding the above fix for problem one did give:

--scala_out: java.io.IOException: Illegal seek

java.io.IOException: Illegal seek

at sun.nio.ch.FileChannelImpl.position0(Native Method)

at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:264)

at sun.nio.ch.ChannelInputStream.available(ChannelInputStream.java:116)

at com.google.protobuf.CodedInputStream.readRawBytesSlowPath(CodedInputStream.java:1192)

at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:519)

at com.google.protobuf.DescriptorProtos$SourceCodeInfo$Location.<init>(DescriptorProtos.java:32942)

at com.google.protobuf.DescriptorProtos$SourceCodeInfo$Location.<init>(DescriptorProtos.java:32855)

at com.google.protobuf.DescriptorProtos$SourceCodeInfo$Location$1.parsePartialFrom(DescriptorProtos.java:34667)

at com.google.protobuf.DescriptorProtos$SourceCodeInfo$Location$1.parsePartialFrom(DescriptorProtos.java:34662)

at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:497)

at com.google.protobuf.DescriptorProtos$SourceCodeInfo.<init>(DescriptorProtos.java:32532)

at com.google.protobuf.DescriptorProtos$SourceCodeInfo.<init>(DescriptorProtos.java:32486)

at com.google.protobuf.DescriptorProtos$SourceCodeInfo$1.parsePartialFrom(DescriptorProtos.java:36320)

at com.google.protobuf.DescriptorProtos$SourceCodeInfo$1.parsePartialFrom(DescriptorProtos.java:36315)

at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:497)

at com.google.protobuf.DescriptorProtos$FileDescriptorProto.<init>(DescriptorProtos.java:1248)

at com.google.protobuf.DescriptorProtos$FileDescriptorProto.<init>(DescriptorProtos.java:1124)

at com.google.protobuf.DescriptorProtos$FileDescriptorProto$1.parsePartialFrom(DescriptorProtos.java:4633)

at com.google.protobuf.DescriptorProtos$FileDescriptorProto$1.parsePartialFrom(DescriptorProtos.java:4628)

at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:497)

at com.google.protobuf.compiler.PluginProtos$CodeGeneratorRequest.<init>(PluginProtos.java:249)

at com.google.protobuf.compiler.PluginProtos$CodeGeneratorRequest.<init>(PluginProtos.java:186)

at com.google.protobuf.compiler.PluginProtos$CodeGeneratorRequest$1.parsePartialFrom(PluginProtos.java:1630)

at com.google.protobuf.compiler.PluginProtos$CodeGeneratorRequest$1.parsePartialFrom(PluginProtos.java:1625)

at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:197)

at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:215)

at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)

at com.google.protobuf.GeneratedMessageV3.parseWithIOException(GeneratedMessageV3.java:291)

at com.google.protobuf.compiler.PluginProtos$CodeGeneratorRequest.parseFrom(PluginProtos.java:625)

at protocbridge.frontend.PluginFrontend$$anonfun$runWithInputStream$2.apply(PluginFrontend.scala:55)

at protocbridge.frontend.PluginFrontend$$anonfun$runWithInputStream$2.apply(PluginFrontend.scala:54)

at scala.util.Try$.apply(Try.scala:161)

at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:54)

at protocbridge.frontend.PosixPluginFrontend$$anonfun$prepare$1.apply$mcV$sp(PosixPluginFrontend.scala:25)

at protocbridge.frontend.PosixPluginFrontend$$anonfun$prepare$1.apply(PosixPluginFrontend.scala:23)

at protocbridge.frontend.PosixPluginFrontend$$anonfun$prepare$1.apply(PosixPluginFrontend.scala:23)

at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)

at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)

at scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)

at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

So I'm guessing adding that is causing the weird 141 exit status with no other information, but without it, the protos are not compiled and added to the classpath?

[Dustin Chaloupka]

Trying some more things... I'm wondering if this is a problem with trying to compile the googleapis-common-protos that I had added at the same time. I reverted back to an older version of ScalaPB and am getting the same kind of error.

[Dustin Chaloupka]

As a note, the proto file I am trying to bring into my project is https://github.com/googleapis/googleapis/blob/master/google/api/annotations.proto (which requires two other proto files, one of which is already included in the scalapb runtime as a third party proto).

On Wednesday, October 12, 2016 at 4:13:19 PM UTC-5, Dustin Chaloupka wrote:

[Nadav Samet]

Hi Dustin,

Adding  "com.trueaccord.scalapb" %% "scalapb-runtime" % "0.5.43" % "protobuf" only extract the jars somewhere and adds them to the search path, so you can compile other protos that import them. It doesn't attempt to compile them. In fact, ScalaPB jar already includes a compiled version of its protos, and Google's protobuf-java includes the compiled version of Google's standard protos.

It's normal not to see scala_out in protoc's usage instructions (even when ran through scalapbc), since it's added as a third-party plugin.

Can you share a minimal project that demonstrates the problem you are seeing?

[Dustin Chaloupka]

Sure thing: https://github.com/DustinChaloupka/scalapb-test

When compiling that project the 141 exit code is the error given.

[Nadav Samet]

Thank you. I was able to reproduce. I wonder why this bug hasn't been discovered earlier. Looks like there's a code path where Google's CodedInputStream is calling InputStream.available() and that throws this IOException if protoc has already closed the pipe. sbt-protoc 0.99.2 depends on the right version of protoc-bridge.

The other thing was to set sbt-protoc to compile the external dependency (Google APIs proto)

This can be achieved by adding this to build.sbt:

PB.protoSources in Compile := Seq(file("src/main/protobuf"), file("target/protobuf_external/google/api"))

PB.includePaths in Compile := Seq(file("src/main/protobuf"), file("target/protobuf_external"))

--
You received this message because you are subscribed to the Google Groups "ScalaPB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalapb+unsubscribe@googlegroups.com.
To post to this group, send email to sca...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scalapb/d850b244-40a6-4545-bfe3-6e62885a0975%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

-Nadav

[Dustin Chaloupka]

Awesome, thanks Nadav!

To unsubscribe from this group and stop receiving emails from it, send an email to scalapb+u...@googlegroups.com.

To post to this group, send email to sca...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scalapb/d850b244-40a6-4545-bfe3-6e62885a0975%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

-Nadav