java_protobuf_library depending on protos in a nexus jar file

[Eric Ayers]

I've got a .jar file that contains .proto files uploaded to nexus.  I believe I'm doing the right thing by including it with a jar() dependency, but the protobuf compiler doesn't seem to see them.   Is this supposed to work? 

Here's the jar from my maven repo and the file I'm trying to get at:

jar -tvf /Users/zundel/.m2/repository//com/squareup/foo/foo-common-protos/HEAD-20140325T205108-1ea6eae/foo-common-protos-HEAD-20140325T205108-1ea6eae.jar | grep -e '\.proto'

  3949 Tue Mar 25 20:51:42 EDT 2014 squareup/foo/adjustment.proto

 Looking at the protoc invocation from -ldebug, I don't see any directories from .pants.d included other than the stuff in my source tree.  From looking at the code, it seems that only source_root() entries for other targets are included.   In fact, I'm not even sure its downloading any  artifacts before compiling the protos.  I never see the step 'ivy' output in the console.  

--

(I slightly altered the output and replaced some strings with 'foo' and 'bar' below.)

$ ./pants goal bundle example

19:59:31 00:00 [main]
               (To run a reporting server: ./pants server)
19:59:31 00:00   [setup]
19:59:31 00:00     [bootstrap]
19:59:31 00:00     [parse]
19:59:31 00:00   [bootstrap]
19:59:31 00:00     [bootstrap-jvm-tools]
19:59:31 00:00   [gen]
19:59:31 00:00     [thrift]
19:59:31 00:00     [scrooge]
19:59:31 00:00     [protoc]
                   Invalidated 25 targets containing 123 source files.squareup/foo/adjustment.proto: File not found.
squareup/foo/bar.proto: Import "squareup/foo/adjustment.proto" was not found or had errors.
squareup/foo/bar.proto:839:12: "squareup.foo.Foo" is not defined.
squareup/foo/bar.proto:833:15: "squareup.foo.Foo" is not defined.
...
FAILURE: /Users/zundel/.pants.d/bin/protobuf/mac/10.9/2.4.1/protoc ... exited non-zero (1)

               FAILURE

My BUILD file:

java_protobuf_library(name='proto',
    sources = [ rglobs('*.proto') ],
      dependencies = [
        pants('all-protos/protos/api/src/main/proto:proto'),
        #... more proto dependencies from my source tree ...
        jar(org='com.squareup.foo',
            name='foo-common-protos',
            rev='HEAD-20140325T205108-1ea6eae').with_sources(),
    ]
)

[Eric Ayers]

I had an side conversation and wanted to fold it back into the thread:

1) Apparently this isn't supposed to be working in pants right now for protobufs, but some work has been done for thrift.  

2) Why do we need this?  Our situation is that some of our protobufs are defined in a project hosted in another git repo.  Inside our repo that uses pants we want to create a new protobuf that uses definitions from the one in the repo.  The protoc compiler needs to have all of the .proto files available.  Also, we want to support other languages than just Java, so they need to be able to use the .proto directly to write access wrappers in other languages.

[Benjy Weinberger]

[Adding pl, as this is affected by his refactor]

Patric's refactor formalizes the concept of "payload" on a target. A
FileSet (the thing returned by globs()) is payload, but so, for
example, is a jar_dependency (whereas in our current regime a
jar_dependency is sort-of like a target, in that it's a node in the
dep graph).

So it sounds like what you need pretty much falls naturally out of
Patrick's refactoring work (and also validates that design decision).

So you'd have something like:

java_protobuf_library(name='foo', sources=jar(...))

Does that make sense?

[Eric Ayers]

I really only want to pull the protos out of the jar for a java_protobuf_library sources, there is a lot of other crap in there.  Just so long as we can add a filter on it.

jar -tvf /Users/zundel/.m2/repository//com/squareup/inventory/inventory-common-protos/HEAD-20140325T205108-1ea6eae/inventory-common-protos-HEAD-20140325T205108-1ea6eae.jar

     0 Tue Mar 25 20:51:44 EDT 2014 META-INF/

   359 Tue Mar 25 20:51:42 EDT 2014 META-INF/MANIFEST.MF

     0 Tue Mar 25 20:51:42 EDT 2014 com/

     0 Tue Mar 25 20:51:42 EDT 2014 com/squareup/

     0 Tue Mar 25 20:51:42 EDT 2014 com/squareup/inventory/

     0 Tue Mar 25 20:51:42 EDT 2014 com/squareup/inventory/validation/

     0 Tue Mar 25 20:51:42 EDT 2014 squareup/

     0 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/

     0 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/inventory/

     0 Tue Mar 25 20:51:42 EDT 2014 squareup/inventory/

   911 Tue Mar 25 20:51:42 EDT 2014 com/squareup/inventory/validation/InventoryValidationModule.class

  1180 Tue Mar 25 20:51:42 EDT 2014 com/squareup/inventory/validation/AdjustmentLinesValidator$1.class

  1449 Tue Mar 25 20:51:42 EDT 2014 com/squareup/inventory/validation/OnlyOneFieldValidator.class

  3511 Tue Mar 25 20:51:42 EDT 2014 com/squareup/inventory/validation/AdjustmentLinesValidator.class

  5154 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/inventory/InventoryAdjustmentProtos.class

 16595 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/inventory/InventoryAdjustmentProtos$InventoryAdjustment.class

 13993 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/inventory/InventoryAdjustmentProtos$InventoryAdjustmentLine.class

  1299 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/inventory/InventoryAdjustmentProtos$InventoryQuantityState$1.class

  1413 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/inventory/InventoryAdjustmentProtos$InventoryAdjustmentLineOrBuilder.class

 20123 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/inventory/InventoryAdjustmentProtos$InventoryAdjustmentLine$Builder.class

  2067 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/inventory/InventoryAdjustmentProtos$InventoryAdjustmentOrBuilder.class

  1305 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/inventory/InventoryAdjustmentProtos$InventoryAdjustmentType$1.class

  4178 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/inventory/InventoryAdjustmentProtos$InventoryQuantityState.class

  4178 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/inventory/InventoryAdjustmentProtos$InventoryAdjustmentType.class

 27191 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/inventory/InventoryAdjustmentProtos$InventoryAdjustment$Builder.class

  3226 Tue Mar 25 20:51:42 EDT 2014 squareup/protos/inventory/InventoryAdjustmentProtos$1.class

  3949 Tue Mar 25 20:51:42 EDT 2014 squareup/inventory/adjustment.proto

     0 Tue Mar 25 20:51:44 EDT 2014 META-INF/maven/

     0 Tue Mar 25 20:51:44 EDT 2014 META-INF/maven/com.squareup.inventory/

     0 Tue Mar 25 20:51:44 EDT 2014 META-INF/maven/com.squareup.inventory/inventory-common-protos/

  1332 Tue Mar 25 20:51:12 EDT 2014 META-INF/maven/com.squareup.inventory/inventory-common-protos/pom.xml

   153 Tue Mar 25 20:51:42 EDT 2014 META-INF/maven/com.squareup.inventory/inventory-common-protos/pom.properties

[Benjy Weinberger]

Yeah, that shouldn't be hard to do. In many cases the tasks themselves
ignore sources they don't know how to handle.