Pants to Bazel conversion

[Krish Narukulla]

Hi Team,

We have finally decided to move forward with Bazel. Currently we do use Pants build system which is similar to Bazel.

We are planning to write translation layer for the discrepancies between Pants and Bazel.

Project: https://github.com/krisnaru/pants-bazel

Currently I am blocked on 3rd party dependencies on how to transform Pants targets to Bazel.

I have written macro which does translation from  label: 3rdparty/java/org/log4j2:log4j-core to @maven//:org_apache_logging_log4j_log4j_core. How can I load the maven jar from macro?   File: https://github.com/krisnaru/pants-bazel/blob/master/tools/build_rules/java_library.bzl#L44  .

I have tried with the java_import rule, but could not load rules from macros. Build Error

---------- -------------

in jars attribute of java_import rule //3rdparty/java/org/log4j2:log4j-core: should not refer to Java rules. Since this rule was created by the macro 'jar_library', the error might have been caused by the macro implementation

--------------------

[Krish Narukulla]

I am coming up with translation layer , which is has been tracked as https://github.com/krisnaru/pants-bazel

[Yi Cheng]

Hi Krish,

On a high level, one major difference between Pants and Bazel is that Bazel needs to know where ALL 3rdparty dependencies are located prior to / at the very beginning of the build. 

To accommodate this, we use https://github.com/twitter/bazel-multiversion to convert all Pants' 3rdparty JVM definitions (3rdparty/java/:: in your case) to a format (.bzl file) Bazel understands, and automating this process by wrapping it via a repository rule.

The tool itself will call Pants to get the relevant info. I think the general steps would look like:

1. make sure the tool works

* calls Pants correctly (actually we have an internal Pants task for it that simply converts 3rdparty/java information to a json. This can can be open sourced / shared, but we don't have a process for it yet),

* outputs .bzl file correctly

2. automate this process via repository rule

If the philosophy sounds promising for your use, we can discuss how to get you try it out.

Thanks,

Yi

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/34dd5bdf-43c9-4fda-9ce9-f057ac011771n%40googlegroups.com.

--

Yi

[Yi Cheng]

and to add, https://github.com/twitter/bazel-multiversion will also do the resolution, since bazel does need to know ALL the 3rdparty dependencies transitively.

--

Yi

[Krish Narukulla]

Thanks Yi.

We have copied 3p dependencies from 3rdparty directory and supply dependencies to ruleset: https://github.com/bazelbuild/rules_jvm_external

We are spending energy for doing translation from pants to bazel. would you share translation layer codebase?

[Yi Cheng]

Ok if https://github.com/bazelbuild/rules_jvm_external fits your need, then please go for it. We are using a separate mechanism because of many other complications.

Regarding the translation layer, I took a glance at https://github.com/krisnaru/pants-bazel, and that's exactly the same of what we are doing essentially. We have a bit more organization, again due to complexities in our code base.

For example, there can be common transformations for `sources` -> `srcs`. I can share some code snippets, but I don't think wholesale would be useful because there are twitter specific pants code, and this will also require a lot of processes internally.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/2bb05d88-5f8f-4d39-beac-077292a09fe4n%40googlegroups.com.

--

Yi

[Yi Cheng]

For some I will get back to you with code snippets, as they are a bit involved.

1. jvm_binary - we translate directly to scala_binary, because there's 0.01% chance in our code base that a deployable is java only. YMMV.

* shading: we have not looked into it yet.

2. jvm_app is a bit tricky. We essentially made a separate rule for it, which does 

* for the "binary" field, create a "scala_binary" target

* once the <scala_binary>.deploy_jar (X) is built, this rule will make an archive putting X and "bundles" loose files together. NOTE: the final bundle will have a different structure than Pants' bundle, but the runtime behavior should be the same.

* tricky part is that bazel cannot glob past package boundaries, if files under bundles happen to fall under other packages (BUILD), this will need to be fixed first.

3. java_protobuf_library, on a high level, we are creating two targets in bazel - proto_library and java_proto_library, then have the latter depend on the former.

load("@rules_proto//proto:defs.bzl", "proto_library")
load("@rules_java//java:defs.bzl", "java_proto_library")

def java_protobuf_library(sources, name, dependencies=None, **kwargs):

    proto_name = "{}_proto".format(name)
    proto_kwargs = {
        "name": proto_name,
        "sources": sources,
        "strip_import_prefix": "//" + native.package_name(), // this is dependent on your repo structure
    }

    proto_library(**proto_kwargs) // I modified this slightly because internally we use a helper function.

    proto_dep = ":{}".format(proto_name)
    java_proto_kwargs = {
        "name": name,
        "dependencies": [proto_dep],
    }
    java_proto_kwargs.update(kwargs)

    java_proto_library(**java_proto_kwargs) // I modified this slightly because internally we use a helper function.
    

On Wed, May 5, 2021 at 4:09 PM Krish Narukulla <kris...@gmail.com> wrote:

Yes, could you share code snippets on how you make the following targets work in the translation layer?

1. Bundles

 

jvm_app(name='ria-bundle',

dependencies=[':ria-server'],

bundles=[

bundle(relative_to='resources/', fileset=rglobs('resources/*')),

bundle(fileset=['ria_server.sh', 'log_config'])

]

)

2. protobufs

java_protobuf_library(name='user-profile',

sources=['userprofile.proto'],

)

3. Shading

jvm_binary(name='dim-lookup-client-1.0.0-SNAPSHOT',

dependencies=[':dimlookup',

'3rdparty/java/org/codehaus/jackson:core',

'3rdparty/java/org/codehaus/jackson:mapper'

],

shading_rules=[shading_relocate('com.maxmind.**', 'com.xxx.relocated.maxmind.@1'),

shading_relocate('com.google.**', 'com.xxx.relocated.google.@1'),

]

)

Thanks

Krish
669-241-9356

--

Yi

[Krish Narukulla]

Yes, could you share code snippets on how you make the following targets work in the translation layer?

1. Bundles

 

jvm_app(name='ria-bundle',

dependencies=[':ria-server'],

bundles=[

bundle(relative_to='resources/', fileset=rglobs('resources/*')),

bundle(fileset=['ria_server.sh', 'log_config'])

]

)

2. protobufs

java_protobuf_library(name='user-profile',

sources=['userprofile.proto'],

)

3. Shading

jvm_binary(name='dim-lookup-client-1.0.0-SNAPSHOT',

dependencies=[':dimlookup',

'3rdparty/java/org/codehaus/jackson:core',

'3rdparty/java/org/codehaus/jackson:mapper'

],

shading_rules=[shading_relocate('com.maxmind.**', 'com.xxx.relocated.maxmind.@1'),

shading_relocate('com.google.**', 'com.xxx.relocated.google.@1'),

]

)

Thanks

Krish
669-241-9356

[Krish Narukulla]

Yi Cheng,

Sorry to bother you on this. How did you make lombok work in translation?

I have implemented a new macro for jar_library which does java_library with exported plugins.

Codebase:

https://github.com/krisnaru/pants-bazel

https://github.com/krisnaru/pants-bazel/blob/master/tools/build_rules/jar_library.bzl

common-lib builds are successful. But App builds fail consistently(unable to find lombok getters).

src/main/java/com/app/Test.java:7: error: cannot find symbol
        System.out.println(d.getProperty());
                            ^
  symbol:   method getProperty()

Thanks

Krish
669-241-9356