Implementing file-based filtering and resource transformation during packaging

[ig...@wix.com]

I'm researching possible implementation of functionality similar to maven-shade-plugin.

One of the required features is to exclude/filter out sets of files (e.g. META-INF/*.config)

In addition, a way to perform file transformation, e.g. concat all resources/licenses into one file.

One idea is to add this functionality to the SingleJar utility to do this, extending its capabilities with this functionality.

SingleJar already does many similar things, so it looks like a good place for it.

We want to implement it and send a PR. Would that be acceptable?

Thanks,

Igal @ Wix.com

[Lukács T. Berki]

Hey there,

Sorry for the late reply -- I should really keep my inbox in a better shape.

In principle, please go ahead, but first a few questions in the interest of ensuring that the functionality you are planning to implement is at least somewhat reusable:

1. What functionality do you exactly need?
2. Are you planning to hard-wire this into singlejar or do you want to add some sort of extension mechanism?
3. How do you plan to tackle the issue that singlejar currently has two implementations, a Java and a C++ one? (for Windows and everything else, respectively)
4. We already provide a binary called jarjar to rename classes (@bazel_tools//third_party/jarjar). Would you be able to re-use some functionality from there?

--
You received this message because you are subscribed to the Google Groups "Bazel/JVM Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-sig-jv...@googlegroups.com.
To post to this group, send email to bazel-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-sig-jvm/63d590f3-94a6-4891-87d5-22ee4f8119c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

[ig...@wix.com]

Thank you for your reply!

1. We're looking to implement specifically two features: filtering (exclude files by regex/wildcard), and Resource transformation (in particular, resource appending, similar to AppendingTransformer in maven-shade)

2. The plan was to hardwire this into singlejar, however,

3. This was just foiled as I learned about the C++ version :)

    My C++ is a bit weak, how about a possibility of using just the java version for this, regardless of the OS?

4. I know about jarjar, but it solves slightly different set of problems, so we didn't consider it for this task.

With regards to 1 - what would you recommend?

Thanks!

[Lukács T. Berki]

On Wed, Jun 27, 2018 at 2:42 PM igalt via Bazel/JVM Special Interest Group <bazel-...@googlegroups.com> wrote:

Thank you for your reply!

1. We're looking to implement specifically two features: filtering (exclude files by regex/wildcard), and Resource transformation (in particular, resource appending, similar to AppendingTransformer in maven-shade)

Can this be done as a custom post-processing step after the singlejar invocation, i.e. genrule(srcs=["foo_deploy.jar"]) or something? I'd prefer to keep singlejar a fairly simple tool, as far as that's possible.

 

2. The plan was to hardwire this into singlejar, however,

3. This was just foiled as I learned about the C++ version :)

    My C++ is a bit weak, how about a possibility of using just the java version for this, regardless of the OS?

I'd prefer not to; IIRC we don't even ship the Java version to platforms for which the C++ one is used. I'd rather we use the C++ one *everywhere*, but I don't remember off the bat why we use the Java version on Windows.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-sig-jvm/629de32d-6aaf-4b85-8f13-eead682a656c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Liam Miller-Cushon]

On Wed, Jun 27, 2018 at 7:45 AM 'Lukács T. Berki' via Bazel/JVM Special Interest Group <bazel-...@googlegroups.com> wrote:

Can this be done as a custom post-processing step after the singlejar invocation, i.e. genrule(srcs=["foo_deploy.jar"]) or something? I'd prefer to keep singlejar a fairly simple tool, as far as that's possible.

+1

 

I'd rather we use the C++ one *everywhere*, but I don't remember off the bat why we use the Java version on Windows.

The plan is to use the c++ one on windows too: https://github.com/bazelbuild/bazel/issues/2241

[László Csomor]

>> 3. This was just foiled as I learned about the C++ version :)
>>     My C++ is a bit weak, how about a possibility of using just the java version for this, regardless of the OS?
>

> I'd prefer not to; IIRC we don't even ship the Java version to platforms for which the C++ one is used. I'd rather we use the C++ one *everywhere*, but I don't remember off the bat why we use the Java version on Windows.

Yes, please use the C++ version if possible.

The reason Bazel on Windows uses the Java implementation of Singlejar is because the C++ on needs to be ported to Windows and we haven't done that yet.

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

[ig...@wix.com]

A quick update! After playing around in the C++ singlejar implementation, I found it already supports most (all?) of the scenarios we need!

In particular, merging META-INF/spring.handlers and .schemas, as well as services, and filtering out *.RSA resources.

This is great because no changes need to be made :)

Thanks again for the heads-up!