Any chance to get something like GRADLE-1980 into the core?

[Oliver Nautsch]

Hello,

I have the problem that I have to manipulate the structure of a XML file during a copy task. This means a line based filtering is not enough. I need the whole content from a file as a String and would like hand over the String to a XmlSlurper.

And I found this issue:

Provide a copy filter where closure transforms entire file text content

Is there any chance to get it into the core. I would like to contribute tests etc to the project if you are thinking this is something valuable.

What do you think?
Is this the right place to ask this question?

Regards
Oliver

[Mark Vieira]

Oliver,

This is a super interesting topic if you step back and look at it. I believe there is a bit of overlap here with this discussion on the forums as well: https://discuss.gradle.org/t/iterate-over-a-copyspec/13030/4.

I'd like to know more about your use case. Specifically, what kinds of transformation are you doing to these files, what are the files used for (what are they an input to) and where do the originals come from? I believe this notion of a file collection transform is a very common use case. However, I think it's gotten muddled with the notion of a copy operation. The question being, is this something that should be part of the `CopySpec` API, or should we separate it out and model it as its own thing. In general I'd say builds should aim for the latter. Tasks should do the most discrete bit of work as possible. There are a number of benefits to doing this, primarily because of incremental builds. Modeling file transformations as separate tasks, rather than in terms of being part of a copy operation, benefit from:

1. Faster incremental builds since changes to non-transformed files will not require the transformation (which could be expensive) to be done again.

2. Implementing the transformation as a custom task allows you to leverage more fine grained incrementalness via IncrementalTaskInputs. That is, only retransform the files that are new or have changed.

Another question remains, which is more related to the topic I posted, which is, should transformation tasks be considered a subtype of copy tasks. As in, should they share an API, specifically, the `CopySpec` API. In essence, we are simply treating a transform as a copy that _also_ does some work to the files. In many ways this is true but we have already polluted the copy API with transformation hooks. Consider if `JavaCompile` as to extend `Copy` or even simply take a `CopySpec` as an input. What would we do with transforms declared using `filter()` or `expand()`? We likely need a clearer separation of how we describe an operation whose purpose is to "move stuff from a to b" from "transform a into b".

Really, a transformation task boils down to declaring inputs, configuring what should happen to them and producing an output. The last two bits are likely to be very transformation specific so it makes sense for the individual task to declare that API. However, the first part is a pretty common thing. The question is whether we should provide some convenient plumbing for build authors to make this easier and if so, what might that look like? Part of this is potentially identifying what bits of the copy operation (filtering files to include, traversing a collection, declaring a destination, etc) that might be more generally available to these types of tasks. Essentially, some kind of `MinimalCopySpec` which is basically a `CopySpec` that doesn't implement `ContentFilterable`.

--
You received this message because you are subscribed to the Google Groups "gradle-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gradle-dev+...@googlegroups.com.
To post to this group, send email to gradl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/e17de802-2cb4-46fe-9409-77b1555549bd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mark Vieira

Principal Engineer

Gradle, Inc.

ma...@gradle.com

[Oliver Nautsch]

Hello Mark,

thanks for you quick response. First my use case:

We have a lot of existing maven projects and I want to convert them to gradle without changing the sources except adding the gradle build files. There is a existing maven plugin which changes the beans.xml (Java CDI) based on given properties. As far as I can see, this properties are defined inside the pom.xml file. The beans.xml file are located in the src/main/resources directory and are changed during the process-resources phase of maven. The plugin adds new interceptors into the beans.xml file based on the properties.

The idea to separate copy and transform is very appealing and I see the advantages. The output of this task can be a part of the input of a copy task like processResources. Or better: part of the output of this copy task is the input for the transformation.

I think a 'MinimalCopySpec' is good as a starting point. But you need some kind of hook to do something with the content - e.g. filtering ;) . Which leads to something like filter hook in a more abstract way. Is this what you are thinking about?

A other way to think about this kind of task is to think it like the unix pipe mechanism. But may be this is too much...

Oliver