Binaries in the new software model
52 views
Skip to first unread message
Peter Ledbrook
Feb 14, 2016, 9:11:01 AM2/14/16
to gradl...@googlegroups.com
Hi,
I've been trying to migrate the Asciidoctor Gradle plugin to the new software model as it seems like a really good fit. The new 2.11 features have certainly helped, but it looks like the binary mechanism seems too limited. I apologise in advance for the lengthy email, but I think it's important to get as much detail in as possible.
The perceived limitations of the software model could be due to a lack of understanding on my part, so I want to give you an idea of what I want to achieve and how I'd like to achieve it. Perhaps it will turn out that there are no such limitations, but perhaps not.
The way I want the Asciidoctor plugin to work is quite straightforward. You have multiple components, perhaps representing a user guide, a reference guide, some HOWTOs, etc. Each component has one and only one associated Asciidoc source set. The binaries produced for a component can be one or more of things like HTML, PDF, and Docbook. This model should then produce tasks of the form
build<Component>AsciidocAs<Binary>
For example, `buildDocsAsciidocAsPdf`. I'd also like to allow other plugin or build authors to register additional binary types (Latex perhaps).
From the user's perspective, they should just need to configure which binaries to produce for a component, and possibly configure the location(s) of the source files. I'd like to do this via a DSL like this:
model {
docs(AsciidocDocument) {
binaries {
pdf(AsciidocPdf)
}
}
}
My current attempts to achieve the above have hit a few problems. My primary concern is that it seems impossible to selectively enable or disable the binary types for a component. As I understand it, all registered binaries are created for all components. Is that correct? If so, I feel this needs to change.
The other significant issue relates to the binary definitions themselves. I'm trying to use managed objects, but the normal rules of Java classes don't apply. Every binary has a single property representing the backend, e.g. "pdf", "html5", etc. These are kind of internal to Asciidoctor. I prefer types in this case. But the following model doesn't work:
@Managed
interface AsciidoctorBinary extends BinarySpec {
String getBackend()
}
@Managed
abstract AsciidocPdf implements AsciidoctorBinary {
String getBackend() { return "pdf" }
}
The problem may have been due to trying to incorporate internal views for both of these as well, but I need an internal view so that I can set the component for the binary. I need to do that so I can generate tasks with the required names.
Would it not make sense to allow specialisations of managed interfaces? Not everything should be configurable to the user.
So, does this all seem reasonable? Is it currently achievable while still using managed objects? If not, could the model be enhanced to make it easier to achieve what I want?
Thanks,
Peter
Andrew Oberstar
Feb 14, 2016, 10:33:22 AM2/14/16
to gradl...@googlegroups.com
The devs can answer your questions better than I, but being in the middle of my own model plugin (for Clojure) I might be of some help. Responses inline.
On Sun, Feb 14, 2016 at 8:11 AM Peter Ledbrook <pe...@cacoethes.co.uk> wrote:
Hi,I've been trying to migrate the Asciidoctor Gradle plugin to the new software model as it seems like a really good fit. The new 2.11 features have certainly helped, but it looks like the binary mechanism seems too limited. I apologise in advance for the lengthy email, but I think it's important to get as much detail in as possible.The perceived limitations of the software model could be due to a lack of understanding on my part, so I want to give you an idea of what I want to achieve and how I'd like to achieve it. Perhaps it will turn out that there are no such limitations, but perhaps not.The way I want the Asciidoctor plugin to work is quite straightforward. You have multiple components, perhaps representing a user guide, a reference guide, some HOWTOs, etc. Each component has one and only one associated Asciidoc source set. The binaries produced for a component can be one or more of things like HTML, PDF, and Docbook. This model should then produce tasks of the formbuild<Component>AsciidocAs<Binary>For example, `buildDocsAsciidocAsPdf`. I'd also like to allow other plugin or build authors to register additional binary types (Latex perhaps).
You can get the task names created through the binary's task collection, which I thing would get you something close to what you desired, and at worst still follows the convention of other binary tasks.
binary.getTasks().taskName("build")
From the user's perspective, they should just need to configure which binaries to produce for a component, and possibly configure the location(s) of the source files. I'd like to do this via a DSL like this:model {docs(AsciidocDocument) {binaries {pdf(AsciidocPdf)}}}
I'm not aware of a way to do this, but my impression has been that they expect the properties of the component to determine which binaries are created (such as how adding different java platforms triggers different jar binaries). Maybe something like this. However, I do see your desired approach being helpful.
docs(AsciidocDocument) {
format 'pdf'
format 'html'
}
My current attempts to achieve the above have hit a few problems. My primary concern is that it seems impossible to selectively enable or disable the binary types for a component. As I understand it, all registered binaries are created for all components. Is that correct? If so, I feel this needs to change.The other significant issue relates to the binary definitions themselves. I'm trying to use managed objects, but the normal rules of Java classes don't apply. Every binary has a single property representing the backend, e.g. "pdf", "html5", etc. These are kind of internal to Asciidoctor. I prefer types in this case. But the following model doesn't work:@Managedinterface AsciidoctorBinary extends BinarySpec {String getBackend()}@Managedabstract AsciidocPdf implements AsciidoctorBinary {String getBackend() { return "pdf" }}
Maybe there are other properties you left out for brevity, but could you alternately have only one base type with the "String getBackend()" or skip that property by just relying on the type?
The problem may have been due to trying to incorporate internal views for both of these as well, but I need an internal view so that I can set the component for the binary. I need to do that so I can generate tasks with the required names.
This part I'm not as familiar with, I had my own struggles trying to get internal views to work the way I expected and failed. However, as noted above, you might get what you need for the names using the binary task collection.
Would it not make sense to allow specialisations of managed interfaces? Not everything should be configurable to the user.So, does this all seem reasonable? Is it currently achievable while still using managed objects? If not, could the model be enhanced to make it easier to achieve what I want?Thanks,Peter
--
You received this message because you are subscribed to the Google Groups "gradle-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gradle-dev+...@googlegroups.com.
To post to this group, send email to gradl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/CANBeYQQ2pztc%3DKvcEM21aCYJT%2B_yC2dx%2BgwVV7VncLE2iL%3DATg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Peter Ledbrook
Feb 15, 2016, 3:10:42 AM2/15/16
to gradl...@googlegroups.com
Thanks for the responses. Replies inline.
For example, `buildDocsAsciidocAsPdf`. I'd also like to allow other plugin or build authors to register additional binary types (Latex perhaps).You can get the task names created through the binary's task collection, which I thing would get you something close to what you desired, and at worst still follows the convention of other binary tasks.binary.getTasks().taskName("build")
I don't think this incorporates the component name.
I'm not aware of a way to do this, but my impression has been that they expect the properties of the component to determine which binaries are created (such as how adding different java platforms triggers different jar binaries). Maybe something like this. However, I do see your desired approach being helpful.docs(AsciidocDocument) {format 'pdf'format 'html'}
I did consider doing this, but I don't think it works with distinct binary types. It would work fine with a generic Asciidoctor binary type though.
@Managedabstract AsciidocPdf implements AsciidoctorBinary {String getBackend() { return "pdf" }}Maybe there are other properties you left out for brevity, but could you alternately have only one base type with the "String getBackend()" or skip that property by just relying on the type?
The only other property right now is a reference to the owning component. That should also be publicly read-only, i.e. writable only through the internal view.
Peter
Cedric Champeau
Feb 15, 2016, 3:28:48 AM2/15/16
to gradle-dev
Hi Peter!
2016-02-14 16:33 GMT+01:00 Andrew Oberstar <ajobe...@gmail.com>:
From the user's perspective, they should just need to configure which binaries to produce for a component, and possibly configure the location(s) of the source files. I'd like to do this via a DSL like this:model {docs(AsciidocDocument) {binaries {pdf(AsciidocPdf)}}}I'm not aware of a way to do this, but my impression has been that they expect the properties of the component to determine which binaries are created (such as how adding different java platforms triggers different jar binaries). Maybe something like this. However, I do see your desired approach being helpful.docs(AsciidocDocument) {format 'pdf'format 'html'}
Yes, this is definitely an option, but not the only one. The software model is designed so that Gradle has to configure as little as possible when executing a particular task. There are, therefore, several possible approaches with regards to configuring binaries. In the Java and native software models, binaries are considered outputs of a library, and created directly by Gradle (thanks to the `@ComponentBinaries` annotation). The number of binaries as well as their configuration depend on the configuration of the library and their variants. In short, the Java plugin author knows how many binaries to create, depending on the configuration of the library. It's not the responsibility of the user to create binaries.
Second, you mustn't mix the DSL (the model { ... } block) with the capabilities of the software model. The DSL has limited abilities compared to the global software model, and should mostly be used for configuring things. In particular, it offers a consistent DSL for creating model nodes, and in your example, you are leveraging this dsl to create a new binary. This is totally legit, even if you have to wonder if this is how you want the user to configure the outputs. In particular, I don't see any particular reason why you wouldn't create all binaries (pdf, html, ...) directly in the plugin, and just let the user configure them:
model {
docs(AsciidocDocument) {
binaries {
pdf {
imageCompressionLevel = 0.9
}
}
}
}
Note that since your plugin would create the "pdf" binary, you don't need to provide the type anymore. Similarily, you *could* decide that your plugin would also create the top-level "docs" component too. However, if you do so, be aware that you should probably choose an alternate, less potential conflicting name. You might wonder why creating all binaries if in the end, the user only wants one. There are several things to this:
- is the creation of a binary model element cheap? It should be cheap, otherwise, it probably means you are initializing too much in there (heavyweight operations should be deferred at execution time)
- do you want the user to see the potential export formats in the model report? If so, yes, create all binaries, even if they are not used.
- it gives you the opportunity to create a "hat" task that creates all exports in a single call
- still gives you the opportunity to call a single export task
My current attempts to achieve the above have hit a few problems. My primary concern is that it seems impossible to selectively enable or disable the binary types for a component. As I understand it, all registered binaries are created for all components. Is that correct? If so, I feel this needs to change.The other significant issue relates to the binary definitions themselves. I'm trying to use managed objects, but the normal rules of Java classes don't apply. Every binary has a single property representing the backend, e.g. "pdf", "html5", etc. These are kind of internal to Asciidoctor. I prefer types in this case. But the following model doesn't work:@Managedinterface AsciidoctorBinary extends BinarySpec {String getBackend()}@Managedabstract AsciidocPdf implements AsciidoctorBinary {String getBackend() { return "pdf" }}Maybe there are other properties you left out for brevity, but could you alternately have only one base type with the "String getBackend()" or skip that property by just relying on the type?
That's almost the right approach. Your "backend" property is typically an internal property. That is to say, something that you, as a plugin author, need, but doesn't have to be exposed to the end user. As such, you need to declare such an internal interface, and use it when you create your binary. For example:
interface AsciidocDocument extends ComponentSpec {}
class DefaultAsciidocDocument extends BaseComponentSpec implements AsciidocDocument {}
@Managed
interface AsciidoctorBinary extends BinarySpec {}
@Managed
interface AsciidoctorBinaryInternal extends AsciidoctorBinary {
String getBackend()
void setBackend(String backend)
}
class AsciidoctorRules extends RuleSource {
@ComponentType
void registerComponentType(TypeBuilder<AsciidocDocument> builder) {
builder.defaultImplementation(DefaultAsciidocDocument)
}
@BinaryType
void registerAsciidoctorBinary(TypeBuilder<AsciidoctorBinary> builder) {
builder.internalView(AsciidoctorBinaryInternal)
}
@Defaults
void createExportBinaries(AsciidocDocument docs) {
docs.binaries.create("pdf", AsciidoctorBinaryInternal) {
binary.backend = 'pdf'
}
}
@Model
void docs(AsciidocDocument docs) {}
}
Does that help?
Peter Ledbrook
Feb 15, 2016, 10:13:14 AM2/15/16
to gradl...@googlegroups.com
Hi Cédric,
Thanks for the detailed reply. You say that "It's not the responsibility of the user to create binaries", but in my case I think it's the responsibility of the user to say which binaries they want if the default isn't appropriate.
Note that since your plugin would create the "pdf" binary, you don't need to provide the type anymore. Similarily, you *could* decide that your plugin would also create the top-level "docs" component too. However, if you do so, be aware that you should probably choose an alternate, less potential conflicting name. You might wonder why creating all binaries if in the end, the user only wants one. There are several things to this:- is the creation of a binary model element cheap? It should be cheap, otherwise, it probably means you are initializing too much in there (heavyweight operations should be deferred at execution time)
It is certainly cheap, yes.
- do you want the user to see the potential export formats in the model report? If so, yes, create all binaries, even if they are not used.
This would probably be useful.
- it gives you the opportunity to create a "hat" task that creates all exports in a single call- still gives you the opportunity to call a single export task
This is where I disagree a little. I think that the types of binaries should be part of the component definition. If someone executes `build` or `assemble` for my build, they should get the standard binaries for that build. HTML and PDF are fairly common, Docbook less so. But Docbook should probably still be registered as a binary type, just not built by default.
Also, you need to factor in the time that it takes to generate each binary. In general, I disable the PDF in the build until I want to publish it because it slows the process down too much.
As I see it, the current model seems to assume that the plugin defines the binary types for a component, whereas I think the model should be extended to allow binary types as part of the component definition. If that makes sense.
@Managedinterface AsciidoctorBinaryInternal extends AsciidoctorBinary {String getBackend()void setBackend(String backend)}
This is similar to what I tried, but I only put the setter in the internal view. I'm not sure I really want to make the getter private.
@Defaultsvoid createExportBinaries(AsciidocDocument docs) {docs.binaries.create("pdf", AsciidoctorBinaryInternal) {binary.backend = 'pdf'}}@Modelvoid docs(AsciidocDocument docs) {}}Does that help?
I understand that this is an option, but having different types representing different types of binary makes the most sense to me. This may just be a limitation of the way managed objects work, but I'd like the team to think about whether it's possible to support fixed implementations of methods on derived managed types. I hope I'm using the vocabulary correctly :)
So in summary, your email seems to confirm my current understanding of how the new software model works. I just don't think it's sufficient for cleanly modelling Asciidoctor builds. Happy to continue discussing the validity and veracity of this view!
Cheers,
Peter
Cedric Champeau
Feb 15, 2016, 10:54:30 AM2/15/16
to gradle-dev
This is where I disagree a little. I think that the types of binaries should be part of the component definition. If someone executes `build` or `assemble` for my build, they should get the standard binaries for that build. HTML and PDF are fairly common, Docbook less so. But Docbook should probably still be registered as a binary type, just not built by default.
Sure, but it's totally up-to-you to attach a binary to the build task or not. That is to say that by default, you might want to attach only HTML and PDF, but it's really up to you, as the plugin author, to create the dependency between the build task and the binary. If you don't attach *all* binaries to the build task, then, you need to provide the user with a way to do it (could be by declaring the export formats explicitly).
Also, you need to factor in the time that it takes to generate each binary. In general, I disable the PDF in the build until I want to publish it because it slows the process down too much.As I see it, the current model seems to assume that the plugin defines the binary types for a component, whereas I think the model should be extended to allow binary types as part of the component definition. If that makes sense.@Managedinterface AsciidoctorBinaryInternal extends AsciidoctorBinary {String getBackend()void setBackend(String backend)}This is similar to what I tried, but I only put the setter in the internal view. I'm not sure I really want to make the getter private.
The question you have to ask yourself is if the user needs to access the name of the backend. As far as I see it, a user doesn't need to, so can be private I think.
@Defaultsvoid createExportBinaries(AsciidocDocument docs) {docs.binaries.create("pdf", AsciidoctorBinaryInternal) {binary.backend = 'pdf'}}@Modelvoid docs(AsciidocDocument docs) {}}Does that help?I understand that this is an option, but having different types representing different types of binary makes the most sense to me. This may just be a limitation of the way managed objects work, but I'd like the team to think about whether it's possible to support fixed implementations of methods on derived managed types. I hope I'm using the vocabulary correctly :)
I think it makes perfect sense if each format has its own configuration options (which will likely be the case). In that case, just look at this updated example:
interface AsciidocDocument extends ComponentSpec {}
class DefaultAsciidocDocument extends BaseComponentSpec implements AsciidocDocument {}
@Managed
interface AsciidoctorBinary extends BinarySpec {}
@Managed
interface AsciidoctorBinaryInternal extends AsciidoctorBinary {
String getBackend()
}
@Managed
interface AsciidoctorPdf extends AsciidoctorBinary {
double getCompression()
void setCompression(double d)
}
@Managed
abstract class AsciidoctorPdfInternal implements AsciidoctorPdf, AsciidoctorBinaryInternal {
String getBackend() { 'pdf' }
}
class AsciidoctorRules extends RuleSource {
@ComponentType
void registerComponentType(TypeBuilder<AsciidocDocument> builder) {
builder.defaultImplementation(DefaultAsciidocDocument)
}
@BinaryType
void registerAsciidoctorBinary(TypeBuilder<AsciidoctorBinary> builder) {
builder.internalView(AsciidoctorBinaryInternal)
}
@Defaults
void createExportBinaries(AsciidocDocument docs) {
docs.binaries.create("pdf", AsciidoctorPdfInternal)
}
@Model
void docs(AsciidocDocument docs) {}
}
apply plugin: AsciidoctorRules
model {
docs {
binaries.named('pdf') {
compression 0.9
}
}
}
Then your binary type can have specific configuration options. Note that the fact that I use "binaries.named(...)" because "binaries { pdf { ... } }" currently fails with an NPE on master (investigating).
Peter Ledbrook
Feb 16, 2016, 4:26:13 AM2/16/16
to gradl...@googlegroups.com
Sure, but it's totally up-to-you to attach a binary to the build task or not. That is to say that by default, you might want to attach only HTML and PDF, but it's really up to you, as the plugin author, to create the dependency between the build task and the binary. If you don't attach *all* binaries to the build task, then, you need to provide the user with a way to do it (could be by declaring the export formats explicitly).
The answer by Andreas on this thread:
suggests that all binaries are attached to `assemble` by default. Is that not correct?
I was hoping to avoid having a `formats` property on the component, but there seems to be no way round that.
The question you have to ask yourself is if the user needs to access the name of the backend. As far as I see it, a user doesn't need to, so can be private I think.
I don't yet know. But regardless, should the software model support public getters with internal setters? I'm guessing so.
@Managedinterface AsciidoctorPdf extends AsciidoctorBinary {double getCompression()void setCompression(double d)}@Managedabstract class AsciidoctorPdfInternal implements AsciidoctorPdf, AsciidoctorBinaryInternal {String getBackend() { 'pdf' }}
I tried something like this, but it didn't work. If Gradle reports an error for the above, should I raise an issue for it?
Anyway, thanks for the feedback. I think I have enough to do a bit more experimentation.
Peter
Cedric Champeau
Feb 16, 2016, 4:49:08 AM2/16/16
to gradle-dev
2016-02-16 10:26 GMT+01:00 Peter Ledbrook <pe...@cacoethes.co.uk>:
Sure, but it's totally up-to-you to attach a binary to the build task or not. That is to say that by default, you might want to attach only HTML and PDF, but it's really up to you, as the plugin author, to create the dependency between the build task and the binary. If you don't attach *all* binaries to the build task, then, you need to provide the user with a way to do it (could be by declaring the export formats explicitly).The answer by Andreas on this thread:suggests that all binaries are attached to `assemble` by default. Is that not correct?
It is correct if you use the `@BinaryTasks` annotation. But nothing prevents you from wiring yourself without this annotation.
I was hoping to avoid having a `formats` property on the component, but there seems to be no way round that.The question you have to ask yourself is if the user needs to access the name of the backend. As far as I see it, a user doesn't need to, so can be private I think.I don't yet know. But regardless, should the software model support public getters with internal setters? I'm guessing so.
It's more specific than that I think. Setters can make sense in the public API, or the private API. So we support them both on internal and public view.
@Managedinterface AsciidoctorPdf extends AsciidoctorBinary {double getCompression()void setCompression(double d)}@Managedabstract class AsciidoctorPdfInternal implements AsciidoctorPdf, AsciidoctorBinaryInternal {String getBackend() { 'pdf' }}I tried something like this, but it didn't work. If Gradle reports an error for the above, should I raise an issue for it?
This works on master (I tested it), but yeah, it's in heavy development, 2.12 will likely be a huge step forward in unifying/cleaning up the way the software model is built.
Anyway, thanks for the feedback. I think I have enough to do a bit more experimentation.Peter
--
You received this message because you are subscribed to the Google Groups "gradle-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gradle-dev+...@googlegroups.com.
To post to this group, send email to gradl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/CANBeYQSYonPWjw0xorq7XO7-Uq1-y4tDf6p02yEN42SzDwSjVA%40mail.gmail.com.
Peter Ledbrook
Feb 16, 2016, 7:30:50 AM2/16/16
to gradl...@googlegroups.com
The answer by Andreas on this thread:suggests that all binaries are attached to `assemble` by default. Is that not correct?It is correct if you use the `@BinaryTasks` annotation. But nothing prevents you from wiring yourself without this annotation.
So the @BinaryTasks annotation takes the tasks your rule creates and attaches them to `assemble`?
@Managedabstract class AsciidoctorPdfInternal implements AsciidoctorPdf, AsciidoctorBinaryInternal {String getBackend() { 'pdf' }}I tried something like this, but it didn't work. If Gradle reports an error for the above, should I raise an issue for it?This works on master (I tested it), but yeah, it's in heavy development, 2.12 will likely be a huge step forward in unifying/cleaning up the way the software model is built.
I just tried with a 2.12 nightly and it's still saying that internal views must be interfaces:
> Exception thrown while executing model rule: AsciidoctorPlugin.Rules#registerHtmlBinaryType
> AsciidoctorPlugin.Rules#registerHtmlBinaryType is not a valid binary model rule method.
> Internal view org.asciidoctor.gradle.model.HtmlBinarySpecInternal must be an interface.
We can keep that discussion to the bug issue though.
Cheers,
Peter
Cedric Champeau
Feb 16, 2016, 8:23:29 AM2/16/16
to gradle-dev
You cannot register an internal view that is an abstract class (that's a limitation, not a bug, of the current implementation). In my example (attached here), only an interface is declared as an internal view.
--
You received this message because you are subscribed to the Google Groups "gradle-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gradle-dev+...@googlegroups.com.
To post to this group, send email to gradl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/CANBeYQRBWB0m3UnbtqfGUtOFV19S8j9dDZONXoAP-8HHPwnaMQ%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages