build bootstrap

[Adam Murdoch]

Hi,

Andrew has written up some interesting notes on build initialization or bootstrap here: https://github.com/gradle/gradle/pull/402

I have some comments/thoughts on this:

So, the goal would be to have a workflow something like this:

1. User has a version of Gradle installed, possibly via the wrapper or IDE.

2. User requests that Gradle bootstrap a project. The user would tell Gradle what kinds of capabilities they’d like the project to have: for example, I want to build a library from Groovy source, I want to build an application from C and C++.

3. Optionally, the user provides some information about the project. This might include project names, language versions, maybe package names, etc.

4. Gradle generates a working project, that you can build, test and run (or whatever makes sense for the project).

This generated project is essentially a working sample, possibly with some ‘real’ values substituted into various places in the build and source files.

The key aspect here is what is missing from the workflow - I’ve asked for some capability that the core does not provide, let’s say building an Android library, but I haven’t had to download or install anything other than the Gradle distribution to use this capability. Gradle has taken care of this.

There are other really interesting things we could do here with the workflow, but I think we can park those for now.

Generally, bootstrapping a project involves some behaviour, not just data. Our standard pattern for publishing and sharing behaviour and data is to package these things as a plugin. So, I would like to base the implementation heavily around plugins. The approach would be something like this:

1. Plugins declare that they provide some capability - in this case bootstrapping a certain kind of project

2. When invoked, Gradle discovers which plugin provides this capability.

3. Gradle downloads and applies the plugin.

4. Gradle invokes the plugin to do the work.

For discovery, I would like to use the plugin portal. A plugin’s meta-data would describe the build capabilities it can bootstrap, and Gradle would ask the plugin portal to provide the mapping from capability to plugin instance. Whatever solution we implement for using the plugin portal from behind the firewall would also apply to this project bootstrap.

I think we could tackle the work in this way:

1. Change the structure of the build init plugin so that it discovers the plugins that provide bootstrap capabilities. Just for the core plugins, initially. Currently the build init plugin is structured so that it knows about all the templates up front. This is just an internal refactoring to enable the next steps.

2. Provide a way to implement some custom bootstrapping in a plugin, but without providing any remote discovery. You would instead, let’s say, use an init script to apply the bootstrapping plugin. The init script could be remote and might live with the plugin in a repo. Or not, at this stage the implementation doesn’t care.

3. Bake the custom bootstrapping capability into the plugin-dev plugin, so that it takes care of whatever packaging and meta-data that needs to be generated to package up a bootstrapping plugin.

4. Ship this meta-data to the plugin portal when publishing the plugin.

5. Possibly change the plugin portal UI to give you some snippet to use to apply the bootstrapping plugin, maybe pointing to some automatically generated init script. At this point, I would be able to write a plugin that does something useful, but also include some templates for generating a sample project that uses my plugin. A user would go to the plugin portal and see not just information about my plugin, but instructions about how to try it out.

6. Remove the need for the init script, so that Gradle queries the plugin portal to discover which bootstrapping plugins are required. This step would simplify the steps I need to follow to bootstrap a project, and also allow me to do everything from the command-line (and hence from the IDE as well, via the tooling API).

--

Adam Murdoch
Gradle Co-founder
http://www.gradle.org
CTO Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com

[Andrew Oberstar]

Overall, I like the idea of using the plugin infrastructure to provide the bootstrapping functionality. Two questions/comments:

1. Assuming capabilities would be something like "android project" or "c++ library", there will likely be multiple plugins providing bootstrapping of the same capability. Any reason it couldn't just rely on plugin ids instead of capabilities? Otherwise it would need to prompt with a list of candidate templates that provide the capability, right?

2. This still doesn't address how a plugin should bootstrap a project. There's definitely benefit in the flexibility of just allowing any old plugin to declare it supports bootstrapping. However, a lot of templating/bootstrapping use cases are pretty straightforward and are mostly (if not entirely) data based, rather than behavior based. To ease those use cases, it would be nice for the core to provide a structured approach to bootstrapping (without precluding wide-open plugins). It's possible structured approaches could come from the community, but I guess it depends on how this all fits together.

Andrew Oberstar

--
You received this message because you are subscribed to the Google Groups "gradle-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gradle-dev+...@googlegroups.com.
To post to this group, send email to gradl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/0EF57C68-9567-46AC-B9D9-6B6C3B75581E%40gradleware.com.
For more options, visit https://groups.google.com/d/optout.

[Hans Dockter]

I'm much in favor to make bootstrapping plugin based.

- Can be independently versioned to Gradle.

- Makes it easy for the community to contribute. For me bootstrapping must be driven by the community as there it includes so much domain knowledge we do not have.

- @Andrew: We can make it very simple to combine a structured approach with plugins. E.g. provide a zip bootstrapper base plugin where people just need to point to a zip with a directory structure. Or a plugin that provides a builder DSL for templates. The plugin-dev plugin can also make it very easy to publish such simple templates as a plugin without the complexity of generic plugin publishing. 

- Even for simple templates it is always good to have the power & flexibility to evolve them.

- Treating bootstrapping as first class citizen in the plugin portal will rock.

Hans 

To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/CAJnGkD4A0BuEAizizFk_bj1CWRHkdZ8bqrBO4bLmzESrfjc6Rw%40mail.gmail.com.

[Lóránt Pintér]

Hi,

we also needed something like this, and after looking at existing bootstrappers like Yeoman and Giter8, ended up creating https://github.com/prezi/grub. That said, I’d be a lot happier if Gradle had built-in functionality for bootstrapping, and we could throw Grub out.

A few random thoughts, based on our use of this tool:

Bootstrapping should run from the command-line, and it should be simple for the end-user. Something along the lines of “gradle bootstrap give-me-a-typical-java-project”. If you need to first create a build.gradle first to create a new project, or supply a dozen parameters on the command-line, people will get confused.

Using plugins (Plugin<Bootstrap>?) for this sounds like a natural fit. I especially like the ability that I can apply one plugin from another.

For our typical use-case at Prezi, this grouping of plugins would be very important. Our build users would always want to bootstrap via a single ‘umbrella' plugin (e.g. “com.prezi.myteam:my-team’s-usual-project-template:1.+”), which would apply other, more general bootstrap plugins (“org.gradle:javascript-bootstrap:1.7” etc.). This would also allow the umbrella plugin to pre-configure some conventions (e.g. it could pre-select the option in the JS bootstrapper to use the Mocha framework for testing, because in my team we always use Mocha).

Putting the bootstrap plugins on the plugin portal would be great for discoverability. In an enterprise environment we would also need to be able to put our internal bootstrap plugins on a similar internal portal.

It would be great if bootstrapping didn’t stop at creating a new project from scratch, but you could alter existing ones as well. Adding more subprojects to an existing project, or adding more capabilities. Using Gradle here gives an advantage over other tools like Yeoman, because we can understand what is already available in an existing build via the tooling API. This makes it possible to add more nuanced changes, which is great. (Although serializing those changes back to, say, build.gradle could still be a challenge.) I can imagine things like “add a new widget to my Android app project” as well.

Resolving variables in file and folder names in the bootstrap template (as in “src/main/java/$packageName/App.java”, like Giter8 does) is a big time-saver, and it makes it a lot easier to find out what happens when a project is bootstrapped.

Versioning templates is another benefit of using plugins for this, but the most usual use-case will be I guess to ask for the latest version of the template. Dynamic versions and defaulting to “+” would be a great way to solve this I think.

—
Lóránt

To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/CAPsC-PJ73D9EY15Bu8ti%3Dqhgtk6g_ZaybaUuA%2BL_VhNpe9MmXQ%40mail.gmail.com.

[Adam Murdoch]

On 12 Feb 2015, at 1:44 pm, Andrew Oberstar <ajobe...@gmail.com> wrote:

Overall, I like the idea of using the plugin infrastructure to provide the bootstrapping functionality. Two questions/comments:

1. Assuming capabilities would be something like "android project" or "c++ library", there will likely be multiple plugins providing bootstrapping of the same capability. Any reason it couldn't just rely on plugin ids instead of capabilities?

It is certainly an option. However, capability and plugin aren’t necessarily the same thing. For example, the cpp-lang plugin provides support for both c++ libraries and c++ executables - you apply the plugin and then declare in the DSL whether you’re building a library or an executable (or both or neither).

It feels to me that we should give the template an identity that is separate to the id of the plugin that provides the template.

Otherwise it would need to prompt with a list of candidate templates that provide the capability, right?

I think we’d need to deal with this either way. There can be a bunch of plugins that do the same thing, so you if you want that thing, you’re going to have to select one of the candidates - either by selecting a template from a list, or a plugin from a list.

2. This still doesn't address how a plugin should bootstrap a project. There's definitely benefit in the flexibility of just allowing any old plugin to declare it supports bootstrapping. However, a lot of templating/bootstrapping use cases are pretty straightforward and are mostly (if not entirely) data based, rather than behavior based. To ease those use cases, it would be nice for the core to provide a structured approach to bootstrapping (without precluding wide-open plugins). It's possible structured approaches could come from the community, but I guess it depends on how this all fits together.

Absolutely. I think we should provide 2 options:

1. A programmatic API that Gradle invokes. The plugin can pretty much do whatever it likes through this.

2. Some convention for where to look for templates in the plugin bundle (jar). We would define some templating engine, a bunch of variables that are available, and Gradle runs the files through the templating engine to produce some files.

Or just #1 and make #2 a convenience on the API.

To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/CAJnGkD4A0BuEAizizFk_bj1CWRHkdZ8bqrBO4bLmzESrfjc6Rw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Adam Murdoch]

On 12 Feb 2015, at 6:42 pm, Lóránt Pintér <lorant...@gmail.com> wrote:

Hi,

we also needed something like this, and after looking at existing bootstrappers like Yeoman and Giter8, ended up creating https://github.com/prezi/grub. That said, I’d be a lot happier if Gradle had built-in functionality for bootstrapping, and we could throw Grub out.

A few random thoughts, based on our use of this tool:

Bootstrapping should run from the command-line, and it should be simple for the end-user. Something along the lines of “gradle bootstrap give-me-a-typical-java-project”. If you need to first create a build.gradle first to create a new project, or supply a dozen parameters on the command-line, people will get confused.

This is exactly the goal. A user should not have to edit anything in order to bootstrap a project. The workflow would be something like:

1. Download and install Gradle.

2. Visit the plugin portal to learn about which templates are available, and copy some snippet to run on the command-line.

3. Run `gradle init <some-stuff>` from the command-line, or paste the snippet.

4. Maybe answer some questions about your project.

Now you end up with a working sample.

There are a bunch of variations on this workflow. For example:

- Step 2 should also be possible from the command-line at some point.

- Instead of steps 1-4, I can instead use some wizard on the plugin portal and receive a zipped-up git repo containing the working sample instead (with the wrapper wired up).

- Instead of steps 1-4, I can click ‘new project’ in the IDE and use a wizard to drive the bootstrap process.

Using plugins (Plugin<Bootstrap>?) for this sounds like a natural fit. I especially like the ability that I can apply one plugin from another.

For our typical use-case at Prezi, this grouping of plugins would be very important. Our build users would always want to bootstrap via a single ‘umbrella' plugin (e.g. “com.prezi.myteam:my-team’s-usual-project-template:1.+”), which would apply other, more general bootstrap plugins (“org.gradle:javascript-bootstrap:1.7” etc.). This would also allow the umbrella plugin to pre-configure some conventions (e.g. it could pre-select the option in the JS bootstrapper to use the Mocha framework for testing, because in my team we always use Mocha).

Absolutely. It should be possible for an organisation to package up a template and plugin that defines that organisation’s conventions.

Putting the bootstrap plugins on the plugin portal would be great for discoverability. In an enterprise environment we would also need to be able to put our internal bootstrap plugins on a similar internal portal.

It would be great if bootstrapping didn’t stop at creating a new project from scratch, but you could alter existing ones as well. Adding more subprojects to an existing project, or adding more capabilities. Using Gradle here gives an advantage over other tools like Yeoman, because we can understand what is already available in an existing build via the tooling API. This makes it possible to add more nuanced changes, which is great. (Although serializing those changes back to, say, build.gradle could still be a challenge.) I can imagine things like “add a new widget to my Android app project” as well.

I’d love to do this kind of thing too. We’ve already started to some degree with the `wrapper` tasks, which will update the build.

I would drive this using the IDE use cases. There are a bunch of build mutations that we’d like to provide wizards for in the IDE, and whatever capability we add there can also be used by the bootstrap plugins.

Resolving variables in file and folder names in the bootstrap template (as in “src/main/java/$packageName/App.java”, like Giter8 does) is a big time-saver, and it makes it a lot easier to find out what happens when a project is bootstrapped.

Versioning templates is another benefit of using plugins for this, but the most usual use-case will be I guess to ask for the latest version of the template. Dynamic versions and defaulting to “+” would be a great way to solve this I think.

Absolutely.

Something that would be really interesting here is some way to evolve a build from one version of a template to a newer one, particularly for templates that define an organisation’s conventions.

To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/1423726955879.61df8adc%40Nodemailer.

For more options, visit https://groups.google.com/d/optout.

--

[KARR, DAVID]

Does “LazyBones” come close to what you’re thinking of? https://github.com/pledbrook/lazybones

--

[Andrew Oberstar]

Honestly, both LazyBones and Grub's functionality meet my needs. Grub has a very similar (though more thought through) approach to the one I was fumbling my way into at the end of the spec. However, I do prefer the LazyBones approach to distributing templates.

I'll try to run through my thought process to see if I'm actually thinking about this differently than Adam or if it's just a wording difference.

A package published to the plugin portal needs to declare via static attributes what plugins it provides (seemingly geared towards Plugin<Project> right now).

If I'm interpreting Adam right, it sounds like a Plugin<Project> could also provide one or more bootstrap capabilities. Given the programmatic nature of plugins and the statically declaration requirements of the portal, that coupling doesn't seem appropriate to me.

If a new Plugin<Bootstrap> approach is available, this would align exactly with providing a static list of plugins in the Bintray attributes used by the portal. The only necessary addition might be declaring whether a particular plugin ID is applied to Project, Init, Settings, or Bootstrap. That would allow Gradle to restrict which types of plugins can be applied through different interfaces (e.g. only Bootstrap plugins can be applied from "gradle init <plugin id>").

A single JAR package could still provide multiple related Plugin<Project> (cpp-lang) and Plugin<Bootstrap> (cpp-lib and cpp-exe) implementations. Plugin<Project>s commonly apply other plugins, and the same approach could be used to mix Plugin<Bootstrap>s together.

Aligning them all as plugins, it seems that the same logic that led to the requirement of qualified plugin IDs would still apply to a Plugin<Bootstrap>. The use case of providing a list of candidate plugins can be met via a search capability on the command line without needing to make the IDs generic and potentially overlapping.

If that all holds true, I think Adam's original list of tasks still lines up pretty well with the addition of a Plugin<Bootstrap> implementation that provides the standardized template structure.

I'm not sure how well this aligns with altering a project after the initial bootstrap.

Andrew Oberstar

To unsubscribe from this group and stop receiving emails from it, send an email to gradle-dev+unsubscribe@googlegroups.com.

To post to this group, send email to gradl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/0EF57C68-9567-46AC-B9D9-6B6C3B75581E%40gradleware.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "gradle-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gradle-dev+unsubscribe@googlegroups.com.

To post to this group, send email to gradl...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/B8D164BED956C5439875951895CB4B2226BC4212%40CAFRFD1MSGUSRIA.ITServices.sbc.com.