Gradel Plugin For Cloud Foundry - Wish List

[jim northrop]

Ref. https://github.com/cloudfoundry/cf-java-client/tree/master/cloudfoundry-gradle-plugin

having used this plugin for several months, these are my pain points :

1. CF-DELETE will fail the complete gradle build if the provisioned app does not exist - a provision for a 'warning' but not a complete failure would allow a gradle build to continue if deemed appropriate

2. CF-DELETE of a provisioned app does NOT remove the associated services. These services become orphans and must be remove manually. It is possible that a service, say redis for example, might be in use by more than a single app, so care would need to be taken to auto-remove services with more than one attachment. Services named by {random-word} feature seem particularly prone to this issue. Have not tested fixed-named services yet.

3. If no app exists on the target, then a CF-PUSH with appropriate gradle closure declarations succeeds as expected assigning the URL as expected. 

When the same app is CF-PUSH-d even a minute later, the push is successful but the app is NOT re-assigned any URL and must be done manually thru the Cloud Foundry console.

4. If cloud foundry credentials are stored in .gradle/gradle.properties, all activity thru gradle plugin works as expected following a CF-LOGIN command. In cases where credentials also are included in the gradle plugin closure, these credentials do NOT take precedence over those declared in gradle.properties. Suggest that hard-coded credentials in the gradle script take precedence over  gradle.properties. This is particularly useful when pushing an app to more than one target PaaS than Pivotal.

5. Have used this plugin to push apps to both Pivotal and www.anynines.com ; see: https://support.anynines.com/home thus confirming the viability of the standard implementation of Cloud Foundry across a number of PaaS provider. Hope to further test other PaaS targets time permitting. 

[Scott Frederick]

Responses inline. 

On Thursday, February 6, 2014 8:56:13 AM UTC-6, jim northrop wrote:

Ref. https://github.com/cloudfoundry/cf-java-client/tree/master/cloudfoundry-gradle-plugin

having used this plugin for several months, these are my pain points :

1. CF-DELETE will fail the complete gradle build if the provisioned app does not exist - a provision for a 'warning' but not a complete failure would allow a gradle build to continue if deemed appropriate

That's a good suggestion.  I added an issue for that: https://github.com/cloudfoundry/cf-java-client/issues/133. 

2. CF-DELETE of a provisioned app does NOT remove the associated services. These services become orphans and must be remove manually. It is possible that a service, say redis for example, might be in use by more than a single app, so care would need to be taken to auto-remove services with more than one attachment. Services named by {random-word} feature seem particularly prone to this issue. Have not tested fixed-named services yet.

Automatically deleting a service when a bound app is deleted is dangerous and potentially costly. Marketplace services on Pivotal Web Services (run.pivotal.io) start billing when a service instance is created. Repeatedly deleting and re-creating service instances could lead to an unnecessarily large bill at the end of the month, as opposed to re-using a service that was previously created. You would not want to delete any service that contained persistent data, like a database, if you intend to re-push the app later. 

I think it would be safer to require the user to run the "cf-delete-service" task before running the "cf-delete" task if they want to delete services. "cf-delete-service" deletes all services specified in the Gradle build file. You could create an "app-delete" task in your build file that depends on "cf-delete-service" and "cf-delete" to automate the calling of these two tasks together every time. 

Note that this won't work when services are named with ${random-word}, as a new random word will get created with each run of Gradle. This would make it impossible for the plugin to find a service named in a previous run. The ${random-word} feature is really intended to be used on routes, which must be unique across the entire CF deployment. Service names only need to be unique to a space within an org, so I don't see the need to randomize the names there. What's your use case for that? 

 

3. If no app exists on the target, then a CF-PUSH with appropriate gradle closure declarations succeeds as expected assigning the URL as expected. 

When the same app is CF-PUSH-d even a minute later, the push is successful but the app is NOT re-assigned any URL and must be done manually thru the Cloud Foundry console.

I can't re-create this problem. I just tested it by pushing an app, changing the Gradle config to add a second URL, then pushing again. After the second push, both URLs were bound as expected. This code should always apply the current URL settings on each push: https://github.com/cloudfoundry/cf-java-client/blob/master/cloudfoundry-gradle-plugin/src/main/groovy/org/cloudfoundry/gradle/tasks/PushCloudFoundryHelper.groovy#L39.

Can you provide some steps to re-create what you are seeing? 

4. If cloud foundry credentials are stored in .gradle/gradle.properties, all activity thru gradle plugin works as expected following a CF-LOGIN command. In cases where credentials also are included in the gradle plugin closure, these credentials do NOT take precedence over those declared in gradle.properties. Suggest that hard-coded credentials in the gradle script take precedence over  gradle.properties. This is particularly useful when pushing an app to more than one target PaaS than Pivotal.

The order of precedence between system properties, gradle.properties, and build script settings is a pretty well-defined convention, and is consistent between the Gradle and Maven plugins. The intention with the current order is to allow an individual user to override what's in the build script by setting properties in any of the Gradle-supported forms (http://www.gradle.org/docs/current/userguide/build_environment.html#sec:gradle_configuration_properties). If we reversed this to give the script precedence, then a user would have no way to override the script but would be forced to modify it. So I think it is best to leave the order of precedence as-is.

Unfortunately gradle.properties doesn't allow conditionals, so it is hard to solve the kind of problem you are trying to solve just using that mechanism. Here's an idea for how to do what you are trying to do: 

The README for the Gradle plugin has some tips for making parts of the configuration dependent on a property: https://github.com/cloudfoundry/cf-java-client/tree/master/cloudfoundry-gradle-plugin#conditional-configuration-based-on-properties. You could use this technique to separate the parts of the configuration that are different for each cloud provider. If you don't want to store the username and password configuration in the main build.gradle file, you could pull that section out into a separate .gradle file and include it in build.gradle. If you created a file named "cf-env.gradle" with the conditional configuration and put that file in your "~/.gradle" directory, then you could do this to include it: 

def cfEnvScript = file("${System.getProperty("user.home")}/.gradle/cf-env.gradle")

if (cfEnvScript.exists()) {

    println "Applying environment settings from ${cfEnvScript.path}"

    apply from: cfEnvScript

}

 

5. Have used this plugin to push apps to both Pivotal and www.anynines.com ; see: https://support.anynines.com/home thus confirming the viability of the standard implementation of Cloud Foundry across a number of PaaS provider. Hope to further test other PaaS targets time permitting. 

That's great to hear, thanks!

Scott 

[jim northrop]

Morning Scott

Thank you for those notes. i can certainhly see the reasoning behind those choice. Here's some further thoughts :

1. CF-DELETE - would love that, even if it's optionally available by some further notation in the plugin closure.

2. Delete existing services - well did not consider the costs provisions behind service additions/removals - ouch  !  In our case, we are using redis persistence as a caching tool to boost performance, so it really is quite useful to be able to auto-remove the data content on a full re-deployment. Would there be any way to add another service attribute on redeployment ? Something along the lines of:

cloudfoundry {

  // www.anynines.com: cf target https://api.de.a9s.eu

  target='https://api.de.a9s.eu'

  organization = 'northrop_orange_fr'  // for anynines

  space = 'development'  // for anynines

  username='****'

  password='***'

  uri = 'caelyfdoctor.de.a9sapp.eu'

  file = new File('build/libs/asciidoctor-project-1.1.2.war')

  application = 'caelyfdoctor'

  memory = 384

  instances = 1

  serviceInfos 

  {

        "rediscloud-asciidoctor" 

  {

            label = 'rediscloud'

            provider = 'Garantiadata'

            version = 'n/a'

            plan = '25mb'

            bind = true

       redeployment:'keep'  // 'remove' as another hint to the cf-delete process as to what to do with this service

        }

  } // end of serviceInfos

} // end of cloudfoundry

3. since i cannot find a feature like 'cf update' we used to have, it makes it necessary to use cf-push when 're-pushing'. if i do this in a terminal session with absolutely NO changes to anything, just 2 pushes one after the other :

gradlew cf-push

gradlew cf-push

the first push works as long as a previous app of same name is not there. The url taken from the closure(i'm guessing) is used as expected. The 2nd push unbinds the url from the app(i'm guessing) to allow the second push to success - which it does and shows 'running 100%' on pivotal console but without the URL having been re-attached to the new instance. This may not be a valid use case. but in the absence of a cf-update, my work-around is cf-delete cf-push then manual rebind of app to new url using pivotal console. I can't even rebind to the same URL as it's in use !! :-P

4. ok, i can dig the rational for using gradle.properties first, so my work-around here is to hard-code credentials in the script closure and stop using gradle.properties. That way i can cf-push to several targets(yes i have to tweek the script) and still make all of them happy credentials-wise :-} 

sure liked the idea of a gradle tweek so will see if that can be used as it's kinda along the lines i was thinking too.

5. am pleased too. it did work ok using cf gradle plugin 1.0.1 but have an issue with 1.0.2 - will post that under separate cover.

more soon

thx

jim

[Scott Frederick]

Responses inline. 

On Sunday, February 9, 2014 3:20:40 PM UTC-6, jim northrop wrote:

Morning Scott

Thank you for those notes. i can certainhly see the reasoning behind those choice. Here's some further thoughts :

1. CF-DELETE - would love that, even if it's optionally available by some further notation in the plugin closure.

This is now the default in version 1.0.2. 

 

2. Delete existing services - well did not consider the costs provisions behind service additions/removals - ouch  !  In our case, we are using redis persistence as a caching tool to boost performance, so it really is quite useful to be able to auto-remove the data content on a full re-deployment. Would there be any way to add another service attribute on redeployment ? Something along the lines of:

cloudfoundry {

  // www.anynines.com: cf target https://api.de.a9s.eu

  target='https://api.de.a9s.eu'

  organization = 'northrop_orange_fr'  // for anynines

  space = 'development'  // for anynines

  username='****'

  password='***'

  uri = 'caelyfdoctor.de.a9sapp.eu'

  file = new File('build/libs/asciidoctor-project-1.1.2.war')

  application = 'caelyfdoctor'

  memory = 384

  instances = 1

  serviceInfos 

  {

        "rediscloud-asciidoctor" 

  {

            label = 'rediscloud'

            provider = 'Garantiadata'

            version = 'n/a'

            plan = '25mb'

            bind = true

       redeployment:'keep'  // 'remove' as another hint to the cf-delete process as to what to do with this service

        }

  } // end of serviceInfos

} // end of cloudfoundry

Another option would be to simply add the following line to your build file, which will cause the 'cf-delete-service' task to always run whenever the 'cf-delete' task is run. 

tasks.'cf-delete'.dependsOn << 'cf-delete-service'

 

3. since i cannot find a feature like 'cf update' we used to have, it makes it necessary to use cf-push when 're-pushing'. if i do this in a terminal session with absolutely NO changes to anything, just 2 pushes one after the other :

gradlew cf-push

gradlew cf-push

the first push works as long as a previous app of same name is not there. The url taken from the closure(i'm guessing) is used as expected. The 2nd push unbinds the url from the app(i'm guessing) to allow the second push to success -

The URLs are not unbound from the app before being pushed. If the app already exists, the URLs specified in the build file are updated in the app's meta-data, but they are not removed. https://github.com/cloudfoundry/cf-java-client/blob/master/cloudfoundry-gradle-plugin/src/main/groovy/org/cloudfoundry/gradle/tasks/PushCloudFoundryHelper.groovy#L39

 

which it does and shows 'running 100%' on pivotal console but without the URL having been re-attached to the new instance. This may not be a valid use case.

It is valid use case, and I do it all the time without losing the URL like you are seeing. If you can share a project with me that has a configured build.gradle file I can try to re-create with your setup since I can't re-create with mine. 

 

but in the absence of a cf-update, my work-around is cf-delete cf-push then manual rebind of app to new url using pivotal console. I can't even rebind to the same URL as it's in use !! :-P

4. ok, i can dig the rational for using gradle.properties first, so my work-around here is to hard-code credentials in the script closure and stop using gradle.properties. That way i can cf-push to several targets(yes i have to tweek the script) and still make all of them happy credentials-wise :-} 

sure liked the idea of a gradle tweek so will see if that can be used as it's kinda along the lines i was thinking too.

Did you try the conditional configuration suggestion (https://github.com/cloudfoundry/cf-java-client/tree/master/cloudfoundry-gradle-plugin#conditional-configuration-based-on-properties) to solve the problem of different credentials for different target? Not sure if that's what you mean by "gradle tweek" of if you are referring to something else. 

[mana]

I would suggest to rename all the gradle cf tasks since the naming is horrible. In gradle, you can modify tasks, extend them and so on by using the task name as a variable and do things as:

task.dependsOn()

task.finalizedBy()

but this is not possible since the task names have a dash ('-') in their name. There is no way to work around this in the gradle.build file :?

Please change the naming pattern so it becomes a valid groovy/java variable.

The other parts of the plugin work great so far. Thanks for you efforts, Scott!

Matthias

[Scott Frederick]

Mattias, 

I agree about the task naming. That was inherited from a previous version of the plugin, and I didn't make then change when I had the best chance during the migration to v2. I created an issue for comments on changing this. Feel free to +1 or comment there with additional thoughts about task renaming. 

That said, it is possible to use the current task names in expressions, you just have to quote the name like this: 

tasks.'cf-delete'.dependsOn << 'cf-delete-service'

It's ugly, but it does work. 

Scott

[Scott Frederick]

The task renaming issue/proposal is here: https://github.com/cloudfoundry/cf-java-client/issues/136. 

Scott

[James Bayer]

don't bury the lead! camel case ftw!

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--

Thank you,

James Bayer

[jim northrop]

scott -

is there any way to get the task completion code/message back from a task like cf-delete cf-login ? am thinking of something i could see 'programatically' in the gradle script ?

thx

[jim northrop]

could u pls explain camelcase ? i'm bad on naming conventions.

thx

[jim northrop]

debug joblog for issue 137

[Scott Frederick]

Continuing this in the issue comments: https://github.com/cloudfoundry/cf-java-client/issues/137. 

Scott