working with runtime properties

[RichardK]

Hi all, sorry if this question has an obvious answer, but I can't seem to get it right. 

I have a plugin that does some rest api calls to an external application. The rest calls needs to be in sequence where some are dependent on the runtime properties produced by other tasks called from a different node in the blueprint. 

I my blueprint, I try to resolve this by creating two nodes, where one node is dependent on the other to complete first. in the first node, set some runtime properties based on the return values of my api calls. Now, in the second node, I try to reed these runtime properties and this is where I am getting stuck :(. Reading through the documentation, it appears to me that the intrinsic function 'get_attribute' is what I need but I can't seem to get it to work. 

Does anyone have an sample blueprint that shows how to configure this that could help me a long? 

Also, in the plugin, for the dependent tasks, do I retrieve these values using ctx.instance.runtime_properties['some property']?

 

Any help is greatly appreciated, 

note: I was not sure which group to post this in so I posted the same message in cloudily-developers as well in case you are signed up with both groups just like me. 

Thanks, 

Richard 

[Trammell -]

Hi Richard,

It's not exactly clear to me what you are asking for - apart from an example. So here is an example. This is a blueprint that illustrates using Puppet apply. There are three operations.

Operation1: Set agenthostname property. Get the hostname store.

Operation2: Retrieve property. Create certificate with hostname.

Operation3: Retrieve property. Delete certificate for hostname.

It's important to understand the order that operations occur in Cloudify. For example create - > preconfigure* -> configure -> postconfigure* -> start -> establish*. The operations that I added an asterisk to, are "relationship operations" which mean they occur not on a node, but between nodes. Remember that if you have multiple instances of a node, the operation will only update the runtime properties of a particular instance - not all instances.

See the docs for more information.

I would suggest a pattern like this, for extra care:

```

# Get current value of property

property = ctx.instance.runtime_properties.get('property')

if property is None:

    # if property is not set, give property a default value.

    property = {}

# update property value with something new

property.update({'new_key': 'new_value'})

# update the runtime properties

ctx.instance.runtime_properties['property'] = property

```

This is a basic process and wont cover all scenarios.

If you want to share your current work I can try to give you more specialized advice.

[RichardK]

Thank you Trammell for replying. Just some additional information. 

What i have created is a cloudify plugin for adding a new applications into okta (using the okta rest api) . The idea is that people when people are creating blueprints for application that uses okta they will be able to use this plug into to add a new application to okta and bypass the manual steps otherwise needed. The task in the plugin that adds the application to okta returns the okta generated appid and metadata for the newly added application. Right now I am just adding it to the outputs, but I want to be able to read this values from other nodes in the blueprint, hence, I tried the runtime properties. 

Now, from what I understand, (since I am new to cloufiy, I may be wrong on this), runtime properties are only available within the node, unless you establish a relationship with another node. Is this correct? 

Now, I don't have another app yet to test with, however I got another condition that somewhat would work in its stead for now. When a user wants to add an app to okta, I have added tasks for creating an optional default group as well. this tasks needs the appid as well to tie the group with the application. (now, I can for sure solve this in a programmatically by adding the option to the add application task, but I would still have the problem with how to capture the appid and metadata when adding a new node that wants to call the plugin. 

I hope this makes more sense, if not let me know. In short, this is what want to be able to do. 

Node A

  sets runtime variable x which is created during execution time. 

Node B

  depends on A to finish first. 

  reads runtime variable x

You asked me to share the code. Here is part of my attempt, I have been fiddling with this for a while, so the below is definitely incorrect. However, may it may shed some more light onto what I am trying to accomplish. 

blueprint.yaml

tosca_definitions_version: cloudify_dsl_1_3
imports:
  - http://www.getcloudify.org/spec/cloudify/4.1/types.yaml
  - plugin.yaml


# Add this block to import your plugin
plugins:
    #any name for your plugin
    fico-octa-plugin:
        # Give the same details as mentioned while creating the plugin
        executor: central_deployment_agent
        package_name: oktautil
        package_version: '0.1'
        distribution_release: ''
        distribution_version: ''
        supported_platform: ''
        distribution: ''



node_templates:
  okta_app:
    type: fico.nodes.okta.app
    properties:
      okta_domain: somehost.com
      okta_key: <some generated key>
      application_json: { get_input: application_json }


  okta_group:
    type: fico.nodes.okta.group
    relationships:
      - type: connnect_to_app
        target: okta_app
        properties:
          appid: { get_attrbute: [okta_app, appid] }

    properties:
      okta_domain: somehost.com
      okta_key: <some generated key>
      group_name: { get_input: group_name }
      group_description: { get_input: group_description }

plugin.yaml

inputs:
  okta_domain:
    description: The Okta domain
  okta_key:
    description: Okta authentication token
  application_json:
    description: application configuration json as string
  group_name:
    description: name of okta group
  group_description:
    description: Group description, give empty string if no description

node_types:

  fico.nodes.okta.group:
    derived_from: cloudify.nodes.Root
    properties:
      okta_domain:
        type: string
      okta_key:
        type: string
      group_name:
        type: string
      group_description:
        type: string
      appid:
        type: string
    interfaces:
      cloudify.interfaces.lifecycle:
        start: ## three operations exists for install workflow ('create', 'configure', and 'start'
          implementation: fico-okta-plugin.plugin.tasks.create_group
        delete: ## 'delete' is an operation for the uninstall workflow.
          implementation: fico-okta-plugin.plugin.tasks.delete_group

  fico.nodes.okta.app:
    derived_from: cloudify.nodes.Root
    properties:
      application_json:
        type: string
      okta_domain:
        type: string
      okta_key:
        type: string
      appid:
        type: string
    interfaces:
      cloudify.interfaces.lifecycle:
        start: ## three operations exists for install workflow ('create', 'configure', and 'start'
          implementation: fico-okta-plugin.plugin.tasks.create_app
        delete: ## 'delete' is an operation for the uninstall workflow.
          implementation: fico-okta-plugin.plugin.tasks.delete_app

[Trammell -]

Hi Richard, You are on the right track. (And by the way your plugin sounds extremely useful.) I believe the confiusion arises from the nomenclature of "properties" within a blueprint relationship. In a blueprint relationship "properties" has basically only one legal value (as of now, and most likely for perpetuity) - "connection_type", which defines whether the relationship is all_to_all or all_to_one. This defines whether the relationship will be to all target node instances or one target node instance. Here are the docs on that. It looks like you understood that here "properties" was like a auxiliary of the node's properties. This is a completely natural mistake.

If you want to take a runtime property of one node-instance and assign it to another node instance, you would need to do that in a relationship interface (in code).

For example, let's say I have this blueprint:

```

node1: type: cloudify.nodes.Root interfaces: cloudify.interfaces.lifecycle:

configure: scripts/configure.py

node2: type: cloudify.nodes.Root relationships: - type: cloudify.relationships.connected_to target: node1 source_interfaces: cloudify.interfaces.relationship_lifecycle: postconfigure: implementation: scripts/postconfigure.py

```

Let's say that scripts/configure.py has the following code:

```

#!/usr/bin/env python

from cloudify import ctx

if __name__ == '__main__':

ctx.instance.runtime_properties['my_property'] = "my value"

```

The scripts/postconfigure.py could read that property and store it like this:

```

#!/usr/bin/env python

from cloudify import ctx

if __name__ == '__main__':

target_node_value = ctx.target.instance.runtime_properties['my_property']

ctx.source.instance.runtime_properties['target_node_value'] = target_node_value

```

Note that I am using here the script plugin instead of a custom plugin, but the same rules apply (except that the operations occur in __main__ instead of in a python function.

Does this help?

Trammell

[RichardK]

Thank you very much!! I think this is exactly what I need and I will give this a try. I may have some follow up questions if/once I get this to work to make sure I understand things correctly. 

thanks again, very much appreciated. 

Richard 

[RichardK]

Ok, so I am sorry, but I must still be missing something or am not understanding what I am doing, just can't see what :(. 

Now I am getting this error when trying to execute:

2017-10-23 18:18:31.605  CFY <test> 'install' workflow execution failed: Workflow failed: Task failed 'plugin.tasks.create_group' -> ctx.source/ctx.target can only be used in a relationship-instance context but used in a node-instance context.

company policy prevents me from sharing the whole code, we are into process of getting approval to share to the community. However, here is a code snippet from my plugin that I am calling from the blueprint:

... this is the task that creates the app in okta

@operation
def create_app(**kwargs):

... omitting the call to create the app but from the response I am able to get the appid and some other data. Next I take the id and 

... add it to the runtime properties:

if 'id' in r:
    appid = r['id']
    ctx.instance.runtime_properties['appid'] = appid
    ctx.logger.info("Application was successfully added to okta with id: {}".format(appid))

...doing some error checking and validation here

@operation
def create_group(**kwargs):
    if DEBUG:
        debugGroup_func('create_group')

    gname = ctx.node.properties['group_name']
    gdesc = ctx.node.properties['group_description']
    okta = OktaClient(ctx.node.properties['okta_key'], ctx.node.properties['okta_domain'], 'https')
    target_node_value = ctx.target.instance.runtime_properties['appid']
    ctx.source.instance.runtime_properties['appid'] = target_node_value

And here is now my blueprint and plugin.yaml

---

tosca_definitions_version: cloudify_dsl_1_3
imports:
  - http://www.getcloudify.org/spec/cloudify/4.1/types.yaml
  - plugin.yaml


# Add this block to import your plugin
plugins:
    #any name for your plugin
    fico-octa-plugin:
        # Give the same details as mentioned while creating the plugin
        executor: central_deployment_agent
        package_name: oktautil
        package_version: '0.1'
        distribution_release: ''
        distribution_version: ''
        supported_platform: ''
        distribution: ''



node_templates:
  okta_app:
    type: fico.nodes.okta.app
    properties:

      okta_domain: <somepassword>
      okta_key: <somekey>

      application_json: { get_input: application_json }


  okta_group:
    type: fico.nodes.okta.group
    relationships:

      - type: cloudify.relationships.depends_on
        target: okta_app
    properties:
      okta_domain: <somepassword>
      okta_key: <somekey>

      group_name: { get_input: group_name }
      group_description: { get_input: group_description }

      appid: { get_attribute: [okta_app, appid] }

string
      application_json:
        type: string
    interfaces:
      cloudify.interfaces.lifecycle:
        configure:  ### not used. Had hopped I could create an okta object to be used by all the other tasks. 
          implementation: fico-okta-plugin.plugin.tasks.configure_app
        

start: ## three operations exists for install workflow ('create', 'configure', and 'start'
          implementation: fico-okta-plugin.plugin.tasks.create_app
        delete: ## 'delete' is an operation for the uninstall workflow.
          implementation: fico-okta-plugin.plugin.tasks.delete_app

[Trammell -]

Hi Richard,

I'll avoid the abstract rules and talk about what's going on in your blueprint/plugin. In the create_group function, this is the problematic line:

    target_node_value = ctx.target.instance.runtime_properties['appid']

This should be:

    target_node_value = ctx.instance.runtime_properties['appid']

Now are some requirements:

1. Your okta_group node needs a relationship to okta_app node, such as the "connnect_to_app" that appears in one of your screenshots. In the blueprint, the relationship should be defined something like this:

  connect_to_app:

    derived_from: cloudify.relationships.connected_to

    source_interfaces:

      cloudify.interfaces.relationship_lifecycle:

        preconfigure:

          implementation: plugin.path.to.connect_to_app

2. The relationship should call a function that does something similar to this:

def connect_to_app(**_):

    # Get appid property from target app node and store it in the source group node.

    ctx.source.instance.runtime_properties['appid'] = ctx.target.instance.runtime_properties['appid']

3. That relationship operation must occur before the "create_group" function. In step 1, above, I defined "connect_to_app" as preconfigure. This step happens after create and before configure on the source node. So your app and group node types should be defined something like this:

    

  okta_app_type:

    derived_from: cloudify.nodes.Root

    interfaces:

      cloudify.interfaces.lifecycle:

        configure: plugin.path.to.create_app

  okta_group_type:

    derived_from: cloudify.nodes.Root

    interfaces:

      cloudify.interfaces.lifecycle:

        configure: plugin.path.to.create_group

If you just have three functions, this should illustrate the "data flow".

[RichardK]

Dude, you are awesome!

Thanks a lot, it worked like a charm and I was able to expand on it and refactor the code some. 

Now to sum it up, why did this work? 

Is there a doc that explains/documents the lifecycle operations for the different types, or am I bound to dig in the source code to figure this out?  I guess I am having difficulties grasping the concept :( and it was not clear to me from looking at the documentation how this is all hanging together. Maybe something more need/could be said about context, instances, and relationships that ties it all together for people new to cloudify. I will try to get this feedback to the cloudify folks, but I am sure they monitor this forum :). 

I can thank you enough, you have been very helpful and I have learned a lot from these message. 

Just to conclude and maybe help others, I have posted the relevant resulting code that worked below:

@operation
def connect_to_app(**kwargs):
    if DEBUG:
        ctx.logger.info('Linking appid, okta_domain, and okta_key betweeen source and target.')

    # preconfigure method to use when connecting two nodes with relation ships.

    # Get appid property from target app node and store it in the source group node.

    ctx.logger.info("Setting up relationship properties and attributes. ")

    ctx.source.instance.runtime_properties['appid'] = ctx.target.instance.runtime_properties['appid']

    ctx.source.instance.runtime_properties['okta_domain'] = ctx.target.instance.runtime_properties['okta_domain']
    ctx.source.instance.runtime_properties['okta_key'] = ctx.target.instance.runtime_properties['okta_key']

@operation
def configure_app(**kwargs):
    if DEBUG:
        debugApp_func('configure_app')
    ctx.instance.runtime_properties['okta_key'] = ctx.node.properties['okta_key']
    ctx.instance.runtime_properties['okta_domain'] = ctx.node.properties['okta_domain']

@operation
def create_group(**kwargs):
    if DEBUG:
        debugGroup_func('create_group')
    gname = ctx.node.properties['group_name']
    gdesc = ctx.node.properties['group_description']

    okta_key = ctx.instance.runtime_properties['okta_key']
    okta_domain = ctx.instance.runtime_properties['okta_domain']
    okta = OktaClient(okta_key, okta_domain, 'https')
    appid = ctx.instance.runtime_properties['appid']
    res = OktaClient._add_okta_group(okta, gname, gdesc)

blueprint

tosca_definitions_version: cloudify_dsl_1_3
imports:
  - http://www.getcloudify.org/spec/cloudify/4.1/types.yaml
  - plugin.yaml

# Add this block to import your plugin
plugins:
    #any name for your plugin
    fico-octa-plugin:
        # Give the same details as mentioned while creating the plugin
        executor: central_deployment_agent
        package_name: oktautil
        package_version: '0.1'
        distribution_release: ''
        distribution_version: ''
        supported_platform: ''
        distribution: ''

node_templates:
  okta_app:
    type: fico.nodes.okta.app
    properties:

      okta_domain: { get_input: okta_domain }
      okta_key: { get_input: okta_key }

      application_json: { get_input: application_json }
  okta_group:
    type: fico.nodes.okta.group
    relationships:

      - type: connect_to_app
        target: okta_app
    properties:

      group_name: { get_input: group_name }
      group_description: { get_input: group_description }

---
plugins:
  fico-okta-plugin:

    executor: central_deployment_agent
    package_name: oktautil
    package_version: '0.1'

inputs:
  okta_domain:
    description: The Okta domain
  okta_key:
    description: Okta authentication token
  application_json:
    description: application configuration json as string
  group_name:
    description: name of okta group
  group_description:
    description: Group description, give empty string if no description


node_types:

  fico.nodes.okta.app:
    derived_from: cloudify.nodes.Root
    properties:
      okta_domain:
        type: string
      okta_key:
        type: string
      application_json:
        type: string
    interfaces:
      cloudify.interfaces.lifecycle:
        configure:

          implementation: fico-okta-plugin.plugin.tasks.configure_app
        start: ## three operations exists for install workflow ('create', 'configure', and 'start'
          implementation: fico-okta-plugin.plugin.tasks.create_app
        delete: ## 'delete' is an operation for the uninstall workflow.
          implementation: fico-okta-plugin.plugin.tasks.delete_app

  fico.nodes.okta.group:
    derived_from: cloudify.nodes.Root
    properties:

      group_name:
        type: string
      group_description:
        type: string

    interfaces:
      cloudify.interfaces.lifecycle:
          start: ## three operations exists for install workflow ('create', 'configure', and 'start'
            implementation: fico-okta-plugin.plugin.tasks.create_group
          delete: ## 'delete' is an operation for the uninstall workflow.
            implementation: fico-okta-plugin.plugin.tasks.delete_group

relationships:

  connect_to_app:
    derived_from: cloudify.relationships.connected_to
    source_interfaces:
      cloudify.interfaces.relationship_lifecycle:
        preconfigure:

          implementation: fico-okta-plugin.plugin.tasks.connect_to_app

outputs:
  # example output the could be used to simplify assertions by test
  application_metadata:
    description: Metadata for the application added to okta
    value:
      saml: { get_attribute: [okta_app, saml_metadata] }
      appid: { get_attribute: [okta_app, appid] }
      groupid: { get_attribute: [okta_group, groupid] }

Thank you. 

On Friday, October 20, 2017 at 10:26:33 AM UTC-7, RichardK wrote:

[Trammell -]

Hi Richard, Great! I am glad to have been of help. The docs that we have on these concepts are those that I've already sent you:

Built-In Workflows (touches on the types of plugin operations and their order): http://docs.getcloudify.org/4.1.0/workflows/built-in-workflows/#the-install-workflow

Relationships (touches on how to use relationships and map them to plugin operations): http://docs.getcloudify.org/4.1.0/blueprints/spec-relationships/#relationship-interfaces

We also have a guide on how to write a plugin: http://docs.getcloudify.org/4.1.0/plugins/creating-your-own-plugin/, but it doesn't address relationship operations.