Flow/FlowDefinition Metadata Retrieval

[Chris Stark]

Hey there,

I'm trying to pull down metadata for some Flows/FlowDefinitions specified in my package.xml like so:

 

    <!-- flows -->
    <types>
        <name>Flow</name>
        <members>HROD_Service_Process</members>
    </types>

    <types>
        <name>FlowDefinition</name>
        <members>HROD_Service_Process</members>
    </types>

 

When I perform a Retrieve Metadata command against my sandbox, IlluminatedCloud shows both the Flow and the FlowDefinition, like so:



However, when the retrieve is actually performed, the Flow itself is not present, even though the FlowDefinition is:



Have I run bug or am I doing something wrong here? When I perform a Metadata API retrieval outside of IlluminatedCloud, the Flow files have the version number appended to the end of the file (ex: HROD_Service_Process-3.flow).

Thanks in advance for any thoughts you might have!

[Scott]

Chris, Flows and FlowDefinitions don't seem to work quite right.  I helped another user through this a few months back.  Here's the dialog:

https://bitbucket.org/RoseSilverSoftware/illuminatedcloud/issues/280/retrieval-of-active-flows-made-in-process

Take a look at that and see if it doesn't help you get things working.

Regards,

Scott

[Chris Stark]

This makes sense given how the Metadata API is treating Flows, although I can retrieve the latest version of the Flow file itself by doing a retrieve with the following package.xml:

    <!-- flows -->
    <types>
        <name>Flow</name>

        <members>HROD_Service-Process-5</members>

    </types>

    <types>
        <name>FlowDefinition</name>
        <members>HROD_Service_Process</members>
    </types>

In this example, Version 5 is the current active version of this flow. When I perform a retrieval outside of IlluminatedCloud (using the jsforce toolset, if it matters).


Using the wildcard (*) for Flows, as detailed in the post you linked, I can retrieve all previous versions of the Flow via IlluminatedCloud but not the currently-active version:

    <!-- flows -->
    <types>
        <name>Flow</name>

        <members>HROD_Service-Process-5</members>

    </types>

    <types>
        <name>FlowDefinition</name>
        <members>HROD_Service_Process</members>
    </types>

Actually performing the retrieval/diff, the only files that are actually retrieved are the older versions (1-4).

For all I know the jsforce toolset is doing some extra work to deal with the goofiness of Flows vs. FlowDefinitions, but it still does appear there is a disconnect here. Any further thoughts would be appreciated!

I appreciate the quick response and the tools you've built. They have made development in Salesforce substantially more reasonable and have helped standardize my team on development tools.

[Scott]

Chris, if you want to see more details about what IC is doing on retrieve, add the following to Help>Debug Log Settings:

#com.illuminatedcloud.intellij.builder.ForceComMetadataRetriever

After you perform the retrieval, look in idea.log (Help>Show Log in Explorer/Finder) and it should give you a good idea of what it did.  If there's a way to see the package.xml being used when performing the retrieval with jsforce, we could compare the two and see what's different.

Regards,

Scott

[Chris Stark]

Scott,

Cool, I ran this test with the following package.xml:

    <!-- flows -->
    <types>
        <name>Flow</name>

        <members>*</members>

    </types>

    <types>
        <name>FlowDefinition</name>
        <members>HROD_Service_Process</members>
    </types>

Here is what I have in the logs:

2016-09-09 16:37:07,491 [  46550]   INFO - ntellij.builder.RetrieveAction - Retrieving metadata from requested connection HROD - Dev into module HROD
2016-09-09 16:37:07,492 [  46551]  DEBUG - lder.ForceComMetadataRetriever - Retrieving metadata for module HROD
2016-09-09 16:37:07,499 [  46558]  DEBUG - lder.ForceComMetadataRetriever - Creating a retrieve request for API version 37.0
2016-09-09 16:37:07,501 [  46560]  DEBUG - lder.ForceComMetadataRetriever - Retrieving all metadata for src/package.xml (25 items).
2016-09-09 16:37:07,748 [  46807]  DEBUG - lder.ForceComMetadataRetriever - Submitted retrieve request with async process ID 09S4C000000Cec9UAC
2016-09-09 16:37:07,748 [  46807]  DEBUG - lder.ForceComMetadataRetriever - Polling the status of the retrieve request.
2016-09-09 16:37:07,834 [  46893]  DEBUG - lder.ForceComMetadataRetriever - Status: InProgress
2016-09-09 16:37:07,834 [  46893]  DEBUG - lder.ForceComMetadataRetriever - Retrieve request is still processing. Waiting 5000 milliseconds and trying again.
2016-09-09 16:37:12,834 [  51893]  DEBUG - lder.ForceComMetadataRetriever - Polling the status of the retrieve request.
2016-09-09 16:37:13,180 [  52239]  DEBUG - lder.ForceComMetadataRetriever - Status: Succeeded
2016-09-09 16:37:13,181 [  52240]  DEBUG - lder.ForceComMetadataRetriever - Retrieve complete.
2016-09-09 16:37:13,428 [  52487]  DEBUG - lder.ForceComMetadataRetriever - Retrieved an archive of size 32328 bytes.
2016-09-09 16:37:13,434 [  52493]  DEBUG - lder.ForceComMetadataRetriever - Wrote retrieved archive to C:\Users\cstark12\AppData\Local\Temp\HROD_216e6999_2e75_40c1_a6f7_490220ac3ea4.zip
2016-09-09 16:37:13,544 [  52603]  DEBUG - lder.ForceComMetadataRetriever - Extracted retrieved archive to C:\Users\cstark12\AppData\Local\Temp\HROD_216e6999_2e75_40c1_a6f7_490220ac3ea4
2016-09-09 16:37:13,544 [  52603]  DEBUG - lder.ForceComMetadataRetriever - Removing intermediate 'unpackaged' directory.
2016-09-09 16:37:13,657 [  52716]  DEBUG - lder.ForceComMetadataRetriever - Extracting static resource bundles.
2016-09-09 16:37:13,658 [  52717]  DEBUG - lder.ForceComMetadataRetriever - No static resources directory was found in the retrieved metadata.
2016-09-09 16:37:13,658 [  52717]  DEBUG - lder.ForceComMetadataRetriever - Removing archive file.
2016-09-09 16:37:13,667 [  52726]  DEBUG - lder.ForceComMetadataRetriever - Refreshing the virtual file system before applying retrieval substitution rules.
2016-09-09 16:37:13,679 [  52738]  DEBUG - lder.ForceComMetadataRetriever - Refreshing the virtual file system before comparison.
2016-09-09 16:37:13,681 [  52740]  DEBUG - lder.ForceComMetadataRetriever - Showing a directory diff for module = HROD, extractDir = C:\Users\cstark12\AppData\Local\Temp\HROD_216e6999_2e75_40c1_a6f7_490220ac3ea4, onWindowClose = null
2016-09-09 16:37:13,681 [  52740]  DEBUG - lder.ForceComMetadataRetriever - sourceRoots = [C:/git/jhu-malone/src]
2016-09-09 16:37:13,682 [  52741]  DEBUG - lder.ForceComMetadataRetriever - virtualExtractDir = C:/Users/cstark12/AppData/Local/Temp/HROD_216e6999_2e75_40c1_a6f7_490220ac3ea4
2016-09-09 16:37:13,682 [  52741]  DEBUG - lder.ForceComMetadataRetriever - sourceRoot = C:/git/jhu-malone/src
2016-09-09 16:37:13,682 [  52741]  DEBUG - lder.ForceComMetadataRetriever - Running directory diff on C:/Users/cstark12/AppData/Local/Temp/HROD_216e6999_2e75_40c1_a6f7_490220ac3ea4 and C:/git/jhu-malone/src.

The temp directories reflect the same thing shown in the DIFF tool in terms of files:

HROD_216e6999_2e75_40c1_a6f7_490220ac3ea4/

   flows/

      HROD_Service_Process-1.flow
      HROD_Service_Process-2.flow
      HROD_Service_Process-3.flow
      HROD_Service_Process-4.flow

  flowDefinitions/

      HROD_Service_Process.flowDefinition

Does this help? Nothing there looks terribly out of the ordinary.

Chris

 

 

[Scott]

Chris, just so I understand what you're seeing vs. what you're expecting, the issue is that you're not seeing HROD_Service_Process-5.flow?  Sorry if I'm missing/confusing the issue...

Regards,

Scott

[Chris Stark]

Yep you've got it, that's exactly the issue. It seems like I can retrieve that version of the flow when using other tools (jsforce), but not using IlluminatedCloud.

This leads me to think (perhaps incorrectly) that there is a bug/feature gap in IlluminatedCloud that prevents the retrieval of the currently-active version of the flow. I think what we have here supports the findings in the previous thread you linked, but it doesn't appear like its a hard limitation of the Metadata API itself?

Of course the jsforce toolset might be doing some kind of workaround here but I don't know for sure and obviously cant expect you to answer for them.

Chris

[Scott]

It's also possible (likely, in fact) that the package.xml I'm generating for retrieval is listing out things explicitly instead of wildcarding even though the package.xml you've fed to it has a wildcard.  That's very possible given that I don't use your package.xml on retrieve since it generally expresses everything and a selective retrieval is for a subset of that.  Let me take a look...minimally I need to log the package.xml that's being used (I thought I was, but evidently just on deployment) which would be a smoking gun here.

I won't likely get a chance to take a close look until later this weekend.  I just found out that my son has a fever, so I need to scoot home and help out there.  My guess at this point is that it's probably just a wildcard thing, though.  I'll let you know what I learn and when you can expect to see a fix.

Regards,

Scott

[Chris Stark]

Cool, hope everything is ok and you have a good weekend.

I appreciate the turnaround time and the level of effort you put into this product, it really shows.

Chris

[Scott]

Okay, I did look into this a bit today.  There seems to be a bug in the Salesforce metadata API when working with Flow metadata.

The issue is that Salesforce's listMetadata() API returns an incorrect entry for the active version of the FlowDefinition.  The returned name and filename of the active version don't include the "-version" part.  For example, I have a flow with three versions and the second one is active.  The listMetadata() API returns them as "MyFlow-1", "MyFlow", and "MyFlow-3".  If I change the active version to 3, I get back "MyFlow-1", "MyFlow-2", and "MyFlow".  However, when you actually retrieve the metadata using a wildcard, you get back "MyFlow-1.flow", "MyFlow-2.flow", and "MyFlow-3.flow".

I've updated Illuminated Cloud to be able to work around this issue.  Basically when your metadata subscription includes Flows, after I use listMetadata() to enumerate metadata of that type, I also run a SOQL query against the Flow SObject.  I then correct the listMetadata() response so that your metadata subscription is correct.  As a result, the package.xml that I create for metadata retrieval is also correct without having to resort to wildcards that might include things you don't want.

The fix will be in the next build.  After updating the plugin, on your next retrieve do the follow to correct the metadata subscription:

1. In the retrieval dialog, if you're using an explicit selection vs. Package.xml or All/Package:
1. Click the pencil icon beside Contents to open the metadata subscription editor.
2. Click the Refresh button beside Selected.
3. Scroll down to the node for Flow and ensure that the things you want are selected.
4. Click OK to return to the retrieval dialog.
2. Now in the retrieval dialog, click the Refresh button on the toolbar.
3. Scroll down to the node for flows and make sure the things you want are selected.
4. Click OK to retrieve the metadata.

Hopefully that will give you back exactly what you're expecting with fine-grained control over your metadata subscription.

I'll try to get the new version uploaded tomorrow.  I'll post another comment here once I've done so.  Please let me know your findings after updating.

Regards,

Scott

[Scott]

Okay, this is released.  Just update to 1.7.3.8 and let me know if it doesn't work better.  Also, I've noticed that I can't deploy this metadata.  It complains about the version being active.  It looks like it's deploying properly to me.  I'll follow up with Salesforce on that and see if there's some other voodoo required on the deployment side!

Regards,

Scott

[Chris Stark]

Scott,

Looks great! Thanks for the quick turnaround.

The flow metadata retrieval in general is discouraging (from Salesforce, not IC). Its handled so differently from how the other metadata types are, in a manner that really hurts any CI/Automated Deployment processes. It pretty much needs its own separate special handling logic. This might wind up being a place where changesets are still required, unfortunately.

Chris

[Christian Anderson]

Just want to say thanks Scott.  Found this thread extremely helpful.  I'm amazed at how quick you are to troubleshoot, fix and inform. Thank for all the work in Illuminated Cloud.  

[Scott]

Glad to hear you found it useful, Christian.  This one definitely falls into the weird category.  I'll make sure that the folks at Salesforce know about it since it definitely doesn't seem to behave as expected.

Regards,

Scott