Using ingest action in a flow

[Anthony Weaver]

We have a flow that currently does some data transfer and compute and we'd like  to add an ingest step.  

We already have:

1. a search index setup in Globus

2. Working python code to extract metadat

3. Manually uploaded the ingest document through the web interface and verified that we can search on that metadata

My question is about the ingest flow action.  When I look at the documentation for the flow ingest action (and some example flows that use it), it seems like I can only ingest one subject at a time.  Is that a correct understanding?  I'm asking because for our use case, we'd like to ingest  multiple subjects at a time as part of the flow, similar to what happens when you do it manually in the web interface by uploading a JSON file.  If we can't use the flow action in that way then I guess we'd need a second compute component and have that do the ingest using the APIs?

Thank you

Tony

[Kurt McKee]

Hello Anthony,

You're correct -- the Search Ingest action provider only supports a single subject at a time.

There are two possible ways to ingest documents with multiple subjects. The first is what you already wrote -- adding a Compute component to ingest the entire document directly to Search.

The second option would be to use a loop defined in the flow. I've put together a sample flow definition and input schema demonstrating how you might take a GMetaList document and chop that up into individual Search Ingest action provider calls. You can view it in the Flows IDE, or you can refer to the files attached to this email.

The core mechanics of the flow definition are:

• Use an initial flow state to define the starting index (0) and the length of the "gmeta" list.
• Just in case the length of the list is 0, evaluate whether to exit the loop using a Choice state.
• Ingest the nth item in the "gmeta" list, as referred to by the current loop index value.
• Increment the index, and jump back to the "exit-loop" Choice state.

Please let me know if you have additional questions!

Sincerely,

Kurt McKee

--

Kurt McKee

Software Engineer

Globus, University of Chicago

[Anthony Weaver]

Just an update for anyone who comes across this thread looking to do something similar.

I did get my code setup to work with my flow but ran into issues running the Globus ingest action in my flow because

the amount of ingest data I'm producing exceeds the limits for what can be returned from a globus compute action.  My particular

use case might analyze 2k-3k images and extract metadata based on the image analysis results.  The amount is under the limit if I produce an ingest 

file and manually upload it, but is too big to return as a compute result.

I'm moving on to option two, generating an ingest file and then programmatically uploading that