Using ingest action in a flow

[Anthony Weaver]

We have a flow that currently does some data transfer and compute and we'd like  to add an ingest step.  

We already have:

1. a search index setup in Globus

2. Working python code to extract metadat

3. Manually uploaded the ingest document through the web interface and verified that we can search on that metadata

My question is about the ingest flow action.  When I look at the documentation for the flow ingest action (and some example flows that use it), it seems like I can only ingest one subject at a time.  Is that a correct understanding?  I'm asking because for our use case, we'd like to ingest  multiple subjects at a time as part of the flow, similar to what happens when you do it manually in the web interface by uploading a JSON file.  If we can't use the flow action in that way then I guess we'd need a second compute component and have that do the ingest using the APIs?

Thank you

Tony

[Kurt McKee]

Hello Anthony,

You're correct -- the Search Ingest action provider only supports a single subject at a time.

There are two possible ways to ingest documents with multiple subjects. The first is what you already wrote -- adding a Compute component to ingest the entire document directly to Search.

The second option would be to use a loop defined in the flow. I've put together a sample flow definition and input schema demonstrating how you might take a GMetaList document and chop that up into individual Search Ingest action provider calls. You can view it in the Flows IDE, or you can refer to the files attached to this email.

The core mechanics of the flow definition are:

• Use an initial flow state to define the starting index (0) and the length of the "gmeta" list.
• Just in case the length of the list is 0, evaluate whether to exit the loop using a Choice state.
• Ingest the nth item in the "gmeta" list, as referred to by the current loop index value.
• Increment the index, and jump back to the "exit-loop" Choice state.

Please let me know if you have additional questions!

Sincerely,

Kurt McKee

--

Kurt McKee

Software Engineer

Globus, University of Chicago