Best Connector to Retry Work Entries After Feed Iterator Completes

[bpevans]

We're considering an HR Feed scenario for identity management where certain errors would have us skip the current entry with the intent of retrying that entry after the remainder of the csv feed file has completed. We would like to do so in a simple TDI-friendly fashion.

The original idea was handling the error by pushing the current work object onto an array variable. And then using another Feed Iterator following the CSV File Iterator to iterate through each of those Array-stored-work objects and take them through the same/Standard Data Flow.

Worst case we're considering, is writing the entry to a second CSV file (Passive CSV File Connector) and then adding another File Iterator in the feed section to then read through any entries in that retry-csv-file.

Any connector types suited to simply pulling a work entry off an array for this type of retry requirement? Would the MemQueue connector be suited to this perhaps - or examples of it's use?

[Eddie Hartman]

Writing to a secondary CSV file has the advantage that you re-run this data set if you need to. If you're happy pushing the Entries to an Array and want to iterate over this I would suggest making the second Iterator a FormEntry Connector.

The FormEntry is like a File Connector in that it reads a byte stream and passes it to a Parser. Unlike the File Connector, FormEntry has its byte stream in a Connections tab parameter: Raw Data Text. However, you don't care what data is configured for this component, but will instead code the Override GetNext Hook.

This Hook will prevent the Connector Interface from actually engaging to return data. Here you can return one item at a time in your Array. For example:
-----
// Lazy man's init
if (typeof(counter) == "undefined") {
counter = 0;
}

if (counter >= entryArray.length) {
result.setStatus(0);
} else {
work.merge(entryArray[counter]);
result.setStatus(1);
}

// Inc counter
counter++;
-----
Hope this helps!

/Eddie

[bpevans]

Thanks Eddie, that works as we hoped!

Saw some oddness with .pop() vs .shift() on the array, code works perfectly as intended using .shift() as the data return but .pop() was always the last data entry for as many iterations of the

Working Override GetNext solution:

// Lazy man's init
if (typeof(counter) == 'undefined') {
counter = 0;
}

if (counter >= gRetry.length) {
result.setStatus(0);
} else {
// shift() works, pop() did not.
work.merge(gRetry.shift());