Regular "mirroring" of data from external site

[Jeff Theobald]

Hello,

Background: large, multi-site REDCap project (external sites all separated by DAGs)

This is a longitudinal project, with repeating events and repeating instruments

For complicated reasons, one of the external sites does not login in to the central REDCap project for data entry. Instead, they need to enter data into a local copy of the REDCap project (we provided them the XML) on their instance of REDCap -- and then their local data needs to be imported into the central REDCap project. 

The central project needs the external data at least monthly. 

Previously-entered data is expected to be changed occasionally (for example, if errors are found, study sites should correct the error); for this reason, we have taken the approach of asking the external site for a complete copy of their entire database each month -- the thought being that if they send us what they have, we will have an exact duplicate of their data! But I think the import process is getting in the way of that goal.

The strategy to date for handling this external site's data has been:

1. The external site exports all data, in raw CSV format.
2. Site submits their CSV file to the central project coordinator.
3. Central project coordinator imports site's CSV file into the central REDCap project
   1. the "Allow blank values to overwrite existing saved values?" is set to "YES" - to account for cases where the site needs to delete a text field entry (e.g., months later, they realize that a question should have been left blank)
   2. Originally, we used the standard "Data Import Tool", but the files became too big for our REDCap to handle. Instead of manually splitting into smaller files, we have been using the "Big Data Import" External Module for the past several imports.
4. Aside from some annoyances (e.g., making sure the external site's records get assigned to their DAG in the central project), we thought this was working well.

Problem: We (central REDCap) are not always seeing the exact same data that they (external site) see in their local copy. I think this has to do with repeating forms. 

Example: 

• Import#1 from external site had a record with 5 instances of a repeating Medication form. 
• After Import #1, the local site realized they made an erroneous data entry -- so they have deleted Medication instance #4 from their local copy. 
• At the time of Import #2, Medication #4 is no longer in the external site's CSV export -- but that doesn't mean it will be deleted from the central project upon Import #2: a row for Medication #4 isn't in the CSV, so nothing about it gets updated in the import. Thus, the external site has the "right" data (Meds #1,2,3,5), but the central site has incorrect data (Meds #1-5; Med #4 persists).

First, does the described behavior match your understanding of how multiple CSV data imports would work with repeating forms?

Based on this, it seems an import would only be guaranteed to result in the central REDCap having an "exact copy" of the external site's data if the central REDCap was essentially a "blank slate" for the site; e.g., if the DAG's entire set of records was deleted (or at least all values set to null) prior to each import. Does that seem right?

Even though the data is saved in multiple places, I'm not thrilled with the idea of deleting all of the site's current data prior to importing the newest data. And I'm not even sure how I'd "zero out" this DAG in an efficient way (hundreds of records, repeating forms & events).

I'd appreciate any suggestions on how to handle this -- preferably via file imports, but maybe API is the best answer.

Thanks all,

Jeff

[Peter Macisaac (POP)]

So I assume they delete the no 4 medication repeating form entirely - so nothingg to trigger an update at home base

If they just delete the med details or replace it with “drug deleted you should have a form instance with blank that will work

Or you could tweak the record to have a drug inactivation  field and date and reason inactivated - that would provide a clearly visible audit trail

If so just a training exercise for their project admin

But:

But why do we research informaticians feel the need to accede to bizarre and unnecessary and complicated cluges to meet unrealistic researcher expectation/demands?

It might be better to say "we cant do that" - suck it up and use the common redcap project if you want to be part of this research. It is not possible to be all things to all men

Ahhh now I feel better!

Peter Macisaac

 

--
You received this message because you are subscribed to the Google Groups "Redcap Open" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redcap_open...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/redcap_open/2a798496-3b0e-4583-9ba2-061a5be7d455n%40googlegroups.com.