redo-unless-change automates data integrity verification

30 views
Skip to first unread message

José I. Rojas Echenique

unread,
Oct 14, 2020, 1:43:42 PM10/14/20
to redo
I use redo to track dependencies when doing data analysis and have
coded up this little tool to automate data integrity verification
using checksums.

It's a somewhat common practice for institutions that run FTP servers
hosting large data files to include md5sums of the files so that users
can verify the integrity of their downloads. I want redo to check the
sum of a data file before overwriting an analysis that depends on it.

The tool is simple, when called from a do script:

redo-unless-change data

redo-unless-change first checks if a file data.md5 exists. If data.md5
exists, redo-unless-change sums data and ensures that the sum matches
the sum in data.md5. If it does, the do script is executed as normal,
otherwise, redo stops and warns the user. If data.md5 does not exist,
redo-unless-change sums data to create data.md5 and prints a message
telling the user that the integrity of data will be checked against
this sum in any future builds.

My simple redo implementation has this tool built in but I'd rather
recommend a more mature implementation to new users. I hope some of
you will find a tool like redo-unless-change useful enough to
implement.

Nils Dagsson Moskopp

unread,
Jul 31, 2022, 1:28:56 AM7/31/22
to José I. Rojas Echenique, redo
José I. Rojas Echenique <jose.i...@gmail.com> writes:

> The tool is simple, when called from a do script:
>
> redo-unless-change data
>
> redo-unless-change first checks if a file data.md5 exists. If data.md5
> exists, redo-unless-change sums data and ensures that the sum matches
> the sum in data.md5. If it does, the do script is executed as normal,
> otherwise, redo stops and warns the user. If data.md5 does not exist,
> redo-unless-change sums data to create data.md5 and prints a message
> telling the user that the integrity of data will be checked against
> this sum in any future builds.

Why did you need an extra tool for it? As I understand this, it seems to
be just a shortcut for exiting the buildrule with a statis code that
indicates error when some hash is not matching.

Greetings,
Nils
signature.asc

J

unread,
Jul 31, 2022, 6:26:04 AM7/31/22
to Nils Dagsson Moskopp, redo
Yes, I now think that's right.
Please ignore my previous email.
Reply all
Reply to author
Forward
0 new messages