Removing intermediate (temp) files

[Itai Rusinek]

Hello,

I have a stage in my ruffus pipeline where I split a very large file to many smaller separate files, process each one and merge the results. If I understood correctly that is more or less the classiccflow with ruffus.

I would like to remove those split temp files, as they are taking up a lot of unnecessary space. I understand that one way would be to empty them and then touch the pipeline up to that point, but it seems awkward.

Is there a more elegant way to remove temp files without cauaing the pipeline to run from the start next time?

Thanks

[Clare Sloggett]

Hi Itai,

We dealt with this by "zeroing out" the intermediate files, i.e.
overwriting them with an empty file (preserving their timestamps).
Here's some very old Python code that does that. Use at your own risk
- it is really dangerous as you can easily target multiple files. If
you zero out the wrong file, it's gone and on some backup systems this
is even more damaging than deleting it as the backup gets overwritten
as well!

And, I have not used it for a long time and underlying libraries could
have changed; please test it first on something harmless and check
that it is doing what you expect.


import os
import sys

# truncate a file to zero bytes, and preserve its original modification time
def zeroFile(file):
if os.path.exists(file):
# save the current time of the file
timeInfo = os.stat(file)
try:
f = open(file,'w')
except IOError:
pass
else:
f.truncate(0)
f.close()
# change the time of the file back to what it was
os.utime(file,(timeInfo.st_atime, timeInfo.st_mtime))

if __name__=='__main__':
filelist = sys.argv[1:]
for file in filelist:
print("zeroing %s" % file)
zeroFile(file)


Cheers,
Clare

> --
> You received this message because you are subscribed to the Google Groups "ruffus_discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ruffus_discus...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

Clare Sloggett
Research Fellow / Bioinformatician
Melbourne Bioinformatics
University of Melbourne, Parkville Campus
187 Grattan St, Carlton VIC 3053, Australia
Ph: 03 903 53357 M: 0414 854 759
In-office days: Tue, Wed, Thurs, Fri