rclone logging commands

[e.so...@utoronto.ca]

Hi all,

I'm downloading some files from a Sharepoint instance using rclone, and have been experimenting with what I want to be collecting to log the transfer internally. For those who are using rclone in your work, would you mind sharing what you collect?

Right now I've been generating three files: checksum file, list of files (output to a csv), and log file when copying:

• rclone hashsum sha256 sharepoint:folderName --output-file /home/rclone/checksum.txt
• rclone lsf -R --format ptsm --files-only --csv sharepoint:folder > /home/rclone/newTransfer_list.csv
• rclone copy sharepoint:folder /home/rclone/newTransfer -MP --log-file /home/rclone/newTransfer_log.txt --log-level DEBUG

But yeah, I'm curious what commands others are doing.

Thanks!

-Emily

[Lara Friedman-Shedlov]

Hi Emily,

I don't have a ton of experience yet with rclone, but I'm wondering why you have a separate command to create the list of checksums (rclone hashsum) when they are already generated as part of the log that is created during the copy operation  when you include "--log-level DEBUG".  Is it because you specifically want sha256 rather than MD5?  Are they formatted in a way that makes them more useable for a specific purpose when you use the hashsum command?

/ Lara Friedman-Shedlov

--
You received this message because you are subscribed to the Google Groups "BitCurator Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcurator-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bitcurator-users/342e28a2-e77e-4792-aaad-af8c5ac90c09n%40googlegroups.com.

--

Lara D. Friedman-Shedlov    (she / they)  (hear my name)

Digital Records Archivist | Archives & Special Collections 

University of Minnesota Libraries | lib.umn.edu | 612.626.7972

[Emily Sommers]

Hi Lara,

 

I prefer to generate sha256 because that’s what our Archivematica instance uses. Though the checksums are in the copy log, it’s nice to have a separate checksum file that I can then reuse to check the files once they’ve been copied locally – in the directory where the downloaded files are: sha256sum --check /home/rclone/checksum.txt > /home/rclone/checksumCheckOutput.txt

 

Not sure I will keep the output of the checksum checking, it’s rather just a step to make sure nothing when wrong when copying from A to B

I forgot to add the --download flag in my previous hashsum command. It should read:
rclone hashsum sha256 sharepoint:folderName --download --output-file /home/rclone/checksum.txt

--
You received this message because you are subscribed to a topic in the Google Groups "BitCurator Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bitcurator-users/ZRXNrTtirc8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bitcurator-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bitcurator-users/CAKWpb_Zf%3DUvN%3DcPx-dixh1xWaPyaRY5GkFeSYMsL8gYsHkqPUQ%40mail.gmail.com.

[Shelly Black]

Hi Emily! I use a version of your third command: rclone copy with the logging option and -P. I haven't used the -M option because I was copying from Google Drive, where the system metadata wasn't retained. But it looks like I should use -M if I copy from SharePoint?

Then, instead of generating a checksum file, I use rclone check after my sync completes.

Shelly

---

From: 'Emily Sommers' via BitCurator Users <bitcurat...@googlegroups.com>
Sent: Thursday, September 24, 2026 1:41 PM
To: bitcurat...@googlegroups.com <bitcurat...@googlegroups.com>
Subject: RE: rclone logging commands

 

  [EXTERNAL]

To view this discussion visit https://groups.google.com/d/msgid/bitcurator-users/YQ1PR01MB1103525CD818BCDA957DE60A87E3812%40YQ1PR01MB110352.CANPRD01.PROD.OUTLOOK.COM.

[Emily Sommers]

Oh thanks Shelly! I missed the ‘check’ option in my initial reading of the docs. I will give that a try.

 

I’m not sure whether the -M (for metadata) flag is necessary when copying from Sharepoint, but I figure might as well include it? I’m a bit unclear about what metadata exactly it preserves. If anyone else knows, please chime in!

 

Thanks,

-Emily

 

From: bitcurat...@googlegroups.com <bitcurat...@googlegroups.com> On Behalf Of Shelly Black
Sent: Thursday, September 24, 2026 4:03 PM
To: bitcurat...@googlegroups.com
Subject: Re: rclone logging commands

 

You don't often get email from syb...@unm.edu. Learn why this is important

To view this discussion visit https://groups.google.com/d/msgid/bitcurator-users/BN7PR07MB4194DFDCE8AA24ECF3550847BB812%40BN7PR07MB4194.namprd07.prod.outlook.com.

[Joe Carrano]

I do very similar things as your commands. I make a copy of the cloud service provided non-standard checksums if available as well using hashsum (i.e. hashsum dropbox)

For Google drive, I record which files were google objects, since those are exported as derivatives:

rclone lsf --csv --format ptm -R --files-only --drive-show-all-gdocs --metadata-include "vnd.google-apps.*" googledrive:"folder" > /path/to/export/orig_in_gdrive_formats.csv

To view this discussion visit https://groups.google.com/d/msgid/bitcurator-users/YQ1PR01MB11035285B3DF3CDD5568B0B2ADE3812%40YQ1PR01MB110352.CANPRD01.PROD.OUTLOOK.COM.

[James Truitt]

At the risk of going slightly off-topic, I'm just going to put in a good word for the rclone ncdu command, which I've found the be a very helpful tool for exploring remotes and getting a handle on where the bulk of storage is being used.