Rclone is a command-line program to manage files on cloud storage. Itis a feature-rich alternative to cloud vendors' web storageinterfaces. Over 70 cloud storage products supportrclone including S3 object stores, business & consumer file storageservices, as well as standard transfer protocols.
Rclone really looks after your data. It preserves timestamps andverifies checksums at all times. Transfers over limited bandwidth;intermittent connections, or subject to quota can be restarted, fromthe last good file transferred. You cancheck the integrity of your files. Wherepossible, rclone employs server-side transfers to minimise localbandwidth use and transfers from one provider to another withoutusing local disk.
Rclone is mature, open-source software originally inspired by rsyncand written in Go. The friendly supportcommunity is familiar with varied use cases. Official Ubuntu, Debian,Fedora, Brew and Chocolatey repos. include rclone. For the latestversion downloading from rclone.org is recommended.
First, you'll need to configure rclone. As the object storage systemshave quite complicated authentication these are kept in a config file.(See the --config entry for how to find the configfile and choose its location.)
rclone normally syncs or copies directories. However, if the sourceremote points to a file, rclone will just copy that file. Thedestination remote must point to a directory - rclone will give theerror Failed to create file system for "remote:file": is a file not a directory if it isn't.
If you supply extra configuration to a backend by command line flag,environment variable or connection string then rclone will add asuffix based on the hash of the config to the name of the remote, eg
Starting with rclone version 1.61, any Unicode numbers and letters are allowed,while in older versions it was limited to plain ASCII (0-9, A-Z, a-z). If you usethe same rclone configuration from different shells, which may be configured withdifferent character encoding, you must be cautious to use characters that arepossible to write in all of them. This is mostly a problem on Windows, wherethe console traditionally uses a non-Unicode character set - definedby the so-called "code page".
rclone uses : to mark a remote name. This is, however, a validfilename component in non-Windows OSes. The remote name parser willonly search for a : up to the first / so if you need to act on afile or directory like this then use the full path starting with a/, or use ./ as a current directory prefix.
Rclone only supports a one-time sync of metadata. This means thatmetadata will be synced from the source object to the destinationobject only when the source object has changed and needs to bere-uploaded. If the metadata subsequently changes on the source objectwithout changing the object itself then it won't be synced to thedestination object. This is in line with the way rclone syncsContent-Type without the --metadata flag.
Options that take parameters can have the values passed in two ways,--option=value or --option value. However boolean (true/false)options behave slightly differently to the other options in that--boolean sets the option to true and the absence of the flag setsit to false. It is also possible to specify --boolean=false or--boolean=true. Note that --boolean false is not valid - this isparsed as --boolean and the false is parsed as an extra commandline argument for rclone.
Note that the units are Byte/s, not bit/s. Typicallyconnections are measured in bit/s - to convert divide by 8. Forexample, let's say you have a 10 Mbit/s connection and you wish rcloneto use half of it - 5 Mbit/s. This is 5/8 = 0.625 MiB/s so you woulduse a --bwlimit 0.625M parameter for rclone.
If this flag is set then in a sync, copy or move, rclone will doall the checks to see whether files need to be transferred beforedoing any of the transfers. Normally rclone would start runningtransfers as soon as possible.
If both --check-first and --order-by are set when doing rclone movethen rclone will use the transfer thread to delete source files whichdon't need transferring. This will enable perfect ordering of thetransfers and deletes but will cause the transfer stats to have moreitems in than expected.
The fact that an existing file rclone.conf in the same directoryas the rclone executable is always preferred, means that it is easyto run in "portable" mode by downloading rclone executable to awritable directory and then create an empty file rclone.conf in thesame directory.
If the location is set to empty string "" or path to a filewith name notfound, or the os null device represented by value NUL onWindows and /dev/null on Unix systems, then rclone will keep theconfig file in memory only.
The file format is basic INI:Sections of text, led by a [section] header and followed bykey=value entries on separate lines. In rclone each remote isrepresented by its own section, where the section name defines thename of the remote. Options are specified as the key=value entries,where the key is the option name without the --backend- prefix,in lowercase and with _ instead of -. E.g. option --mega-hard-deletecorresponds to key hard_delete. Only backend options can be specified.A special, and required, key type identifies the storage system,where the value is the internal lowercase name as returned by commandrclone help backends. Comments are indicated by ; or # at thebeginning of a line.
Note that passwords are in obscuredform. Also, many storage systems uses token-based authentication insteadof passwords, and this requires additional steps. It is easier, and safer,to use the interactive command rclone config instead of manuallyediting the configuration file.
To reduce risk of corrupting an existing configuration file, rclonewill not write directly to it when saving changes. Instead it willfirst write to a new, temporary, file. If a configuration file alreadyexisted, it will (on Unix systems) try to mirror its permissions tothe new file. Then it will rename the existing file to a temporaryname as backup. Next, rclone will rename the new file to the correct name,before finally cleaning up by deleting the backup file.
If the configuration file path used by rclone is a symbolic link, thenthis will be evaluated and rclone will write to the resolved path, insteadof overwriting the symbolic link. Temporary files used in the process(described above) will be written to the same parent directory as thatof the resolved configuration file, but if this directory is also asymbolic link it will not be resolved and the temporary files will bewritten to the location of the directory symbolic link.
This option allows rclone to return exit code 9 if no files were transferredbetween the source and destination. This allows using rclone in scripts, andtriggering follow-on actions if data was copied, or skipping if not.
When using rclone via the API rclone caches created remotes for 5minutes by default in the "fs cache". This means that if you dorepeated actions on the same remote then rclone won't have to build itagain from scratch, which makes it more efficient.
Note that if you use some flags, eg --backup-dir and if this is setto 0 rclone may build two remotes (one for the source or destinationand one for the --backup-dir where it may have only built onebefore.
Normally rclone will look at modification time and size of files tosee if they are equal. If you set this flag then rclone will checkonly the modification time. If --checksum is set then it onlychecks the checksum.
With this option set, files will be created and deleted as requested,but existing files will never be updated. If an existing file doesnot match between the source and destination, rclone will give the errorSource and destination exist but do not match: immutable file modified.
Without --inplace (the default) rclone will first upload to atemporary file with an extension like this, where XXXXXX represents arandom string and .partial is --partial-suffix value(.partial by default).
Comma separated list of log format options. Accepted options are date,time, microseconds, pid, longfile, shortfile, UTC. Any otherkeywords will be silently ignored. pid will tag log messages with processidentifier which useful with rclone mount --daemon. Other acceptedoptions are explained in the go documentation.The default log format is "date,time".
This shouldn't need to be changed from the default in normal operations.However, if you get a lot of low level retries you may wishto reduce the value so rclone moves on to a high level retry (see the--retries flag) quicker.
On the local disk, rclone preallocates the file (usingfallocate(FALLOC_FL_KEEP_SIZE) on unix or NTSetInformationFile onWindows both of which takes no time) then each thread writes directlyinto the file at the correct place. This means that rclone won'tcreate fragmented or sparse files and there won't be any assembly timeat the end of the transfer.
Don't set Accept-Encoding: gzip. This means that rclone won't askthe server for compressed files automatically. Useful if you've setthe server to return files with Content-Encoding: gzip but youuploaded compressed files.
If you are only copying a small number of files (or are filtering mostof the files) and/or have a large number of files on the destinationthen --no-traverse will stop rclone listing the destination and savetime.
If this flag is used when rclone comes to upload a file it will checkto see if there is an existing file on the destination. If this filematches the source with size (and checksum if available) but has adiffering timestamp then instead of re-uploading it, rclone willupdate the timestamp on the destination file. If the checksum does notmatch rclone will upload the new file. If the checksum is absent (e.g.on a crypt backend) then rclone will update the timestamp.
Normally if you are doing a modification time sync rclone will updatemodification times without --refresh-times provided that the remotesupports checksums and the checksums match on the file. However if thechecksums are absent then rclone will upload the file rather thansetting the timestamp as this is the safe behaviour.
35fe9a5643