s3ql VS Rclone

[amit kertis]

can u please some one tell me what is the different between the s3ql VS Rclone

and also more simple explain about the different between s3ql VS s3fs

thanks amit 

 

[Super Strobi]

Op dinsdag 15 oktober 2019 11:18:37 UTC+2 schreef amit kertis:

can u please some one tell me what is the different between the s3ql VS Rclone

and also more simple explain about the different between s3ql VS s3fs

In short: s3ql provides a filesystem on local and cloud storage back-ends, in which you have to put your files/data (eg by tools like rsync, cp or rclone), whereas rclone is a synchronisation tool to local and cloud storage back-ends.

s3ql does provide, within that filesystem space, the option to encrypt of these files, but also deduplicate of these files. Rclone just "copies".

s3fs is a fuse based filesystem which only "works" on an s3 bucket, whereas s3ql runs on ample more cloud storage back-ends.

Hope this helps!

[amit kertis]

thanks

this is very helpful

so if i looking for the best tool for having my swift object storage mount to my local pc 

which one u recommended or u can give me something else to look on 

again thanks

Amit  

[Super Strobi]

Op woensdag 16 oktober 2019 09:13:51 UTC+2 schreef amit kertis:

thanks

this is very helpful

so if i looking for the best tool for having my swift object storage mount to my local pc 

which one u recommended or u can give me something else to look on 

Follow http://www.rath.org/s3ql-docs/backends.html#openstack-swift

Once implemented, you use rsync/rclone/... to "fill" that filesystem.

Good luck!

[Daniel Jagszent]

Hello Amit,

[...]so if i looking for the best tool for having my swift object storage mount to my local pc 

which one u recommended or u can give me something else to look on [...]

Then S3QL is not what you are looking for. When you use S3QL you *only* can access your data with S3QL.
S3QL cannot see anything that might already be there on your storage.
You cannot directly access files you uploaded with S3QL (That is you can but you will only see encrypted 10MB blobs)

You could use rclone and its FUSE filesystem feature (https://rclone.org/commands/rclone_mount/).
Another nice option for desktops (only Win/Mac) is Mountain Duck (https://mountainduck.io/)

If you do not need integration in your file system but only need something like an FTP-Program to access your data, try Cyberduck (https://cyberduck.io/).

[amit kertis]

i little bit confuse now

@Daniel say something else

i dont understand why u sat the s3ql is not good for me 

why i cant use it for mounting my swift to my local pc

can u please explain more 

S3QL doesn't give me a regular mount 

if i will do ls on my local pc i will not see the files on the swift storage ??

[Daniel Jagszent]

Hello Amit,

i little bit confuse now

@Daniel say something else

i dont understand why u sat the s3ql is not good for me 

why i cant use it for mounting my swift to my local pc

can u please explain more [...]

S3QL is a full blown file system that uses e.g. OpenStack Swift for saving data. Everything that S3QL stores on Swift will be encrypted. S3QL works with a database that holds vital metadata about your file system (e.g. the database holds the information that object "s3ql_data_2345" on Swift is the first 10 MB of the file "Vacation.mp4") without this metadata (and the encryption passphrase) you cannot make sense of the data that S3QL stores in Swift.

As I understand you, you want to store files in a Swift container (like the "Vacation.mp4" file from above) directly.  So that if you examine the contents of this container (e.g. with Cyberduck) you will directly see the file/ can download the file. That simply is not what S3QL does. Nearly all of the features of S3QL couldn't be done if it would work that way.

Maybe just try S3QL with a fresh OpenStack Swift Container. You will see what I mean then. Or if it is too complicated for a quick test, use the "Local" backend https://www.rath.org/s3ql-docs/backends.html#local and compare the contents of the mount point and the local storage backend directory.

[...] if i will do ls on my local pc i will not see the files on the swift storage ??

correct.

Maybe tell us what you want to do in more detail? What problem should S3QL solve for you?

[amit kertis]

Thanks for the details information
I will try to configure local S3QL and check it out With our swift
Hope not to get into configuration problems :)

In general what I am looking for is a stable tool that will give me the possibility to work with swift storage as file system that mean that I will be able to work with “cp” “mkdir” “ls” “rm” etc....
so I won’t need to use swift command For those action and if I will use different object storage every thing will continue to work with no need of changing the code

Thanks for the help

[Anthony DeRobertis]

With S3QL, you access you data only via S3QL. You would no more directly
access data in the Swift object storage than you would directly access
the data on /dev/sda1 if you were using a traditional filesystem. A file
on an S3QL filesystem does not map to an object in the underlying
storage (Swift). This allows S3QL to do things like de-duplicate blocks
and to have semantics much closer to a traditional filesystem.

Also, S3QL supports being mounted from at most one computer at a time.

S3FS presents S3 objects as files. If you put a file in S3FS, it'll be
available as a single S3 object. I think you can also have multiple S3FS
mounts of the same S3 objects, on different computers. S3FS does not act
like a traditional filesystem in a bunch of ways (see the "Limitations"
sections in their readme).

rclone is similar to S3FS.

[amit kertis]

Thanks again ,
So if I understand correctly, with s3ql if I will mount my local folder (Home/test ) to a swift container , and I will do “cp “ to a file to this folder , then if I will enter the folder ( cd home/test) I will NOT Be able to open The file or to copy it to other folder ?

Regarding “ A file on an S3QL filesystem does not map to an object in the underlying storage (Swift)“

That mean that I loss all the benefit of swift ?

May be u have some video that show what u try to explain me ?
Or any other video that will put more light on the S3qL behavior


If I try more simplify my needs , so I need that the developer will have the ability to mount their local folder to a swift container
And to continue working like they do a regular mount to a file storage

Thanks for the help
Amit

[Anthony DeRobertis]

On 10/18/19 1:43 AM, amit kertis wrote:
> Thanks again ,
> So if I understand correctly, with s3ql if I will mount my local folder (Home/test ) to a swift container , and I will do “cp “ to a file to this folder , then if I will enter the folder ( cd home/test) I will NOT Be able to open The file or to copy it to other folder ?
>
> Regarding “ A file on an S3QL filesystem does not map to an object in the underlying storage (Swift)“
>
> That mean that I loss all the benefit of swift ?

Let me try again...

If you mount an S3QL filesystem somewhere, you can copy files to it. You
can copy files from it. You can move around and rename files. It acts
like a normal filesystem, with all the behaviors you expect from that.

However, the objects S3QL will store on Swift are blocks, not whole
files. So if you access the Swift object store directly (using the
Object Storage API), you'll see s3ql_data_1, s3ql_data_2, s3ql_data_3,
etc. Those are the blocks that store your files. So you can't (for
example) make your file public with a URL. Or access it in any other
Swift-supporting application.

If you want to access your files *only* using S3QL, it works well for that.

If you want to access your files with some other Object Storage API
client, or via URLs, pick one of the other options.

[amit kertis]

Thank you !!! Now I understand it

If you want to access your files with some other Object Storage API
client, or via URLs, pick one of the other options.

What is the other options you recommend me ??