FTP virtual filesystem and encryption support
58 views
Skip to first unread message
Lukáš Jirkovský
Jul 18, 2016, 3:20:15 PM7/18/16
to SabreDAV Discussion
Hello everyone!
First of all I would like to thank anyone involved in SabreDAV, it's a great piece of software. I must compliment how well the source code is documented and structured. Thanks to that, I was able to start hacking in no time, despite having absolutely no knowledge of the modern PHP ecosystem. But back to the reason why I'm writing.
I have implemented a virtual filesystem that is backed by a FTP server [1] and I was thinking that somebody might find it useful. Would it be desirable if I integrated it into SabreDAV properly and created a pull request for this?
Now a real life use case of this virtual filesystem. I have a webhosting that doesn't allow use as a disk storage (presumably because the webhosting is hosted on SSDs, so the disk space is expensive). However, they will also give you a few gigabytes on a separate slower server that can be accessed using FTP for the file storage needs. With SabreDAV and this virtual file system I get a nice WebDAV interface to the FTP server, so I can use one of the many WebDAV-based synchronization tools.
On a related note, I began implementing file encryption support. Actually, the only thing remaining before publishing the code is to test it in the wild. I was thinking about making a plugin, but it seems it's not possible to change the file name in an event, so it would not be possible to encrypt the file names. So instead I implemented a virtual filesystem whose encrypted directory and file implementation encrypt file and directory names and file contents before passing them to a wrapped file or directory implementation. Would you be interested in a pull request for such feature?
[1] https://bitbucket.org/stativ/sabredav-ftpfs
Lukas
First of all I would like to thank anyone involved in SabreDAV, it's a great piece of software. I must compliment how well the source code is documented and structured. Thanks to that, I was able to start hacking in no time, despite having absolutely no knowledge of the modern PHP ecosystem. But back to the reason why I'm writing.
I have implemented a virtual filesystem that is backed by a FTP server [1] and I was thinking that somebody might find it useful. Would it be desirable if I integrated it into SabreDAV properly and created a pull request for this?
Now a real life use case of this virtual filesystem. I have a webhosting that doesn't allow use as a disk storage (presumably because the webhosting is hosted on SSDs, so the disk space is expensive). However, they will also give you a few gigabytes on a separate slower server that can be accessed using FTP for the file storage needs. With SabreDAV and this virtual file system I get a nice WebDAV interface to the FTP server, so I can use one of the many WebDAV-based synchronization tools.
On a related note, I began implementing file encryption support. Actually, the only thing remaining before publishing the code is to test it in the wild. I was thinking about making a plugin, but it seems it's not possible to change the file name in an event, so it would not be possible to encrypt the file names. So instead I implemented a virtual filesystem whose encrypted directory and file implementation encrypt file and directory names and file contents before passing them to a wrapped file or directory implementation. Would you be interested in a pull request for such feature?
[1] https://bitbucket.org/stativ/sabredav-ftpfs
Lukas
Allon Moritz
Jul 18, 2016, 3:27:06 PM7/18/16
to sabredav-discuss
I would be really interested in a pr!
--
You received this message because you are subscribed to the Google Groups "SabreDAV Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sabredav-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sabredav-discuss/c6b2bbbc-21bc-4da4-a890-3fa4a9f75bb6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Evert Pot
Jul 18, 2016, 3:53:06 PM7/18/16
to SabreDAV Discussion
Hi Lukáš,
A few things:
- Did you know you could also just use Sabre\DAV\FS\Directory('ftp://....') ? If so, any reason why didn't work?
- Instead of integrating this into sabre/dav, simply publish it as a separate composer package
- Don't use the Sabre/ PHP namespace!
As for encryption, it sounds like you are primarily talking about encryption on the storage-level, but send back the files unencrypted to the server. Why not do this on the Directory/File level? This might be a better place than a plugin, no?
Evert
Lukas Jirkovsky
Jul 19, 2016, 6:00:56 PM7/19/16
to sabredav...@googlegroups.com
On 18 July 2016 at 21:53, Evert Pot <ever...@gmail.com> wrote:
> Hi Lukáš,
PHP Warning: Sabre\DAV\FS\Directory::getChildren(): stream does not
support seeking in
/tmp/SabreDAV/vendor/sabre/dav/lib/DAV/FS/Directory.php on line 103
This can be solved by wrapping the \FilesystemIterator in \NoRewindIterator.
However, the bigger issue is that it is incredibly ineffective. It's
so slow to a point that it becomes absolutely unusable.
I did a simple test, where I used a directory with 100 small text
files containing "Hello World!". I started up wireshark and started
capturing the FTP packets and tried listing the directory. The
operation takes 90 seconds and 4029 FTP packets are sent. With FTPFS I
get the listing after 20 seconds and 1614 packets. The main problem is
that the ftp:// wrapper creates a new connection for every file. FTPFS
could be even faster, but because the server tends to close the
connection after checking every few files, I added a one-seconds sleep
and reconnect if a function call fails before calling the function
again. With larger number of files (~200) the wrapper version usually
fails with a timeout after several minutes.
There's also a little issue that the ftp:// wrapper requires
allow_url_fopen, which may be disabled by some servers.
> Instead of integrating this into sabre/dav, simply publish it as a separate
> composer package
Does that mean this feature is outside the scope of SabreDAV and as
such, pull request would be rejected?
> Don't use the Sabre/ PHP namespace!
I think if I created a pull request the namespace should be kept as
/Sabre as it would become part of the SabreDAV project. But if the
FTPFS is kept as a separate project, I will definitely change the
namespace to something else.
> As for encryption, it sounds like you are primarily talking about encryption
> on the storage-level, but send back the files unencrypted to the server. Why
> not do this on the Directory/File level? This might be a better place than a
> plugin, no?
Yes, it's on the storage level. It's basically a wrapper (or
decorator, if you want) around the existing File and Directory
implementation. The basic usage is like this:
$root = new \Sabre\DAV\FS\Directory('/webdav/');
$encryptedRoot = new \FSencrypt\Directory($root);
The $encryptedRoot will itself wrap the children in
\FSencrypt\Directory or \FSencrypt\File as needed. Both encrypted
Directory and File handle encryption and decryption of file names and
file data before passing them to a wrapped Directory/File
implementation.
Lukas
> Hi Lukáš,
> A few things:
>
> Did you know you could also just use Sabre\DAV\FS\Directory('ftp://....') ?
> If so, any reason why didn't work?
This does not work as is. The reason is this warning:
>
> Did you know you could also just use Sabre\DAV\FS\Directory('ftp://....') ?
> If so, any reason why didn't work?
PHP Warning: Sabre\DAV\FS\Directory::getChildren(): stream does not
support seeking in
/tmp/SabreDAV/vendor/sabre/dav/lib/DAV/FS/Directory.php on line 103
This can be solved by wrapping the \FilesystemIterator in \NoRewindIterator.
However, the bigger issue is that it is incredibly ineffective. It's
so slow to a point that it becomes absolutely unusable.
I did a simple test, where I used a directory with 100 small text
files containing "Hello World!". I started up wireshark and started
capturing the FTP packets and tried listing the directory. The
operation takes 90 seconds and 4029 FTP packets are sent. With FTPFS I
get the listing after 20 seconds and 1614 packets. The main problem is
that the ftp:// wrapper creates a new connection for every file. FTPFS
could be even faster, but because the server tends to close the
connection after checking every few files, I added a one-seconds sleep
and reconnect if a function call fails before calling the function
again. With larger number of files (~200) the wrapper version usually
fails with a timeout after several minutes.
There's also a little issue that the ftp:// wrapper requires
allow_url_fopen, which may be disabled by some servers.
> Instead of integrating this into sabre/dav, simply publish it as a separate
> composer package
such, pull request would be rejected?
> Don't use the Sabre/ PHP namespace!
/Sabre as it would become part of the SabreDAV project. But if the
FTPFS is kept as a separate project, I will definitely change the
namespace to something else.
> As for encryption, it sounds like you are primarily talking about encryption
> on the storage-level, but send back the files unencrypted to the server. Why
> not do this on the Directory/File level? This might be a better place than a
> plugin, no?
decorator, if you want) around the existing File and Directory
implementation. The basic usage is like this:
$root = new \Sabre\DAV\FS\Directory('/webdav/');
$encryptedRoot = new \FSencrypt\Directory($root);
The $encryptedRoot will itself wrap the children in
\FSencrypt\Directory or \FSencrypt\File as needed. Both encrypted
Directory and File handle encryption and decryption of file names and
file data before passing them to a wrapped Directory/File
implementation.
Lukas
Evert Pot
Jul 19, 2016, 9:52:11 PM7/19/16
to SabreDAV Discussion
This explanation makes total sense. Still think we should fix the seeking issue, but I can definitely see that you get quite a bit of optimizations by doing it this way.
> Instead of integrating this into sabre/dav, simply publish it as a separate
> composer package
Does that mean this feature is outside the scope of SabreDAV and as
such, pull request would be rejected?
If we were to adopt it in the project, I would still want to do this as a separate package. There's other packages (such as sabre/xml, sabre/vobject), so it makes a lot of sense that this would too.
However, I don't think it's at the quality level of a core package yet. In the meantime I would suggest to publish it under your own namespace / vendor prefix.
If you really want to aim to make this a sabre.io package, then it should
* Follow all the same coding standards
* Be 100% unittested
* Have documentation written for sabre.io
I don't have a ton of time, so I can't do this for you, but if those tasks are all done I would be happy to make this official. However, if you choose not to, for whatever reason, I would still be more than happy to dedicate a page on sabre.io discussing how to setup and use your package. So ultimately it's up to you.
Here's an example of a different package that also uses sabre/dav for a different purpose:
https://github.com/1afa/sambadav
Evert
Lukas Jirkovsky
Jul 20, 2016, 4:07:42 PM7/20/16
to sabredav...@googlegroups.com
Hello Evert
but getting it covered with unit tests will certainly take some time,
as I don't expect the ftp_* functions to be easily testable. Until
then I'll use my own prefix.
> If you really want to aim to make this a sabre.io package, then it should
>
> * Follow all the same coding standards
> * Be 100% unittested
> * Have documentation written for sabre.io
>
> I don't have a ton of time, so I can't do this for you, but if those tasks
> are all done I would be happy to make this official. However, if you choose
> not to, for whatever reason, I would still be more than happy to dedicate a
> page on sabre.io discussing how to setup and use your package. So ultimately
> it's up to you.
I don't expect you to do this work for me, it's my itch after all :-)
It's nice to hear that it would be welcome addition once it meets all
quality requirements.
> Here's an example of a different package that also uses sabre/dav for a
> different purpose:
>
> https://github.com/1afa/sambadav
>
Thank you for the link! I'm sure it would be helpful to see how more
experienced ones handle packages for SabreDAV
Lukas
>
> If we were to adopt it in the project, I would still want to do this as a
> separate package. There's other packages (such as sabre/xml, sabre/vobject),
> so it makes a lot of sense that this would too.
> However, I don't think it's at the quality level of a core package yet. In
> the meantime I would suggest to publish it under your own namespace / vendor
> prefix.
This sounds reasonable. I tried to follow the code style as possible,
> If we were to adopt it in the project, I would still want to do this as a
> separate package. There's other packages (such as sabre/xml, sabre/vobject),
> so it makes a lot of sense that this would too.
> However, I don't think it's at the quality level of a core package yet. In
> the meantime I would suggest to publish it under your own namespace / vendor
> prefix.
but getting it covered with unit tests will certainly take some time,
as I don't expect the ftp_* functions to be easily testable. Until
then I'll use my own prefix.
> If you really want to aim to make this a sabre.io package, then it should
>
> * Follow all the same coding standards
> * Be 100% unittested
> * Have documentation written for sabre.io
>
> I don't have a ton of time, so I can't do this for you, but if those tasks
> are all done I would be happy to make this official. However, if you choose
> not to, for whatever reason, I would still be more than happy to dedicate a
> page on sabre.io discussing how to setup and use your package. So ultimately
> it's up to you.
It's nice to hear that it would be welcome addition once it meets all
quality requirements.
> Here's an example of a different package that also uses sabre/dav for a
> different purpose:
>
> https://github.com/1afa/sambadav
>
experienced ones handle packages for SabreDAV
Lukas
Reply all
Reply to author
Forward
0 new messages