ANNOUNCE: WinFsp - Windows File System Proxy
1,512 views
Skip to first unread message
Bill Zissimopoulos
Apr 26, 2016, 12:24:30 AM4/26/16
to Dokan
I am releasing today a new open source project called WinFsp, the Windows File System Proxy. This is a new solution for developing user mode file systems for Windows, which I have been developing for the past 5 1/2 months.
Some of the benefits and features that WinFsp supports out of the box are listed below:
* Allows for easy development of file systems in user mode. There are no restrictions on what a process can do in order to implement a file system (other than respond in a timely manner to file system requests).
* Support for disk and network based file systems.
* Support for NTFS level security and access control.
* Support for memory mapped files, cached files and the NT cache manager.
* Support for file change notifications.
* Support for file locking.
* Correct NT semantics with respect to file sharing, file deletion and renaming.
This project is currently in "works in my VM" alpha status. It passes its own test suite and some general use testing. Additional testing by a wider audience will be required before the project can be considered stable.
The project is released under the AGPLv3 and is available at this location:
More information about the project, a design document and the API reference can be found at the project's website at this location:
A few final notes regarding this effort vs Dokan/Dokany. Some of you may question why I chose to create a new solution vs working to improve Dokany. Some of you may remember that I even contributed some code to Dokany. The truth is that I spend quite some time weighing the pros and cons of both approaches. In the end I felt that I had the best chance of success if I created a project separate from Dokany.
Bill Zissimopoulos
Maxime C
Jun 4, 2016, 5:11:22 PM6/4/16
to Dokan
Hi Bill,
I finally got some time to review your project for real. First thanks for sharing, and let's be honest it's good work.
Before all, Dokany initiative is an attempt to unify the community who died between 2011-2014 and I can say it's a success today. So of course I'm not that happy with your choice to go on a new project instead of trying to improve Dokan, the Windows kernel open source community being already very small and you was an asset on Dokan being knowledgeable and with free time. But well... it's your choice.
Anyway, let's be constructive here ;).
I would be interested to know this pros/cons list if it differs from GitHub issues list, to be at least aware of it as your project was out of the blue for me.
I can see you made some radically different architecture choices compared to Dokan, could you argument? I can guess few of them (launcher, cache manager, ...) but I believe making it publicly explicit here would also benefits to general knowledge.
I finally got some time to review your project for real. First thanks for sharing, and let's be honest it's good work.
Before all, Dokany initiative is an attempt to unify the community who died between 2011-2014 and I can say it's a success today. So of course I'm not that happy with your choice to go on a new project instead of trying to improve Dokan, the Windows kernel open source community being already very small and you was an asset on Dokan being knowledgeable and with free time. But well... it's your choice.
Anyway, let's be constructive here ;).
I would be interested to know this pros/cons list if it differs from GitHub issues list, to be at least aware of it as your project was out of the blue for me.
I can see you made some radically different architecture choices compared to Dokan, could you argument? I can guess few of them (launcher, cache manager, ...) but I believe making it publicly explicit here would also benefits to general knowledge.
Bill Zissimopoulos
Jun 6, 2016, 6:42:02 PM6/6/16
to do...@googlegroups.com
Hi, Maxime:
I finally got some time to review your project for real. First thanks for sharing, and let's be honest it's good work.
Thank you.
Before all, Dokany initiative is an attempt to unify the community who died between 2011-2014 and I can say it's a success today. So of course I'm not that happy with your choice to go on a new project instead of trying to improve Dokan, the Windows kernel open source community being already very small and you was an asset on Dokan being knowledgeable and with free time. But well... it's your choice.
I agree that Dokan is a success. But I also believe that competition is a good thing. It spurs people to create greater and better things. I expect that both Dokan and WinFsp will improve because of competition rather than the opposite.
Anyway, let's be constructive here ;).
I would be interested to know this pros/cons list if it differs from GitHub issues list, to be at least aware of it as your project was out of the blue for me.
It is hard to present such a list in a sincere manner without ruffling feathers and perhaps starting a flame war. So please keep in mind that everything I say below is just personal (and perhaps uninformed) opinion and nothing else.
In August 2015 I was looking for a way to port one of my file systems to Windows. I had not done any Windows work in a long time (since I left Microsoft about 8 years ago) and was therefore quite happy when I found Dokany. I quickly ported my file system
to Windows and Dokany and thought I would have to fix a few things before I could say "JOB DONE".
But then I started trying to use the ported file system and started discovering interesting breakages. For example, I would try to copy files in Windows Explorer and files with empty names would appear. Or I could delete directories using del in the command
prompt. Or memory mapped files would not work. Or... or... or...
At first I blamed my file system, but soon I realized that these problems originated with Dokan. No problem, let's take a look at the code and try to fix some bugs. I think at that point I engaged the Dokan community for the first time. I started sending
some bug reports, later some patches and pull requests.
At some point I decided that Dokan had some very serious issues:
- Broken file system semantics that went beyond simple bug fixing. Some of them seemed to be a fundamental misunderstanding of how the Windows file system works.
- A lot of design decisions in it appeared to be the result of stumbling along rather than rigorous thinking of how to achieve a goal (e.g. notification thread).
- The code base looked like it was originally written by a brilliant, but inexperienced intern.
Obviously my confidence in the project plummeted and I started considering my alternatives:
- Abandon this effort altogether.
- Try to fix what I considered broken.
- A rather difficult and unwelcome task as by this time I considered the Dokany code base fundamentally flawed.
- How could I get the issues fixed at the pace I wanted anyway? Would my changes even get accepted? I did not control the project after all.
- Perhaps I should just fork the project? But then I would have no community support and I would still be constrained by Dokan's architecture and API.
- Maybe I should create a new independent project?
- Bad idea. I no longer develop for Windows and I have never written an FSD anyway.
- Plus it would take so long!
In the end the last choice won and I created WinFsp, primarily because this would give me complete control to take the project where I wanted to take it. Admittedly I thought it would take me 2-3 months to get it going and it took 4-5.
I can see you made some radically different architecture choices compared to Dokan, could you argument? I can guess few of them (launcher, cache manager, ...) but I believe making it publicly explicit here would also benefits to general knowledge.
WinFsp currently provides two main areas of functionality:
- A design and API for writing user mode file systems. Similar to Dokan in functionality although rather different under the hood.
- A small framework for writing user mode file systems as Windows services. This framework is optional, a user mode file system does not have to use it.
WinFsp consists of an FSD and DLL components. Other than feature support (e.g. support for the Cache Manager, Filter Manager, etc.) WinFsp differs from Dokan in two important areas:
- Opening the WinFsp control device creates a "volume device object" and either creates a virtual volume device object or registers with the MUP. Closing the control device handle automatically frees the associated device objects (and MUP connections). There is no need for an explicit unmount operation. If the user mode file system process dies, the kernel mode file system structures get automatically cleaned up.
- WinFsp has a capable but simple mechanism for marshaling IRP's to a user mode file system. The mechanism supports performing actions in the context of the originating process and/or the user mode file system process: this allows for zero-copy reads/writes, easily marshaling security context, etc. In the worst case scenario (ignoring pathological cases where it has to retry for locks, etc.) it will do no more than two context switches per file system request (in practice it does less because it uses ideas from IOCP's in its fundamental FSP_IOQ mechanism).
I encourage people who want to understand more about the WinFsp architecture to read the following documents:
- WinFsp Design. Describes the FSD and DLL components.
- WinFsp Service Architecture. Describes the (optional) service model.
>>
Maxime, I understand that my criticism of Dokan above may appear harsh and I also understand that it invites criticism of my own work. Despite this I want to say that I believe that you and your team have done an outstanding job so far in breathing new
life into this project and it is proven by the traction that your project enjoys within the open source community. And if it was not for Dokan, there would be no WinFsp either. I would perhaps have looked at the landscape last August and said: "oh, so there
is no FUSE for Windows; oh, well...", perhaps for the benefit of everyone involved.
Cheers.
Bill
Maxime C
Jul 22, 2016, 2:00:56 AM7/22/16
to Dokan
Hi Bill,
No offense here don't worry and thank you for taking the time to answer.
I generally agree(d) with your remarks, it was more or less the same ones I did myself when we tried to bring Dokan project back to life. But at this time we decided that fixes and improvement could be done instead of starting from ground zero again. I believe we are now getting better and closer to a good and stable user mode file system solution.
I saw your posts on reddit recently, I bet you're trying to have some testers for your releases now and to build a community. Trying to push WinFSP to Cygwin through your FUSE wrapper was also a nice move, but I would hold one. Even if you got hundreds of stars on your project recently, almost no one tried it. Tests, code review and even use are done by yourself *only* whereas Dokan on its side is tested by hundreds of people with thousands of active installations. Pushing it to Cygwin is far too early and if this is accepted by Cygwin I will have to get involved so that users can choose between WinFSP or Dokan.
We will see in the long run if you took the right decision with WinFSP architecture compared to Dokan and where your project will go when some people will really start to use it.
I'm not raising for a competition here just an attempt to have a good user mode file system driver platform for Windows ; this is all what Dokan project revival was about anyway.
No offense here don't worry and thank you for taking the time to answer.
I generally agree(d) with your remarks, it was more or less the same ones I did myself when we tried to bring Dokan project back to life. But at this time we decided that fixes and improvement could be done instead of starting from ground zero again. I believe we are now getting better and closer to a good and stable user mode file system solution.
I saw your posts on reddit recently, I bet you're trying to have some testers for your releases now and to build a community. Trying to push WinFSP to Cygwin through your FUSE wrapper was also a nice move, but I would hold one. Even if you got hundreds of stars on your project recently, almost no one tried it. Tests, code review and even use are done by yourself *only* whereas Dokan on its side is tested by hundreds of people with thousands of active installations. Pushing it to Cygwin is far too early and if this is accepted by Cygwin I will have to get involved so that users can choose between WinFSP or Dokan.
We will see in the long run if you took the right decision with WinFSP architecture compared to Dokan and where your project will go when some people will really start to use it.
I'm not raising for a competition here just an attempt to have a good user mode file system driver platform for Windows ; this is all what Dokan project revival was about anyway.
Bill Zissimopoulos
Jul 22, 2016, 3:39:13 AM7/22/16
to Dokan
On Thursday, July 21, 2016 at 11:00:56 PM UTC-7, Maxime C wrote:
Even if you got hundreds of stars on your project recently, almost no one tried it. Tests, code review and even use are done by yourself *only* whereas Dokan on its side is tested by hundreds of people with thousands of active installations.
You are right! The reviews and emails I have received so far are in general very favorable, but WinFsp needs a lot more testing. I invite all people on this list to try it and let me know what they think and if they find something broken I will fix it:
If someone wishes to discuss specific issues about WinFsp they can also use the Google Group:
Finally I would be happy to work with Dokany if you believe we could come up with a plan that would benefit both our projects and the community.
Bill
timofonic timofonic
Sep 19, 2016, 9:00:18 PM9/19/16
to do...@googlegroups.com
Hello to everyone.
Excuse me. I'm just a lurker and an user, but I got tired of certain attitudes.
" Pushing it to Cygwin is far too early and if this is accepted by
Cygwin I will have to get involved so that users can choose between
WinFSP or Dokan."
I see a lot of ego's here, very typical in FOSS and even more these
days (unfortunately). Why not have an objective analysis about what
the best approach (WinFSP vs Dokan) and put the best features on that
one? Why waste resources on a project that has scarce ones? That's why
there're GIANT AMOUNTS of incomplete FOSS out there!
Why does WinFSP/Dokan need to be Windows specific? Can't they get
merged into FUSE and become the Windows backends of that technology?
FUSE had a boom for some time. Despite of that, these days too many
projects are abandoned or slowly maintained,
I see some wrong things in FOSS that nobody seems to be able to
address, I hope someday able to learn programming and contribute to
solve it:
- An implementation of $whatever technology requires extra effort to
port it or developers only care about their own platform:
1. Works Linux but not on BSD.
2. Works on Windows but not on other platforms
(https://en.wikipedia.org/wiki/Category:Windows-only_free_software).
This may be seen as a minioritary attitude but it happens in projects,
some of them are very interesting. SumatraPDF is a interesting
document viewer as it's really lightweight and supports usually more
formats than Evince and Okular.
- Projects too much attached to certain toolkit or not having a clear
multiplatform nature, despite having a potential multiplatform nature
to avoid vendor lock-in and actively promote interoperability.
* Libpurple/Pidgin plus their KDE (Telepathy) and Gnome (Empathy)
counterparts are half-baked and severly underfundedl. As a result,
they aren't really available outside of Linux/BSD (that GSoC Android
port is from 2015 and suffering the effects of bitrotting!!!) and lack
important features like VoIP and file transfer for most of their
protocols.
- Projects with too much polarized & zealotist ecosystems that are
essentially the same. This finally made projects with inefficient but
brute force approachs to succeed to most of the userbase, many times
being *proprietary software*:
I consider the Text Wars to be really stupid and destructive: Vim vs
Emacs? They are outdated and limited pieces of software, using
scripting languages that aren't attractive to old (there are, but less
each time) & new school people that is able to program.
* I think they ought to distilling the best of their ideas and form a
disruptive project as they were in their first days, instead living
from their legacy glory. There's hope in NeoVim as it wants to replace
VimL to Lua, but they're in early WIP and they don't consider the VimL
to Lua translator and actively go to ditch VimL to not be a top
priority.
Despite I'm mostly a Linux user, I prefer FOSS not have restrictions
in usage and go to the multiplatform path. That's why LibreOffice is
so successful.
Dukan/WinFSP/FUSE are frameworks that can be used by power users and
end-user applications, so making things as easy as possible to
developers should be a top priority.
Kind regards.
> --
> You received this message because you are subscribed to the Google Groups
> "Dokan" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dokan+un...@googlegroups.com.
> To post to this group, send email to do...@googlegroups.com.
> Visit this group at https://groups.google.com/group/dokan.
> For more options, visit https://groups.google.com/d/optout.
Excuse me. I'm just a lurker and an user, but I got tired of certain attitudes.
" Pushing it to Cygwin is far too early and if this is accepted by
Cygwin I will have to get involved so that users can choose between
WinFSP or Dokan."
days (unfortunately). Why not have an objective analysis about what
the best approach (WinFSP vs Dokan) and put the best features on that
one? Why waste resources on a project that has scarce ones? That's why
there're GIANT AMOUNTS of incomplete FOSS out there!
Why does WinFSP/Dokan need to be Windows specific? Can't they get
merged into FUSE and become the Windows backends of that technology?
FUSE had a boom for some time. Despite of that, these days too many
projects are abandoned or slowly maintained,
I see some wrong things in FOSS that nobody seems to be able to
address, I hope someday able to learn programming and contribute to
solve it:
- An implementation of $whatever technology requires extra effort to
port it or developers only care about their own platform:
1. Works Linux but not on BSD.
2. Works on Windows but not on other platforms
(https://en.wikipedia.org/wiki/Category:Windows-only_free_software).
This may be seen as a minioritary attitude but it happens in projects,
some of them are very interesting. SumatraPDF is a interesting
document viewer as it's really lightweight and supports usually more
formats than Evince and Okular.
- Projects too much attached to certain toolkit or not having a clear
multiplatform nature, despite having a potential multiplatform nature
to avoid vendor lock-in and actively promote interoperability.
* Libpurple/Pidgin plus their KDE (Telepathy) and Gnome (Empathy)
counterparts are half-baked and severly underfundedl. As a result,
they aren't really available outside of Linux/BSD (that GSoC Android
port is from 2015 and suffering the effects of bitrotting!!!) and lack
important features like VoIP and file transfer for most of their
protocols.
- Projects with too much polarized & zealotist ecosystems that are
essentially the same. This finally made projects with inefficient but
brute force approachs to succeed to most of the userbase, many times
being *proprietary software*:
I consider the Text Wars to be really stupid and destructive: Vim vs
Emacs? They are outdated and limited pieces of software, using
scripting languages that aren't attractive to old (there are, but less
each time) & new school people that is able to program.
* I think they ought to distilling the best of their ideas and form a
disruptive project as they were in their first days, instead living
from their legacy glory. There's hope in NeoVim as it wants to replace
VimL to Lua, but they're in early WIP and they don't consider the VimL
to Lua translator and actively go to ditch VimL to not be a top
priority.
Despite I'm mostly a Linux user, I prefer FOSS not have restrictions
in usage and go to the multiplatform path. That's why LibreOffice is
so successful.
Dukan/WinFSP/FUSE are frameworks that can be used by power users and
end-user applications, so making things as easy as possible to
developers should be a top priority.
Kind regards.
> You received this message because you are subscribed to the Google Groups
> "Dokan" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dokan+un...@googlegroups.com.
> To post to this group, send email to do...@googlegroups.com.
> Visit this group at https://groups.google.com/group/dokan.
> For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages