Testing whether Dokany+Mirror is crazy-solid

[John Oberschelp]

To see if Dokany+Mirror is crazy-solid, I am trying to run GIMP Portable off of a mirror of my C drive.

I can install it (pretty good!), but I can't run it.

Details: Here is the cleanest way I reproduce the problem:

1) Install fresh Windows 8.1 x64 on my test machine and do all Windows updates.

2) Install Visual Studio runtime dlls by going to...

         www.microsoft.com/en-US/download/details.aspx?id=40784

... and downloading and running both the x86 and x64 installers.

3) Download the Dokany installer (DokanInstall_0.7.3-RC.exe) and run it.

4) In an admin Command Prompt window, go to...

         C:\Program Files\Dokan\DokanLibrary\sample\mirror

… and run...

         mirror.exe /r c:\ /l m

4) Download the Portable GIMP installer (GIMPPortable_2.8.14-fix.paf.exe) to the test machine and run it from...

        M:\Users\johno\Downloads

Up to here, all is well with the world.

Go to folder...

         M:\Users\johno\Downloads\GIMPPortable

… and try to launch GIMPPortable.exe

I get a dialog...

GNU Image Manipulation Program

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

GNU Image Manipulation Program has stopped working

The Event Viewer says Application Error, Event ID 1000, Task Category 100

Of course, the GIMP Portable app works via the C drive. I have tried many variations but I am basically shooting in the dark.

Any idea what I should do/try/report/test/log next?

Is Dokany+Mirror robust enough for this?

I don't want to report it as a bug if I am just doing something stupid or if this is not reproducible by others.

Alternatively, is there another torture test others are using?

[John Oberschelp]

28 minutes after I posted I thought of trying with only 1 thread (mirror.exe /r c:\ /l m /t 1).
Portable GIMP runs (with non-critical errors (related to only having one filesystem thread?)) from my mirrored M: drive!
So is there non-threadsafe code in dokany or mirror?

[maxime....@gmail.com]

Mhh working on my side with GIMP Portable and this with 1 or more thread.
But according to your tests it seems to be a thread issue yes (maybe your environment is faster than my). Could you enable debug output on mirror.exe when using multiple threads to see where it hangs?

Looks like a bug yes (btw this issue reported on DokanX seems to be related: https://github.com/BenjaminKim/dokanx/issues/16).

[Alejandro Exojo]

The mirror code I've seen seems thread safe. That said, an application
can be prone to race conditions since the filesystem (in general) is
like a big global data structure, and by default operations are not
atomic. Coding the filesystem in user space as a non-privileged
application is even trickier, and there might be things that are
impossible to do. There are examples of deadlocks mentioned in the FUSE
documentation, so this is not like a Dokan problem per se:

http://fuse.sourceforge.net/doxygen/


That said, I would be very interested in the log as well. I learned that
one thing was implementing the DokanOperations in isolation, and a
different one making real world uses of the file system work as expected.

Greetings.

[John Oberschelp]

"Curiouser and curiouser!" Cried Alice (she was so much surprised, that for the moment she quite forgot how to speak good English).”
-- Lewis Carroll, Alice's Adventures in Wonderland & Through the Looking-Glass

In short, I turned on the debug flag, and it worked perfectly.
Then I turned the debug flag off, and it hung as before at the start of the app launch.

As usual, I removed the GIMPPortable folder, rebooted, but this time I mounted mirror with...
        mirror.exe /r c:\ /l m /d
... that is with debug output.
As usual, from M:\Users\johno\Downloads, I ran the installer...
        GIMPPortable_2.8.14-fix.paf.exe
... and entered...

        M:\Users\johno\Downloads\GIMPPortable

... and ran GIMPPortable.exe
It worked. GIMP started fine. Not a single byte went to the mirror.exe debug console window.

So I exited GIMP and closed the cmd.exe window that launched my mirror.
And as usual, I removed the GIMPPortable folder, but this time, after reboot, I mounted mirror with...

mirror.exe /r c:\ /l m

... that is without debug output.
Ran the installer from M fine.
Tried to run GIMPPortable.exe

Instantly brought up the dialog...
GIMP Portable (PortableApps.com Launcher)
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
GIMP Portable (PortableApps.com Launcher) has stopped working

And under View Problem Details
Problem Event Name APPCRASH
Application Name GIMPPortable.exe
Application VFersion 2.2.0.0
Application Timestamp 4f47e2df
exception Code c0000005
Exception Offset 00003a00
OS Version 6.3.9600.2.0.0.768.101

I think last week I tried turning on GIMPPortable's debug with its command line switch, and also got nothing, but I will try that again.
Thanks for the replies. Any suggestions on what I should try next?

[John Oberschelp]

Maybe GIMPPortable was a poor choice on my part for stress testing, as GIMPPortable uses pythonw which may have its own threading issues.
I would love to know what process others have used to test dokany+mirror before using it as a base for their code, particularly any threading test.
   
Here are more concise test results.
Never reinstalling GIMPPortable, I can always run it from the C drive or single-threaded M drive, back and forth, over and over, with or without reboot.
If I run with default-threaded M, I get either:
    "This app can't run on your PC" or

    "GNU Image Manipulation Program has stopped working"

If I add the /d switch (mirror.exe /r c:\ /l m /d), I get a dialog...
pythonw.exe
-=-=-=-=-=-=-=-=
pythonw.exe has stopped working

After closing this error dialog, the launch of GIMPPortable completes!

As promised, I retried the GIMP --verbose switch but I die before any output.
i.e. Launching mirror with default threads and running a batch file with...
    M:
    cd M:\Users\johno\Downloads\GIMPPortable
    GIMPPortable.exe --verbose >C:\Users\johno\jao_dokany\log.txt
... produces an empty log.txt file.

[Liryna]

Some month ago, I made the same test as you but with my implementation not the mirror.exe.
I was using ProcessExplorer as portable exe. https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
ProcessExplorer extract him self in 32 or 64 bits in the same folder. So you will get a good amount of read/write procceed and if some failed the process will crash :)

I remember that I had to fix some code and make it thread safe to handle the heavy test.

[Alejandro Exojo]

On 23 June 2015 at 19:41, John Oberschelp <jo...@oberschelp.net> wrote:
> I would love to know what process others have used to test dokany+mirror
> before using it as a base for their code, particularly any threading test.

This seems a good enough base to, at least, test all the possible operations:

https://github.com/BenjaminKim/FileSystemTest

Note that even without using threads in the mirror application, I
would say that you could have race conditions that the Dokan library
doesn't protect you from (or so I think). For example, you could have
a call to DeleteDirectory, and you check that the directory is
deletable because is empty, so you return success. Then you get a call
to create a file inside it, before you were able to remove the
directory in Cleanup.

[Liryna]

For example, you could have
a call to DeleteDirectory, and you check that the directory is
deletable because is empty, so you return success. Then you get a call
to create a file inside it, before you were able to remove the
directory in Cleanup.

 

If you implementation create the condition that your directory have to be empty for delete, it is normally that it is you who have to handle that your directory need to keep clean.

Otherwise if you keep being simple as the mirror, you should have no problem.
The mirror only forward driver call to user space IO that handle race.

[John Oberschelp]

Thank you for the feedback Liryna and Alejandro. I just noticed it going here to post, so I have not processed your posts yet.

I am pretty sure there is an intermittent multithreading bug in Dokany+Mirror at job initiation.

Could someone please try to reproduce this erratic behavior?

Setup

Install fio (http://git.kernel.dk/?p=fio.git) (Windows binaries at http://www.bluestop.org/fio/)

Save this 8-line test script next to fio.exe as...

trivial.fio

-=-=-=-=-=-==-=-=-

[global]

ioengine=windowsaio

rw=readwrite

size=1g

directory=C\:\fio

thread=1

[trivial-readwrite-1g]

-=-=-=-=-=-=-=-=-=-=-=-

Note that in the line “directory=C\:\fio”, that first backslash is an intentional escape character.

Tests

Each test I ran either:

Worked: Runs fine without error.

Stopped: Immediately produces the error dialog “fio.exe has stopped working”.

SystemError: Immediately stops with “fio.exe – System Error, The program can't start because *.dll is missing from your computer” where the asterisk was actually a bullet.

Just to make sure it works on the C drive, go to fio.exe on your C drive, and repeatedly run “fio.exe trivial.fio”. My results: Worked x 25 (Worked every time.)

Start Mirror with 1 thread, go to fio.exe on your M drive, and repeatedly run ”fio.exe trivial.fio” again.

My results: Worked x 25

Restart Mirror with default threads, go to fio.exe on your M drive, and repeatedly run ”fio.exe trivial.fio” again.

My results: Worked, Stopped x 2, Worked, Stopped, Worked x 3, Stopped x 3, Worked

Restart Mirror with default threads in debug mode, go to fio.exe on your M drive, and repeatedly run ”fio.exe trivial.fio”.

My results: Stopped, Worked x 11, Stopped (no debug output at all)

Restart Mirror with 9 threads, go to fio.exe on your M drive, and repeatedly run ”fio.exe trivial.fio”.

My Results: SystemError, Stopped, Worked x 5

[John Oberschelp]

Progress!
I captured and have attached logs of both a successful and a failed run of fio under Dokany+Mirror
It is easy to reproduce (though it only fails for me about 20% of the time) with...
        mirror.exe /r c:\ /l m /d /s
        fio.exe trivial.fio
(fio.exe and trivial.fio are explained in my previous post. For those not familiar, fio is an excellent I/O test tool.)

The two logs no longer compare at all (using WinMerge to compare) starting at line 1189, where the one that fails has...
GetFileSecurity c:\\Users\johno\Desktop\fio-2.2.8-x64\fio.exe
        invalid handle

Possibly related, possibly not, both report an ERROR_INSUFFICIENT_BUFFER earlier.

Again, could someone reproduce please? Or help me track down the problem?