DokanNet: ReadFile readBytes offset Issue

[tombi]

Hello there,

I'm writing a filesystem application with DokanNet, which fetches data
over bluetooth from a device with a sd card So it's not very fast.
When I try to copy a file from my fs, the ReadFile function is called
with various buffer lengths. I have noticed that there is some
timeout, which aborts the copy process of my file, if my application
can not receive the data to be copied in the buffer fast enough. So I
decided to retrieve for example just 2048 bytes if the buffer length
is 32768. Of course I set the parameter readBytes to 2048. The next
time the ReadFile function is called, the offset is not incremented by
2048 (as it should be) but by 32768. After the copy process has
finished, my file has the right size, but is filled with zeros at the
positions which were skipped. I have detected that this problem only
occurs with buffer lengths less than 65536. With buffer length 65536
the offset is incremented correctly by 2048.

I have also compiled the driver and DokanNet from svn and watched the
kernel debug messages with DebugView. In this way I have detected that
the increment problem only occurs if the IRP flags IRP_PAGING_IO and
IRP_NOCACHE are set.

This issue is reproducible with the DokanNet mirror implementation if
you modify the ReadFile method so that it sets readBytes to a smaller
value than the buffer length.

So my question is: Is there a way to limit the buffer to a given
value, for example 2048 bytes or can I perhaps turn paging I/O off, so
that the offset will be incremented correctly by the value of
readBytes instead of the buffer length.

Thanks in advance!

tombi

[Peter Siegel]

Correct me of i am wrong,but as gar As i know when you return less
Bytes than expected , the App thinks its the end of the file...?

> --
> You received this message because you are subscribed to the Google
> Groups "Dokan" group.
> To post to this group, send email to do...@googlegroups.com.
> To unsubscribe from this group, send email to dokan+un...@googlegroups.com
> .
> For more options, visit this group at http://groups.google.com/group/dokan?hl=en
> .

[Hiroki Asakawa]

Hi, tombi

There was a bug in CurrentByteOffset in sys/read.c and I fixed it today.
But the bug only affects non paging IO.

> I have detected that this problem only
> occurs with buffer lengths less than 65536. With buffer length 65536
> the offset is incremented correctly by 2048.

Does buffer length mean the third parameter of ReadFile, NumberOfBytesToRead?

[tombi]

Hi,

thanks for your answers.

> <siegelpeter2...@gmail.com> wrote:
> > Correct me of i am wrong,but as gar As i know when you return less Bytes
> > than expected , the App thinks its the end of the file...?

I don't think so, because windows doesn't stop the copy process.
In read.c of the dokan.dll you can see the following:

if (status < 0) {
eventInfo->Status = STATUS_INVALID_PARAMETER;
} else if(readLength == 0) {
eventInfo->Status = STATUS_END_OF_FILE;
} else {
eventInfo->Status = STATUS_SUCCESS;
eventInfo->BufferLength = readLength;
}

So dokan indicates the EOF only if readLengh == 0.

On 27 Jan., 00:16, Hiroki Asakawa <asa...@gmail.com> wrote:
> There was a bug in CurrentByteOffset in sys/read.c and I fixed it today.
> But the bug only affects non paging IO.

I tried the new version, but, as you said, it doesn't solve my problem
unfortunately.

> Does buffer length mean the third parameter of ReadFile, NumberOfBytesToRead?

exactly.

I will try to change the timeout, as you wrote here:
http://groups.google.com/group/dokan/browse_thread/thread/4290b7d36b1bf677/049451d75e009cf0

Is there still no way to tell windows not to use paging IO or to limit
the NumberOfBytesToRead?

tombi

[tombi]

Hi again,

while trying out some things I made out that the copy process doesn't
time out if the read request is paging IO. In DebugView I see that
there is some timeout from dokan, but windows tries to read every 20
secounds again and again as you can see in the following kernel log (I
haven't changed the timeout, it's still 15.). I don't know if this is
a bug. Whatever, if the copy process uses ReadFile without paging IO
and there is a timeout I get the message "Insufficient system
resources exist to complete the requested service" and the copy
process stops.

==> DokanRead
ProcessId 4068
FileName:\test\51k_file.txt
ByteCount:32768 ByteOffset:0
Paging IO
Nocache
Synchronous IO
STATUS_PENDING
<== DokanRead
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
timeout Irp #7D
<== ReleaseTimeoutPendingIRP
==> DokanRead
ProcessId 4068
FileName:\test\51k_file.txt
ByteCount:4096 ByteOffset:0
Paging IO
Nocache
Synchronous IO
STATUS_PENDING
<== DokanRead
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
timeout Irp #7E
<== ReleaseTimeoutPendingIRP
==> DokanRead
ProcessId 4068
FileName:\test\51k_file.txt
ByteCount:4096 ByteOffset:0
Paging IO
Nocache
Synchronous IO
STATUS_PENDING
<== DokanRead
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> DokanCreate
ProcessId 2684
FileName:\
IrpSp->Flags = 0
<== DokanCreate
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
timeout Irp #7F
<== ReleaseTimeoutPendingIRP
==> DokanRead
ProcessId 4068
FileName:\test\51k_file.txt
ByteCount:4096 ByteOffset:0
Paging IO
Nocache
Synchronous IO
STATUS_PENDING
<== DokanRead
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
timeout Irp #80
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
timeout Irp #81
<== ReleaseTimeoutPendingIRP
==> DokanRead
ProcessId 4068
FileName:\test\51k_file.txt
ByteCount:4096 ByteOffset:0
Paging IO
Nocache
Synchronous IO
STATUS_PENDING
<== DokanRead
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
<== ReleaseTimeoutPendingIRP
==> ReleaseTimeoutPendingIRP
timeout Irp #82
<== ReleaseTimeoutPendingIRP
==> DokanRead
ProcessId 4068
FileName:\test\51k_file.txt
ByteCount:4096 ByteOffset:0
Paging IO
Nocache
Synchronous IO
STATUS_PENDING
<== DokanRead

By the way I made out that the two defines DOKAN_NOTIFICATION_TIMEOUT
and DOKAN_UNMOUNT_NOTIFICATION_TIMEOUT aren't used in the whole dokan
project from svn.

My workaround for now is to check if pagingIO in the fileinfo struct
is set. If it's set I always fill the buffer with the whole
NumberOfBytesToRead and set NumberOfBytesRead to the same value. If
the timeout is hit in the meantime, windows mostly reduces the
NumberOfBytesToRead to 4096. Then the requested bytes will be
transferred mostly before the timeout is hit. If pagingIO is not set,
I can read less bytes as NumberOfBytesToRead without any issues.

Whatever, it would be nice if the filesystem application can decide
everytime itself how much of the NumberOfBytesToRead will actually be
read. I think this would be possible if pagingIO can be turned off.

tombi

On 27 Jan., 10:54, tombi <to...@gmx.li> wrote:
> Hi,
>
> thanks for your answers.
>
> > <siegelpeter2...@gmail.com> wrote:
> > > Correct me of i am wrong,but as gar As i know when you return less Bytes
> > > than expected , the App thinks its the end of the file...?
>
> I don't think so, because windows doesn't stop the copy process.
> In read.c of the dokan.dll you can see the following:
>
>         if (status < 0) {
>                 eventInfo->Status = STATUS_INVALID_PARAMETER;
>         } else if(readLength == 0) {
>                 eventInfo->Status = STATUS_END_OF_FILE;
>         } else {
>                 eventInfo->Status = STATUS_SUCCESS;
>                 eventInfo->BufferLength = readLength;
>         }
>
> So dokan indicates the EOF only if readLengh == 0.
>
> On 27 Jan., 00:16, Hiroki Asakawa <asa...@gmail.com> wrote:
>
> > There was a bug in CurrentByteOffset in sys/read.c and I fixed it today.
> > But the bug only affects non paging IO.
>
> I tried the new version, but, as you said, it doesn't solve my problem
> unfortunately.
>
> > Does buffer length mean the third parameter of ReadFile, NumberOfBytesToRead?
>
> exactly.
>

> I will try to change the timeout, as you wrote here:http://groups.google.com/group/dokan/browse_thread/thread/4290b7d36b1...

[Hiroki Asakawa]

Hi, tombi

I could reproduce your error by setting ReadLength less than BufferLength
in MirrorReadFile. I think this should succeed. I'm investigating..

Oh, you are right, those two constants are not used anymore.
Instead, DOKAN_IPR_PENDING_TIMEOUT is used.

[Hiroki Asakawa]

Hi, tombi

Today, I added DokanResetTimeout function. This function allows you
to extend the timeout of the current IO operation.
For example, you can extend 30 seconds by the following command.
DokanResetTimeout(1000 * 30, DokanFileInfo);

I will update Dokan.NET library too.

Thanks,
Hiroki Asakawa

[Thomas]

Hi Hiroki,

I've looked at the fix, and I'd like to backport it to our 0.4.2
codebase. Could you explain what the problem was exactly?

Thomas

On 27 Jan., 00:16, Hiroki Asakawa <asa...@gmail.com> wrote:

> Hi, tombi
>
> There was a bug in CurrentByteOffset in sys/read.c and I fixed it today.
> But the bug only affects non paging IO.
>
> > I have detected that this problem only
> > occurs with buffer lengths less than 65536. With buffer length 65536
> > the offset is incremented correctly by 2048.
>
> Does buffer length mean the third parameter of ReadFile, NumberOfBytesToRead?
>
> On Tue, Jan 26, 2010 at 10:54 PM, Peter Siegel
>

[Hiroki Asakawa]

Hi, Thomas

On Tue, Feb 2, 2010 at 5:18 PM, Thomas <t.s.m...@gmail.com> wrote:
> Hi Hiroki,
>
> I've looked at the fix, and I'd like to backport it to our 0.4.2
> codebase. Could you explain what the problem was exactly?

I changed a lot since 0.4.2. I think it is difficult to backport. Why you
want to do?

[Thomas]

Because we run against 0.4.2+fixes (www.wuala.com, btw.). We NEED a
stable baseline that we can run and apply fixes to. If you eventually
release a new version, we will probably adopt that. And then we'll
test it for 2 months, and apply any necessary fixes to that exact
baseline. Running from HEAD is simply not an option if you ship a
product to real users. FUSE, for example also has a stable and a
development branch.

We would greatly appreciate it if you would open a bugfix branch for
the next release and I'm sure we as a company (and the community)
could help out with maintaining it (that's what we're doing anyway, in
our repository).

The fix looks like you just moved the handling of the
CurrentByteOffset to the sys driver, but the behaviour looks exactly
the same to me. Or was the problem in the mirror sample?

Thomas

On 2 Feb., 21:21, Hiroki Asakawa <asa...@gmail.com> wrote:
> Hi, Thomas
>

[Hiroki Asakawa]

Hi, Thomas

On Wed, Feb 3, 2010 at 9:16 AM, Thomas <t.s.m...@gmail.com> wrote:
> Because we run against 0.4.2+fixes (www.wuala.com, btw.). We NEED a
> stable baseline that we can run and apply fixes to. If you eventually
> release a new version, we will probably adopt that. And then we'll
> test it for 2 months, and apply any necessary fixes to that exact
> baseline. Running from HEAD is simply not an option if you ship a
> product to real users. FUSE, for example also has a stable and a
> development branch.

I changed a lot since the last release. Some of them are small bug
fix, but others are big change. I sometimes needs to change large
amount of Dokan code in oder to fix a bug. And at the same time,
I changed the dokan code to support new features.
I totally agree your opinion that stable version is required.
Currently, it is difficult for me to maintain two branches.
I think Dokan should be released much frequently. More than
one year cycle is too long. Probably, you need two branches
because of less frequency of Dokan release.

Now I'm preparing for the new release. Most of the issues are fixed
these days. I'm writing new installer since I can't compile the
previous installer my new machine :-)
I also want to automate the release process, and testing etc
to achieve frequent release.

BTW, I'm now in Zurich near your company.
And this bug, http://bugs.wuala.com/view.php?id=801
Dokan supports network filesytem type. See:
http://groups.google.com/group/dokan/browse_thread/thread/8bb46ff0cda279fd/

> We would greatly appreciate it if you would open a bugfix branch for
> the next release and I'm sure we as a company (and the community)
> could help out with maintaining it (that's what we're doing anyway, in
> our repository).
>
> The fix looks like you just moved the handling of the
> CurrentByteOffset to the sys driver, but the behaviour looks exactly
> the same to me. Or was the problem in the mirror sample?

Yes, I changed the way of handing of CurrentByteOffset. I think
the previous way is right, I introduced a bug at some point.

[tombi]

Hi Hiroki,

On 29 Jan., 22:36, Hiroki Asakawa <asa...@gmail.com> wrote:
> Today, I added DokanResetTimeout function. This function allows you
> to extend the timeout of the current IO operation.
> For example, you can extend 30 seconds by the following command.
> DokanResetTimeout(1000 * 30, DokanFileInfo);
>
> I will update Dokan.NET library too.

Today, I've testet the new DokanResetTimeout method with my .net app.
But it didn't work, the method always returned false. After some
debugging I found out the bug: In my ReadFile method, the
DokanFileInfo.DokanContext is always 0. I traced the bug in DokanNet
and made out that the rawFileInfo.DokanContext is most of the time 0
in the CreateFileProxy method. So as a workaround I removed "readonly"
from DokanContext in the DokanFileInfo class and added the line
"info.DokanContext = rawInfo.DokanContext;" to the ConvertFileInfo
method. Now it works fine, thanks. :)

tombi

[tombi]

Hi,

I just found out that my workaroud also fixed a bug where I couldn't
delete or copy files from the command line. I always got the error
"incorrect parameter".

tombi

[Hiroki Asakawa]

Hi, tombi

Thank you for your confirmation. There is a bug in dokan\create.c
I didn't set DokanContext in CreateFile.
http://code.google.com/p/dokan/source/detail?r=77