Re: Issue 50 in mp4v2: Extremely Poor Performace with Native Win32 - Std Fileprovider

[mp...@googlecode.com]

Updates:
Owner: kidjan

Comment #2 on issue 50 by kidjan: Extremely Poor Performace with Native
Win32 - Std Fileprovider
http://code.google.com/p/mp4v2/issues/detail?id=50

jeffreyloden,

Could you try this again with the most recent trunk? And is there some way
you could give me your test code so I could try and repeat it? There were
some more changes to the native windows functions (mostly to deal with
unicode), but I'd like to know if you still have the same issues.

One thought I had is to improve the file provider abstraction so users can
more easily create their own read/write routines (in particular, I'd like
to write one using Qt).

Thanks,
kidjan

[mp...@googlecode.com]

Comment #3 on issue 50 by danahins...@gmail.com: Extremely Poor Performace

with Native Win32 - Std Fileprovider
http://code.google.com/p/mp4v2/issues/detail?id=50

Just to see, I tried this with the most current rev, and the problem still
exists. Not sure if there are any developers that run on Windows, but if
so, my fix greatly speeds up reads and writes. I would be glad to work
with anyone that would be interested in getting this into the sources.

Very easy to test, on Windows run a read and a write with and without the
changes and notice the speedup. I've got thousands of users using my
version and I've never had a problem with it.

Dan

[mp...@googlecode.com]

Comment #4 on issue 50 by kid...@gmail.com: Extremely Poor Performace with

Native Win32 - Std Fileprovider
http://code.google.com/p/mp4v2/issues/detail?id=50

What happens if you change the flags used? For example, see:

http://msdn.microsoft.com/en-us/library/aa363858.aspx#caching_behavior

...in particular,

"Specifying the FILE_FLAG_SEQUENTIAL_SCAN flag can increase performance for
applications that read large files using sequential access. Performance
gains can be even more noticeable for applications that read large files
mostly sequentially, but occasionally skip forward over small ranges of
bytes. If an application moves the file pointer for random access, optimum
caching performance most likely will not occur. However, correct operation
is still guaranteed."

Looks like the code is currently using FILE_ATTRIBUTE_NORMAL (see
http://code.google.com/p/mp4v2/issues/attachmentText?id=50&aid=-802417126500239040&name=File_win32.cpp&token=4idgdzVy-fM3qodZaF02-OC04Xs%3A1332118380013#46
); might be interesting to see what happens if you put in
FILE_FLAG_SEQUENTIAL_SCAN since most MP4 file reading probably "read(s)
large files mostly sequentially, but occasionally skip forward over small
ranges of bytes"

Not opposed to the patch, but I'd prefer a solution that leveraged the
existing file API rather than caching in user land.

[mp...@googlecode.com]

Comment #5 on issue 50 by kid...@gmail.com: Extremely Poor Performace with

Native Win32 - Std Fileprovider
http://code.google.com/p/mp4v2/issues/detail?id=50

Also, this sort of sheds some light on this API:

http://blogs.msdn.com/b/oldnewthing/archive/2012/01/20/10258690.aspx

So, the weird thing is if an application passes in neither
FILE_FLAG_SEQUENTIAL_SCAN or FILE_FLAG_RANDOM_ACCESS, then the underlying
API goes into some sort of heuristic mode where it tries to optimize
performance based on the calls coming in. Also, it seems that this
behavior may not be consistent across Windows OSs.

It has to be said: what a *shit* API this is. The documentation alone
really shows you how convoluted it is. I wouldn't be opposed to going back
to the more portable way (fopen/fread/fseek) and doing away with this
windows specific file provider all together.

[mp...@googlecode.com]

Comment #6 on issue 50 by dbyr...@gmail.com: Extremely Poor Performace with

Native Win32 - Std Fileprovider
http://code.google.com/p/mp4v2/issues/detail?id=50

Going back to fopen/fread/fseek has some pretty big drawbacks too: can't
read filenames with non-ascii characters and can't read filenames that live
in long paths.

I've been super delinquent in not trying Dan's patch for ages. There may
be some light at the end of the tunnel for me but given my past
performance, is there someone else who can take a look at his patch and
work through any changes (if there are any) with him and get it committed?

Thanks much.

-DB

[mp...@googlecode.com]

Comment #7 on issue 50 by jnor...@logitech.com: Extremely Poor Performace

with Native Win32 - Std Fileprovider
http://code.google.com/p/mp4v2/issues/detail?id=50

Before we do any patches, somebody needs to at least try
FILE_FLAG_SEQUENTIAL_SCAN to see if that resolves the performance issues.

[mp...@googlecode.com]

Comment #8 on issue 50 by kid...@gmail.com: Extremely Poor Performace with

Native Win32 - Std Fileprovider
http://code.google.com/p/mp4v2/issues/detail?id=50

@dbyron,

I was under the impression that fopen accepts UTF8? If that's true, seems
like you could read filenames with non-ascii characters, but maybe I'm
misinformed about this.

[mp...@googlecode.com]

Comment #9 on issue 50 by dbyr...@gmail.com: Extremely Poor Performace with

Native Win32 - Std Fileprovider
http://code.google.com/p/mp4v2/issues/detail?id=50

I don't think fopen accepts UTF-8. See the attached test program. I
created three text files corresponding to the three files it tries to
open. fopen opens the one with the ascii name, but not the ascii ones.

Attachments:
main.c 1.8 KB

[mp...@googlecode.com]

Comment #10 on issue 50 by ktakata6...@gmail.com: Extremely Poor Performace

with Native Win32 - Std Fileprovider
http://code.google.com/p/mp4v2/issues/detail?id=50

Hi,

I use _wfopen() to handle Unicode filenames.
_wfopen() can't handle very long paths (longer than MAX_PATH characters),
but I
think it's acceptable because Explorer can not also handle such long paths.
--
Ken Takata

Attachments:
File_win32.cpp 5.7 KB

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[mp...@googlecode.com]

Comment #11 on issue 50 by danahins...@gmail.com: Extremely Poor Performace

with Native Win32 - Std Fileprovider
http://code.google.com/p/mp4v2/issues/detail?id=50

I just tested out using _wfopen, and it has as good a performance as my
fix, and since it's an OS API instead of my home rolled buffering, it seems
like the right answer. Can someone put this in the next release?

Dan

[mp...@googlecode.com]

Comment #12 on issue 50 by kid...@gmail.com: Extremely Poor Performace with

Native Win32 - Std Fileprovider
http://code.google.com/p/mp4v2/issues/detail?id=50

happily, if I can get it as a diff?

[mp...@googlecode.com]

Comment #13 on issue 50 by ktakata6...@gmail.com: Extremely Poor Performace

with Native Win32 - Std Fileprovider
http://code.google.com/p/mp4v2/issues/detail?id=50

OK, I have attached the diff.
--
Ken Takata

Attachments:
use_wfopen.diff 6.6 KB

[mp...@googlecode.com]

Comment #14 on issue 50 by ktakata6...@gmail.com: Extremely Poor Performace

with Native Win32 - Std Fileprovider

https://code.google.com/p/mp4v2/issues/detail?id=50

Hi,

I have updated the patch for R507.
--
Ken Takata

Attachments:
use_wfopen-r507.diff 6.5 KB