VERY slow writing to network shared drive, but NOTEPAD faster

[Cháo]

Hi all,

I've got a Ubuntu shared directory mapped as a drive, k drive, in Windows XP and whenever I try to save (:w) it takes about 8 to 12 seconds to complete the operation for a 133,124 bytes size text file - not acceptable slow.  IF I save under Notepad with the same file, it takes about 2 seconds!!

The share drive is connected through a VPN from work.  I don't have swapfiles / backup turn on, so I am not sure what else is causing this delay.  I was surprised that notepad had beaten VIM in this situation!  I want VIM to kick notepad anytime, so if anyone has any suggestions please let me know and I will try it out.  It's been bugging me for some days, is this a defect?  In today's network speed, I don't think there would be this much delays for saving text files.

Note: if I write the same size bytes to a new file, it only takes about 6 seconds, so writing to an existing file over a mapped share drive has more overheads / delays.

Go Vim!

Thanks,
Cháo

Woody Allen  - "Money is better than poverty, if only for financial reasons."

[Dominique Pellé]

Hi

Things worth trying:

- What's the speed when running Vim with: vim -u NONE
(to rule out anything unexpected hat could be done by a plugin)
- How is the time difference with small files vs large files?
Is the speed difference proportional to the file size? Or is
there a constant slow down regardless of the file size?
It might be a combination of both.
It would be useful to have a graph showing write time vs file
size when using Vim and Notepad.
- things worth trying to check if they help:
:set noswapfile
:set nobackup
:set nowritebackup
:set nofsync

I tried running Vim with strace on Linux (I don't know any
equivalent on Windows XP) and I see that Vim writes the
file by chunks of 8192 bytes (which sounds reasonable to
me). It corresponds to BUFSIZE macro in src/fileio.c:

#define BUFSIZE 8192 /* size of normal write buffer */

I wonder whether making BUFSIZE bigger could help.
If you can recompile Vim, it might be worth trying to replace
BUFSIZE with something bigger (as an experiment) such as:

#define BUFSIZE 1024*32 /* size of normal write buffer */

Other things I see with strace:

- before writing file, Vim stats the file 4 times (not sure why
4 times are needed). For example I did ":w! /tmp/foo.txt" and
strace -r shows:

...snip...
0.000091 chdir("/tmp") = 0
0.000101 getcwd("/tmp", 4096) = 5
0.000094 fchdir(6) = 0
0.000086 close(6) = 0
!!! 0.000097 stat64("/tmp/foo.txt", {st_mode=S_IFREG|0644,
st_size=423510, ...}) = 0
0.000185 open(".", O_RDONLY|O_LARGEFILE) = 6
0.000097 fchdir(6) = 0
0.000085 chdir("/tmp") = 0
0.000096 getcwd("/tmp", 4096) = 5
0.000092 fchdir(6) = 0
0.000083 close(6) = 0
!!! 0.000090 stat64("/tmp/foo.txt", {st_mode=S_IFREG|0644,
st_size=423510, ...}) = 0
0.000345 stat64("/tmp/foo.txt", {st_mode=S_IFREG|0644,
st_size=423510, ...}) = 0
0.000191 readlink("/tmp/foo.txt", 0xbf836fe4, 4095) = -1 EINVAL
(Invalid argument)
!!! 0.000125 stat64("/tmp/.foo.txt.swp", 0xbf837f88) = -1 ENOENT (No
such file or directory)
0.000180 write(1, "\33[?25l\"/tmp/foo.txt\" \33[34;16H\33[K", 32) = 32
!!! 0.000133 stat64("/tmp/foo.txt", {st_mode=S_IFREG|0644,
st_size=423510, ...}) = 0
0.000178 access("/tmp/foo.txt", W_OK) = 0
0.000206 getxattr("/tmp/foo.txt", "system.posix_acl_access",
0xbf838f10, 132) = -1 EOPNOTSUPP (Operation not supported)
0.000209 open("/tmp/foo.txt",
O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0644) = 6

... and then follow a bunch of writes of 8192 bytes on file
descriptor 6 (/tmp/foo.txt)...

0.000384 fsync(6) = 0
0.024673 close(6) = 0

Cheers
-- Dominique

[Dominique Pellé]

> > Hi all,
> > I've got a Ubuntu shared directory mapped as a drive, k drive, in Windows XP
> > and whenever I try to save (:w) it takes about 8 to 12 seconds to complete
> > the operation for a 133,124 bytes size text file - not acceptable slow. IF
> > I save under Notepad with the same file, it takes about 2 seconds!!

...snip...

> I tried running Vim with strace on Linux (I don't know any
> equivalent on Windows XP) and I see that Vim writes the
> file by chunks of 8192 bytes

...snip...

Google search "strace Windows XP" shows something similar
to strace on Windows. You could try to use it to see what
system calls make Notepad and Vim when writing a file on
your shared mapped drive "k:". The difference might be helpful
to explain the speed difference.

Cheers
-- Dominique

[Cháo]

Thanks Dominique for the suggestions.  One thing I overlooked was to set nowritebackup, I thought set nobackup was all the backup.  So this shaved a couple of seconds off!

Ok, I totally forgot to use the trusted tools from sysInternals!  Strace should be nice, but not familiar with it and it captures lots of extra info like mouse movement or what not, so I used File Monitor from sysinternal and I see the difference now between Notepad and GVim.

I've attached the comparisons in a zip files with 2 formats for those without Excel :)

First up was Notepad - about 2 seconds time to save: it's pretty quick, it just writes and done.  But I was wondering why it WRITE in huge chunk size of 133125 and then query stuff and the do some more WRITE operations in chuck size of 65536.  OH well.  It looks like Notepad writes in bigger chunks (65536) than gvim.

2nd is gvim with a file on the root of a mapped drive - about 3 seconds.  In this situation it's mirrors Notepad very close, but gvim queries the heck out the file before the WRITE op.

And finally, the 3rd and true problem case, gvim with a file that's deep in the directory structures - about 10 seconds!!   Now with FileMon tool, I can see that most of the time it's spending on querying info about the nested directories!  I've highlighted them in red and also repeated queries of the same directory structures in GREEN background.  Lots of traversing the network shared drive and all its subdirectories - each time taking about 2-3 seconds!  The actual WRITE data is using 8192 chuck size and not bad performance - about 2 seconds - guess could speed up a bit more by allowing it to save bigger chunks of data at once like Notepad.  So final thought is that GVIM tries to make sure everything is in order before anything happens?

Is there an option to disable the traversing from the root of the drive to each of the subdirs when saving a file?  If not, I might have to compile myself a copy with some changes.

Go VIM!

Thanks again,
Cháo

W. C. Fields  - "If I had to live my life over, I'd live over a saloon."

2009/6/13 Dominique Pellé <dominiq...@gmail.com>

[Dominique Pellé]

Thanks for the detailed report. That rings a bell now. Take a look
at this thread from January:

http://groups.google.com/group/vim_use/browse_thread/thread/ced1795b324d1b4e?pli=1

It looks similar to what you're experiencing.

User reported in above thread said that with 'set noshellslash' then
accessing //computer/c$/test.txt is fast (see :help shellslash)

Can you confirm whether it also helps in your case?

Cheers

PS: (reminder) in this mailing list, the convention is to bottom post
i.e. reply below the original message rather than above it.

-- Dominique

[Cháo]

2009/6/14 Dominique Pellé <dominiq...@gmail.com>

Thanks for the tip about posting reply at the bottom - it's gmail that's being funny plus I always did that :)

So I also emailed another one to vim_dev list aboout slowness when switch buffers of network mapped files and it seems that's related to writing/saving such files as well.  And Bram was quick to point out the following:

This is probably caused by the code that figures out the full path name
on Unix.  This is required to avoid any trouble with symbolic links.
Especially editing the same file under different names.

Your directory is actually on a network share that doesn't support
symbolic links.  But that's very difficult to detect.

I'm not sure if there is any way to avoid this without compromising
reliability.  Perhaps the expansion can be postponed or done in the
background.  That would make it more complex.

That "shellslash" didn't work for me.  I think in the thread that you sent, they were also working at close to the root of the drive, so performance wasn't as bad.  This all subdirs traversing business is causing slowness in proportion to the number of subdirs for a file.  I use vimSession to load my "project" from the network mapped drive and it took about 2 mins to finish loading 13 files!!!  So I am hoping a way to disable this behavior of visiting each subdir when doing anything to a network mapped drive file.

Thanks,
--

Cháo

Groucho Marx  - "I was married by a judge. I should have asked for a jury."

[Cesar Romani]

Cháo wrote:
> 2009/6/14 Dominique Pellé <dominiq...@gmail.com
> <mailto:dominiq...@gmail.com>>

I'm experiencing the same problem by editing a file on a network shared
drive with Windows XP. In that case notepad is faster.

Regards,
Cesar