Opening large files?

[Adrian Nagle]

'm helping another user open a large, 3Gb, file. The standard windows
editors balk, so I recommended VIM. Unfortunately, even vim crashes
after scrolling some amount. For instance, he can't go straight to
the end of file.

The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
any settings to modify to make vim more stable with large files, or is
there some Windows performance limitation and just out of luck?

Thanks.
Adrian
--
Adrian Nagle (ana...@gmail.com)
"There can be no thought of finishing, for 'aiming at the stars,' both
literally and figuratively, is a problem to occupy generations, so
that no matter how much progress one makes, there is always the thrill
of just beginning." Dr. Goddard to H.G. Wells, 1932

[Tim Chase]

On 2014-04-16 09:58, Adrian Nagle wrote:
> 'm helping another user open a large, 3Gb, file. The standard
> windows editors balk, so I recommended VIM. Unfortunately, even
> vim crashes after scrolling some amount. For instance, he can't go
> straight to the end of file.
>
> The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
> any settings to modify to make vim more stable with large files, or
> is there some Windows performance limitation and just out of luck?

This is pretty common, so your system's specs should be able to
handle it. Hints can be found here:

http://vim.wikia.com/wiki/Faster_loading_of_large_files

and a plugin/script to facilitate it can be found here:

http://www.vim.org/scripts/script.php?script_id=1506

-tim

[Dominique Pellé]

That plugin is definitely useful to speed up Vim with large files
and to use less memory.

Vim has to load the entire file in memory (unlike a few editors
that can load on demand). However, I would not expect crashes
or running out of memory on a 3GB file with 32GB of RAM (and
possibly more of virtual memory).

Which version of Vim are you using?
Can you get a stack trace where it crashes?
Make sure you use the latest version of Vim to avoid wasting time
on bugs possibly already fixed. I see a recent version of Vim
for Windows here:

http://sourceforge.net/projects/cream/files/Vim/

Regards
Dominique

[Ken Takata]

Hi,

2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
> 'm helping another user open a large, 3Gb, file. The standard windows
> editors balk, so I recommended VIM. Unfortunately, even vim crashes
> after scrolling some amount. For instance, he can't go straight to
> the end of file.
>
> The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
> any settings to modify to make vim more stable with large files, or is
> there some Windows performance limitation and just out of luck?

There is a related item in the todo.txt:

| Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
| pointer in long and seek offset in 64 bit var.

I wrote some patches to fix this, but they seem to be still unstable.

https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default

Regards,
Ken Takata

[Charles Campbell]

The LargeFile plugin turns the swapfile option for large files, so it
would help avoid this problem; TIm has already provided a link to it, so
I won't repeat that here.

Is your "another user" using a FAT32 partition? That filesystem is
limited to 4G files, and so possibly the swapfile is exceeding that limit.

Regards,
Chip Campbell

[Tim Chase]

On 2014-04-16 09:58, Adrian Nagle wrote:

> 'm helping another user open a large, 3Gb, file. The standard
> windows editors balk, so I recommended VIM. Unfortunately, even
> vim crashes after scrolling some amount. For instance, he can't go
> straight to the end of file.
>
> The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
> any settings to modify to make vim more stable with large files, or
> is there some Windows performance limitation and just out of luck?

In addition to the LargeFile tips, for certain types of edits, using
a stream editor such as "sed" will usually let you operate on
arbitrary-sized files, as it only processes one line at a time
(modulo what you keep in the hold-space). If you detail the types of
transformations/changes that you are hoping to make, there might a
way to do it that can be done regardless of the file-size.

-tim

[Charles Campbell]

Sorry about the omission and typo: should be "...turns the swapfile
option off...". And, instead of "TIm", it should be "Tim".

Chip Campbell

[Adrian Nagle]

Folks,

I appreciate the help.  I will look into the suggestions.  We use NTFS.

Thanks!

Adrian

[Tim Chase]

On 2014-04-17 09:44, Charles Campbell wrote:
> And, instead of "TIm", it should be "Tim".

No problem, CHip ;-)

-tim

[Bram Moolenaar]

Did you make progress on this?
Can we also add some tests to verify the fix?

--
From "know your smileys":
:-E Has major dental problems

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Ken Takata]

Hi Bram,

2014/4/29 Tue 22:04:23 UTC+9 Bram Moolenaar wrote:
> Ken Takata wrote:
>
> > 2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
> > > 'm helping another user open a large, 3Gb, file. The standard windows
> > > editors balk, so I recommended VIM. Unfortunately, even vim crashes
> > > after scrolling some amount. For instance, he can't go straight to
> > > the end of file.
> > >
> > > The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
> > > any settings to modify to make vim more stable with large files, or is
> > > there some Windows performance limitation and just out of luck?
> >
> > There is a related item in the todo.txt:
> >
> > | Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
> > | pointer in long and seek offset in 64 bit var.
> >
> > I wrote some patches to fix this, but they seem to be still unstable.
> >
> > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
> > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default
>
> Did you make progress on this?

I wrote "still unstable", but it seems that it was my mistake.
Now I think that the patches are OK.

Sometimes Vim (without the patches) freezes when I open a very big file
(about 2 GB) and scroll up and down using scroll bar. After applying the
patches, Vim sometimes takes very long time for scrolling up and down, but
it wasn't a freeze. (I misunderstood that.)

BTW, I found that 32-bit Vim couldn't handle a very big file properly when
":set noswapfile". In my understanding, this is an expected(?) behavior
because Vim tries to load the whole file into the memory when 'swapfile' is
off, and a 32-bit program can't allocate larger than 2-GiB memory.
(Actually, a 32-bit program can get 3-GiB user space if /LARGEADDRESSAWARE
option is specified for 'link'.)

> Can we also add some tests to verify the fix?

I'm thinking what is the best way to test this.
Something like this?

" Make sure that a line break is 1 byte.
:set ff=unix
:set undolevels=-1
" Input 99 'A's. The line becomes 100 bytes including a line break.
99iA<Esc>
yy
" Put 19,999,999 times. The file becomes 2,000,000,000 bytes.
19999999p
" Moving around in the file randomly.
G
10%
90%
50%
gg
...
" Edit some lines.
...
" Extract some lines and write them to test.out.
...

Regards,
Ken Takata

[Bram Moolenaar]

Although it would be good to test this way, I think it should not be
part of "make test", since it will probably fail on smaller systems.
Perhaps we should make a "make bigtest" target, for more testing.

Generally, I think we need to test that the patch doesn't break anything
for normal editing. But perhaps the tests we already have are
sufficient for that. If you look at your patch, are there any commands
that would not be used by the existing tests?

--
"The sun oozed over the horizon, shoved aside darkness, crept along the
greensward, and, with sickly fingers, pushed through the castle window,
revealing the pillaged princess, hand at throat, crown asunder, gaping
in frenzied horror at the sated, sodden amphibian lying beside her,
disbelieving the magnitude of the frog's deception, screaming madly,
"You lied!"
- Winner of the Bulwer-Lytton contest (San Jose State University),
wherein one writes only the first line of a bad novel

[Ken Takata]

Hi Bram,

I agree.

> Generally, I think we need to test that the patch doesn't break anything
> for normal editing. But perhaps the tests we already have are
> sufficient for that. If you look at your patch, are there any commands
> that would not be used by the existing tests?

I think that the existing tests cover almost all part, but maybe they
doesn't cover the following functions, commands and features:
* getfsize(), getfperm(), getftime(), getftype()
* :checktime
* +cscope, +netbeans_intg

BTW, I have updated the patch:
https://bitbucket.org/k_takata/vim-ktakata-mq/src/1ded06658094d26a5eef3ffe42e5edc94d93f59c/support-largefiles-on-windows.patch?at=default (same as before)
https://bitbucket.org/k_takata/vim-ktakata-mq/src/1ded06658094d26a5eef3ffe42e5edc94d93f59c/use-stat_T.patch?at=default
* Define HAVE_STAT64 in vim.h and use it in os_mswin.c.
* There was no need to use stat_T in os_unix.c. stat_T should be used with
mch_stat().

Regards,
Ken Takata

[Bram Moolenaar]

So is the patch now ready to be included, or did you still have a
problem to fix?

--
hundred-and-one symptoms of being an internet addict:
142. You dream about creating the world's greatest web site.

[Ken Takata]

Hi Bram,

2014/5/13 Tue 3:40:37 UTC+9 Bram Moolenaar wrote:
> So is the patch now ready to be included, or did you still have a
> problem to fix?

I think it's ready. I don't see any regressions with the patch (at least in my
use cases).
Updating the tests is another todo item.

Regards,
Ken Takata

[Bram Moolenaar]

Well, it's better to have tests before including it. The tests might
find a flaw.

--
hundred-and-one symptoms of being an internet addict:

149. You find your computer sexier than your girlfriend

[Ken Takata]

Hi Bram,

2014/5/13 Tue 20:12:59 UTC+9 Bram Moolenaar wrote:
> Ken Takata wrote:
>
> > 2014/5/13 Tue 3:40:37 UTC+9 Bram Moolenaar wrote:
> > > So is the patch now ready to be included, or did you still have a
> > > problem to fix?
> >
> > I think it's ready. I don't see any regressions with the patch (at least in my
> > use cases).
> > Updating the tests is another todo item.
>
> Well, it's better to have tests before including it. The tests might
> find a flaw.

OK, I wrote very simple tests for getfsize(), getfperm(), getftime(),
getftype() and :checktime.
https://bitbucket.org/k_takata/vim-ktakata-mq/src/09e276a96d721ded06d8a140e730638ffdfb18ca/add-stat-test.patch

Regards,
Ken Takata

[Ken Takata]

Hi Bram,

I have updated the tests for 7.4.315. (Just renamed from test107 to test108.)
https://bitbucket.org/k_takata/vim-ktakata-mq/src/tip/add-stat-test.patch

Regards,
Ken Takata

[Ken Takata]

Hi Bram,

After 7.4.399, support-largefiles-on-windows.patch couldn't be applied because of
conflicts. So I have updated the patch.
The latest patches are:

https://bitbucket.org/k_takata/vim-ktakata-mq/src/be6f89cd8a129025d2a456eb60b946265d90dffe/support-largefiles-on-windows.patch?at=default
https://bitbucket.org/k_takata/vim-ktakata-mq/src/be6f89cd8a129025d2a456eb60b946265d90dffe/use-stat_T.patch?at=default
https://bitbucket.org/k_takata/vim-ktakata-mq/src/be6f89cd8a129025d2a456eb60b946265d90dffe/add-stat-test.patch?at=default

Regards,
Ken Takata