Very big file gets truncated

[anst...@gmail.com]

I have a text file (dc.epd) that is nearly 8GB. When I open it in gvim.exe version 1353 the display shows "dc.epd" 0 lines, 0 characters and the file is truncated to zero characters. I have compiled with Mic. Visual Studio 2015 compiler using standard makefile. I start gvim with the command: vim_git\src\gvim.exe dc.epd
Afterwards it looks like this:
dc.epd _____________________________________________________________ 0 19. 2.2016 11:28:28
The .swp file:
.dc.epd.swp ________________________________________________________ 0 19. 2.2016 11:28:28 H

The start of the file which contains chess positions look like this:
1B1b1k1Q/1p2r1p1/5p2/p2q2p1/3P2P1/8/PP3P1P/4R1K1 b - - bm Kf7; ce 0; acd 79;
1b1B1k1Q/1r3p2/2p1r1qp/1pPp1R2/pP6/4PR1P/P5P1/6K1 b - - bm Qg8; ce 0; acd 20;
1B1b1k2/1b1n1ppp/p1p1p3/1p6/P3P3/2N2P1P/1PP1B1PK/8 w - - bm Bd6+; ce 0; acd 20;
1B1b1k2/1B3npp/1p6/p1p5/8/1P4P1/P6P/7K b - - bm Ke7; ce 0; acd 30;
1B1b1k2/1B3npp/1p6/p1p5/8/1P6/P5PP/7K w - - bm g3; ce 0; acd 32;
1B1b1k2/1B3npp/1pp5/p7/8/1P6/P5PP/7K b - - bm c5; ce 0; acd 26;
1B1b1k2/1b6/1p6/p1pN1p2/P1P2P2/1P1R4/7r/3K4 w - - bm Re3; ce 0; acd 25;
1B1b1k2/1b6/1p6/p1pN1p2/P1P2P2/1P2R3/7r/3K4 b - - bm Kf7; ce 0; acd 23;
1B1b1k2/1p1r1p2/6p1/1P1P3r/2P1R3/1N4P1/6K1/8 b - - bm Be7; ce 0; acd 21;
1B1b1k2/1p1r1p2/6p1/NP1P3r/2P1R3/6P1/6K1/8 w - - bm Nb3; ce 0; acd 24;
1B1b1k2/1p2r1p1/5p2/p2q2pQ/3P2P1/8/PP3P1P/4R1K1 w - - bm Qh8+; ce 0; acd 39;
1B1b1k2/1p3npp/2p1B3/p7/8/1P6/P5PP/7K w - - bm Bc8; ce 0; acd 33;
1b1B1k2/1p3r2/p5p1/P1PR4/1P3P2/5KP1/8/8 b - - bm Kg7; ce 0; acd 23;
1b1B1k2/1p3r2/p5p1/P1PR4/1P3P2/6P1/5K2/8 w - - bm Kf3; ce 0; acd 22;
1b1B1k2/1p6/1P1p4/P2P1K2/3P4/8/5P2/8 w - - bm Bc7; ce 0; acd 20;

Regards
Andreas

[Ken Takata]

Hi Andreas,

Could you try this series of patches?
https://groups.google.com/d/msg/vim_dev/4EkET3c3oVA/bonA8fAcCwAJ

If the patches don't cleanly applied download the latest patches from here:
https://bitbucket.org/k_takata/vim-ktakata-mq/src

Regards,
Ken Takata

[Tony Mechelynck]

Vim loads the whole editfile in memory: for this, it needs enough
memory both available and addressable. In 32-bit Vim, including "Vim
without Cream" even when loaded (via WOW64) on a 64-bit Windows OS,
this means it is not possible to edit a file longer than 2 GiB, or
even several files whose total length exceeds 2 GiB.

4 GiB might be addressable but I doubt that 32-bit Vim would accept
it; anything higher (including your 8-GiB file) requires a higher
addressing mode (typically 64-bit) in all three of processor, OS and
program.

See
:help limits
:help 'maxmem'
:help 'maxmemtot'

But IMHO truncating the file is a bug. 64-bit Vim ought to edit it
normally (if there is enough live + swap memory), and 32-bit Vim ought
to give an error and refuse to edit the file.


Best regards,
Tony.

[Ken Takata]

Hi Tony,

It's not true when swapfile is enabled. Vim can edit a file larger than the
memory size. But unfortunately, Vim cannot handle swapfile larger than 2 GiB
on Windows, because the size is handled with 32-bit integer even on 64-bit
Windows.
My patches fix this problem by using 64-bit integer (even on 32-bit Vim).

Regards,
Ken Takata

[anst...@gmail.com]

Thank you for the patches. I had very little time to try them but I managed to run most of them and then try the new version briefly.
The new version managed to read the file without truncating it and it read a lot of lines. I think the character count was wrong though, but I didn't have time to check more. The patches was a clear improvement !
I will probably have more time on monday.

Thanks again
Andreas

[anst...@gmail.com]

I was runing on a machine with 16GB of memory so any way it should not be a problem. By the way it also has an SSD-disk witch might be significant.

Regards
Andreas

[Andreas]

An update.
I tried the new version again and it read all my 111239727 lines OK.
The only error i have found is that it shows the 32-bit truncated value of number of characters read - i.e. it shows 4145396456 characters in stead of the real 4145396456 + 2^32 = 8440363752 characters.
So these patches did the trick even though I didn't manage to do them all :)

Thanks
Andreas

[Ken Takata]

Hi Andreas,

Thank you for testing my patches.
Did you apply 0002_fix-off_T-display.patch? Without this, character count
will not be shown properly when a file has been read/written.
Or did you check the character count in another way?

Regards,
Ken Takata

[Ken Takata]

Hi Andreas,

2016/2/22 Mon 19:44:25 UTC+9 Ken Takata wrote:
> Thank you for testing my patches.
> Did you apply 0002_fix-off_T-display.patch? Without this, character count
> will not be shown properly when a file has been read/written.
> Or did you check the character count in another way?

Ah okay, I found that g<C-G> doesn't show the character count properly.
It's just a display problem, and it doesn't have any side effects when editing
a large file.
I think it's better to fix it with my another series of patches (num64 patches):
https://groups.google.com/d/msg/vim_dev/p8Fl_vJDGy8/WgoprhIdCwAJ
I need time to fix this.

Regards,
Ken Takata

[Andreas]

I reported the number of characters shown immediately after the file is loaded, but it is probably because I didn't manage to include all the patches.
If I use G<C-G> I get the same, but now with a signed 32-bit so the number of characters reported is -149570840 which should be (2^32-149570840)+2^32 = 8440363752.

Thanks a lot
Andreas