[win32][patch] Updated large file patches

[Ken Takata]

Hi,

(The original thread was in vim_use:
https://groups.google.com/d/topic/vim_use/RLM8Vqa_aME/discussion )

I have updated the patches for supporting large files on Windows.
I also converted the test to the new style. And I also added a new test for
large files: test_largefile.vim. This creates a 4GB file, moving around it,
and write it to disk. This uses over 8GB disk spaces, and consumes CPU
powers. So I don't add this test to src/testdir/Make_all.mak. I confirmed
that when running the test without my fix, gvim.exe (32-bit) stopped with
"E342: Out of memory!" (takes about 10min), but it succeeded with my fix
(takes about 5min).
Please check the attached patches.

Note: Please apply the following patch before them to avoid conflicts.
https://groups.google.com/d/topic/vim_dev/JHbE4twU4dk/discussion

Regards,
Ken Takata

[Ken Takata]

Hi,

I have updated the patches again, because they conflict with 7.4.1133.
I also added a test for the following patch:
https://groups.google.com/d/msg/vim_dev/JHbE4twU4dk/o_wsCJ_RBgAJ
(The test is included in add-stat-test.patch.)

Regards,
Ken Takata

[Bram Moolenaar]

Thanks. In what order do the patches need to be applied?
Looks like the stat_T patch comes after the others?

For the test, we can add a "torture" target.

I have this old note in the todo file:

Win32: patch to use 64 bit stat() if possible. (Ken Takata, 2014 May 12)
More tests May 14. Update May 29. Update Aug 10.

I assume that is related to this?


--
hundred-and-one symptoms of being an internet addict:
9. All your daydreaming is preoccupied with getting a faster connection to the
net: 28.8...ISDN...cable modem...T1...T3.

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Ken Takata]

Hi Bram,

2016/1/20 Wed 3:33:18 UTC+9 Bram Moolenaar wrote:
> Ken Takata wrote:
>
> > (The original thread was in vim_use:
> > https://groups.google.com/d/topic/vim_use/RLM8Vqa_aME/discussion )
> >
> > I have updated the patches for supporting large files on Windows.
> > I also converted the test to the new style. And I also added a new test for
> > large files: test_largefile.vim. This creates a 4GB file, moving around it,
> > and write it to disk. This uses over 8GB disk spaces, and consumes CPU
> > powers. So I don't add this test to src/testdir/Make_all.mak. I confirmed
> > that when running the test without my fix, gvim.exe (32-bit) stopped with
> > "E342: Out of memory!" (takes about 10min), but it succeeded with my fix
> > (takes about 5min).
> > Please check the attached patches.
> >
> > Note: Please apply the following patch before them to avoid conflicts.
> > https://groups.google.com/d/topic/vim_dev/JHbE4twU4dk/discussion
>
> Thanks. In what order do the patches need to be applied?
> Looks like the stat_T patch comes after the others?

Yes.
use-stat_T.patch must be applied after support-largefiles-on-windows.patch.
Other two are independent.

> For the test, we can add a "torture" target.
>
> I have this old note in the todo file:
>
> Win32: patch to use 64 bit stat() if possible. (Ken Takata, 2014 May 12)
> More tests May 14. Update May 29. Update Aug 10.
>
> I assume that is related to this?

Yes, the stat_T patch is a patch for this.
Without this patch, getfsize() returns a broken value (e.g. -294967296), if
the size of a file is larger than 2 GiB and smaller than 4 GiB. And if the
size is larger than 4 GiB, getfsize() and other stat functions return -1.
MSVC's 32-bit stat() always returns error if the filesize is larger than 4 GiB.

Regards,
Ken Takata

[Ken Takata]

Hi,

I wrote an additional patch to fix display of off_T value.
add-stat-test.patch is updated to fix conflicts with 7.4.1142.
Please apply in the following order:

1. support-largefiles-on-windows.patch (previous mail)
2. fix-off_T-display.patch (this mail)
3. use-stat_T.patch (previous mail)
4. add-stat-test.patch (this mail)
5. test_largefile.patch (previous mail)

Regards,
Ken Takata

[Bram Moolenaar]

Thanks!

--
hundred-and-one symptoms of being an internet addict:

25. You believe nothing looks sexier than a man in boxer shorts illuminated
only by a 17" inch svga monitor.

[Ken Takata]

Hi,

Here are the updated large files patch for 7.4.1202.

Regards,
Ken Takata

[Ken Takata]

Hi,

And... this is for 7.4.1205.

Regards,
Ken Takata

[Ken Takata]

Hi,

Oops, I sent my update too early.
I have to wait all the style changes have completed.

Regards,
Ken Takata

[Ken Takata]

Hi,

I think that all the style changes have completed.
Here are the updated large file patches for 7.4.1229.

Regards,
Ken Takata

[Ken Takata]

Hi,

2016/2/1 Mon 22:34:43 UTC+9 Ken Takata wrote:
> I think that all the style changes have completed.
> Here are the updated large file patches for 7.4.1229.

I have updated the large file patches for 7.4.1665.
Please check the attached five patches.

Bram, I'm afraid that you are mistaking some of my patches.
Attached patches are related to the following items from the todo.txt:

L170-173

> Win32: patch to use 64 bit stat() if possible. (Ken Takata, 2014 May 12)
> More tests May 14. Update May 29. Update Aug 10.

> Now part of large file patches. (Ken Takata, 2016 Feb 1)
> Two patches now? New update Feb 24.

L1929-1931
> Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
> pointer in long and seek offset in 64 bit var.
> Patches from Ken Takata might help (2014 Apr 17)

The large file patches are also available from the following URLs:
https://bitbucket.org/k_takata/vim-ktakata-mq/src/tip/0001_support-largefiles-on-windows.patch
https://bitbucket.org/k_takata/vim-ktakata-mq/src/tip/0002_fix-off_T-display.patch
https://bitbucket.org/k_takata/vim-ktakata-mq/src/tip/0003_use-stat_T.patch
https://bitbucket.org/k_takata/vim-ktakata-mq/src/tip/0004_add-stat-test.patch
https://bitbucket.org/k_takata/vim-ktakata-mq/src/tip/0005_test_largefile.patch
(I will keep them up to date in the same URLs.)

I think these patches are rather important, as some people reported that very
large file can be truncated or Vim can crash:
https://groups.google.com/d/topic/vim_dev/8epHlRuIAHc/discussion
https://groups.google.com/d/topic/vim_use/RLM8Vqa_aME/discussion

Note: these patches doesn't fix some display problems (e.g. g<C-G>) because
64-bit integers are needed. To fix them, the num64 patch is needed.
https://groups.google.com/d/topic/vim_dev/p8Fl_vJDGy8/discussion
(Additionally, the 0002_fix-off_T-display.patch can be simpler when the num64
patch is included.)

Regards,
Ken Takata

[Bram Moolenaar]

Ken Takata wrote:

> 2016/2/1 Mon 22:34:43 UTC+9 Ken Takata wrote:
> > I think that all the style changes have completed.
> > Here are the updated large file patches for 7.4.1229.
>
> I have updated the large file patches for 7.4.1665.
> Please check the attached five patches.
>
> Bram, I'm afraid that you are mistaking some of my patches.
> Attached patches are related to the following items from the todo.txt:
>
> L170-173
> > Win32: patch to use 64 bit stat() if possible. (Ken Takata, 2014 May 12)
> > More tests May 14. Update May 29. Update Aug 10.
> > Now part of large file patches. (Ken Takata, 2016 Feb 1)
> > Two patches now? New update Feb 24.
>
> L1929-1931
> > Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
> > pointer in long and seek offset in 64 bit var.
> > Patches from Ken Takata might help (2014 Apr 17)
>
>
> The large file patches are also available from the following URLs:
> https://bitbucket.org/k_takata/vim-ktakata-mq/src/tip/0001_support-largefiles-on-windows.patch
> https://bitbucket.org/k_takata/vim-ktakata-mq/src/tip/0002_fix-off_T-display.patch
> https://bitbucket.org/k_takata/vim-ktakata-mq/src/tip/0003_use-stat_T.patch
> https://bitbucket.org/k_takata/vim-ktakata-mq/src/tip/0004_add-stat-test.patch
> https://bitbucket.org/k_takata/vim-ktakata-mq/src/tip/0005_test_largefile.patch
> (I will keep them up to date in the same URLs.)

So those patches are supposed to be included in order and sent out as
separate patches? Or could I just include all of these and send out one
big patch?

> I think these patches are rather important, as some people reported that very
> large file can be truncated or Vim can crash:
> https://groups.google.com/d/topic/vim_dev/8epHlRuIAHc/discussion
> https://groups.google.com/d/topic/vim_use/RLM8Vqa_aME/discussion
>
> Note: these patches doesn't fix some display problems (e.g. g<C-G>) because
> 64-bit integers are needed. To fix them, the num64 patch is needed.
> https://groups.google.com/d/topic/vim_dev/p8Fl_vJDGy8/discussion
> (Additionally, the 0002_fix-off_T-display.patch can be simpler when the num64
> patch is included.)

I hope to get back to including patches soon. Just a bit more cleanup
for new features to do.

--
hundred-and-one symptoms of being an internet addict:

152. You find yourself falling for someone you've never seen or hardly
know, but, boy can he/she TYPE!!!!!!

[Ken Takata]

Hi Bram,

I separate them mainly for readability. They can be merged as one patch.

More detail:

* 0001: This is a core part of them. This should fix the reported two problems:
> > https://groups.google.com/d/topic/vim_dev/8epHlRuIAHc/discussion
> > https://groups.google.com/d/topic/vim_use/RLM8Vqa_aME/discussion
* 0002: Only fixes one display problem. This can be considered as a part of
0001.
* 0003: This fixes another problem: 32-bit stat() returns an error for a file
larger than 4 GiB. :checktime, getfperm() or other stat functions
might fail for large files.
* 0004: Adds basic tests for stat functions. This can be considered as a part
of 0003.
* 0005: Adds tests for 0001 and 0003. This is not added as a normal test

target. A special target would be needed as you said:
> For the test, we can add a "torture" target.

> > I think these patches are rather important, as some people reported that very
> > large file can be truncated or Vim can crash:
> > https://groups.google.com/d/topic/vim_dev/8epHlRuIAHc/discussion
> > https://groups.google.com/d/topic/vim_use/RLM8Vqa_aME/discussion
> >
> > Note: these patches doesn't fix some display problems (e.g. g<C-G>) because
> > 64-bit integers are needed. To fix them, the num64 patch is needed.
> > https://groups.google.com/d/topic/vim_dev/p8Fl_vJDGy8/discussion
> > (Additionally, the 0002_fix-off_T-display.patch can be simpler when the num64
> > patch is included.)
>
> I hope to get back to including patches soon. Just a bit more cleanup
> for new features to do.

Regards,
Ken Takata

[Andreas]

I tried these pathces on version 1685 on windows 10 using Visual Studio 2015.
It didn't manage to patch src/proto/ fileio.pro, memline.pro, os_mswin.pro and src/testdir/Make_all.mak.
Got some warnings of the type "warning C4133: 'function': incompatible types - from 'stat_T *' to 'stat *'" from the compiler.

Tried the resulting gvim.exe on 8GB file and it worked nicely. The only problem I found was that the number of charatcers read is 32-bit and also the byte offset if you use G + CTRL-G.

I haven't tested editing the file but if you want I can do some tests.

Regards
Andreas

[Ken Takata]

Hi Andreas,

Thank you for trying the patches!

I don't know why patching failed, but it might be the cause of the warnings.
The patches apply cleanly in my environment (7.4.1689), and I don't see such
warnings (with VC2010, VC2015 and MinGW-w64).

> Tried the resulting gvim.exe on 8GB file and it worked nicely. The only problem I found was that the number of charatcers read is 32-bit and also the byte offset if you use G + CTRL-G.

Yes, this is a known problem. If you apply the following patch, the number
should be shown properly:
https://groups.google.com/d/msg/vim_dev/p8Fl_vJDGy8/CKSZ1ytQHAAJ

Regards,
Ken Takata

[Andreas]

Is is probably because I'm not used to downloading patches. I apply them by just typing "patch.exe < <patch file>" in the src directory. I guess I should use some Git function instead.

Regards
Andreas

[Andreas]

I remember now why. When I try to do the patch using TortoiseGit I get the message:
git.exe am --3way --ignore-space-change --keep-cr "C:\files\0001_support-largefiles-on-windows(1).patch"
Patch does not have a valid e-mail address.

Fail

Regards
Andreas

[Ken Takata]

Hi Andreas,

2016/4/1 Fri 17:11:12 UTC+9 Andreas wrote:
> I remember now why. When I try to do the patch using TortoiseGit I get the message:
> git.exe am --3way --ignore-space-change --keep-cr "C:\files\0001_support-largefiles-on-windows(1).patch"
> Patch does not have a valid e-mail address.

Ah, that's because my patches are not generated by `git format-patch`.
You can use `git apply` instead of `git am` to apply the patches.
(I don't know how to perform `git apply` from TortoiseGit.)

Regards,
Ken Takata

[Andreas]

Tried gitext with the same result, but then I started the command window and did git apply and it worked.

Now even the number of characters displayed on input is right though if I do G + Ctrl-G it seems the byte count is still 32-bit.

Thank you very much
Andreas

[Ken Takata]

Hi Andreas,

You must have forgotten applying the following patch:

[Andreas]

The num64 patch worked nicely.

Thanks again
Andreas