[vim/vim] gvim 9.0 cannot open big files (Issue #12620)
razvaneduard
Steps to reproduce
Up till some weeks ago I used gvim 8.2 downloaded from vim.org (gvim82.zip + vim82rt.zip)
With it I can open large files 2Gb, 4Gb even larger.
I found version 9.0 is out and got it from vim.org (gvim90.zip + vim90rt.zip)
It cannot open large files, it cause high CPU and "program not responding"
I found these nightly builds and I tested different versions:
these doesn't work (high CPU, file never loaded, program not responding)
gvim90 x86 (gvim90.zip + vim90rt.zip)
gvim_9.0.1671_x64
gvim_9.0.1671_x86
gvim_8.2.5171_x64_signed
gvim_8.2.5171_x86_signed
only this works
8.2 x86 (gvim82.zip + vim82rt.zip)
Thanks
Expected behavior
If 8.2 32bit base version open large file,
I expected 64bit version or 9.0 to open large files
Version of Vim
9.0 - 9.0.1671
Environment
windows 11 pro
16Gb RAM
Intel i7-1185G7 @ 3.00GHz
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Tony Mechelynck
What do your problematic versions of Vim answer to
verbose set mm? mmt? swf?
? On my Linux64 system with 33.4 GB of memory available after the kernel is loaded, both maxmem and maxmemtot are set to 16327166 (in KiB) and swapfile is on. If the maxmem setting is exceeded and a buffer needs more memory, Vim will free some memory elsewhere; and the maxmemtot setting defines the maximum memory size to use for all buffers together. The help also recommends not to use 'noswapfile' for big files because all their text will then be in memory.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
razvaneduard
I've made a mistake in isolating this issue, I don\t think is related to file size.
I discovered that gvim_9.0.1671_x64 can open only a single file all of the big files I have.
The only file that it can open is 3.8Gb in size, the rest which cannot be opened are around 1.7 - 1.8Gb
So, the problem is not related to file size, maybe related to the content of the file.
gvim 8.2 x86 still open all of them .
Files should be unicode (they are some log and traces) so I though maybe some invalid unicode char can cause this.
I used
grep -axv '.*' file
to find invalid unicode characters (after a short search on internet).
The bad one have 8 lines with � in them, the good one has only 1 line with with �.
From the bad file I excluded the lines with �
grep -ax '.*' bad.log >bad2good.log
and I can open bad2good.log with gvim_9.0.1671_x64
I remember seeing in vim this invalid unicode sign �, so normally vim detects invalid unicode.
But in case of these files which it could not open
the lines with � were too many or in a combination that make vim choke...
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Tony Mechelynck
Normally Vim should only edit in Unicode files which contain only valid Unicode data. What is the answer to
:verbose set fencs?
Open the problematic file with
:new ++enc=latin1 filename.ext
(replacing filename.ext above by the actual pathfilename, and assuming that 'encoding' is set to UTF-8) then if you do 8g8 Vim will position the cursor on the next invalid UTF-8 byte sequence at or after the cursor. (It won't move if it is already on an invalid byte sequence.) This way you can determine what exactly these non-UTF-8 bytes were.
Best regards,
Tony.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
razvaneduard
:verbose set fencs
fileencodings=ucs-bom,utf-8,default, latin1
8g8 will find the first invalid unicode æ then next invalid unicode å on this line
ORA-00001: entydig begrænsning (MIRMOD.COLLECT$SQL_BINDS_UK) er overtrådt
It's the first line I found previously with
grep -axv '.*' file
Thanks
Razvan
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
Please try vim --clean Does that work?
Also, if you create a simple file, just containing ASCII and a few of those invalid characters, does the problem also occur?
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
Oh, and does the cygwin vim can handle those files?
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
razvaneduard
I started gvim_9.0.1671_x64
gvim.exe --clean
then loaded bad.log, still high CPU and then 'program not responding.
in cygwin vim 8.2.4372 the same behavior with bad log, high CPU and file never opens.
however in gvim8.2 x86 (gvim82.zip + vim82rt.zip) bad.log open ok
I outputed the lines with invalid unicode
grep -axv '.*' bad.log > bad-small.log
bad-small.log has only 8 lines, 8g8 will find the same invalid unicode as in original bad.log
gvim_9.0.1671_x64 open just fine this bad-small.log
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
okay, so would it be possible to share that big file?
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
razvaneduard
I need to remove from this file line containing words like host or name.
I tried
grep -iv "name\|host" bad.log >bad-new.log
but bad-new.log opens fine in vim, 8g8 does not find any invalid utf-8
any suggestions on how to sanitize this file without changing encoding of the characters?
first invalid unicode character appear in bad.log at line 25.082449 so I tried
split -l 25000000 bad.log bad
first file bada has no invalid utf8 and 1Gb
second file badb has 800Mb and 8g8 found the invalid characters.
So the size is relevant in this issue
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
razvaneduard
I found it
:e ++enc=latin1 bad.log
:g/host/d
790 fewer lines
:g/name/d
6513 fewer lines
:sav bad-new.log
bad-new.log hang when opened
andl shows invalid unicode when searched with 8g8
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
razvaneduard
I reproduced the issue by creating a 1.3Gb file having all lines with the same text.
To this big file I appended the lines with invalid utf-8 characters I extract from my original file.
vim hang trying to open this file with 'program not responding'.
If I open it with ++enc=latin1 it shows invalid utf8 when I search with 8g8.
I attached it here
Thanks,
Razvan
big3.zip
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
Hm,
I need to further check:
chrisbra@debian ~/code/vim-src/src (git)-[last-non-blank_char]- % time ./vim --clean -c ':q!' big3
./vim --clean -c ':q!' big3 517.27s user 12.63s system 89% cpu 9:50.34 total
chrisbra@debian ~/code/vim-src/src (git)-[last-non-blank_char]- % time ./vim --clean --cmd ':set enc=latin1' -c ':q!' big3
./vim --clean --cmd ':set enc=latin1' -c ':q!' big3 584.91s user 13.02s system 85% cpu 11:39.65 total
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
razvaneduard
yes, vim does not really hang, but is very very slow, unusable slow, when open a big file with invalid utf-8 chars
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
I guess we need to do some profiling to find out what happens. How long does your vim82 take to load the file? And how long does vim9 need? (You can check by running the commands I have shown in my previous comment.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
razvaneduard
I used cygwin
this is 9.0.1672 64bit
time ./vim90/vim --clean -c ':q!' big3
real 3m55,567s
user 0m0,000s
sys 0m0,000s
this is 8.2.5171 64bit
time ./vim82/vim --clean -c ':q!' big3
real 3m27,685s
user 0m0,000s
sys 0m0,015s
this is 8.2 32bit
time ./vim82/vim --clean -c ':q!' big3
real 0m3,844s
user 0m0,000s
sys 0m0,000s
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
oh, that is drastic. :/ unfortunately, I saw the same slow down on a self-compiled 8.2.0. I'll re-check in a bit.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
Note, that your test version of 8.2 is 32bit, while the other builds are 64 bit. So I went down the rabbit hole and compiled a 32bit vim version of v9.0.1378 (that is the latest version available in Debian sid) and see the same:
- vim 9.0.1378 32bit:
root@bookworm:/tmp/buildd/vim-9.0.1378/src/vim-basic# file vim vim: ELF 32-bit LSB pie executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, BuildID[sha1]=1985d979b0b1b2ef1ab401dde4d793055ac42165, for GNU/Linux 3.2.0, with debug_info, not stripped root@bookworm:/tmp/buildd/vim-9.0.1378/src/vim-basic# time ./vim --clean -c ':q!' big3 real 0m18.142s user 0m2.439s sys 0m1.916s
- vim 9.0.1378 64bit (self-compiled, using almost same config flags):
chrisbra@bookworm ~/code/vim-src/src (git)-[tags/v9.0.1378]- % file vim vim: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=4958700e77a0e2ad59fb376fd8bf556b1dc518d2, for GNU/Linux 3.2.0, with debug_info, not stripped chrisbra@bookworm ~/code/vim-src/src (git)-[tags/v9.0.1378]- % time ./vim --clean -c ':q!' /var/cache/pbuilder/build/188456/tmp/buildd/vim-9.0.1378/src/vim-basic/big3 ./vim --clean -c ':q!' 289.22s user 8.27s system 62% cpu 7:53.37 total
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
so it rather seems to be an issue of 32bit vs: 64bit builds.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
razvaneduard
yes, I said from the beginning of the thread that
8.2 x86 (gvim82.zip + vim82rt.zip)
open the file just fine
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
yes and perhaps I misread it, but it was not initially clear to me that you meant 32bit vs. 64bit and not a specific regression introduced by a patch between v8.2 and v9.0.1671 (which seem to be around 7000 patches)
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.