[vim/vim] Vim Stuck on Large file size when All the data is on single line (#7403)
adeelcap15
Bug Detail
I was trying to open 1.8 GB JSON file. All the text is on single line. VIM was not responding. When press gj it takes 12 seconds to come to next line. The interesting thing is once I formatted that JSON by below command.
jq . myfile.json > formattedFile.json
Opening this formmattedFile.json, I can navigate without any problem though the file size increased to 2.5 GB.
Even opening VIM with all the plugins I have, the `formattedFile.json' was working fine.
To Reproduce
Detailed steps to reproduce the behavior:
- Run
vim --clean(orgvim --clean, etc.) - Open
filenamewith all text on single line - It will not responding '....'
Expected behavior
VIM should be responsive even all the content is on single line.
Environment:
- Vim version 8.2
- Ubuntu 20.04
- GNOME Terminal
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Christian Brabandt
how much time did you give vim to become responding again? Have you tried just to wait a bit? Or perhaps just press Ctrl-C a couple of times (might be that the syntax highlighting becomes slow), this should make vim start to become responding again.
Try with a bare vim without any syntax highlighting: vim -u NONE -N and perhaps do some profiling of what vim script is slow. Possibly also check the output of :syntime.
Also, what was the exact vim version please?
adeelcap15
@chrisbra VIM became stuck and the syntax highlighting was also off. I have opened vim without any plugin as mentioned above.
My vim version is:
`VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Nov 30 2020 10:19:08)
Included patches: 1-2069
`I have compiled from source code.
Christian Brabandt
so hitting CTRL-C did not make vim re-active? You might need to do some profiling of the vim source to find out what is happening. Does it happen with any large file (single line) or just this particular one?
K.Takata
I think that this is a well known long standing issue.
Handling a very very long line on Vim causes terrible slow down.
I'm not sure about the root cause, though.
adeelcap15
so hitting
CTRL-Cdid not make vim re-active? You might need to do some profiling of the vim source to find out what is happening. Does it happen with any large file (single line) or just this particular one?
@chrisbra It only happens with large file (single line) only.
Christian Brabandt
okay, so you should probably do some profiling the vim source to see what is happening.
adeelcap15
Here is the profile which I got by using this command
vim --startuptime profile_oneline.log -u NONE bigFile.json
It takes 12 minutes in case if file is not formatted and all the text is in single line.
`
Big File with only one Line :
times in msec
clock self+sourced self: sourced script
clock elapsed: other lines
000.019 000.019: --- VIM STARTING ---
000.376 000.357: Allocated generic buffers
000.584 000.208: locale set
000.597 000.013: window checked
001.725 001.128: inits 1
001.934 000.209: parsing arguments
001.939 000.005: expanding arguments
001.974 000.035: shell init
002.501 000.527: Termcap init
002.524 000.023: inits 2
002.744 000.220: init highlight
002.746 000.002: sourcing vimrc file(s)
002.761 000.015: inits 3
002.778 000.017: setting raw mode
002.794 000.016: start termcap
002.831 000.037: clearing screen
757119.572 757116.741: opening buffers
757119.577 000.005: BufEnter autocommands
757119.579 000.002: editing files in windows
757119.591 000.012: VimEnter autocommands
757119.592 000.001: before starting main loop
791200.137 34080.545: first screen update
791200.140 000.003: --- VIM STARTED ---
`
Big File with multiple lines
It takes 7 seconds in case if file is formatted and text is NOT in single line
In case of formattedFile.json, it has following log.
`
times in msec
clock self+sourced self: sourced script
clock elapsed: other lines
000.020 000.020: --- VIM STARTING ---
000.234 000.214: Allocated generic buffers
000.355 000.121: locale set
000.361 000.006: window checked
001.053 000.692: inits 1
001.198 000.145: parsing arguments
001.202 000.004: expanding arguments
001.255 000.053: shell init
002.086 000.831: Termcap init
002.137 000.051: inits 2
002.678 000.541: init highlight
002.684 000.006: sourcing vimrc file(s)
002.720 000.036: inits 3
002.755 000.035: setting raw mode
002.786 000.031: start termcap
002.833 000.047: clearing screen
7325.757 7322.924: opening buffers
7325.760 000.003: BufEnter autocommands
7325.761 000.001: editing files in windows
7325.772 000.011: VimEnter autocommands
7325.773 000.001: before starting main loop
7325.984 000.211: first screen update
7325.985 000.001: --- VIM STARTED ---
`
Here it takes 7 seconds.
vim-dev ML
Christian Brabandt (Vim Github
Repository) wrote:
okay, so you should probably do some profiling the vim source
to see what is happening.
I used a 15.8Mb file instead of an over-1G file (so I wouldn't need
to wait as long).
Then:
./vim -clean tmp +'q'
gprof -clean ./vim tmp > vim.prof
Results:
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
38.76 0.81 0.81 15829833 0.00 0.00 regtry
14.35 1.11 0.30 15856456 0.00 0.00
re_multi_type
8.61 1.29 0.18 142468553 0.00 0.00 regnext
6.94 1.44 0.15 79149165 0.00 0.00
regstack_push
5.74 1.56 0.12 1 0.12 0.14
chk_modeline
4.31 1.65 0.09
dbcs_ptr2char
3.35 1.72 0.07 1 0.07 1.68
bt_regexec_both
2.87 1.78 0.06 110979252 0.00 0.00
utf_ptr2char
2.39 1.83 0.05 47537430 0.00 0.00 ga_grow
2.39 1.88 0.05 18076529 0.00 0.00
utfc_ptr2len
1.91 1.92 0.04 1 0.04 0.29 readfile
1.91 1.96 0.04
getdecchrs
1.44 1.99 0.03 33042 0.00 0.00
ga_grow_inner
1.20 2.01 0.03 26385 0.00 0.00
nfa_did_time_out
0.96 2.03 0.02 47898024 0.00 0.00
fast_breakcheck
0.96 2.05 0.02 15830166 0.00 0.00
vim_isspace
0.48 2.06 0.01 134586 0.00 0.00
line_breakcheck
0.48 2.07 0.01 75221 0.00 0.00
find_term_bykeys
0.48 2.08 0.01 13592 0.00 0.00
hash_lookup
0.48 2.09 0.01 1741 0.00 0.00 do_map
0.00 2.09 0.00 816492 0.00 0.00
profile_passed_limit
I dropped the many calls that didn't appreciably change the time
used.
Regards,
Chip Campbell
Bram Moolenaar
> > so hitting `CTRL-C` did not make vim re-active? You might need to do
> > some profiling of the vim source to find out what is happening. Does
> > it happen with any large file (single line) or just this particular
> > one?
>
> @chrisbra It only happens with large file (single line) only.
one piece of allocated memory. That doesn't work well for lines that
are several Mbyte long. It breaks down for a line of a Gbyte.
There might be ways to improve this, but the assumption that a text line
is one contiguous piece of memory is all over the code. Thus it's going
to be an enormous amount of work. And it helps only very few users.
--
[Autumn changed into Winter ... Winter changed into Spring ... Spring
changed back into Autumn and Autumn gave Winter and Spring a miss and
went straight on into Summer ... Until one day ...]
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD
/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
—
You are receiving this because you commented.
