vim: format each long lines and insert just one break afterward

[ping]

tested long time no good results...

so the goal here sounds simple: say I have

* a long text file (over 50K lines), and
* each line by itself is a nature paragraph, so sometime is very long
(over 1000 charactors).
* there are sometimes one of more empty line in between, sometimes not

I wanted to format it into:

1. if 2 lines are adjoining, add an empty line to as a break
2. if 2 lines already have more than 1 empty lines in between, remove
the extra and make sure only one empty line left.
3. trunk the long lines to multiple lines with each of 80 charactors

initially I though its simple, just a ranged :g can do:

:'<, '>global/.*/exec "normal! gqq"

but this can only do task 3 above. for the task 1 and 2.
I don't know of how to pass the lines found by :g to a function and some
more magic there.

so I turned to a normal function, with which I can read each line and do
some adjustment there:

func! GQeachline(...) range "ping: comments
let lnum=a:firstline

while lnum <= a:lastline

let line = getline(lnum)
let l1len = len(line)
echom "linenum: " . lnum . "length " . l1len

if l1len >= 80
echom "gq this line (" . l1len . " > tw!"
exec "normal! gqqj"
elseif line =~ '^\s*$' "if it's an empty line
echom "this line is empty"
else
echom "this line is short " . l1len
endif

let lnum=lnum+1

endwhile

endfunc

""line number all changed
" let lnum=a:firstline
" while lnum <= a:lastline
"
" let line = getline(lnum)
" let l1len = len(line)
"
" let nlinenum=lnum+1
" let nline=getline(nlinenum)
"
" if nline =~ '^\s*$'
" else
" exec "normal! o\<esc>"
" endif
"
" let lnum=lnum+1
"
" endwhile


command! -range=% -nargs=* GQeachline :<line1>,<line2>call
GQeachline(<q-args>)

map ,gQ :GQeachline<CR>


these doesn't work well either --- once I did a gqq whenver got a
longline, I changed the "current" text lines and so the next iteration
might not work well.
this is especially hard when I tried to insert empty lines based on what
are the current lines
I did some research and didn't find an existing solution on this.

I feel this is a very generic task --- think about you got a ure text
book (bible) and you just want to format it better...

are there a good way to do it?



thanks!

regards
ping

[Gary Johnson]

On 2013-05-18, ping wrote:
> tested long time no good results...
>
> so the goal here sounds simple: say I have
>
> * a long text file (over 50K lines), and
> * each line by itself is a nature paragraph, so sometime is very long
> (over 1000 charactors).
> * there are sometimes one of more empty line in between, sometimes not
>
> I wanted to format it into:
>
> 1. if 2 lines are adjoining, add an empty line to as a break
> 2. if 2 lines already have more than 1 empty lines in between, remove
> the extra and make sure only one empty line left.

This is equivalent to replacing a sequence of one or more newlines
with two newlines.

:%s/\n\+/\r\r/

> 3. trunk the long lines to multiple lines with each of 80 charactors

:set tw=80
gggqG

Regards,
Gary

[ping]

Gary , thanks (again)!
I got it worked in these steps finally (sequence MATTERS):

1) :'<,'>g/.*/exec "normal! o\<esc>"
unconditionally insert a new line after each line (this may bring a lot
of empty lines , if the original book already contains empty lines)

2) format all each line seperately.
:'<,'>g/.*/exec "normal! gqq"
this is different than gggqG (this will mess up everything :D )

3) :'<,'>s/^\s*\n\n\+/\r/gc
use your method, delete all extra empty lines

[Tim Chase]

On 2013-05-18 20:22, ping wrote:
> 1) :'<,'>g/.*/exec "normal! o\<esc>"
> unconditionally insert a new line after each line (this may bring a
> lot of empty lines , if the original book already contains empty
> lines)

Any line matching ".*" could be written as just "^", and the
"normal..." bit can be done with an ex command, making a simpler
version:

:'<,'>g/^/put_

(put the null "_" register)

> 2) format all each line seperately.
> :'<,'>g/.*/exec "normal! gqq"
> this is different than gggqG (this will mess up everything :D )

and again here, using "^" instead of ".*"

-tim

[ping]

hi tim:
thanks for the improvement, I tested and your commands works.

I have 2 questions here:
1) about the magic "put_"
I couldn't understand, how come "put" a null register "_" will put a new
line?
my understanding is if it's a null then it shouldn't do anything "in
theory" right...
what's the idea behind this?

:[line]pu[t] [x] Put the text [from register x] after [line] (default
current line). This always works linewise, thus
this command can be used to put a yanked block
as new
lines.
The cursor is left on the first non-blank in
the last
new line.
The register can also be '=' followed by an
optional
expression. The expression continues until the
end of
the command. You need to escape the '|' and '"'
characters to prevent them from terminating the
command. Example: >
:put ='path' . \",/test\"
< If there is no expression after '=', Vim uses the
previous expression. You can see it with ":dis =".

2) now with the new way I have:

'<,'>g/^/put_
'<,'>g/^/exec "normal! gqq"

'<,'>s/^\s*\n\n\+/\r/

since I'm satisfied with this solution and I want to make it an easier
bind, so I make this:

function! FormatBook(...) range
exec a:firstline . "," . a:lastline . "g/^/put_"
exec a:firstline . "," . a:lastline . "g/^/exec " . "normal! gqq"
exec a:firstline . "," . a:lastline . "s/^\s*\n\n\+/\r/"
endf

com! -range=% -nargs=* FormatBook :<line1>,<line2>call
FormatBook(<f-args>)
map ,bf :FormatBook<CR>

but when I tested it report errors:

14 more lines
Error detected while processing function FormatBook:
line 2:
E121: Undefined variable: normal
E15: Invalid expression: normal! gqq
line 3:
E486: Pattern not found: ^s*^@^@+

seems something wrong with 2nd line:
exec a:firstline . "," . a:lastline . "g/^/exec " . "normal! gqq"

I know this is weired , but how to generalize this to make it work with
range via a simple key map?

thanks!

[Gary Johnson]

Isn't that explained in those two lines? _ is an empty register.
":put _" puts the contents of that register in a new line
("linewise") after the current line. Hence, ":put _" creates a new,
empty line.

I don't understand why you're going through all these gyrations to
perform what appears to be a simple task, as I answered originally.
If you want a set of lines to be formatted as paragraphs, with each
line as its own paragraph and separated by single blank lines, all
you have to do is first separate each line from the others by single
blank lines, which can be done by this:

:s/\n\+/\r\r/

with any range that you like. If some of your blank lines are not
empty (i.e., contain spaces), use this instead:

:s/\(\s*\n\)\+/\r\r/

Once all your paragraph lines are separated by blank lines, there is
no need to format each one individually; you can reformat them all
at once. Further, if you previously selected the lines visually,
you can reformat that same visual region with this normal-mode
command:

gvgq

Putting those together with a visual-mode mapping yields this:

:vnoremap ,bf :s/\(\s*\n\)\+/\r\r/<CR>gvgq

I don't care what solution you finally adopt. Vim offers many ways
to solve the same problem. I do care that you understand what
you're doing. I also care that I understand what I'm doing, so when
a solution I propose doesn't seem to work, I like to understand why.

Regards,
Gary

[ping]

this look much simpler, quick and works great. good to learn!

On 05/19/2013 02:30 PM, Gary Johnson wrote:

> Putting those together with a visual-mode mapping yields this:
>
> :vnoremap ,bf :s/\(\s*\n\)\+/\r\r/<CR>gvgq

still, just for a learning purpose, I'm curious about what's wrong with
the following script?

since I'm satisfied with this solution and I want to make it an
easier bind, so I make this:

function! FormatBook(...) range
exec a:firstline . "," . a:lastline . "g/^/put_"
exec a:firstline . "," . a:lastline . "g/^/exec " . "normal! gqq"
exec a:firstline . "," . a:lastline . "s/^\s*\n\n\+/\r/"
endf

com! -range=% -nargs=* FormatBook :<line1>,<line2>call
FormatBook(<f-args>)
map ,bf :FormatBook<CR>

specifically, I want to learn what's the right way to re-write the
following line in a ranged func ?

exec a:firstline . "," . a:lastline . "g/^/exec " . "normal! gqq"

thanks!

regards
ping

[Gary Johnson]

The action of the first exec is to expand and execute its arguments,
which in this case is a single argument, the string

a:firstline . "," . a:lastline . "g/^/exec " . "normal! gqq"

Assuming for sake of a concrete example a firstline of 123 and a
lastline of 456, that string expands to this:

123,456g/^/exec normal! gqq

Note that this second exec sees "normal!" and "gqq" not as strings
but as variables. That is what is causing the error message you
reported previously.

You don't need that second "exec" at all, so a better way to write
that command is this:

a:firstline . "," . a:lastline . "g/^/normal! gqq"

Regards,
Gary

[ping]

On 05/20/2013 01:03 PM, Gary Johnson wrote:
> The action of the first exec is to expand and execute its arguments,
> which in this case is a single argument, the string
>
> a:firstline . "," . a:lastline . "g/^/exec " . "normal! gqq"
>
> Assuming for sake of a concrete example a firstline of 123 and a
> lastline of 456, that string expands to this:
>
> 123,456g/^/exec normal! gqq
>
> Note that this second exec sees "normal!" and "gqq" not as strings
> but as variables. That is what is causing the error message you
> reported previously.
>
> You don't need that second "exec" at all, so a better way to write
> that command is this:
>
> a:firstline . "," . a:lastline . "g/^/normal! gqq"

I Got it now, thanks!