An interesting little poser

[Chris Jones]

I have a bunch of text files where the footnotes are in the following
format:

<----- START OF FILE ----->
Nobis delectus adipisci nostrum vel. Consequuntur ducimus repellat
saepe. Itaque [1] et sit voluptate non sint. Ab labore sapiente voluptatem
ratione illum voluptatem reprehenderit. Corporis sint exercitationem
doloremque impedit possimus laboriosam [2] nesciunt. Ex enim tenetur aut
dolorem sequi et nihil.

Explicabo ea itaque beatae. Earum [3] tenetur voluptatem tempora commodi.
Porro quibusdam vel quia repudiandae corporis ducimus.

[1]
STUFF THAT REFERS TO ITAQUE.¹
[2]
STUFF EXT THAT REFERS TO LABORIOSAM.
[3]
STUFF THAT REFERS TO EARUM.
...
<----- END OF FILE ----->

The footnotes are at the end of each file and I need to grab/delete them
in sequence and move/substitute each footnote's contents to the spot
where the corresponding anchor lives in the body of the text.

After substitution the first paragraph looks like this:

Nobis delectus adipisci nostrum vel. Consequuntur ducimus repellat
saepe. Itaque^[STUFF THAT REFERS TO ITAQUE.]² et sit voluptate non sint.
Ab labore sapiente voluptatem ratione illum voluptatem reprehenderit.
Corporis sint exercitationem doloremque impedit possimus
laboriosam^[STUFF THAT REFERS TO POSSIMUS LABORIOSAM] nesciunt. Ex enim
tenetur aut dolorem sequi et nihil.

Doing it manually is going to be somewhat tedious — not to the risk of
errors.

Seems a straightforward solution might consist in a regex-based search
of the files and populating some kind of associative array (dictionary?)
thusly as I go along:

{ '[1]' 'footnote #1', '[2]' 'footnote #2', ... }

... and proceed to a second pass that substitutes each anchor by the
contents of the corresponding footnote.

Before I do this, I was wondering if anyone had come across a similar
problem and might have come up with an elegant solution - i.e. one that
relies on a clever utilisation of vim's search & replace features rather
than writing code.

Thanks,

CJ

¹ upper case for highlighting/legibility purposes.
² ^[blah...] markdown syntax.

[Dave Bucklin]

This is not a Vim answer, but I've done somethi g in Awk that does something similar. You might be able to port it to VimScript if that's your need.

https://gitlab.com/davebucklin/gopher-tools/-/blob/master/ref2link

[Tim Chase]

On 2020-06-21 19:18, Chris Jones wrote:
> Seems a straightforward solution might consist in a regex-based
> search of the files and populating some kind of associative array
> (dictionary?) thusly as I go along:
>
> { '[1]' 'footnote #1', '[2]' 'footnote #2', ... }

To build this associative array, you might try

:$?^$?,$g/\[\d\+\]/let b:a[getline('.')]=getline(line(".")+1)

This assumes

- the content of each footnote is only one line (snagging multi-line
footnotes becomes an uglier task)

- all your footnotes are compactly at the end of the file with no
blank lines between the start of the footnotes and the end of the
file. If there are, you'd have to manually specify the range to
cover the range of footnotes instead of using the
automatically-determined range "$?^$?,$"

> ... and proceed to a second pass that substitutes each anchor by the
> contents of the corresponding footnote.

You can then use

:1,$?^$?s/\[\d\+]/\='^['.b:a[submatch(0)].']'/g

(again, if you have blank lines among your footnotes, manually
specify the end of the text-range instead of automatically
determining it with "$?^$?") to find all the footnote-references in
your text and replace them with the footnote text with added markup
notation.

You can then do simple clean-up to remove the now-redundant footnote
block:

:$?^$?,$d

and reformat/reflow everything as you see fit, maybe with

gggqG

Hope this helps,

-tim

[Tim Chase]

On 2020-06-21 18:57, Tim Chase wrote:
> To build this associative array, you might try
>
> :$?^$?,$g/\[\d\+\]/let b:a[getline('.')]=getline(line(".")+1)

Whoops, before you do this, you might have to let vim know that b:a
is an array:

:let b:a={}

:$?^$?,$g/\[\d\+\]/let b:a[getline('.')]=getline(line(".")+1)

> You can then use
>
> :1,$?^$?s/\[\d\+]/\='^['.b:a[submatch(0)].']'/g

Oh, this also assumes that all footnote-references have corresponding
entries in the footnote block. If you have a [13] and there's no
[13] footnote at the bottom, that substitute will yell at you about a
"E716: Key not present in Dictionary: {bogus footnote}"

-tim

[Chris Jones]

Congrats! Works out of the box and does exactly what I had in mind:
copy-pasted the commands and the result is spectacular. The clever idea
is to use the :g(lobal) command to build the array.

I posted a little too fast and left out an important third³ footnote at
the bottom of my message... something like 'the footnotes in the example
are only one line long so as not to clutter up your screen... but in my
use case many are multi-line'.

But after playing with my files for a while it's pretty easy to reflow
the footnotes so that they're only one line long:

1. add a null line between the footnotes
2. reflow them with a largee textwidth value
3. add a line break after each [nnn] in column 1
4. remove the null lines added in 1

Having done this I'm back to where I was (but with one-line notes)
& your solution will work the same.

What are all these magical dollar signs in the "g(lobal)" and
"s(ubstitute)" commands? They appear to define a vim command 'range' but
since I've never used ranges — and I'm damned if I can figure out how to
read this particular bit of magic...

Since I couldn't think of a way to highlight the targeted section of the
file I did a:

:$?^$?,$w /tmp/t.txt

... and I saw that the footnotes section at the end of the files was
excluded from the resulting /tmp/t.txt file.

Please explain.

Oh, and how would I go about running this sequence of commands on
a bunch of open file (buffers) in a vim session? what would the best
tactics to do :bufdo of all these commands?

Well... I had a hunch this could be 'done the smart way' simply using
basic vim commands... gladd I asked...

Thanks,

CJ

[Tim Chase]

On 2020-06-22 20:16, Chris Jones wrote:
> Congrats! Works out of the box and does exactly what I had in mind

Great!

> copy-pasted the commands and the result is spectacular. The clever
> idea is to use the :g(lobal) command to build the array.

I do love the power of the :g command and abuse it regularly

> I posted a little too fast and left out an important third³
> footnote at the bottom of my message... something like 'the
> footnotes in the example are only one line long so as not to
> clutter up your screen... but in my use case many are multi-line'.
>
> But after playing with my files for a while it's pretty easy to
> reflow the footnotes so that they're only one line long:
>
> 1. add a null line between the footnotes
> 2. reflow them with a largee textwidth value
> 3. add a line break after each [nnn] in column 1
> 4. remove the null lines added in 1

Glad you were able to hack a solution. Again abusing the :g command,
I might have done the preprocessing as

:$?^$?+,$g/^\[\@!.*\n\[\@!/,/\n\[\|\%$/j

to do the joining for me :-) That uses the range for the footnotes
(see below) and looks for any line that doesn't start with a "["
character ("\[\@!"), followed by whatever on that line (".*"),
followed by a newline ("\n"), followed by the assertion that a
literal "[" doesn't start on this 2nd line either ("\[\@!"). Or
roughly translated "find footnote text that is continued on the next
line". Starting at that point, create a second range from there (that
second ",") that goes through either the newline followed by a
literal "[" (another footnote follows here) or ("\|") the end of file
("\@$") and join those lines together ("j").

It's big & ugly, but it does the job in one go.

> What are all these magical dollar signs in the "g(lobal)" and
> "s(ubstitute)" commands? They appear to define a vim command
> 'range' but since I've never used ranges

A range can be modified relative to each location. So the range of
lines footnote lines we're interested in are

:$?^$?,$

Breaking that down, the first "$" says "starting at the last line of
the file", then ?…? search backwards until we find "^$" (an empty
line), and start the range there. In the above :j(oin) example, I
start the range one line after that ("+" which is the same as "+1").
The comma delineates the start of the range (that line we just found,
roughly defined as either "the blank line preceeding the last line of
the file" or with the "+" it's "the line after the blank line
preceeding the last line of the file"). So now after the comma we
define the end of the range as the last line of the file ("$").

For the body-text of your document, the range is "1" (the first line
in the file) through (",") the "blank line preceeding the last line
in the file" (same as before, "$?^$?").

> Since I couldn't think of a way to highlight the targeted section
> of the file I did a:
>
> :$?^$?,$w /tmp/t.txt

For any of these ranges, you could also use other ways of defining
them. If the last blank line in the file is line 314, then the
footnotes are

:315,$

and the body text would be the range

:1,313

Alternatively you could highlight them in visual mode with the "V"
command and vim would automatically populate the range as

:'<,'>

(i.e. "from the first visually-selected line through the last
visually-selected line").

The advantage to describing the range syntactically is that it
doesn't need to manually find those line-numbers or manually select
things visually, allowing you to automate it across multiple files as
is your next request: ;-)

> Oh, and how would I go about running this sequence of commands on
> a bunch of open file (buffers) in a vim session? what would the best
> tactics to do :bufdo of all these commands?

In this case because each of the commands involves the :g command, it
becomes a bit trickier. I'd likely define a function something like

function! Unfootnote()
" make all the footnotes on one line
$?^$?+,$g/^\[\@!.*\n\[\@!/,/\n\[\|\%$/j
" gather the footnotes into b:a

let b:a={}
$?^$?,$g/\[\d\+\]/let b:a[getline('.')]=getline(line(".")+1)

" find all the footnote references
" and replace them with the corresponding text

1,$?^$?s/\[\d\+]/\='^['.b:a[submatch(0)].']'/g

endfunc

Note how each of those Ex commands we've discussed is a valid command
in the body of a function as well. Yet another obscure corner of vim
that escapes many folks. :-)

You should then be able to invoke that across all the files with
bufdo/argdo, something like

:set hidden
:bufdo call Unfootnote()

and, after reviewing that it did what you wanted, issue

:wall

if they meet your needs.

Note that, while the parts of the function are reasonably tested, this
function itself is largely untested, but *should* be pretty close.

Hopefully this both makes sense and helps level up your vim, letting
you get drunk on this new-found power. :-)

-tim

[Chris Jones]

On Mon, Jun 22, 2020 at 09:40:38PM EDT, Tim Chase wrote:

Sorry for the delay... needed time for this to sink in (and test)

> In this case because each of the commands involves the :g command, it
> becomes a bit trickier. I'd likely define a function something like
>
> function! Unfootnote()
> " make all the footnotes on one line
> $?^$?+,$g/^\[\@!.*\n\[\@!/,/\n\[\|\%$/j
> " gather the footnotes into b:a
> let b:a={}
> $?^$?,$g/\[\d\+\]/let b:a[getline('.')]=getline(line(".")+1)
> " find all the footnote references
> " and replace them with the corresponding text
> 1,$?^$?s/\[\d\+]/\='^['.b:a[submatch(0)].']'/g
> endfunc

Works just fine! I wrapped the function in an if/endif block to make
sure there does exist footnotes in any given file (some of the files do
not have a footnotes section at the bottom)

:normal gg
:let b:s=getpos('.')
" check for the existence of a footnotes
:silent! normal /\[\d\+]
:let b:e=getpos('.')
if b:s[1] != b:e[1]

Testing whether the search was successful which would cause the cursor
to move... Couldn't figure out a nicer way...

> Note how each of those Ex commands we've discussed is a valid command
> in the body of a function as well. Yet another obscure corner of vim
> that escapes many folks. :-)

That at least hadn't escaped me... so much so that I wasted a couple of
hours trying to figure out why a ':normal /pattern' worked at the prompt
and didn't work when I coded it in a function... the cursor just refused
to budge and no matter how much I tested I couldn't figure out why...
until I calmed down and realized that what I coded was NOT what I had
actually typed at the ex prompt:

:normal /pattern
# ex command PLUS <CR> - big difference!

...

> Note that, while the parts of the function are reasonably tested, this
> function itself is largely untested, but *should* be pretty close.

It is perfectly suitable thank you! Without the if/endif block it didn't
break anything... just joined all the lines in the last paragraph of the
file (which didn't matter since I later reflow everything anyway — see
the other function I came up with).

Here's the function I wrote (I could integrate the code to the one you
kindly provided to do it all in one pass):

function! Delfoot()
:normal gg
:let b:s=getpos('.')
:silent! normal /^$\n\[[0-9]\{1,}]
:let b:e=getpos('.')
if b:s[1] != b:e[1]
:let b:f=expand('%:t')
:let b:fn='../ftns/ft'.b:f[2:5].'.txt'
:execute ".,$w " b:fn
:execute ".,$d"
endif
:set tw=78
:normal ggVGgq
endfunc

Nothing clever about this one! :-) ... I backup the foonote section at
the end of each file before I do away with it (and reflow everything to
a sensible textwidth to finish off the job so the files are ready for
edit/proofreading).

> Hopefully this both makes sense and helps level up your vim, letting
> you get drunk on this new-found power. :-)

Made up for that depressing feeling I get whenever I venture into vim
script coding and realize I have forgotten the little I know.

I eventually found a vim fandom article that has some examples of what
can be done with syntaxically built ranges

Thank,

CJ

[Chris Jones]

On Mon, Jun 22, 2020 at 09:40:38PM EDT, Tim Chase wrote:
>

> function! Unfootnote()
> " make all the footnotes on one line
> $?^$?+,$g/^\[\@!.*\n\[\@!/,/\n\[\|\%$/j
> " gather the footnotes into b:a
> let b:a={}
> $?^$?,$g/\[\d\+\]/let b:a[getline('.')]=getline(line(".")+1)
> " find all the footnote references
> " and replace them with the corresponding text
> 1,$?^$?s/\[\d\+]/\='^['.b:a[submatch(0)].']'/g
> endfunc

When I run the above function (as copy-pasted from this thread) I get an
'E16: Invalid range' error on the g(global) command on some files.
Interestingly the function still does the job as if nothing had
happened. I went back to the explanation you gave in your earlier post
and after comparing with other files where I do not get the error
I can't find what's wrong with it. It looks like the range defining the
footnotes section is perfectly valid and that the Vim has
a problem with the pattern used in the g:/pattern/j command (I did
a $?^$?+,$print and it prints the footnotes section to the screen and
does not yell at me).

The interesting point is that the g: command's pattern has two parts
separated by a comma... which makes it suspiciously look like...
a range!

So is Vim confused by the regex that makes up the /pattern/…?

The problem occurs for instance with the following example file:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Debitis neque ipsum eos aut. Voluptatum ratione voluptatum facere voluptas.
Maxime qui ad autem ducimus quae [36]. Officia labore iusto voluptas ad quia
nostrum.


[36]
Ullam consequatur.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

But if I add a carriage return between Ullam & consequatur like so:

[36]
Ullam
consequatur.

... the problem goes away!

Not sure if it's some kind of corner case or this little experimented
has unearthed a bug in Vim's regex logic... and unfortunately I have no
idea how to investigate further.

Assuming you manage to reproduce the error is there any way you could
take a (quick) look?

Thanks,

CJ

[Mateusz Okulus]

I've made the following macro that works for the given example, assuming
1 line of footnote.

Run the following (copy and paste)

:map <F2> G?\[\d\+\]<CR>va[y/<C-r>0<CR>dd0DnF[hr^llPdt]

Press F2 to run this macro. You can also hold the key
and will get a message when it's done.

Explanation:
G - to to the bottom of the file
? - start reverse search (search will look up)
\[\d\+\] - regex - nonzero digits inside square brackets
<CR> - enter to finish entering regex
va[ - select square brackets and digits inside
y - yank them
/ - search (this one will wrap)
<C-r>0 - insert content of register 0, which holds last yank
allowing us to search for the [digits] pattern
<CR> - enter to finish entering search
dd - delete line with [digits]; we are now on footnote line
0 - go to the beginning of the line
D - delete to the end of line, without newline, so when we paste it
won't be with newline
n - go to the the place where the footnote is referenced
F[ - search backward for [
h - 1 left
r^ - replace space with ^
I've noticed the space is removed in example.
ll - go to the first digit, skipping ^[
P - paste footnote before the digits
dt] - delete the digits, without ]

You can enter this manually and to see what's happening.
I think it's much easier to analyze and understand than a function with
weird regexes, and it shows you how the result looks on one-by-one basis.
You might need to manually clean some whitespace at the end after it's finished.

Regards,
mat

[Tim Chase]

[sorry for the belated reply.]

On 2020-06-27 22:46, Chris Jones wrote:
> Not sure if it's some kind of corner case or this little
> experimented has unearthed a bug in Vim's regex logic...

Not a bug in the regex parser. See further diagnosis:

> On Mon, Jun 22, 2020 at 09:40:38PM EDT, Tim Chase wrote:
> > " make all the footnotes on one line
> > $?^$?+,$g/^\[\@!.*\n\[\@!/,/\n\[\|\%$/j

The problem lies here. If you remove the second half of the range

:$?^$?+,$g/^\[\@!.*\n\[\@!/

vim is finding lines that need to be joined. However in your example

> The problem occurs for instance with the following example file:
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

[snip]

> [36]
> Ullam consequatur.
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

that last line ends in a newline that isn't followed by a \[ so my
regex thinks it should join this. But then when it goes to find the
ending of the range to join, it looks for something slightly
different.

Try changing that line from

$?^$?+,$g/^\[\@!.*\n\[\@!/,/\n\[\|\%$/j

to

$?^$?+,$g/^\[\@!.*\n\[\@!./,/\n\[\|\%$/j

(adding that "." after the "make sure a literal [ doesn't happen
here") to ensure that there's something for the line to join.

Seems to resolve the issue for me without breaking my other
test-cases that I threw at it.

You might still get "errors" (more like complaints) if you have a
file containing 0 multi-line footnotes, telling you "Pattern not
found". But that should be ignorable because there simply weren't any
footnote lines for it to join together.

-tim

[Chris Jones]

On Sun, Jun 28, 2020 at 05:37:04PM EDT, Mateusz Okulus wrote:
>
> Run the following (copy and paste)
>
> :map <F2> G?\[\d\+\]<CR>va[y/<C-r>0<CR>dd0DnF[hr^llPdt]

Nice! Really does a spectacular job when run on my toy example...
especially when one presses and holds down <F2> key.

But there appears to be a problem with the macro when the
digit(s)/number that define the footnotes (say '3' for footnote [3]) is
also present somewhere in the body of the text - e.g. in a date such as
'1234'.

I think the problem is caused by the '[3]' string yanked via 'va]' being
interpreted as a regex that is a 'collection' whose sole member is the
single character '3'. So that the ensuing '/<c-r>0' (results in '/[0]'
search) looks for all occurrences of the character '3' instead of the
intended ('['+'3'+']') string. This behaviour can be verified
interactively after adding a date such as '1234' anywhere in the body of
the example and doing a '/[3]' repeatedly (as opposed to a '/\[3]').

Here's a version that appears to fix a couple of problems:

:map <F2> G?\[\d\+\]<CR>va]y/\<C-R>0<CR>ndd0Dnhr^llPldt]
^ ^ ^
(a) (b) (c)

a. escape first '[' when searching for '[3]'.

b. added 'n' to skip the first occurrence of '[3]' (the footnote anchor) so the
cursor is back at the beginning of the footnote proper.

c. extra 'l' to avoid deleting the last character at the end of the
footnote.

I don't know if there's a way to run a macro against a bunch of files in
one pass so I may have to to convert the macro to a function that can be
run via :bufdo/:argdo — 700+ files, with some 20-30 footnotes apiece.

Thanks,

CJ

[Mateusz Okulus]

You are right, I made a mistake. I was originally confused a bit why I
would end on the digit instead of [ after pressing n. I think you can
record the macro in a register and do something like

:argdo normal 99999 @q

I used the map version because it escapes <CR>, while if you wanted to
yank it into e.g. q register you would need ^M (a single character).

Regards,
mat

[aro...@vex.net]

>
> I don't know if there's a way to run a macro against a bunch of files in
> one pass

In that case, I'd suggest using a shell script to execute vim on the files
individually. Obviously, going to use slightly more machine time, since
it's loading vim n times, but it'c going to be more flexible, (e.g. doing
the job in batches), to trade off against human development time.

[Chris Jones]

On Mon, Jun 29, 2020 at 03:34:03PM EDT, Tim Chase wrote:

> Try changing that line from
>
> $?^$?+,$g/^\[\@!.*\n\[\@!/,/\n\[\|\%$/j
>
> to
>
> $?^$?+,$g/^\[\@!.*\n\[\@!./,/\n\[\|\%$/j

This indeed fixes the problem.

Still trying to wrap my head around this bizarre

:g/regex1/,/regex2/command ... syntax.

I eventually found something in the user manual that (sort of) explains
what's going on here... under [edit-paragraph-join].

What I understand is that somewhat in a 'shorthand manner'... with the
:g(lobal) command you can specify a second /patttern2/ (separated from
the 'normal' g /pattern1/ by a comma) defining a range for the command
executed by :global... or in other words that the command executed by
':g' will target not the current line (as is normally the case) but
rather all the lines in the range from 'current line' (the line :g is
currently point at) to the next line in the buffer that matches
/pattern2/...

Rather comical that I spent so much time wondering what that comma
somewhere in the middle of what I assumed was one single regex was
about...! never imagined this comma in '/pattern1/,/pattern2/' was not
part of a regex but rather the range separator... Incidentally this
clarifies vim's 'invalid range' message - i.e. it's the join command's
range he's complaining about... not the global :g range!

In any case the modified Unfootnote() function now does the job without
a glitch.

Thank you very much for your help!

CJ

[Tim Chase]

On 2020-06-30 18:33, Chris Jones wrote:
> On Mon, Jun 29, 2020 at 03:34:03PM EDT, Tim Chase wrote:
>> Try changing that line from
>>
>> $?^$?+,$g/^\[\@!.*\n\[\@!/,/\n\[\|\%$/j
>>
>> to
>>
>> $?^$?+,$g/^\[\@!.*\n\[\@!./,/\n\[\|\%$/j
>
> This indeed fixes the problem.

Great!

> Still trying to wrap my head around this bizarre
>
> :g/regex1/,/regex2/command

It helps to think of broken down. The :g command itself can take one
range:

:{range1}g/regex1/{command}

and that {command} can consist of a command *and an optional range
relative to each of those matches*. So in the above, that {command}
is

{range2}command

and that {range2} is

,/regex2/

which has the start of the range "." implied, so would be the same as

.,/regex2/

The power of this "find lines with a :g command, then operate on a
range of lines relative to each of those matches"


So it might be clearer to write that original as

$?^$?+,$ g/^\[\@!.*\n\[\@!./ .,/\n\[\|\%$/ j

11111111 2222222222222222222 3333333333333 4

where

1 = the range for the footnotes
2 = find lines where the next line is a continuation
3 = a range from this continued-line through the end of the footnote
4 = join that range of lines from #3

To learn this requires pulling together bits from several sections of
the help:

:help range
:help search-offset
:help 10.3 " particularly the ADD AND SUBTRACT section
:help 10.4

so it's certainly not a well-advertised bit of functionality.

But as you can see, it's an amazingly *powerful* bit of functionality
that I use all the time.

> Thank you very much for your help!

Thanks for the fun challenge. :-)

-tim

[Chris Jones]

On Sun, Jun 28, 2020 at 05:37:04PM EDT, Mateusz Okulus wrote:

Above pseudo-code translated to a function wrapped in a while loop.

function! Footnotes2()
:normal G
:while search('^\[\d\+\]$', 'bc')
:normal va]
:normal y
:let b:p='\'.@0
:execute "call search(b:p)"
:execute "call search(b:p)"
:normal dd
:normal 0
:normal D
:execute "call search(b:p)"
:normal h
:normal r^
:normal ll
:normal P
:normal l
:normal dt]
:normal G
:endwhile
endfunc

I prefer a function for flexibility: I can easily add pre-processing
& post-processing code - e.g. to reflow multi-line footnotes... remove
the NULL lines at EOF... etc.

Thanks,

CJ