best library for data forms entry in vim

59 views
Skip to first unread message

hor...@gmail.com

unread,
Nov 17, 2020, 2:40:26 PM11/17/20
to vim_use
All,

I need to quickly enter data for thousands of structured documents - ie: where I know the bulk of the document but there are placeholders for specific variables. Think something like this:

    Dear <person>

    We received your order for <name>, price <price>, quantity <quantity>, total <total>.

    Please look over the above information to make sure that this is correct.

I was hoping vim could do something like this, where it could limit the input for the user to just the categories - leaving the form intact, and where the different inputs could be linked so that some values could be calculated via the values in other inputs.

In other words, an efficient form of excel.

Anyways I was looking at vim.org, and there are a number of forms libraries there but at first glance I'm not sure which one is the most supported, user friendly, or overall functional.

So my question is what is my best bet? this would save many man-months worth of time and I was actually hoping that something like this would be supported natively in vim itself - but at first glance I don't see anything like this.
Any pointers on this would be greatly appreciated.

Ed

(ps - 

 My ideal interface would be something integrated into vim itself where I could embed pieces of perl or python code in the fields - and have the values of variables be set when users enter in these fields. Something like:

Dear {{ $person }},

    We received your order for  {{ $name }}, price {{ $price }}, quantity {{ $quantity }}, total {{ sub { $price * $quantity } }}.

but if there is an external form library that would do the equivalent I'd be happy to use it.
)

rwmit...@gmail.com

unread,
Nov 17, 2020, 3:24:51 PM11/17/20
to vim_use
I would think sed is best choice as the entire substitution could be scripted.

Ruben Safir

unread,
Nov 17, 2020, 3:27:22 PM11/17/20
to vim...@googlegroups.com
On 11/17/20 3:24 PM, rwmit...@gmail.com wrote:
> I would think sed is best choice as the entire substitution could be
> scripted.
>

LISP

> On Tuesday, November 17, 2020 at 2:40:26 PM UTC-5 hor...@gmail.com wrote:
>
> All,
>
> I need to quickly enter data for thousands of structured documents -
> ie: where I know the bulk of the document but there are placeholders
> for specific variables. Think something like this:
>
>     Dear <person>
>
>     We received your order for <name>, price <price>,
> quantity <quantity>, total <total>.
>
>     Please look over the above information to make sure that this is
> correct.
>
> I was hoping vim could do something like this, where it could limit
> the input for the user to just the categories - leaving the form
> intact, and where the different inputs could be linked so that some
> values could be calculated via the values in other inputs.
>
> In other words, an efficient form of excel.
>
> Anyways I was looking at vim.org <http://vim.org/>, and there are a
> number of forms libraries there but at first glance I'm not sure
> which one is the most supported, user friendly, or overall functional.
>
> So my question is what is my best bet? this would save many
> man-months worth of time and I was actually hoping that something
> like this would be supported natively in vim itself - but at first
> glance I don't see anything like this.
> Any pointers on this would be greatly appreciated.
>
> Ed
>
> (ps - 
>
>  My ideal interface would be something integrated into vim itself
> where I could embed pieces of perl or python code in the fields -
> and have the values of variables be set when users enter in these
> fields. Something like:
>
> Dear {{ $person }},
>
>     We received your order for  {{ $name }}, price {{ $price }},
> quantity {{ $quantity }}, total {{ sub { $price * $quantity } }}.
>
> but if there is an external form library that would do the
> equivalent I'd be happy to use it.
> )
>
> --
> --
> You received this message from the "vim_use" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>
> ---
> You received this message because you are subscribed to the Google
> Groups "vim_use" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to vim_use+u...@googlegroups.com
> <mailto:vim_use+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vim_use/4990ea22-e094-48e4-8609-64b47dae68f9n%40googlegroups.com
> <https://groups.google.com/d/msgid/vim_use/4990ea22-e094-48e4-8609-64b47dae68f9n%40googlegroups.com?utm_medium=email&utm_source=footer>.


--
So many immigrant groups have swept through our town
that Brooklyn, like Atlantis, reaches mythological
proportions in the mind of the world - RI Safir 1998
http://www.mrbrklyn.com

DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002
http://www.nylxs.com - Leadership Development in Free Software
http://www2.mrbrklyn.com/resources - Unpublished Archive
http://www.coinhangout.com - coins!
http://www.brooklyn-living.com

Being so tracked is for FARM ANIMALS and and extermination camps,
but incompatible with living as a free human being. -RI Safir 2013

hor...@gmail.com

unread,
Nov 17, 2020, 5:12:01 PM11/17/20
to vim_use
I'm not sure I made myself clear -  this particular application needs to be interactive.

In other words, you enter in values in one location, it calculates derivative values in another location, and you keep working on it until you are satisfied that you are in fact done with it. That is why vim stuck out as a possible solution - it really needs an editor. All my example was meant to do was show what I had in mind.

so have people attempted this? if so which api should I be looking at..


asymptosis

unread,
Nov 17, 2020, 5:14:32 PM11/17/20
to vim...@googlegroups.com
>I need to quickly enter data for thousands of structured documents - ie:
>where I know the bulk of the document but there are placeholders for
>specific variables. Think something like this:
> ...

Without specifying why you are constrained to only using Vim, it's hard to avoid pointing out that there are better ways to do this.

Either m4 or mustach should be your friends unless you can explain why Vim is literally the only program that could possibly be used.

Recommended reading:

https://www.catb.org/~esr/writings/unix-koans/shell-tools.html

hor...@gmail.com

unread,
Nov 17, 2020, 7:46:54 PM11/17/20
to vim_use
mustache and m4 are tools for automation right? Ie: you are reliant on data-structures and can't interact with them - ie you can't have users enter in input and have that input applied correct?

again, the reason I want vim is because it is an editor. Users could enter in values, have derivative values calculated for them and then inserted directly into the same form - in the editor. Again, think excel and macros, except with free text. It needs to be interactive because the users need to be able to play around with given constraints and see which constraints work for them given specific input - before saving that form for a program to later use it.

Since users are going to be potentially editing hundreds or thousands of files, said templates will need to do data checking and should be programmable to do both calculations and these checks- and since vim already has the possibility of having either perl or python embedded into it, vim seemed to be a powerful and natural fit. The fact that people also know vim in my particular environment points to vim as well.

If I'm wrong about this or you know tools to do this please let me know, but I'm pretty sure because my application is interactive I need the power of an editor here.

Shlomi Fish

unread,
Nov 18, 2020, 3:14:41 AM11/18/20
to asymptosis, vim...@googlegroups.com
Hi all!

On Wed, 18 Nov 2020 09:14:13 +1100
asymptosis <asymp...@posteo.net> wrote:

> >I need to quickly enter data for thousands of structured documents - ie:
> >where I know the bulk of the document but there are placeholders for
> >specific variables. Think something like this:
> > ...
>
> Without specifying why you are constrained to only using Vim, it's hard to
> avoid pointing out that there are better ways to do this.
>
> Either m4 or mustach should be your friends unless you can explain why Vim is
> literally the only program that could possibly be used.
>

I can recommend *against* using m4, which although part of the POSIX standard,
is limited and vile in both syntax and behaviour. I have a page on my site with
a roundup of similar text-processing tools and links including preprocessors,
and template systems:

https://www.shlomifish.org/open-source/resources/text-processing-tools/

(Licence is https://creativecommons.org/licenses/by/3.0/ - share and enjoy. The
sources are here:
https://github.com/shlomif/shlomi-fish-homepage/blob/master/src/open-source/resources/text-processing-tools/index.xhtml.tt2
)

> Recommended reading:
>
> https://www.catb.org/~esr/writings/unix-koans/shell-tools.html
>

Heh, nice. Regarding awk and ESR, I wrote this blog post:
https://shlomifish.livejournal.com/1991.html reflecting on ESR's case against
awk on TAOUP.

Regards,

Shlomi Fish


--

Shlomi Fish https://www.shlomifish.org/

Chuck Norris has 99 problems including a bitch.
https://www.shlomifish.org/humour/bits/facts/Chuck-Norris/

Please reply to list if it's a mailing list post - https://shlom.in/reply .

Gabriele

unread,
Nov 18, 2020, 3:37:19 AM11/18/20
to vim...@googlegroups.com
On 18/11/2020 9.14, Shlomi Fish wrote:
> I can recommend *against* using m4, which although part of the POSIX standard,
> is limited and vile in both syntax and behaviour. I have a page on my site with
> a roundup of similar text-processing tools and links including preprocessors,
> and template systems:
>
> https://www.shlomifish.org/open-source/resources/text-processing-tools/

Well as we're at it I can strongly recommend against GPP, which I see
listed first on that page.

I haven't had experience with the linked logological.org version though,
it's not entirely impossible that they improved it a lot.

Of course this has still nothing to do with what the thread author wanted

BPJ

unread,
Nov 18, 2020, 10:51:02 AM11/18/20
to vim_use
Den tis 17 nov. 2020 23:12hor...@gmail.com <hor...@gmail.com> skrev:
I'm not sure I made myself clear -  this particular application needs to be interactive.

In other words, you enter in values in one location, it calculates derivative values in another location, and you keep working on it until you are satisfied that you are in fact done with it. That is why vim stuck out as a possible solution - it really needs an editor. All my example was meant to do was show what I had in mind.

I had the same need a couple of years ago. The script I wrote for the purpose (and just updated for publicizing) does not work from inside an editor but does interactively replace placeholders (format configurable) in a text file with prompted-for values, optionally silently inserting already "known" values on subsequent occurrences of the same placeholder. You can also optionally preload a default key-value mapping from a YAML or JSON file and/or save the collected/modified data to a YAML file.



so have people attempted this? if so which api should I be looking at..


--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vim_use/c3b018f8-bd51-44a6-900f-adbc7dbd22een%40googlegroups.com.

hor...@gmail.com

unread,
Nov 18, 2020, 4:13:41 PM11/18/20
to vim_use
BPJ,

Thanks for the feedback and idea, and i'll check out your tool. You know, that's my fallback plan, to script it up and do substitutions based off of user input if necessary. I've done something very similar before, and its workable.

But I submit to you that being able to do it via an editor would be an order of magnitude more scalable and usable. You could put in helpful keybindings to speed up data processing, use certain keys to calculate derivative windows to help users evaluate their input, and use the editor within a script to present multiple cases to end users for them to process one after another. 

My goal was to have users take about 20 seconds per use case with the help of scripting, in-vim macros, and vim's very useful perl and python bindings. It would reduce error rates as well.

If you had a clean Implementation for this, you could make vim the center for a whole new class of applications based off of data processing.  Again, think excel for unix.

In any case, if I was going to provide low-level support for this in vim itself, I'd make it so that there was a file of type template, where:

1. the state of the default text of the file could be tagged as uneditable - ie: the vim cursor could only edit certain parts of the file based on tags given by the user. All vim keys would 'skip over these uneditable parts and go directly to the next user entry portion
2. the default state of the file would be insert mode, with helpful key macros to go to the next line.
3. The editable parts would have callbacks associated with them. Each callback could be then specified in the file in the form of a python or perl piece of code which either would set input or calculate input.

So my original example would look something like this. Each time a user would enter in value, the callbacks would be re-executed and any updates would be made to the script. If a value is undef, it would show up as a red piece of text.

I maintain that this would be very powerful. My personal application is making training cases for a nn, but I could see all sorts of applications for it.

Ed

 Dear <<<person>>>>

We received your order for <<<name>>>, price <<<price>>>, quantity <<<quantity>>>, total <<<total>>>

Please look over the above information to make sure that this is correct.

<<<email>>>
__CODE__
{
    'person' => sub { $person = $_[0] },
    'name'   => sub { $name = $_[0] },
    'price'.   => sub { $price = $_[0] },
    'quantity' => sub { $quantity = $_[0] },
    'total'.       => sub { $total = $price * $quantity }
    'email'        => sub { if  (($_[0]) && _is_email($_[0])) {  _mailit( $_[0], $text) }

BPJ

unread,
Nov 18, 2020, 4:42:42 PM11/18/20
to vim_use
Den ons 18 nov. 2020 22:14hor...@gmail.com <hor...@gmail.com> skrev:
BPJ,

Thanks for the feedback and idea, and i'll check out your tool.

Thanks. Note that I just pushed a rather vital fix for a bug which I introduced earlier today!

/bpj

Shlomi Fish

unread,
Nov 19, 2020, 4:30:17 AM11/19/20
to Gabriele, vim...@googlegroups.com
Hi Gabriele,

On Wed, 18 Nov 2020 09:36:30 +0100
Gabriele <gb...@tiscali.it> wrote:

> On 18/11/2020 9.14, Shlomi Fish wrote:
> > I can recommend *against* using m4, which although part of the POSIX
> > standard, is limited and vile in both syntax and behaviour. I have a page
> > on my site with a roundup of similar text-processing tools and links
> > including preprocessors, and template systems:
> >
> > https://www.shlomifish.org/open-source/resources/text-processing-tools/
>
> Well as we're at it I can strongly recommend against GPP, which I see
> listed first on that page.
>
> I haven't had experience with the linked logological.org version though,
> it's not entirely impossible that they improved it a lot.
>

Thanks for the information. I should also note that the distinction between
template systems and preprocessors is often blurred, similarly to the
distinction between text editors and IDEs. I used to list m4 first in that
section, but someone thought I somehow recommend or endorse it due to that.

> Of course this has still nothing to do with what the thread author wanted
>

Right, I've only replied due to
https://explainxkcd.com/wiki/index.php/386:_Duty_Calls .
https://youtu.be/GoEn1YfYTBM - Tiffany Alvord - “Fall Together”

Every successful open source project will eventually spawn a sub‐project.
https://www.shlomifish.org/humour/fortunes/osp_rules.html

hor...@gmail.com

unread,
Nov 19, 2020, 1:08:27 PM11/19/20
to vim_use
You know, the more I'm thinking about it, the more I think vim needs an internal templating system. 

As it stands, I get how this is done in vim now - you use vimscript to create the template for you, in other words you need to program the look and feel. It has a lot in common with approaches like asciimatics in python (https://github.com/peterbrittain/asciimatics). 

This is IMO a lot more difficult than it need be. For example, leandro posted a todo application for me to see how its done (https://github.com/freitass/todo.txt-vim) and I looked at it - about 250 lines of vimscript code to get the formatting where you can enter in todo lines. I'd rather have vim itself do the formatting for me, and just worry about the actual logic behind it. So that app would become

todo.tmpl

----
Todo
----
Done                                                                    Name                                                                                       Status
[[ {{ type=box, name=_donebox align=left }} {{ type=string[35], name = _task, align=left, color=red }} {{type=string[35], name=_status, align=left }} ]]



where you could define the elements you want in your application inline and attach pieces of code to them to do any variable manipulation. Arrays of lines could be designated by  [[.   ]],  Any user input could directly map onto python or perl in the form of python/perl arrays, or perhaps arrays of hashes - and each element would have its own coderef associated with it if you needed to process the output in any way.

I've mocked this up by making vim separately called from perl, memoizing all the variables that are defined in the template, and then responding to user input by hack - the user needs to close the actual vim session in order for perl to actually do the recalculation and realignment. This works (after a fashion) but it really is a hack and wouldn't be usable for what i want to do. But i'm hoping that perhaps a templating system could make its way into a newer version of vim.

Ruben Safir

unread,
Nov 19, 2020, 5:53:41 PM11/19/20
to vim...@googlegroups.com
On 11/19/20 1:08 PM, hor...@gmail.com wrote:
> You know, the more I'm thinking about it, the more I think vim needs an
> internal templating system. 

no it doesn't, so don't thnink about it so much.

vim is fine, if not already over built a little.

rwmit...@gmail.com

unread,
Nov 19, 2020, 8:20:46 PM11/19/20
to vim_use
This discussion comes up repeatedly (so shows using the conversation search in google groups).

One suggestion was to use a split panel.  One panel would have user entered keywords and the other the combined document.  When saving the keywords, a script would auto run to do the substitutions and then load the results into the other panel. (well, that is how i interpreted the suggestion)

If your goal is to have them edit a document every 20 seconds, I really can't imagine they'll be doing a lot of editing other than filling in the required fields.

If you plan on having multiple templates, the problem gets more involved and so does the glue code to do the magic of manipulating fields to compute new fields/values.

Back in '88 I worked for a company that used something called Dataflex to generate input forms for phone support to use when responding to customers.  Ultimately you're probably looking at a custom solution.  It is amazing how something can look simple on the outset but quickly becomes much more involved.

hor...@gmail.com

unread,
Nov 24, 2020, 5:23:41 PM11/24/20
to vim_use

Look -  if it comes up repeatedly in conversations, then hey, you know there just MIGHT be something there?

I'd argue it this way - the goal of an editor is to be able for users to edit documents, right? That's the verbatim definition of an editor. 

Now, there is a large class of documents that vim is NOT suited for - and that is structured documents, consisting of data fields. And unfortunately for vim, that is a huge portion of the data-entering world, and is getting larger by the day. And there is no good unix program for handling it.

So you have a large, unfilled use case here - where your goal is to not worry about the formatting, the editing, the keybindings - or anything associated with an editor - and where you want to only worry about the data and can manipulate that data via callbacks.

And yes, you are right, since vim didn't have something like this I ended up writing a custom solution. It sucks. I wrote it in 1500 lines of python using asciimatics when I could have done it within say 50 lines of python with a decent templating system and vim. Again, it royally sucks. 

I took about 3 days to write it when it could have taken - again - about 1 hour or so with a decent templating system. And more to the point it looks less professional than it would have in vim. It doesn't leverage the large code base and I have to train people to use the arbitrary keybindings.

So yeah,  I definitely say that there is a use case for it here and if there are multiple conversations about it that come up from time to time I'd say that this just makes my point for me.

Ed


Reply all
Reply to author
Forward
0 new messages