Re-naming accounts..

[Mark Scannell]

Hello all --

Is it possible to apply a regular expression to rename accounts?

I have a complex structure of three account sets, all with Ass|Lia|Inc|Exp|Equ, and sub-categories in Assets (Current, Pension, and Fixed). I'd like to apply a regular expression to re-order the structure of the accounts for reporting purposes, without changing the underlying data.

I know I can do this in the source data, but I'd rather not do that.

Thanks!

Mark

[Martin Blais]

You could easly write a wrapper script that applies your renames to another file and then runs your report on that.

Always generate & delete the intermediate file.

Beancount's approach would be different: A plugin would be used to rewrite the account names in the transactions data structures directly.

--

---
You received this message because you are subscribed to the Google Groups "Ledger" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ledger-cli+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Martin Michlmayr]

* Mark Scannell <mesc...@gmail.com> [2016-01-02 13:31]:

> I have a complex structure of three account sets, all with
> Ass|Lia|Inc|Exp|Equ, and sub-categories in Assets (Current, Pension, and
> Fixed). I'd like to apply a regular expression to re-order the structure of
> the accounts for reporting purposes, without changing the underlying data.
>
> I know I can do this in the source data, but I'd rather not do that.

I'm not sure what you mean by re-order the structure, but if you want
to change the account names, you can create an account alias.

account Assets:newname
alias Assets:oldname

2015-01-02 * Test
Assets:oldname 10.00 EUR
Equity

$ ledger -f d reg assets
2015-01-02 Test Assets:newname 10.00 EUR 10.00 EUR

--
Martin Michlmayr
http://www.cyrius.com/

[Mark Scannell]

Martin,

I do need to check out more beancount one day.

I'm now trying something like this:

ledger -f main.ledger --limit xyz print | <potential filtering> | ledger -f - -f prices.ledger --display xyz bal

Then I can just use sed or whatever I'd like.

Mark

[Mark Scannell]

Martin,

I was thinking of that, and I may use alias' as I have a lot of abbreviations. The challenge is that I have my partner's books, my books, and then our joint books. As our finances are gradually being treated as one, I'd like to be able to have Assets -> mine, Assets -> hers, Assets -> ours, Expenses -> mine|hers|ours, etc.

A simple regex substitution works fine, so the ledger print | sed -e '//' | ledger -f - type of solution can work.

Mark

[Stefano Zacchiroli]

On Sun, Jan 03, 2016 at 05:40:57AM -0800, Mark Scannell wrote:
> As our finances are gradually being treated as one, I'd like to be
> able to have Assets -> mine, Assets -> hers, Assets -> ours, Expenses
> -> mine|hers|ours, etc.

FWIW I'm in a similar situation, but I've decided to only separate by
owners the Assets hierarchy, and only where's relevant (e.g., bank
accounts that formally belong to a single person). But I haven't
separated the Expenses hierarchy, because I considered that would be
overkill. Instead, I'm using an "Author: " tag to separate transactions
by person. I'm fairly happy with the result, but I'll gladly hear about
different experiences.

> A simple regex substitution works fine, so the ledger print | sed -e '//' |
> ledger -f - type of solution can work.

We've discussed this in the past IIRC, and the general feeling was that
it'd be nice to have a grammar-aware tool to rename accounts
consistently. But also that there is no such tool right now, and
therefore people are using various sed-based workarounds. My own scripts
to that end are attached. (They're provably incomplete/incorrect in
various corner cases, but they work well enough for me, and try hard to
preserve the right amount of horizontal spacing, which was a must for
me.)

Cheers.
--
Stefano Zacchiroli . . . . . . . za...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader . . . . . @zacchiro . . . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »

[Martin Michlmayr]

* Mark Scannell <mesc...@gmail.com> [2016-01-03 05:40]:

> I was thinking of that, and I may use alias' as I have a lot of
> abbreviations. The challenge is that I have my partner's books, my
> books, and then our joint books. As our finances are gradually being
> treated as one, I'd like to be able to have Assets -> mine, Assets
> -> hers, Assets -> ours, Expenses -> mine|hers|ours, etc.

This is something I need to solve as well and I haven't come up with a
solution yet that I'm 100% happy with. I've changed the title of the
thread.

What I came up with so far is this:

- For joint accounts, just use a normal Assets: account, e.g.
Assets:Savings:Bank
- For individual accounts (e.g. credit cards on a person's name) or
accounts at the same bank due to historical reasons, add the name
of the person to the account name, e.g. Assets:Savings:Bank Martin
- I don't think a general Assets:Martin: makes sense because that
defeats th whole purpose of combining finances (it's no longer
"yours" vs "theirs" when you're married).
- However, there are some things that belong to a specific person.
Specifically, I track frequent flyer miles and other reward points
and for that I created Assets:Rewards:Martin since a) all those
rewards accounts are for a specific person and b) usually both
people have the same accounts.

I don't think this scheme is ideal since it's inconsistent
(Assets:Rewards:Martin vs Assets:Savings:Bank Martin instead of
Assets:Savings:Martin:Bank) but unfortunately I cannot think of a
cleaner solution.

Furthermore, while in the US you can do joint tax returns, this is not
how it works in many other countries. So I still have to know income
for each person. I guess I could look for 'Income:' on accounts with
'Martin' in the name. But maybe a cleaner solution would be account
meta tags [1] which would allow me to define an owner.

Also, I think my idea of re-write rules / ledger views [1] would come
in really useful here to define a view that separates stuff out by
person again. Unfortunately this is not possible with leader at the
moment (I guess a beancount plug-in would be possible).

I'd like to hear how other people manage this.

[1] http://bugs.ledger-cli.org/show_bug.cgi?id=1070
[2] http://bugs.ledger-cli.org/show_bug.cgi?id=714

[Stefano Zacchiroli]

On Sun, Jan 03, 2016 at 10:33:32AM -0800, Martin Michlmayr wrote:
> This is something I need to solve as well and I haven't come up with a
> solution yet that I'm 100% happy with. I've changed the title of the
> thread.

Thanks for this.

> What I came up with so far is this:
>
> - For joint accounts, just use a normal Assets: account, e.g.
> Assets:Savings:Bank
> - For individual accounts (e.g. credit cards on a person's name) or
> accounts at the same bank due to historical reasons, add the name
> of the person to the account name, e.g. Assets:Savings:Bank Martin
> - I don't think a general Assets:Martin: makes sense because that
> defeats th whole purpose of combining finances (it's no longer
> "yours" vs "theirs" when you're married).

I'm doing the same. In my hierarchy the person is invariably the leaf
in any given branch of the hierarchy --- although to be fair I think
that just happened naturally, rather than as the result of a deliberate
hierarchy design choice.

> - However, there are some things that belong to a specific person.
> Specifically, I track frequent flyer miles and other reward points
> and for that I created Assets:Rewards:Martin since a) all those
> rewards accounts are for a specific person and b) usually both
> people have the same accounts.
>
> I don't think this scheme is ideal since it's inconsistent
> (Assets:Rewards:Martin vs Assets:Savings:Bank Martin instead of
> Assets:Savings:Martin:Bank) but unfortunately I cannot think of a
> cleaner solution.

I understand the problem, but I don't think it is specific to the
"spouse account" scenario. It is rather a consequence of the more
general fact that forcing accounts into being a hierarchy is an
artifice. With the exception of the tier 1 accounts that come from
double entry theory (assets/liabilities/equities/etc.), accounts are not
necessarily hierarchical. Rather, they are points in a multi-dimensional
vectorial space. Out of it you can extract several possible hierarchies,
neither of which is satisfactory for all use cases.

IIRC Martin Blais had a document discussing this issue in more details,
reaching out to the fact that tags too should sometime be considered
part of the account "hierarchy" and sometime not.

My bottom line here is that you will never be able to find a single
hierarchy that is completely satisfying. So as long as the tools we use
force us to choose a hierarchy, you need to cope with the resulting
frustration the best you can. In this case you should probably just cope
with querying on the ":Martin" substring as you suggested.

> Furthermore, while in the US you can do joint tax returns, this is not
> how it works in many other countries. So I still have to know income
> for each person. I guess I could look for 'Income:' on accounts with
> 'Martin' in the name.

As a work-around for this, I currently rely on the fact that me and my
spouse work for different employer, so I visually distinguish on
Income:Employer1 vs Income:Employer2. But that of course is unsatisfying
as a general solution because spouses can work for the same employer, or
have part of their incomes coming from the same entity in contexts that
do not match the employer/employee scenario.

But I also add "Payer:" and "X-Payee:" [1] tags to all Income:*
postings, so that I can do stuff like:

ledger balance --group-by "tag('x-payee')"
ledger balance --group-by "tag('payer')"

and have incomes grouped by person or paying entity. BTW, if people here
have better suggestions for the naming of the "payer" tag, I'd really
appreciate them.

[1]: the heading "X-" is to avoid the special semantics that ledger
associates to the "Payee" tag, which I find really annoying

> But maybe a cleaner solution would be account meta tags [1] which
> would allow me to define an owner.

That would be very useful in general, and yes also provide another range
of workarounds (but not a general solution IMO) for the single hierarchy
imposition.

> Also, I think my idea of re-write rules / ledger views [2] would come

> in really useful here to define a view that separates stuff out by

> person again. Unfortunately this is not possible with ledger at the

> moment (I guess a beancount plug-in would be possible).

I had forget about this one, but I'll look it up again.

[Ben Finney]

Mark Scannell <mesc...@gmail.com> writes:

> The challenge is that I have my partner's books, my books, and then
> our joint books. As our finances are gradually being treated as one,
> I'd like to be able to have Assets -> mine, Assets -> hers, Assets ->
> ours, Expenses -> mine|hers|ours, etc.

Pete Keen described a way to use Ledger's “virtual transactions” and
“automated transactions” to maintain a split such as you describe, while
allowing the split to be conflated when needing to deal with the “real”
transactions.

Some people decide to do this by dividing the bills up between
roommates. […] Other roommates decide to nominate one person to have
all of the bills in their name and post the amounts due every month
for everyone to see. This is what my girlfriend and have been doing
and it's been working great. All of the bills are in my name and I
give her a summary every month and she hands me a check. Easy peasy.

[…] Ledger has an extremely handy feature named automated
transactions. The basic idea is that you provide a template
transaction and a pattern to match, and ledger will insert the
filled-in template transaction every time the pattern matches.

[…] For various reasons I keep most of my money in [one real
account]. I have most of it allocated away into various "funds",
which are just fake buckets that only exist for me. […] I've
implemented these buckets using ledger's virtual transaction
feature.

[…] On their own, these two features are pretty useful. It's when
you combine them that the awesome power of ledger starts appearing.

<URL:https://www.petekeen.net/program-your-finances-automated-transactions>

--
\ “In the long run nothing can withstand reason and experience, |
`\ and the contradiction which religion offers to both is all too |
_o__) palpable.” —Sigmund Freud |
Ben Finney

[o1bigtenor]

Greetings

I deplore top posting but as I have little specific (albeit general I
do) to say to what is described I will post separately.

Before any comments, I am not an accountant nor an expert on ledger
nor any other record keeping type program!

Even before I got married a number of years ago I found it necessary
to track expenses and all the other goodies for the number of small
entities (not companies by formal incorporation but by activity) and
in the process came up with this solution that is not simple but does
do the job, and I think well.

I started from the GIFI codes (both the USA and Canada use them but
they do somewhat differently). They are a 4 digit code for use by
incorporated entities for reporting financial information to taxation
entities.

What I did was add further digits, like this - - xxxx.yy.yy.yy use
it something like this (using and asset account followed by an expense
account to illustrate). (I'm not sure all 6 digits are necessary but
that was my design choice.)

Asset account
1002.60.40.01

1002 - - - designates Deposits in Canadian banks and institutions -
in Canadian currency
.60 - - - designates a particular institution
.40 - - - - designates a personal savings account (same number is used
with all the different institutions)
.01 - - - designates a particular savings account (some institutions
have multiple accounts available

Expense account
9763.10.01.02

9763 - - - designates Machinery repairs (Agricultural equipment would
be a different number for a non-farm business)
.10 - - - - designates vehicles
.01 - - - - designates trucks
.02 - - - - designates a particular truck

So when I need information for the tax authorities I poll using the 4
digit codes and when I want informaiton for my management (costing and
expenses) I use whatever level needed to supply what I need.

I started my account list using what was provided by the taxation
authorities. From this I developed by document (list) which is now 33
pages in length.

I do my accounts using a textfile program called leafpad which makes
for easy copy and paste which I use often.

Dee

[Martin Blais]

Stay away from virtual postings, they break the accounting equation. You never need to use them, period. They are a crutch that misleads many into not figuring things out. Beancount doesn't even support them and I've doing everything without so far without problems (though it has forced me to learn bookkeeping).

I see at least two ways of accounting for shared expenses:

1. The simple way: You can simply create a Liabilities account. When you book a shared expense, you book a portion of it to the liabilities account, e.g.

2015-12-11 * "CON EDISON       ONLINE PMT"

  Assets:US:Checking                                              -46.51 USD

  Liabilities:AccountsPayable:Caroline                              46.51/2 USD

  Expenses:Home:Electricity

You can then share the account on that liabilities account with your partner. Cut-n-pasting from the web interface into gmail preserves the HTML so that's one easy way share the balance details. However much your roommate/partner owes is the balance of that account. 

It's really convenient because you can use that account for a variety of purposes. It effectively separates the "counting" and the "payment". Counting occurs off the liabilities account; payments occur off the balance, paid to the account.

2. You can create a separate book. I used to do this for joint expenses and assets. I describe this here in much detail in the context of a trip but you can do the same thing, continuously:

http://furius.ca/beancount/doc/sharing

This uses a plugin to split up shared expenses easily, but you can do it manually too.

As for using virtual units to track only a portion of amounts where you feel you need "virtual" units or "virtual postings", the technique you can use that doesn't break the accounting equation is to use a mirror currency instead. I've described this a number of times on this list. There's even a plugin to automate that method in Beancount. Look for mirror accounting in the archives.

http://furius.ca/beancount/doc/sharing

[Martin Blais]

On Mon, Jan 4, 2016 at 4:51 AM, Stefano Zacchiroli <za...@upsilon.cc> wrote:

On Sun, Jan 03, 2016 at 10:33:32AM -0800, Martin Michlmayr wrote:
>  - However, there are some things that belong to a specific person.
>    Specifically, I track frequent flyer miles and other reward points
>    and for that I created Assets:Rewards:Martin since a) all those
>    rewards accounts are for a specific person and b) usually both
>    people have the same accounts.
>
> I don't think this scheme is ideal since it's inconsistent
> (Assets:Rewards:Martin vs Assets:Savings:Bank Martin instead of
> Assets:Savings:Martin:Bank) but unfortunately I cannot think of a
> cleaner solution.

I understand the problem, but I don't think it is specific to the
"spouse account" scenario. It is rather a consequence of the more
general fact that forcing accounts into being a hierarchy is an
artifice. With the exception of the tier 1 accounts that come from
double entry theory (assets/liabilities/equities/etc.), accounts are not
necessarily hierarchical. Rather, they are points in a multi-dimensional
vectorial space. Out of it you can extract several possible hierarchies,
neither of which is satisfactory for all use cases.

That's the right way to look at command-line accounting: the contents of a ledger are really nothing more than a giant stupid table of posting rows. The account is just one particular column in the row. Meta tags are just more implicitly defined columns.

Querying implicitly joins the attributes of the transactions to the tables of postings, so you can use them directly.

Journals are listings of subsets of rows. Balance sheets and income statements are aggregations where the key is "account". That's it.

[John Wiegley]

>>>>> Martin Blais <bl...@furius.ca> writes:

> Stay away from virtual postings, they break the accounting equation. You
> never need to use them, period.

I disagree with this statement. I've used them to good effect in the past.
There are times when you need a bit of flexibility to get exactly the kind of
reports you want. It's not all about accounting, or honoring the accounting
equation; it's about managing finances the way a Ledger user wants to. I think
this pretty much sums up the core philosophical divide between beancount and
Ledger.

--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2

[Douglas Philips]

On 1/4/16 2:15 PM, John Wiegley wrote:
>>>>>> Martin Blais <bl...@furius.ca> writes:
>
>> Stay away from virtual postings, they break the accounting equation. You
>> never need to use them, period.
>
> I disagree with this statement. I've used them to good effect in the past.
> There are times when you need a bit of flexibility to get exactly the kind of
> reports you want. It's not all about accounting, or honoring the accounting
> equation; it's about managing finances the way a Ledger user wants to. I think
> this pretty much sums up the core philosophical divide between beancount and
> Ledger.

So this is interesting timing, I was thinking of using virtual postings to track budget amounts.
Specifically, when I have a credit card expense, I have a posting such as:

2016/01/01 Fix the thing
Expenses:Thing $20.00
Liabilities:Credit Card:Pursuit $-20.00


What I have been doing as an experiment is to add in something additional:

2016/01/01 Fix the thing
Expenses:Thing $20.00
Liabilities:Credit Card:Pursuit $-20.00
Budget:Pursuit $20.00
Assests:Checking $-20.00

So that way I can see what my checking balance will be when I pay the bill for the Pursuit card.
This way I can see the balance of the checking account afterwards. I can't see it before, which is annoying.
And, when I pay off the bill (directly from Checking), I have to go back and delete all those additional lines.
Ugh.
So, I was thinking why not use virtual postings?

Or is there a better way?

Thanks to John for such a cool tool, and Martin for having an alternative.

-=Doug

[John Wiegley]

>>>>> Douglas Philips <douglas...@gmail.com> writes:

> What I have been doing as an experiment is to add in something additional:

> 2016/01/01 Fix the thing
> Expenses:Thing $20.00
> Liabilities:Credit Card:Pursuit $-20.00

> (Budget:Pursuit) $20.00

Now you can just do a balance query of Assets:Checking against Budget and see
what the net result will be. Or use --real to strip away the budget, or just
don't query against Budget.

> Thanks to John for such a cool tool, and Martin for having an alternative.

You're most welcome!

[Jeffrey Brent McBeth]

On Mon, Jan 04, 2016 at 02:04:38PM -0800, John Wiegley wrote:
> >>>>> Douglas Philips <douglas...@gmail.com> writes:
>
> > What I have been doing as an experiment is to add in something additional:
>
> > 2016/01/01 Fix the thing
> > Expenses:Thing $20.00
> > Liabilities:Credit Card:Pursuit $-20.00
> > (Budget:Pursuit) $20.00
>
> Now you can just do a balance query of Assets:Checking against Budget and see
> what the net result will be. Or use --real to strip away the budget, or just
> don't query against Budget.

I'm really happy using the imperative (I forget the term) virtual accounts (surrounded by []), it maintains the accounting equation, I can easily query how much is really in my checking vs how much is unallocated to ther budget things, and I can square things up periodically if I choose.

Jeff

--
"The man who does not read good books has no advantage over
the man who cannot read them."
-- Mark Twain

[Douglas Philips]

On 1/4/16 6:27 PM, Jeffrey Brent McBeth wrote:
> On Mon, Jan 04, 2016 at 02:04:38PM -0800, John Wiegley wrote:
>>>>>>> Douglas Philips <douglas...@gmail.com> writes:
>>
>>> What I have been doing as an experiment is to add in something additional:
>>
>>> 2016/01/01 Fix the thing
>>> Expenses:Thing $20.00
>>> Liabilities:Credit Card:Pursuit $-20.00
>>> (Budget:Pursuit) $20.00
>>
>> Now you can just do a balance query of Assets:Checking against Budget and see
>> what the net result will be. Or use --real to strip away the budget, or just
>> don't query against Budget.
>
> I'm really happy using the imperative (I forget the term) virtual
> accounts (surrounded by []), it maintains the accounting equation, I
> can easily query how much is really in my checking vs how much is
> unallocated to ther budget things, and I can square things up
> periodically if I choose.

Ok, here's where I get confused about Virtual accts, parenthetical or squared-off)

I read through the manual (2.6.3, but the v3 manual is the same)...
There is an example of using virtual accounts and automatic transactions to move money into
a tithing liability account. Then real transactions are used to pay out of that virtual account?
(section 5.22.9 in the v3 documentation pdf)
That's where I get confused. If a report is run with --real then won't the balances be wrogn?


I want to move funds from checking to some kind of budget-y account, yet be able to see the
actual checking amounts and the "diminished by budget-y" amounts.
Ok, I think I can do that with virtual transactions as John showed above.

Then I want to say that there is just one actual payment from the checking to the liability that
"absorbs" the budget account amount (or some part of it), all at once. This is the one transaction
that the bank will see and that will be on the statement for reconciling purposes.
So how do I get the budget amount not to interfere with the payment amount when running register reports
with and without --real. (that is, I don't want to have to avoid --real or have to use it...)

Hmmm, hope that is slightly clearer than mud!

Thanks,
-=Doug

[John Wiegley]

>>>>> Douglas Philips <douglas...@gmail.com> writes:

> I read through the manual (2.6.3, but the v3 manual is the same)... There is
> an example of using virtual accounts and automatic transactions to move
> money into a tithing liability account. Then real transactions are used to
> pay out of that virtual account? (section 5.22.9 in the v3 documentation
> pdf) That's where I get confused. If a report is run with --real then won't
> the balances be wrogn?

--real would just show a positive balance for that liability account, since
the goal without --real is to bring it to zero. No bug deal, the --real report
just looks like I've been spending money to a Liability for no reason.

> Then I want to say that there is just one actual payment from the checking
> to the liability that "absorbs" the budget account amount (or some part of
> it), all at once.

I'd have to see an example of this to completely clear on what you mean...

[Douglas Philips]

On 1/4/16 11:47 PM, John Wiegley wrote:
>>>>>> Douglas Philips <douglas...@gmail.com> writes:
>
>> I read through the manual (2.6.3, but the v3 manual is the same)... There is
>> an example of using virtual accounts and automatic transactions to move
>> money into a tithing liability account. Then real transactions are used to
>> pay out of that virtual account? (section 5.22.9 in the v3 documentation
>> pdf) That's where I get confused. If a report is run with --real then won't
>> the balances be wrogn?
>
> --real would just show a positive balance for that liability account, since
> the goal without --real is to bring it to zero. No bug deal, the --real report
> just looks like I've been spending money to a Liability for no reason.

Ah, OK... (Still wrapping my head around that)

>> Then I want to say that there is just one actual payment from the checking
>> to the liability that "absorbs" the budget account amount (or some part of
>> it), all at once.
>
> I'd have to see an example of this to completely clear on what you mean...

Ok, so let's say that I have a bunch of "small" transactions.
Meals, petrol purchases, etc.
All these move assests (USD) from my Liabilities:Credit Card:CardX account
to various expense accounts. So I can see what I owe to the CardX company based
on that. And for small expenses I can directly subtract that from my checking account balance
to see what I'd have left after paying them off.

But.

For larger expenses (car repairs, house repairs), I want to pay them off slower,
so let's say for the small expenses I add the full amount to the budget account,
but for larger expenses I might only add, say, $100 to the budget account.
Now the statement comes. I can see from my budget account how much I intend to pay,
and thus I know in advance how much of a hit my checking account will take.
Which keeps me from spending that money on something else by accident.

Ok, so the statement comes and I cut a check.
This is recorded as a transaction between my checking account and my Liabilities:Credit Card:CardX
account. This will also be a transaction that shows up on both accounts and can be reconciled.
The budget account transactions have nothing against which they can be reconciled.
However, since I have made a payment to the Liabilities account, I also want to deduct
that from the Budget account so that I know I have relieved some of that obligation and can have
an accurate view of the amount I still holding in reserve.

So maybe a better name would be Reserve:CardX and not Budget:CardX. I'm not sure if that
makes any difference, but at least I'm clearer on what my intentions are.

To put it another way, the budget (or reserve) transaction is a check that I write mentally
to "prepay" the credit card obligation, but it's not a real transaction that can be reconciled
so I can't actually record it as a transaction against checking without really getting confused
when the checking account statement arrives.

-=Doug (needs more coffee this early)

[Douglas Philips]

On 1/5/16 8:04 AM, I wrote:
> To put it another way, the budget (or reserve) transaction is a check that I write mentally
> to "prepay" the credit card obligation, but it's not a real transaction that can be reconciled
> so I can't actually record it as a transaction against checking without really getting confused
> when the checking account statement arrives.

...definitely need that coffeee, forgot to add:
So all those virtual checks get summed and converted into a real reconcilable check.
Thus they need to disappear.
Which is how I had been doing it per my original post.
What I'd like to do is avoid having to edit my ledger file to make those virtual/reserve transactions "go away". :-)

Thanks!
-=Doug

[Martin Blais]

On Mon, Jan 4, 2016 at 2:15 PM, John Wiegley <jo...@newartisans.com> wrote:

>>>>> Martin Blais <bl...@furius.ca> writes:

> Stay away from virtual postings, they break the accounting equation. You
> never need to use them, period.

I disagree with this statement. I've used them to good effect in the past.

There are times when you need a bit of flexibility to get exactly the kind of
reports you want. It's not all about accounting, or honoring the accounting
equation; it's about managing finances the way a Ledger user wants to.

The fact that one can use them does not mean they're a good thing to have; they have a cost too, they are dangerous (they break the elegant balance property) and their availability must be weighed against this cost. All it means is that for solving these particular problems you haven't forced yourself to carry out the same operations within the constraints of the double-entry system. In specific instances, I've shown on this list how you can solve all the same problems within the constraints of balancing transactions, albeit sometimes requiring with minor twists and turns.

The major problem with virtual postings is that they offer a way out of the DE method. Due to its nature, most command-line accounting users aren't accountants and are just learning the method, and as such, they inevitably reach out for the crutch way way earlier than they should, basically as soon as they don't know how to book some particular type of transaction. You may not use it that way, but for most users it's a cop-out. Breaking the rules to do things "the way a user wants to" instead of learning the DE system breaks a fundamental property of the dataset. That bit of flexibility you want is one with costly consequences.

If one truly required virtual postings in order to solve a particular problem, one should wonder whether a spreadsheet would be a more appropriate tool to solve their particular problem. Otherwise they should try harder to frame their problem as a set of double-entry transactions. It's possible: Beancount does not even support virtual postings and I'm solving all the same problems other people do.

I think
this pretty much sums up the core philosophical divide between beancount and
Ledger.

That's oversimplifying it. The philosophical divide runs deeper than this, and I would describe it in three parts.

First, Ledger is optimistic. It assumes it's easy to input correct data by a user. My experience with data entry of the kind we're doing is that it's impossible to do this right without many automated checks. Sign errors on unasserted accounts are very common, for instance. In contrast, Beancount is highly pessimistic. It assumes the user is unreliable. It imposes a number of constraints on the input. For instance, if you added a share of AAPL at $100 to an empty account it won't let you remove a share of AAPL at $101 from it; you just don't have one. It doesn't assume the user is able or should be relied upon to input transactions in the correct order (dated assertions instead of file-order assertions). It optionally checks that proceeds match sale price (sellgains plugin). And it allows you to place extra constraints on your chart of accounts, e.g. a plugin that refuses postings to non-leaf accounts, or that refuses more than one commodity per account, or that requires you declare all accounts with Open directives; choose your level of pedanticity a-la-carte. It adds more automated cross-checks than the double-entry method provides. After all, cross-checking is why we choose to use the DE method in the first place, why not go hardcore on checking for correctness? Beancount should appeal to s/he who does not trust themselves too much. 

And to the point: because of this, it does not provide support for unbalanced/virtual postings; it's not a shortcoming, it's on purpose.

Secondly, there's a design ethos difference. As is evidenced in the user manual, Ledger provides a myriad of options. This surely will be appeal to many, but to me it seems it has grown into a very complicated monolithic tool. How these options interact and some of the semantic consequences of many of these options are confusing and very subtle. Beancount offers a minimalistic approach: while there are some small number of options (https://bitbucket.org/blais/beancount/src/tip/src/python/beancount/parser/options.py?fileviewer=file-view-default#options.py-196), it tries really hard not to have them. And those options that do affect the semantics of transactions always appear in the input file (nothing on the command-line) and are distinct from the options of particular tools. Loading a file always results in the same stream of transactions, regardless of the reporting tool that will consume them. The only command-line options present are those which affect the particular behavior of the reporting tool invoked; those never change the semantics of the stream itself.

Thirdly, Beancount embraces stream processing to a deeper extent. Its loader creates a single ordered list of directives, and all the directives share some common attributes (a name, a date, metadata). This is all the data. Directives that are considered "grammar" in Ledger are defined as ordinary directive objects, e.g. "Open" is nothing special in Beancount and does nothing by itself. It's simply used in some routines that apply constraints (an account appears, have an Open directive been witnessed prior?) or that might want to hang per-account metadata to them. Prices are also specified as directives and are embedded in the stream, and can be generated in this way. All internal operations are defined as processing and outputting a stream of directives. This make it possible to allow a user to insert their own code inside the processing pipeline to carry out arbitrary transformations on this stream of directives--anything is possible, unlimited by the particular semantics of an expression language. It's a mechanism that allows users to build new features by writing a short add-on in Python, which gets run at the core of Beancount, not an API to access its data at the edges. If anything, Beancount's own internal processing will evolve towards doing less and less and moving all the work to these plugins, perhaps even to the extent of allowing plugins to declare the directive types (with the exception of Transaction objects). It is evolving into a shallow driver that just puts together a processing pipeline to produce a stream of directives, with a handy library and functional operations.

[Simon Michael]

Re virtual postings (I think of them as (non-balanced) and
[separately-balanced] postings): I think it's good to allow them - the
non-balanced kind, at least - as an escape valve, since most users are
not expert in double entry bookkeeping starting out.

I also think there are times when they are a valid choice for an
experienced user. For example I use them to note invoices I've sent,
which otherwise would have no place in my cash-basis journal.

These postings can easily be found and replaced with balanced postings
later as you get more sophisticated.

Sometimes strict correctness checking is a higher priority than
short-term productivity. We should provide options to warn
about/disallow them when a user prefers that.

[Simon Michael]

Nice discussion..!

On 1/5/16 5:34 AM, Martin Blais wrote:
> My experience with data entry of the kind we're doing is that it's
> impossible to do this right without many automated checks.

I wouldn't put it quite that strongly, but I agree with you that more
correctness aids are a very good thing.

> number of constraints on the input. For instance, if you added a share of
> AAPL at $100 to an empty account it won't let you remove a share of AAPL at
> $101 from it; you just don't have one.

I want this (being able to declare constraints on accounts, eg that the
balance should never go negative). I'd use it for my budget envelopes.
Though, in this case I'd need to make it a warning rather than an error. :)

> And it allows you to place
> extra constraints on your chart of accounts, e.g. a plugin that refuses
> postings to non-leaf accounts, or that refuses more than one commodity per
> account, or that requires you declare all accounts with Open directives;
> choose your level of pedanticity a-la-carte.

Exactly. All great features, I want these too.

> And to the point: because of this, it does not provide support for
> unbalanced/virtual postings; it's not a shortcoming, it's on purpose.

You said a-la-carte above; by that principle virtual postings should
arguably be at least on the menu..

> Secondly, there's a design ethos difference. As is evidenced in the user
> manual, Ledger provides a myriad of options. This surely will be appeal to
> many, but to me it seems it has grown into a very complicated monolithic
> tool. How these options interact and some of the semantic consequences of
> many of these options are confusing and very subtle. Beancount offers a
> minimalistic approach: while there are some small number of options (
> https://bitbucket.org/blais/beancount/src/tip/src/python/beancount/parser/options.py?fileviewer=file-view-default#options.py-196),
> it tries really hard not to have them.

Yes. The interactions of hledger's features and options get rather
subtle too, even though I work hard to make them consistent and predictable.

It sounds like beancount goes further in this direction. I bet you too
have the complexity challenge though, especially as your plugins multiply.

> And those options that do affect the
> semantics of transactions always appear in the input file (nothing on the
> command-line) and are distinct from the options of particular tools.

...

> Loading a file always results in the same stream of transactions,
> regardless of the reporting tool that will consume them. The only
> command-line options present are those which affect the particular behavior
> of the reporting tool invoked; those never change the semantics of the
> stream itself.

That's interesting, I'll think about this in the context of hledger.

> Thirdly, Beancount embraces stream processing to a deeper extent. Its
> loader creates a single ordered list of directives, and all the directives
> share some common attributes (a name, a date, metadata). This is all the
> data. Directives that are considered "grammar" in Ledger are defined as
> ordinary directive objects, e.g. "Open" is nothing special in Beancount and
> does nothing by itself. It's simply used in some routines that apply
> constraints (an account appears, have an Open directive been witnessed
> prior?) or that might want to hang per-account metadata to them. Prices are
> also specified as directives and are embedded in the stream, and can be
> generated in this way. All internal operations are defined as processing
> and outputting a stream of directives. This make it possible to allow a
> user to insert their own code inside the processing pipeline to carry out
> arbitrary transformations on this stream of directives--anything is
> possible, unlimited by the particular semantics of an expression language.

In hledger we have moved in this direction over time, also - building a
first-class representation of all input, including directives, comments
etc., to be processed later, rather than applying them at parse time.
Beancount's approach sounds nicely general and powerful (and hopefully
you don't lose efficiency from this).

[Martin Blais]

On Thu, Jan 7, 2016 at 1:26 PM, Simon Michael <si...@joyful.com> wrote:

Re virtual postings (I think of them as (non-balanced) and
[separately-balanced] postings): I think it's good to allow them - the
non-balanced kind, at least - as an escape valve, since most users are
not expert in double entry bookkeeping starting out.

I don't understand this logic. Providing an "escape" valve is exactly the wrong thing to do. We can write off this discussion to a difference in attitudes; I prefer not to allow the creation of invalid data and barf instead. It's really not that hard to learn to do things right. Maybe you think of that as "advanced"; I think of that as the starting point.

Separately balanced postings are equivalent to another set of transactions that you can turn on and off, e.g. all associated with a special tag across the entire file. Wholly unnecessary: users can use two transactions instead. If really desired, recreatable in Beancount by using a plugin that filters out postings with a particular per-posting flag.

I also think there are times when they are a valid choice for an
experienced user. For example I use them to note invoices I've sent,
which otherwise would have no place in my cash-basis journal.

These postings can easily be found and replaced with balanced postings
later as you get more sophisticated.

Sometimes strict correctness checking is a higher priority than
short-term productivity.

I'll misquote Orwell:

Productivity is correctness.

Freedom is slavery.

Balance is strength.

[John Wiegley]

>>>>> Martin Blais <bl...@furius.ca> writes:

> I prefer not to allow the creation of invalid data and barf instead.

I think we will always differ, because we use different meanings of "invalid".
You are holding, quite rightly, to the objective standard of the accounting
equation. For this reason, beancount is right in its philosophical grounding
to take this stand.

I hold, however, to whatever the user deems to be useful. I have no objective
standard to claim, and my philosophy allows a hammer to be used on things very
different from nails.

The user must decide what best meets their needs.