Another Ledger user's perspective

[redst...@gmail.com]

Another ledger user's perspective

I've been using double entry bookkeeping continuously for at least a decade and a half. First, via homecooked spreadsheets, then gnucash, then ledger. I'm now considering beancount for a few reasons, and I thought I'd capture what I view as the upsides and downsides for me.

beancount advantages (for me):

------------------------------

(in rough order of importance)

- Stock accounting: big problem with ledger! This is clearly mentioned in beancount's documentation (comparison against ledger).

- Investment analysis: another big problem. I'd like to compute investment returns, and a zillion other things, if possible. Also, I'd like to tag accounts and write account-tag based queries which I can then post process.

- Plugins: I want to take things a step further and have my account system do more, including investment analysis, etc. I'd also like scripting "around" the accounting software that I have to be write, to be cleaner/simpler 

- Stock trades: sucking up rounding errors automatically. For now, I do this on the input-side.

- Balance assertions: order-dependence prevents me from using this feature well

- Python: I'm comfortable in python, and think it's the right "level" of language for accounting

- Parser: non-"proprietary" parser is good for development

beancount downsides (for me):

-----------------------------

- Fixed 5 top-level accounts

  - I use a ZeroSum account as outlined in one of my previous posts. I use it more extensively than that post indicates

  - I rely on a top-level "TODO" account that is important to me as well. I've tried to propagate the TODO account to the 5 accounts, and find that it doesn't work

- Effective dates: I use these extensively. They mostly fall into two main categories:

  - Tax view: self-explanatory. Zillion possible examples and use-cases

  - For meaningful expense spreading: I tend to do a lot of prepaying for things (eg: insurance), and expense analysis is meaningless if I'm not able to spread this. I've tried putting things into a "holding account", but that's kludgy and doesn't work well in the long run.

- Virtual accounts: I know Beancount has a strong no-no towards this, mainly with the "virtual" and the "unbalanced" parts. I don't care about those either. But I do need some solution, and Virtual accounts in ledger solved one of my longstanding problems neatly and elegantly. That problem is tracking my W2 (annual tax form). When I get a W2 now, it matches number for number in my virtual account. Particularly valuable with multiple W2s. I caught an error on an employer's side two years ago based on this.

What I need for this is not necessarily a "virtual account", but:

   - an account outside the 5 top-level accounts. It's fine if account has two parts, and they have to sum up to zero.

   - this account has to live in its own "ecosystem."  In other words, it has to sum up to zero without depending on the other accounts. It should never be considered for any of the reports (like cashflow/balance/net assets/expenses etc.)

   - a way to write automated rules that will fill up these accounts (does Beancount have one?)

Is this possible to do in Beancount as it is currently?

- Small userbase? I seek longevity in my accounting systems. This is a bit of a chicken-egg problem, and is at the bottom of my concerns, but I'm mentioning it anyway.

Other observations:

-------------------

- In general, my sense is, I'll always have a whole bunch of scripts "around" the core accounting software package I pick. For example, ofx/csv/screenscraping downloads and conversion into the journal on the input side. A few dozen analysis tools on the output/post-processing side. The question is, how well does the accounting software play with these? Can queries be handled by it at the level I ask them (and not via rebuilding high-level information from the low-level xml/csv output)?

- I'm having trouble test-driving Beancount because it seems non-trivial to convert years of data I have in ledger journal format into Beancount format. Is there a way to lessen this pain? To xml and back or something similar?

I'll post more as I find more.

[redst...@gmail.com]

Trying to give beancount a shot, and I have a bunch of questions/comments:

I have tons of personal command-line "recipes" in ledger (basically queries) to spit out all the information I want either frequently or occasionally. All these make extensive use of ledger's filtering and aggregation capabilities. So I'm really glad you've implemented at least a first version of bean-query. I was a bit surprised to find out this was recent.

But so far, being able to write SQL-like queries in beancount is a pretty huge step over ledger. I find it difficult to remember or keep track of what each switch and argument does in ledger, and constantly have to keep track of what's there in .ledgerrc at any given point. Seems like I can finally focus on the actual accounting part without fighting with my accounting software. Anyway, nicely done! :)

Questions:

1) Are there any scripts to convert ledger files to beancount? Even if they're still have rough edges?

2) I vaguely remembered seeing a plugin to auto-open non-existing accounts. I don't see it in src/plugins. Am I misremembering?

3) "bean-query ledger" seems to be broken. Is this true?

4) Is there a way to produce a .csv of a) the entire input file b) the output of a query. For (b) please feel free to provide me a pointer to the code, and issues you have already thought of if any, and I'd be happy to contribute. This is almost critical to me because I use excel extensively for more complex queries, analyses including charts, and to hunt down problems.

5) I haven't looked, but you're probably using some readline library (and also $PAGER), and I really like this. One thing that's not working too well is when I resize an xterm. Display works fine and reflows, but input gets confused. Again, if you point me to code, I'll see if it's perhaps a matter of turning on some library options (Strike this for now: this might've been a problem at my end. I'm unable to reproduce it for now).

6) Is there a way to colorize output? A simple coloring of negative amounts is mainly what I need, but I can see the need for more at some point. Whatever library ledger is using to do so works great for me on a fairly complex cygwin/xterm/ssh/tmux setup without issue.

[Martin Blais]

On Thu, Jan 22, 2015 at 4:53 PM, <redst...@gmail.com> wrote:

Another ledger user's perspective

I've been using double entry bookkeeping continuously for at least a decade and a half. First, via homecooked spreadsheets, then gnucash, then ledger. I'm now considering beancount for a few reasons, and I thought I'd capture what I view as the upsides and downsides for me.

beancount advantages (for me):

------------------------------

(in rough order of importance)

- Stock accounting: big problem with ledger! This is clearly mentioned in beancount's documentation (comparison against ledger).

Also one of the main drivers for me to do this in the first place.

Soon: Lot dates will be done correctly too, so you can figure out long vs. short term lots.

- Investment analysis: another big problem. I'd like to compute investment returns, and a zillion other things, if possible. Also, I'd like to tag accounts and write account-tag based queries which I can then post process.

Good idea. I'll support that, I think it's a common case. (Note: In this context let's not call them "tags", that's Ledger terminology and it's badly defined IMO. "Tags" in Beancount as a set of strings attached to transactions. "Metadata" is what I call what you're thinking of.)

- Plugins: I want to take things a step further and have my account system do more, including investment analysis, etc. I'd also like scripting "around" the accounting software that I have to be write, to be cleaner/simpler 

It sound like we're thinking very similarly. I view my corpus of data as something I should be able to play with and manipulate. To me, Beancount is as much a Python library as it is a set of tools built on top of it.

- Stock trades: sucking up rounding errors automatically. For now, I do this on the input-side.

I haven't been able to handle that problem nicely in Ledger... I think the precision handling is broken (I've filed a bug). I don't think the Ledger community views that as a problem, which I don't understand. Given the round problems and the lack of correct balance assertions, I don't understand how someone can have 20 years of Ledger input that works and that does not have errors. I could never do it--I need the automated balance checks and cost basis verification.

- Balance assertions: order-dependence prevents me from using this feature well

- Python: I'm comfortable in python, and think it's the right "level" of language for accounting

- Parser: non-"proprietary" parser is good for development

You mean "generated with a grammar"?

If so, I also agree. With my little grammar file I've been able to turn around impactful changes to the syntax in no time. For instance, adding metadata has been a breeze, very easy. That's just the right thing to do. It makes it almost _too_ easy. Plus it also gives you the guarantee that you could easily port the whole thing to a new computer language implementation without too much pain, or at least easily support the parsing bit, from many languages.

beancount downsides (for me):

-----------------------------

- Fixed 5 top-level accounts

  - I use a ZeroSum account as outlined in one of my previous posts. I use it more extensively than that post indicates

  - I rely on a top-level "TODO" account that is important to me as well. I've tried to propagate the TODO account to the 5 accounts, and find that it doesn't work

I don't understand what doesn't work, can you provide more details? You should be able to put all these transfer accounts under the Equity: category with no problem. Use Equity:TODO if you like. All accounts are treated the same in Beancount, and adding another root account/category would not change anything AFAIK. Very curious to know what the difficult was.

The reason for being strict about the five categories is essentially to be able to build income statements and balance sheets that look realistic and whose sums are consistent and correct. In a way, those categories are just a way to flag accounts as transient (income & expenses) vs. permanent (assets & liabilities), and positive balance (assets & expenses) vs. negative balance (liabiilties & income).  Equity is mainly there to absord retained earnings and conversion entries ("trading accounts").

- Effective dates: I use these extensively. They mostly fall into two main categories:

  - Tax view: self-explanatory. Zillion possible examples and use-cases

I don't understand what's self-explanatory about this bit. Can you provide an explicit example? I haven't completed my section in the Cookbook on how to file taxes, but so far I need no tricks to do this, it's straightforward double-entry method (with per-year Expenses accounts, e.g. Expenses:Taxes:Y2015:Federal).

 

  - For meaningful expense spreading: I tend to do a lot of prepaying for things (eg: insurance), and expense analysis is meaningless if I'm not able to spread this. I've tried putting things into a "holding account", but that's kludgy and doesn't work well in the long run.

I think what you mean here - and it isn't entirely obvious to me - is that you're amortizing an initial expense over time. This can be carried out using accounts and multiple entries but my reading is that you want to do this in a single entry. This would definitely require an implementation of the limbo accounts feature. I'll put that on the list.

If you can provide a realistic example of how you use this, I'll add it to my TODO item description (I'm imagining a different date per posting).

- Virtual accounts: I know Beancount has a strong no-no towards this, mainly with the "virtual" and the "unbalanced" parts. I don't care about those either. But I do need some solution, and Virtual accounts in ledger solved one of my longstanding problems neatly and elegantly. That problem is tracking my W2 (annual tax form). When I get a W2 now, it matches number for number in my virtual account. Particularly valuable with multiple W2s. I caught an error on an employer's side two years ago based on this.

I also compute numbers to match the W2 form exactly. All you have to do is aggregate the expense accounts over the calendar year (taxes are expenses).

You don't need virtual accounts for this.

What I need for this is not necessarily a "virtual account", but:

   - an account outside the 5 top-level accounts. It's fine if account has two parts, and they have to sum up to zero. 

   - this account has to live in its own "ecosystem."  In other words, it has to sum up to zero without depending on the other accounts. It should never be considered for any of the reports (like cashflow/balance/net assets/expenses etc.)

   - a way to write automated rules that will fill up these accounts (does Beancount have one?)

Is this possible to do in Beancount as it is currently?

Oh, I think I see what you're saying. You want to enforce a further constraints on a particular subset of accounts.

(I still don't know how you use it for W2 though, or why it's needed.)

You can do this by writing a plugin, absolutely.

Here's an idea:

- Define yourself a special account root, e.g. Equity:Extra to act as parent for all those special accounts.

- Write a plugin that will ensure that for all transactions that include at least one posting on an Equity:Extra account, the sum of all the weights of these postings is zero (you can reuse the code that's already these to check that transactions balances).

- If you want to automatically fill in missing postings these accounts, you can also do that from a plugin.

- Your plugin should be configurable with the root account you want to make special in that way, in this case "Equity:Extra". See other plugins for how to pass in a configuration.

- You can optionally filter out all those Equity:Extra:* postings in the reports using the FROM syntax. Otherwise the detail of the Equity:Extra accounts in the balance sheet will be pretty harmless anyhow, but you could remove it.

Note that instead of identifying these special postings using a known root account, you could instead trigger that capability by using posting flags.

(Also, note that if all you care about is the balanced virtual accounts, that's entirely equivalent to a second transaction on the same date with the subset of postings to Equity:Extra, or identified by a configurable posting flag. I could be convinced to add that in, that is, a special state for a subset of postings, as a "shadow transaction" whereby the parser splits the single transaction into two separate transactions, adding a tag to the forked transaction so it can be filtered out at will. That could be implemented entirely as a plugin, BTW.)

- Small userbase? I seek longevity in my accounting systems. This is a bit of a chicken-egg problem, and is at the bottom of my concerns, but I'm mentioning it anyway.

Longevity is also one of my main priorities in developing Beancount!  I'll tell you what, there's _nothing_ that frustrates me more when I need to make an update to my ledger than facing broken code. That's why I focused on writing unit tests for _everything_ in the rewrite (see source code, non-withstanding code under beancount.query). That's also why I chose to implement this with Python: Python will work everywhere and for a long time, and all I depend on is a bit of C, no C++ (I'm a little tired of coding in Python personally, I would have coded this in another language to tinker with something new but I _really_ just want to it to _just work_ when I need it, and this is quite a safe bet it will for a long time). I commit the generated C code as well, to make sure even changes in flex/bison don't break users. I'm also _very_ sensitive about limiting the dependencies to simple libraries that I know are well-maintained and which themselves do not depend on much. Avoiding breakage is at the very top of my list. I will be needing to use this for myself for at least another decade, if not more. (In the absolute worse case I'd lose interest for adding new features and it would remain where it's at but I would fix the bugs; but so far the fire keeps me going.)

Other observations:

-------------------

- In general, my sense is, I'll always have a whole bunch of scripts "around" the core accounting software package I pick. For example, ofx/csv/screenscraping downloads and conversion into the journal on the input side. A few dozen analysis tools on the output/post-processing side. The question is, how well does the accounting software play with these? Can queries be handled by it at the level I ask them (and not via rebuilding high-level information from the low-level xml/csv output)?

Absolutely! I think about it the same way. There's just no way one can get by doing something like this without at least a small amount of scripting for imports, and from my POV the more scriptable this dataset is the better. This should be as much a toolkit as it is a set of tools, if you want to write some scripts. I share my ugly import code too (ledgerhub), but realistically I don't expect many others to reuse that, so it's appropriately isolated in that other project so I can keep the Beancount as an independent "clean core".

- I'm having trouble test-driving Beancount because it seems non-trivial to convert years of data I have in ledger journal format into Beancount format. Is there a way to lessen this pain? To xml and back or something similar?

Not that I can imagine, no. I think the pain is mostly due to the stricter requirements around cost basis. From what I can tell from other users you need to do a fair bit of manual information gathering to get your cost bases and correctly disambiguate price conversions from investments (postings held at cost). If you have an old dataset with only the checks that Ledger provides it almost assuredly contains many mistakes. The good news is that doing this conversion work you will likely find those mistakes and result in a more solid record of transactions. You will also be able to automatically convert it back to Ledger if you need to (bean-report ... ledger). (Well... the one showstopper is if you need recurring transaction templates to replace _past_ transactions, I don't support those, those are only useful for forecasting in the future IMO and all I have for now is an example plugin in beancount.plugins.forecast, but that's another discussion).

One thing that might help is to hold off until I have average cost booking and/or automatic default booking method (e.g. FIFO). That will make it a bit easier to automatically deal with cost basis for some accounts.

Finally, feel free to ask specific questions about problems encountered while doing the conversion.

On Fri, Jan 23, 2015 at 11:11 PM, <redst...@gmail.com> wrote:

Trying to give beancount a shot, and I have a bunch of questions/comments:

I have tons of personal command-line "recipes" in ledger (basically queries) to spit out all the information I want either frequently or occasionally. All these make extensive use of ledger's filtering and aggregation capabilities. So I'm really glad you've implemented at least a first version of bean-query. I was a bit surprised to find out this was recent.

But so far, being able to write SQL-like queries in beancount is a pretty huge step over ledger. I find it difficult to remember or keep track of what each switch and argument does in ledger, and constantly have to keep track of what's there in .ledgerrc at any given point. Seems like I can finally focus on the actual accounting part without fighting with my accounting software. Anyway, nicely done! :)

Thanks! Yes, I was mostly using the web interface until not too long ago. I never liked Ledger's command-line options for filtering and aggregation. I find it very confusing and I think there are some incompatible combinations that don't trigger errors (and that should), and it could be better documented. A lot of the information about how it works can only be found on the mailing-list. I find specifying those operations via SQL more suitable and it should familiar to most people. Now that I have it I'm trying to use this exclusively and have mostly stopped using bean-web.

Questions:

1) Are there any scripts to convert ledger files to beancount? Even if they're still have rough edges?

Not from me. I tried to write a Ledger to Beancount converter but after two 4 hour sessions giving it a good shot at trying to get Ledger to build on a vanilla 13.04 Ubuntu box with the Python bindings I gave up. I should probably give it a go using a more recent OS using Docker, but I have too many other things to do.

I think it's more important for me to support a Beancount to Ledger converter (and there is one available from bean-report, although it doesn't support everything, Beancount supports more flexible transactions that cannot be translated in Ledger though most people won't have those) because some people can then convert their files to Beancount syntax and then they should be able to use both with a simple conversion step.

2) I vaguely remembered seeing a plugin to auto-open non-existing accounts. I don't see it in src/plugins. Am I misremembering?

Just put this at the top of your file:

plugin "beancount.ops.auto_accounts"

At the moment, plugins live under beancount.ops and beancount.plugins, but I'll make a grand refactoring of this at some point so there's only one place.

3) "bean-query ledger" seems to be broken. Is this true?

Hmmm, it should mostly work. This is 70% complete code that needs another round of review, so there are a few rough edges. I merged it in the 'default' branch prematurely in order to start letting other people kick the tires on it and for me to focus on more urgent work around booking (see announcement on the mailing list). If you find specific bugs, please add an issue to the bug tracker, I'll handle them.

4) Is there a way to produce a .csv of a) the entire input file b) the output of a query. For (b) please feel free to provide me a pointer to the code, and issues you have already thought of if any, and I'd be happy to contribute. This is almost critical to me because I use excel extensively for more complex queries, analyses including charts, and to hunt down problems.

This is a very important feature for me too, to add to bean-query, but I haven't gotten around to it yet, see missing option here:

https://bitbucket.org/blais/beancount/src/613ca6a7ddf9908c46e3c39936fe04c2ec08e50b/src/python/beancount/scripts/query.py?at=default#cl-19

The text renderer is here:

https://bitbucket.org/blais/beancount/src/613ca6a7ddf9908c46e3c39936fe04c2ec08e50b/src/python/beancount/query/query_render.py?at=default#cl-404

I'm happy to receive contributions, but the easiest way to get those merged into the main code is to accompany them with convincing unit test coverage that covers the majority of cases. (I've tried to be pretty good about that in this codebase myself.)

(I probably need to look into the same attribution process forms used by GNU or what the Clojure community uses for copyright assignments before I merge code changes though.)

5) I haven't looked, but you're probably using some readline library (and also $PAGER), and I really like this. One thing that's not working too well is when I resize an xterm. Display works fine and reflows, but input gets confused. Again, if you point me to code, I'll see if it's perhaps a matter of turning on some library options (Strike this for now: this might've been a problem at my end. I'm unable to reproduce it for now).

I'm not sure what to say... don't resize while doing a query?

6) Is there a way to colorize output? A simple coloring of negative amounts is mainly what I need, but I can see the need for more at some point. Whatever library ledger is using to do so works great for me on a fairly complex cygwin/xterm/ssh/tmux setup without issue.

Not yet, but I'd love to add this at some point.

I think you could make that part of a pager for now, rendering just negative numbers from a stream in red should do it. You can modify this simple script, for example:

    http://furius.ca/tmp/colr/colr

Thanks a million for the interesting exchange.

I don't know what happened today somebody must have been tweeting about Beancount, I had to spend much of the day answer comments on the docs!

[redst...@gmail.com]

Thanks a bunch for the detailed response! And in particular, explaining your vision for it, which is great, and good to know.

I've moved over 90% my journals to beancount to give it a try (the remaining 10% might take 95% of the time though :)), and am liking quite a bit so far. I have a ton of questions which I'll post soon. Most are simple/small ones. Anyway, first, some responses:

 

Soon: Lot dates will be done correctly too, so you can figure out long vs. short term lots.

And I will definitely be using this feature as soon as it's ready!

 

- Investment analysis: another big problem. I'd like to compute investment returns, and a zillion other things, if possible. Also, I'd like to tag accounts and write account-tag based queries which I can then post process.

Good idea. I'll support that, I think it's a common case. (Note: In this context let's not call them "tags", that's Ledger terminology and it's badly defined IMO. "Tags" in Beancount as a set of strings attached to transactions. "Metadata" is what I call what you're thinking of.)

Agree, "metadata" is really what I'm trying to describe.

 

 

- Parser: non-"proprietary" parser is good for development

You mean "generated with a grammar"?

If so, I also agree. With my little grammar file I've been able to turn around impactful changes to the syntax in no time. For instance, adding metadata has been a breeze, very easy. That's just the right thing to do. It makes it almost _too_ easy. Plus it also gives you the guarantee that you could easily port the whole thing to a new computer language implementation without too much pain, or at least easily support the parsing bit, from many languages.

Yes, that's what I meant.

 

beancount downsides (for me):

-----------------------------

- Fixed 5 top-level accounts

  - I use a ZeroSum account as outlined in one of my previous posts. I use it more extensively than that post indicates

  - I rely on a top-level "TODO" account that is important to me as well. I've tried to propagate the TODO account to the 5 accounts, and find that it doesn't work

I don't understand what doesn't work, can you provide more details? You should be able to put all these transfer accounts under the Equity: category with no problem. Use Equity:TODO if you like. All accounts are treated the same in Beancount, and adding another root account/category would not change anything AFAIK. Very curious to know what the difficult was.

The reason for being strict about the five categories is essentially to be able to build income statements and balance sheets that look realistic and whose sums are consistent and correct. In a way, those categories are just a way to flag accounts as transient (income & expenses) vs. permanent (assets & liabilities), and positive balance (assets & expenses) vs. negative balance (liabiilties & income).  Equity is mainly there to absord retained earnings and conversion entries ("trading accounts").

I think this is a longer discussion, but the core of it is the question of whether the 5 accounts fit the user's style. I do see the value in your strictness, and yes, these 5 are the most fundamental ones. But I'm thinking that this can all still be done without having to create a hard-constraint.

 

- Effective dates: I use these extensively. They mostly fall into two main categories:

  - Tax view: self-explanatory. Zillion possible examples and use-cases

I don't understand what's self-explanatory about this bit. Can you provide an explicit example? I haven't completed my section in the Cookbook on how to file taxes, but so far I need no tricks to do this, it's straightforward double-entry method (with per-year Expenses accounts, e.g. Expenses:Taxes:Y2015:Federal).

Again, this might be a question of style, in addition to what one expects of their accounting system. Speaking for myself, minimizing time spent, simplicity, and elegance are frequently the drivers. Adding an account per year Expenses:Taxes:<variable> goes against time spent and simplicity. Simplicity-wise, at the least, this means that my pre-conversion scripts have to be aware of this extra layer. In addition, when I ask for a report of Expenses, I constantly have to filter out Taxes because they are from a different year. In addition, when I ask for a Tax report, I always have to filter out the refund (because it's always a year (or more) away). But wait, it gets worse. I have plenty of cases where for example, tax prepayments made for a quarter are made outside the quarter. Having a separate account for each quarter multiples this problem, and IMHO, causes ugliness to boot. Does this help give a sense of the problem?

 

  

  - For meaningful expense spreading: I tend to do a lot of prepaying for things (eg: insurance), and expense analysis is meaningless if I'm not able to spread this. I've tried putting things into a "holding account", but that's kludgy and doesn't work well in the long run.

I think what you mean here - and it isn't entirely obvious to me - is that you're amortizing an initial expense over time. This can be carried out using accounts and multiple entries but my reading is that you want to do this in a single entry. This would definitely require an implementation of the limbo accounts feature. I'll put that on the list.

If you can provide a realistic example of how you use this, I'll add it to my TODO item description (I'm imagining a different date per posting).

Yes, amortizing is what I meant. Expense reporting is a pain to use without effective dates for me. Example:

2013-07-02 * "ACME Insurance"

   Liabilities:Credit-Cards:MisterCard  -949 USD

   Expenses:Insurance:Renter   (949 USD / 12)  ; [=2013/07/2]

   Expenses:Insurance:Renter   (949 USD / 12)  ; [=2013/08/2]

   Expenses:Insurance:Renter   (949 USD / 12)  ; [=2013/08/2]

   Expenses:Insurance:Renter   (949 USD / 12)  ; [=2013/10/2]

   ... (12 entries, total)

The above syntax is Ledger syntax, and that part of it, I don't like (putting the date inside a comment). Instead, it'd be nice if the posting had a date. I see that this breaks your invariant of having a transaction sum up to zero within the smallest granularity of time (1 day), and I see the rationale for that as well. The limbo accounts would work for this, though I don't see how the invariant is not violated (if you impose the "sum=0 within one day" constraint).

Ledger has its --effective flag to ignore or take into account the effective date, as you know. Just FYI, I run ledger with --effective 99.9% of the time.

 

- Virtual accounts: I know Beancount has a strong no-no towards this, mainly with the "virtual" and the "unbalanced" parts. I don't care about those either. But I do need some solution, and Virtual accounts in ledger solved one of my longstanding problems neatly and elegantly. That problem is tracking my W2 (annual tax form). When I get a W2 now, it matches number for number in my virtual account. Particularly valuable with multiple W2s. I caught an error on an employer's side two years ago based on this.

I also compute numbers to match the W2 form exactly. All you have to do is aggregate the expense accounts over the calendar year (taxes are expenses).

You don't need virtual accounts for this.

It's actually not just W2s, but other related forms as well. And also on the W2, I compute about 10 items that are basically comments, and in addition, all boxes. Here are the problems:

- These computations are extremely regular. An "automated transaction" in Ledger is able to capture this regularity very well

- This means I don't have to put them into each paycheck. Less clutter

- This means I don't have "phantom" accounts to which money flows into and back the same day. Elegant + no need to worry about whether the Phantom account is non-zero at any point.

- When the occasional exception occurs, I can directly influence the virtual account

- If an error occurs and goes unnoticed, it doesn't affect any of my "real" accounts. The virtual account here represents the need for a domain with lower-level of scrutiny

 

What I need for this is not necessarily a "virtual account", but:

   - an account outside the 5 top-level accounts. It's fine if account has two parts, and they have to sum up to zero. 

   - this account has to live in its own "ecosystem."  In other words, it has to sum up to zero without depending on the other accounts. It should never be considered for any of the reports (like cashflow/balance/net assets/expenses etc.)

   - a way to write automated rules that will fill up these accounts (does Beancount have one?)

Is this possible to do in Beancount as it is currently?

Oh, I think I see what you're saying. You want to enforce a further constraints on a particular subset of accounts.

Actually, I might've misled you here, sorry! I don't need the "ecosystem" constraint, I was trying to imagine how to stuff my requirements into beancount/a-hypothetical-system's paradigm.

 

(Also, note that if all you care about is the balanced virtual accounts, that's entirely equivalent to a second transaction on the same date with the subset of postings to Equity:Extra, or identified by a configurable posting flag. I could be convinced to add that in, that is, a special state for a subset of postings, as a "shadow transaction" whereby the parser splits the single transaction into two separate transactions, adding a tag to the forked transaction so it can be filtered out at will. That could be implemented entirely as a plugin, BTW.)

I did think of this idea (and the high-level parts of your previous paras here), but like I mentioned earlier, in this case, I'd prefer to not have all these phantom transactions. Anyway:

Rethinking this completely, without accounts at all perhaps: save a (partial query+computation) in the journal. Example of how this might look:

journal:

reg_income = SELECT LAST(balance) WHERE account ~ "Income:Salary:Regular"

401k = SELECT LAST(balance) FROM  payee ~ "Individual Contribution" WHERE account ~ "Assets:401k"

W2_box99 = reg_income + 401k

Now, a query would look like:

SELECT * FROM type="saved_queries" and year=2014 WHERE name="W2_box99"

What do you think of that?

- I'm having trouble test-driving Beancount because it seems non-trivial to convert years of data I have in ledger journal format into Beancount format. Is there a way to lessen this pain? To xml and back or something similar?

One thing that might help is to hold off until I have average cost booking and/or automatic default booking method (e.g. FIFO). That will make it a bit easier to automatically deal with cost basis for some accounts.

I've converted all the "easy" transactions (which are perhaps 90% of the transactions), and here's what holding me up (mostly FYI):

- value expressions (I use these plenty)

- multiple files via include (really helps with the debug process as I convert)

- convert to ledger easily (I'll file a bug on this as you said, if it still breaks)

- a solution to solve what virtual accounts are solving for me currently

 

4) Is there a way to produce a .csv of a) the entire input file b) the output of a query. For (b) please feel free to provide me a pointer to the code, and issues you have already thought of if any, and I'd be happy to contribute. This is almost critical to me because I use excel extensively for more complex queries, analyses including charts, and to hunt down problems.

This is a very important feature for me too, to add to bean-query, but I haven't gotten around to it yet, see missing option here:

Thanks, will look at this as soon as I can. Noted your preference for unit test coverage (and very much agree!)

 

 

Thanks a million for the interesting exchange.

Thanks for your detailed responses too - they've significantly increased my confidence in seriously considering beancount! 

 

I don't know what happened today somebody must have been tweeting about Beancount, I had to spend much of the day answer comments on the docs!

Heh, this is good!  

[Martin Blais]

On Mon, Jan 26, 2015 at 3:55 AM, <redst...@gmail.com> wrote:

beancount downsides (for me):

-----------------------------

- Fixed 5 top-level accounts

  - I use a ZeroSum account as outlined in one of my previous posts. I use it more extensively than that post indicates

  - I rely on a top-level "TODO" account that is important to me as well. I've tried to propagate the TODO account to the 5 accounts, and find that it doesn't work

I don't understand what doesn't work, can you provide more details? You should be able to put all these transfer accounts under the Equity: category with no problem. Use Equity:TODO if you like. All accounts are treated the same in Beancount, and adding another root account/category would not change anything AFAIK. Very curious to know what the difficult was.

The reason for being strict about the five categories is essentially to be able to build income statements and balance sheets that look realistic and whose sums are consistent and correct. In a way, those categories are just a way to flag accounts as transient (income & expenses) vs. permanent (assets & liabilities), and positive balance (assets & expenses) vs. negative balance (liabiilties & income).  Equity is mainly there to absord retained earnings and conversion entries ("trading accounts").

I think this is a longer discussion, but the core of it is the question of whether the 5 accounts fit the user's style. I do see the value in your strictness, and yes, these 5 are the most fundamental ones. But I'm thinking that this can all still be done without having to create a hard-constraint.

Any ideas? How do I structure a balance sheet then? What do I do with the "other" accounts?

Some ideas:

- Swallow all other root categories under Equity.

- Render them as a separate report only and enforce separately that their balance sums to zero.

 

- Effective dates: I use these extensively. They mostly fall into two main categories:

  - Tax view: self-explanatory. Zillion possible examples and use-cases

I don't understand what's self-explanatory about this bit. Can you provide an explicit example? I haven't completed my section in the Cookbook on how to file taxes, but so far I need no tricks to do this, it's straightforward double-entry method (with per-year Expenses accounts, e.g. Expenses:Taxes:Y2015:Federal).

Again, this might be a question of style, in addition to what one expects of their accounting system. Speaking for myself, minimizing time spent, simplicity, and elegance are frequently the drivers. Adding an account per year Expenses:Taxes:<variable> goes against time spent and simplicity. Simplicity-wise, at the least, this means that my pre-conversion scripts have to be aware of this extra layer. In addition, when I ask for a report of Expenses, I constantly have to filter out Taxes because they are from a different year. In addition, when I ask for a Tax report, I always have to filter out the refund (because it's always a year (or more) away). But wait, it gets worse. I have plenty of cases where for example, tax prepayments made for a quarter are made outside the quarter. Having a separate account for each quarter multiples this problem, and IMHO, causes ugliness to boot. Does this help give a sense of the problem?

I agree that if you have to create a separate set of accounts for each quarter that's too much to ask. But annually is not that big a problem, see how I do it here, for example, look at the Expenses:Taxes accounts:

https://bitbucket.org/blais/beancount/src/tip/examples/example.beancount?at=default

Let's examine how you book it now and see whether it couldn't be carried out this way. I still don't understand why you really need virtual postings.

- Why do you have to "filter out taxes because they are from a different year"? Those expenses are always recorded at the moment they're paid, so I think that you might want (a) a report by calendar year of all expenses (tax expenses paid during this year and the past), or (b) a report over _all_ years narrowed down to the accounts under a particular tax year (to look at the history of tax payments for that year, e.g., in case you get audited, to clear that year).

- "When you ask for a Tax report"... what's a "Tax report"? (Not sure what you mean.) Why do you have to filter out the refund?

- Tax prepayments "for a quarter"... aren't those to be applied to a particular tax year? Why do you need quarter-specific treatment?  If this is an underlying filtering requirement, could you not instead deal with this using metadata or #tags?  (I also don't think you should have a set of separate accounts for each quarter, I'm not suggesting that.)

I think it might be worthwhile at this point to drill down and to enumerate a list of the reports you need to produce and how you use them and let's discuss how those could or could not be produced with the simple annual booking method I suggest. I think we will both learn something from that process, if you have the patience to engage it.

  - For meaningful expense spreading: I tend to do a lot of prepaying for things (eg: insurance), and expense analysis is meaningless if I'm not able to spread this. I've tried putting things into a "holding account", but that's kludgy and doesn't work well in the long run.

I think what you mean here - and it isn't entirely obvious to me - is that you're amortizing an initial expense over time. This can be carried out using accounts and multiple entries but my reading is that you want to do this in a single entry. This would definitely require an implementation of the limbo accounts feature. I'll put that on the list.

If you can provide a realistic example of how you use this, I'll add it to my TODO item description (I'm imagining a different date per posting).

Yes, amortizing is what I meant. Expense reporting is a pain to use without effective dates for me. Example:

2013-07-02 * "ACME Insurance"

   Liabilities:Credit-Cards:MisterCard  -949 USD

   Expenses:Insurance:Renter   (949 USD / 12)  ; [=2013/07/2]

   Expenses:Insurance:Renter   (949 USD / 12)  ; [=2013/08/2]

   Expenses:Insurance:Renter   (949 USD / 12)  ; [=2013/08/2]

   Expenses:Insurance:Renter   (949 USD / 12)  ; [=2013/10/2]

   ... (12 entries, total)

The above syntax is Ledger syntax, and that part of it, I don't like (putting the date inside a comment). Instead, it'd be nice if the posting had a date. I see that this breaks your invariant of having a transaction sum up to zero within the smallest granularity of time (1 day), and I see the rationale for that as well. The limbo accounts would work for this, though I don't see how the invariant is not violated (if you impose the "sum=0 within one day" constraint).

Ledger has its --effective flag to ignore or take into account the effective date, as you know. Just FYI, I run ledger with --effective 99.9% of the time.

Thanks for your example, this is great, this is what I thought you were doing. I also figured the common case was to use the "effective" date (the date on the postings).  I really should implement that transfer account solution (I could use it myself in a few instances actually.)

FWIW I used to have per-posting dates. The syntax I chose was to specify the date as a prefix to the account name (not comments, in Beancount comments are always just comments).

 

- Virtual accounts: I know Beancount has a strong no-no towards this, mainly with the "virtual" and the "unbalanced" parts. I don't care about those either. But I do need some solution, and Virtual accounts in ledger solved one of my longstanding problems neatly and elegantly. That problem is tracking my W2 (annual tax form). When I get a W2 now, it matches number for number in my virtual account. Particularly valuable with multiple W2s. I caught an error on an employer's side two years ago based on this.

I also compute numbers to match the W2 form exactly. All you have to do is aggregate the expense accounts over the calendar year (taxes are expenses).

You don't need virtual accounts for this.

It's actually not just W2s, but other related forms as well. And also on the W2, I compute about 10 items that are basically comments, and in addition, all boxes. Here are the problems:

- These computations are extremely regular. An "automated transaction" in Ledger is able to capture this regularity very well

I don't believe in regular transactions to represent the past. In my experience, these vary a little bit between every paycheck and if you want to get a precise account of expenses you need to enter them manually, even if your life is unchanging (but it's not: you'll get raises, expense repayments, changes in tax situation, and payments that cap throughout the year, e.g. max. amount on Social Security payments). But I do, however, believe a recurring transaction feature could be useful for extending into the future. That's another topic anyhow.

In any case, I don't want to impose my will: you should be able to build some sort of syntax for generating recurring transactions into the past using a plugin. That feature is near the bottom of my list for now, I view all the other stuff as much more important than to build recurring transaction support. A little bit of cut-n-paste is doable in the meantime. How I think this feature should emerge is people with an interest in it should build various prototypes for defining recurring transactions and one of those should eventually get integrated into the mainline. (If help is needed with adding a little bit of syntax enhancement to support that development I'm happy to indulge.)

 

- This means I don't have to put them into each paycheck. Less clutter

- This means I don't have "phantom" accounts to which money flows into and back the same day. Elegant + no need to worry about whether the Phantom account is non-zero at any point. 

- When the occasional exception occurs, I can directly influence the virtual account

- If an error occurs and goes unnoticed, it doesn't affect any of my "real" accounts. The virtual account here represents the need for a domain with lower-level of scrutiny

I'm not clear on what phantom accounts you use/need. Can you provide an explicit example paycheck transactions - you can remove the amounts - with those virtual postings, as an example?  I don't get it, I need an example.  (What I'm reading from this is that you're using the mirror accounting technique as I do for tracking 401k contributions, and that you're likely doing it on these special accounts, which could probably be defined into Assets/Expenses like I do.)

 

(Also, note that if all you care about is the balanced virtual accounts, that's entirely equivalent to a second transaction on the same date with the subset of postings to Equity:Extra, or identified by a configurable posting flag. I could be convinced to add that in, that is, a special state for a subset of postings, as a "shadow transaction" whereby the parser splits the single transaction into two separate transactions, adding a tag to the forked transaction so it can be filtered out at will. That could be implemented entirely as a plugin, BTW.)

I did think of this idea (and the high-level parts of your previous paras here), but like I mentioned earlier, in this case, I'd prefer to not have all these phantom transactions. Anyway:

Rethinking this completely, without accounts at all perhaps: save a (partial query+computation) in the journal. Example of how this might look:

journal:

reg_income = SELECT LAST(balance) WHERE account ~ "Income:Salary:Regular"

401k = SELECT LAST(balance) FROM  payee ~ "Individual Contribution" WHERE account ~ "Assets:401k"

W2_box99 = reg_income + 401k

Now, a query would look like:

SELECT * FROM type="saved_queries" and year=2014 WHERE name="W2_box99"

What do you think of that?

Sounds more complicated to me than it needs to be.

Why can't you just sum over the calendar year?

e.g. using my example file:

beancount> SELECT account, SUM(position) WHERE year = 2013 AND account ~ 'Expenses:Taxes:Y2013' GROUP BY account;

                 account                    sum_position  

------------------------------------------ ---------------

Expenses:Taxes:Y2013:US:SDI                   29.12 USD   

Expenses:Taxes:Y2013:US:Medicare            2772.12 USD   

Expenses:Taxes:Y2013:US:Federal:PreTax401k 17500.00 IRAUSD

Expenses:Taxes:Y2013:US:SocSec              7000.04 USD   

Expenses:Taxes:Y2013:US:Federal            27635.92 USD   

Expenses:Taxes:Y2013:US:CityNYC             4547.92 USD   

Expenses:Taxes:Y2013:US:State               9492.08 USD   

beancount> 

What's not to like about that?

These are the numbers that will appears in the boxes of your W-2.

Remove the "year = 2013" constraint to get the total you've actually paid after you file or get an adjustment notice.

Note that in the example file I'm using the mirror accounts methods to account for 401k contributions (I call this IRAUSD units, should probably rename). This can be automated using a plugin: beancount.plugins.ira_contrib.  I'll modify the example to use that sometime (I already use it in my personal file, works well).

 

- I'm having trouble test-driving Beancount because it seems non-trivial to convert years of data I have in ledger journal format into Beancount format. Is there a way to lessen this pain? To xml and back or something similar?

One thing that might help is to hold off until I have average cost booking and/or automatic default booking method (e.g. FIFO). That will make it a bit easier to automatically deal with cost basis for some accounts.

I've converted all the "easy" transactions (which are perhaps 90% of the transactions), and here's what holding me up (mostly FYI): 

- value expressions (I use these plenty) 

- multiple files via include (really helps with the debug process as I convert)

- convert to ledger easily (I'll file a bug on this as you said, if it still breaks)

- a solution to solve what virtual accounts are solving for me currently

This is very useful feedback overall. Thanks a lot for taking the time to engage! :-)

(1) and (2) are easy to implement and will be done. Knowing someone could use it right away is motivating.

(3) should already work. 

(4) we're discussing above.

Cheers,

[redst...@gmail.com]

- Effective dates: I use these extensively. They mostly fall into two main categories:

  - Tax view: self-explanatory. Zillion possible examples and use-cases

I don't understand what's self-explanatory about this bit. Can you provide an explicit example? I haven't completed my section in the Cookbook on how to file taxes, but so far I need no tricks to do this, it's straightforward double-entry method (with per-year Expenses accounts, e.g. Expenses:Taxes:Y2015:Federal).

Again, this might be a question of style, in addition to what one expects of their accounting system. Speaking for myself, minimizing time spent, simplicity, and elegance are frequently the drivers. Adding an account per year Expenses:Taxes:<variable> goes against time spent and simplicity. Simplicity-wise, at the least, this means that my pre-conversion scripts have to be aware of this extra layer. In addition, when I ask for a report of Expenses, I constantly have to filter out Taxes because they are from a different year. In addition, when I ask for a Tax report, I always have to filter out the refund (because it's always a year (or more) away). But wait, it gets worse. I have plenty of cases where for example, tax prepayments made for a quarter are made outside the quarter. Having a separate account for each quarter multiples this problem, and IMHO, causes ugliness to boot. Does this help give a sense of the problem?

I agree that if you have to create a separate set of accounts for each quarter that's too much to ask. But annually is not that big a problem, see how I do it here, for example, look at the Expenses:Taxes accounts:

https://bitbucket.org/blais/beancount/src/tip/examples/example.beancount?at=default

I used to do this myself until a few years ago, so I'm completely familiar with it, but I switched over to the way I do it now for many reasons (listed above). That is, to use "Expenses:Taxes:Federal", and not sub-account it using the year (":Y2012"). But this is all moot anyway, because I do need quarter-based accounting, and there would exists others in my boat too.

 

Let's examine how you book it now and see whether it couldn't be carried out this way. I still don't understand why you really need virtual postings.

- Why do you have to "filter out taxes because they are from a different year"? Those expenses are always recorded at the moment they're paid, so I think that you might want (a) a report by calendar year of all expenses (tax expenses paid during this year and the past), or (b) a report over _all_ years narrowed down to the accounts under a particular tax year (to look at the history of tax payments for that year, e.g., in case you get audited, to clear that year).

True, I typically want (a) and (b). With my current setup, using --real gets me (a), and --effective gets be (b). So I see this particular point as more of an effective dates issue rather than to do with virtual postings.

 

- "When you ask for a Tax report"... what's a "Tax report"? (Not sure what you mean.) Why do you have to filter out the refund?

By "tax report," I really mean a bunch of questions related to taxes. Your (b) above is one of them. Another is: how much have I paid so far for tax-year 2015? The common thing here being, taxes are paid/refunded for given periods outside that period.

As it stands, let's say all my tax payments, refunds, etc. go into "Expenses:Taxes:Federal." So I don't have a :Y2013 for the reasons mentioned above. This means, when I do:

SELECT * FROM year=2014 where accounts ~ "Expenses:Taxes:Federal"

then I also end up getting these things that I don't want to see:

- estimated payment for 2013's last quarter

- payment/refund for 2013, that appeared this year

But again, this is an effective dates issue, not yet a virtual accounts issue.

 

- Tax prepayments "for a quarter"... aren't those to be applied to a particular tax year? Why do you need quarter-specific treatment?

I don't, the IRS does :). I see you haven't run into this yet. Estimated payments are to be made within the quarter where the income is obtained. If you're employed, your employer sees to it this happens. If you work for yourself, or receive non-employment income, you have to handle this yourself. And it's quarter based. Most states do this as well. 

 

I think it might be worthwhile at this point to drill down and to enumerate a list of the reports you need to produce and how you use them and let's discuss how those could or could not be produced with the simple annual booking method I suggest. I think we will both learn something from that process, if you have the patience to engage it.

Happy to! And I'm always looking for ways to find better, cleaner, ways to do things.

 

FWIW I used to have per-posting dates. The syntax I chose was to specify the date as a prefix to the account name (not comments, in Beancount comments are always just comments).

This has always made me uncomfortable about ledger - that comments might not actually be comments. So I'm glad you're committed to sticking to this.

  

- Virtual accounts: I know Beancount has a strong no-no towards this, mainly with the "virtual" and the "unbalanced" parts. I don't care about those either. But I do need some solution, and Virtual accounts in ledger solved one of my longstanding problems neatly and elegantly. That problem is tracking my W2 (annual tax form). When I get a W2 now, it matches number for number in my virtual account. Particularly valuable with multiple W2s. I caught an error on an employer's side two years ago based on this.

I also compute numbers to match the W2 form exactly. All you have to do is aggregate the expense accounts over the calendar year (taxes are expenses).

You don't need virtual accounts for this.

It's actually not just W2s, but other related forms as well. And also on the W2, I compute about 10 items that are basically comments, and in addition, all boxes. Here are the problems:

- These computations are extremely regular. An "automated transaction" in Ledger is able to capture this regularity very well

I don't believe in regular transactions to represent the past. In my experience, these vary a little bit between every paycheck and if you want to get a precise account of expenses you need to enter them manually, even if your life is unchanging (but it's not: you'll get raises, expense repayments, changes in tax situation, and payments that cap throughout the year, e.g. max. amount on Social Security payments). But I do, however, believe a recurring transaction feature could be useful for extending into the future. That's another topic anyhow.

In any case, I don't want to impose my will: you should be able to build some sort of syntax for generating recurring transactions into the past using a plugin. That feature is near the bottom of my list for now, I view all the other stuff as much more important than to build recurring transaction support. A little bit of cut-n-paste is doable in the meantime. How I think this feature should emerge is people with an interest in it should build various prototypes for defining recurring transactions and one of those should eventually get integrated into the mainline. (If help is needed with adding a little bit of syntax enhancement to support that development I'm happy to indulge.)

Sorry, perhaps I didn't explain this well, and maybe led you completely off-track. I should've emphasized "computations" in "regular computations": I mean that the *computations* are regular, not the transactions/postings. I've never used automated transactions to represent the past, and have no interest in doing so. Let me assure you every paycheck is manually entered. What I meant is this:

= expr account=~/^Income:Employment:Salary/

   (W2:Gross Pay:Salary)   -1

= expr account=~/^Income:Employment:Benefits:LTD-I/

   (W2:Gross Pay:LTD-I)   -1

; deductions

= expr account=~/^ZeroSumAccounts:401k/ and payee=~/Individual Contribution/

   (W2:Deductions:401k)   -1

; limits

= expr account=~/^ZeroSumAccounts:401k/ and payee=~/Individual Contribution/

   (AnnualLimits:401k)   -1

These are automated transactions in ledger that post to virtual accounts. I actually have many more of these (eg: W2:Box1, 1099:xxx), but you get the idea. If you're not familiar with what these mean, see: http://www.ledger-cli.org/3.0/doc/ledger3.html#Automated-Transactions (it's brief). The parenthesis, of course, are the virtual accounts.

  

- This means I don't have to put them into each paycheck. Less clutter

- This means I don't have "phantom" accounts to which money flows into and back the same day. Elegant + no need to worry about whether the Phantom account is non-zero at any point. 

- When the occasional exception occurs, I can directly influence the virtual account

- If an error occurs and goes unnoticed, it doesn't affect any of my "real" accounts. The virtual account here represents the need for a domain with lower-level of scrutiny

I'm not clear on what phantom accounts you use/need. Can you provide an explicit example paycheck transactions - you can remove the amounts - with those virtual postings, as an example?  I don't get it, I need an example.  (What I'm reading from this is that you're using the mirror accounting technique as I do for tracking 401k contributions, and that you're likely doing it on these special accounts, which could probably be defined into Assets/Expenses like I do.)

Phantom accounts: I was referring to what you suggested both in your post and in your documentation. If I understand it right, for IRA limits for instance, you'd suggest:

2015-01-01 * "salary"

    Income:Salary   -100 USD

    Investments:Retirement:401k  20 USD

    Equity:AnnualLimits:401k        20 IRAUSD   ; individual contribution

    Equity:AnnualLimits:Balancer -20 IRAUSD   ; to balance out previous posting

    Assets:Bank

Now at any time, "Equity:AnnualLimits:401k" can be used to compare your contributions to the IRA limit for the year. I was saying that this to me can be much more elegantly expressed with one single what ledger calls an automated transaction:

= expr account=~/^Zero Sum accounts:401k/ and payee=~/Individual Contribution/

   (AnnualLimits:401k)   -1

And thus, the journal remains clean. I guess your mirror method would work for these. But it's nice to be able to write an expression like the above, and have it remain a part of the input data, rather than mess with a plugin, and have it be dependent on the program instead.

 

Sounds more complicated to me than it needs to be.

Why can't you just sum over the calendar year?

Because there are like 25 or more of these, and not just a single monolithic 'Expenses:Taxes:Y2013'. And as my example above shows, some have to be subtracted, and some, added.

 

What's not to like about that?

These are the numbers that will appears in the boxes of your W-2.

I don't see a calculation of gross pay, for example. SS pay, medicacare pay, etc. (not taxes).

So to summarize the issues are:

- effective dates  (this seems to be on your to-do list, so not much more discussion needed)

- automated transactions (hopefully, I've shown why they're useful above. although they can perhaps be solved on the reporting side)

- virtual accounts  (I'm not saying virtual accounts are needed, but it's not clear to me as to how to solve the problems above more elegantly. I believe solutions can be found for them via reporting, but I don't have clarity yet).

- accounts outside the 5 top-level ones (I didn't talk about this much but this might be related to some of the problems above - more later).

It's definitely nice to be able to have a good conversation about these - thanks! :)

[Martin Blais]

On Tue, Jan 27, 2015 at 9:21 PM, <redst...@gmail.com> wrote:

- Effective dates: I use these extensively. They mostly fall into two main categories:

  - Tax view: self-explanatory. Zillion possible examples and use-cases

I don't understand what's self-explanatory about this bit. Can you provide an explicit example? I haven't completed my section in the Cookbook on how to file taxes, but so far I need no tricks to do this, it's straightforward double-entry method (with per-year Expenses accounts, e.g. Expenses:Taxes:Y2015:Federal).

Again, this might be a question of style, in addition to what one expects of their accounting system. Speaking for myself, minimizing time spent, simplicity, and elegance are frequently the drivers. Adding an account per year Expenses:Taxes:<variable> goes against time spent and simplicity. Simplicity-wise, at the least, this means that my pre-conversion scripts have to be aware of this extra layer. In addition, when I ask for a report of Expenses, I constantly have to filter out Taxes because they are from a different year. In addition, when I ask for a Tax report, I always have to filter out the refund (because it's always a year (or more) away). But wait, it gets worse. I have plenty of cases where for example, tax prepayments made for a quarter are made outside the quarter. Having a separate account for each quarter multiples this problem, and IMHO, causes ugliness to boot. Does this help give a sense of the problem?

I agree that if you have to create a separate set of accounts for each quarter that's too much to ask. But annually is not that big a problem, see how I do it here, for example, look at the Expenses:Taxes accounts:

https://bitbucket.org/blais/beancount/src/tip/examples/example.beancount?at=default

I used to do this myself until a few years ago, so I'm completely familiar with it, but I switched over to the way I do it now for many reasons (listed above). That is, to use "Expenses:Taxes:Federal", and not sub-account it using the year (":Y2012"). But this is all moot anyway, because I do need quarter-based accounting, and there would exists others in my boat too.

What you say "quarter-based account" you mean "reporting for the quarter". You can do that regardless of the set of accounts you use (the more general method you describe, or my method with per-year accounts). Both methods are valid IMO, but without the per-year accounts, you'd have a more difficult time to figure out the entire history of payments that apply to a particular tax year. I suppose you could also use tags for this, after all, all that's needed is a method for grouping.  #tax2014, for example.

 

Let's examine how you book it now and see whether it couldn't be carried out this way. I still don't understand why you really need virtual postings.

- Why do you have to "filter out taxes because they are from a different year"? Those expenses are always recorded at the moment they're paid, so I think that you might want (a) a report by calendar year of all expenses (tax expenses paid during this year and the past), or (b) a report over _all_ years narrowed down to the accounts under a particular tax year (to look at the history of tax payments for that year, e.g., in case you get audited, to clear that year).

True, I typically want (a) and (b). With my current setup, using --real gets me (a), and --effective gets be (b). So I see this particular point as more of an effective dates issue rather than to do with virtual postings.

I just want to clarify something: I like the idea of being able to create multiple transactions in a single one, this idea that then requires "transfer accounts" what we called "limbo" earlier in the discussion with mharris, I think it's clearly useful and that with the transfer account solution it keeps the underlying DE nature of the ledger sane. But the concept of "effective date", in the way that Ledger does it, is problematic, because it creates different possible histories of transactions and that is a departure from the current state of things in Beancount. I really like the fact that there is a single assignment of dates at the moment, a single history, and I've placed emphasis on making it easy to provide different views (different filters and aggregations and transformations) of that single, immutable history.

Ledger's --effective feature allows you to effectively create two separate versions of history, and that has led to a lot of confusion, at least judging by the questions on the mailing-list. It would be reasonable if it truly enabled something new, but as far as I can tell it has mainly been used as a way to change the groups of postings aggregations are made with.

IMO creating a separate version of history is a kind of misuse of the feature. In other words, you don't really want the transaction to apply at the "effective" or alternate date, what you want is an aggregation or reporting over a different group of transactions, aggregation which is specified by date. So instead of supporting this odd Ledger feature, I'd rather focus on providing more flexible ways to carry out aggregations on a single version of history.

If you can think of a use case where effective dates enable one to do something realistically useful that cannot be handled by #tags or by a more powerful filtering / aggregation mechanism, I could be swayed, but otherwise adding the feature would break a pretty nice invariant.

 

- "When you ask for a Tax report"... what's a "Tax report"? (Not sure what you mean.) Why do you have to filter out the refund?

By "tax report," I really mean a bunch of questions related to taxes. Your (b) above is one of them. Another is: how much have I paid so far for tax-year 2015? The common thing here being, taxes are paid/refunded for given periods outside that period.

Yes, and that's why I identify those transactions by an account dedicated for that year. Another method could be to use tags, of just strings in the narration, or metadata. Just something to attach to the transaction that allows you to aggregate with later on.

As it stands, let's say all my tax payments, refunds, etc. go into "Expenses:Taxes:Federal." So I don't have a :Y2013 for the reasons mentioned above. This means, when I do:

SELECT * FROM year=2014 where accounts ~ "Expenses:Taxes:Federal"

then I also end up getting these things that I don't want to see:

- estimated payment for 2013's last quarter

- payment/refund for 2013, that appeared this year

But again, this is an effective dates issue, not yet a virtual accounts issue.

The true origin of the problem is that you'd like to select (filter) just those transactions which are intended to affect tax-year 2013. It's a filtering problem, not an "effective dates" problem. The way I proposed to solve that is using account names, but as mentioned above, there are other ways that might suit you better to perform that filter.

- Tax prepayments "for a quarter"... aren't those to be applied to a particular tax year? Why do you need quarter-specific treatment?

I don't, the IRS does :). I see you haven't run into this yet. Estimated payments are to be made within the quarter where the income is obtained. If you're employed, your employer sees to it this happens. If you work for yourself, or receive non-employment income, you have to handle this yourself. And it's quarter based. Most states do this as well. 

Actually I have; I used to have a Canadian corporation and had to make these types of quarterly installments (5 types, at different frequencies). But using the per-year account it wasn't needed that I make aggregations for them... I just eyeballed the journal. 4 payments... done.

In any case, filtering by something else than the date should be solvable how I suggest above. If you like, you can just use tags, e.g. #q1-2015" and have a single date of application (the date when you actually made the payment).

(As a historical note: having to track these annoying installments was one of the reasons I looked for a solution at the time and read up about double-entry accounting and learned what it was...)

I think it might be worthwhile at this point to drill down and to enumerate a list of the reports you need to produce and how you use them and let's discuss how those could or could not be produced with the simple annual booking method I suggest. I think we will both learn something from that process, if you have the patience to engage it.

Happy to! And I'm always looking for ways to find better, cleaner, ways to do things.

 

FWIW I used to have per-posting dates. The syntax I chose was to specify the date as a prefix to the account name (not comments, in Beancount comments are always just comments).

This has always made me uncomfortable about ledger - that comments might not actually be comments. So I'm glad you're committed to sticking to this.

  

- Virtual accounts: I know Beancount has a strong no-no towards this, mainly with the "virtual" and the "unbalanced" parts. I don't care about those either. But I do need some solution, and Virtual accounts in ledger solved one of my longstanding problems neatly and elegantly. That problem is tracking my W2 (annual tax form). When I get a W2 now, it matches number for number in my virtual account. Particularly valuable with multiple W2s. I caught an error on an employer's side two years ago based on this.

I also compute numbers to match the W2 form exactly. All you have to do is aggregate the expense accounts over the calendar year (taxes are expenses).

You don't need virtual accounts for this.

It's actually not just W2s, but other related forms as well. And also on the W2, I compute about 10 items that are basically comments, and in addition, all boxes. Here are the problems:

- These computations are extremely regular. An "automated transaction" in Ledger is able to capture this regularity very well

I don't believe in regular transactions to represent the past. In my experience, these vary a little bit between every paycheck and if you want to get a precise account of expenses you need to enter them manually, even if your life is unchanging (but it's not: you'll get raises, expense repayments, changes in tax situation, and payments that cap throughout the year, e.g. max. amount on Social Security payments). But I do, however, believe a recurring transaction feature could be useful for extending into the future. That's another topic anyhow.

In any case, I don't want to impose my will: you should be able to build some sort of syntax for generating recurring transactions into the past using a plugin. That feature is near the bottom of my list for now, I view all the other stuff as much more important than to build recurring transaction support. A little bit of cut-n-paste is doable in the meantime. How I think this feature should emerge is people with an interest in it should build various prototypes for defining recurring transactions and one of those should eventually get integrated into the mainline. (If help is needed with adding a little bit of syntax enhancement to support that development I'm happy to indulge.)

Sorry, perhaps I didn't explain this well, and maybe led you completely off-track. I should've emphasized "computations" in "regular computations": I mean that the *computations* are regular, not the transactions/postings. I've never used automated transactions to represent the past, and have no interest in doing so. Let me assure you every paycheck is manually entered. What I meant is this:

= expr account=~/^Income:Employment:Salary/

   (W2:Gross Pay:Salary)   -1

= expr account=~/^Income:Employment:Benefits:LTD-I/

   (W2:Gross Pay:LTD-I)   -1

; deductions

= expr account=~/^ZeroSumAccounts:401k/ and payee=~/Individual Contribution/

   (W2:Deductions:401k)   -1

; limits

= expr account=~/^ZeroSumAccounts:401k/ and payee=~/Individual Contribution/

   (AnnualLimits:401k)   -1

These are automated transactions in ledger that post to virtual accounts. I actually have many more of these (eg: W2:Box1, 1099:xxx), but you get the idea. If you're not familiar with what these mean, see: http://www.ledger-cli.org/3.0/doc/ledger3.html#Automated-Transactions (it's brief). The parenthesis, of course, are the virtual accounts.

Thanks for the clarification.

There are two rather distinct aspects to what you're doing, which are to be intended to be handled a bit differently with Beancount. (I'll be a bit verbose here and perhaps to some extent repeat some of the aspects of what I might have said before. Sorry for some of the repeat.)

The first is that you're using a custom language to automate the insertion of postings on existing transactions. Ledger includes this mini-language that is convenient for specifying many types of simple transactions like this, i.e., the syntax provide access to some variables in order to derive the desired amount, and there is some expression matching syntax to determine which transactions the transformations are intended to be applied to (I hope this is the same syntax as the query syntax, BTW, but I don't know). While the expression syntax is convenient (and it's also nice that it lives in the same file), it's clear to me it cannot possibly support all desired transformations. 

Beancount has no such expression language. Beancount is intended to be a little more "low-level" in that sense. The reason for not including a transformation language is that you should be able to carry this out with plugins that you write (similar to writing the matching expressions above, except a bit more verbose and imperative), and then you can do _anything_ you want in code, including many transformations which are impossible in Ledger. For example, adjusting the cost basis off of matching transactions (you have to find two separate transactions that occur at different points in time and pair them up and then transform them) is something you could not specify with the expression syntax in Ledger. In a Beancount plugin, you receive the entire stream of directives and your plugin outputs a modified stream of directives--you may modify, add or delete directives at will. You can do a lot of crazy stuff. I don't want to put too many limits on the kinds of transformations me or a user might want to carry out... let your imagination run wild. Ledger's approach to this has been an attempt at specifying the nature of the transformations and as such its specifications puts limits on what you can do. So it's a little harder in Beancount: you have to write code. Note that the particulars of the configuration for the plugin may still live in your input file (you insert a blob of Python code that will get evaluated and passed on to your config object).

The second is that you're using virtual postings to count flows for particular events. I have argued previously that this can be handled with the "mirror accounting" method (I apologize, for I have yet to write a full doc on this, but it's in the various emails on the list), whereby you can use an alternative currency to represent these flows. For example, I use "IRAUSD" for IRA contributions. Then you can have corresponding Income, Assets and Expenses account for those contributions, and you can respect the double-entry method while doing so. The accounts even make a lot of sense: the income account represents how many allowed contribution you receive as a "right" from the govt for the year, the asset account represents the amount of allowed contributions you have remaining, and the expenses account counts the number of contributions you've made so far. It's very sensible.

I'll agree that the simplicity of having these special accounts just to count things without having to worry about balancing at all is nice, but then it forces one system to deal with the balancing aspect separately... What do we then do with all these virtual postings? Are they to be included in aggregations? Where do they show up on the balance sheet? Should they be restricted to some accounts (e.g. some account that can _only_ receive virtual postings and such postings may only be made onto those special accounts)? There are many ideas...  I think the feature was just added to Ledger without regard to figuring out what should be done with these. IMO it's adding more complexity than it's worth, so I have so far chosen to shun virtual accounts in order to maintain the balancing property, because there exists a fairly elegant solution that solves the problem and so it's not urgent. I'll admit it might be nice to have these...hmm, I'll call them "utility accounts", that don't balance, but because they will definitely require special treatment and clearly defined rules for how they are to be handled, and I'd want to spec out really clearly how they would be handled in all cases before adding them in.

These two aspects, and a working solution for the specific application you have, can be seen in the beancount.plugins.ira_contribs plugin. The plugin I provide is likely not general enough for all cases (mharris reported so) but you can certainly modify if to make it do what you need it to. You could even prototype a similar expression language in the input configuration of your plugin that does exactly what you describe above (minus the balancing aspect), but that's probably more work than you need to do. In your shoes I would just make a plugin with hardcoded configuration (the plugin file does not have to live in Beancount, anywhere under your PYTHONPATH will do fine).

  

- This means I don't have to put them into each paycheck. Less clutter

- This means I don't have "phantom" accounts to which money flows into and back the same day. Elegant + no need to worry about whether the Phantom account is non-zero at any point. 

- When the occasional exception occurs, I can directly influence the virtual account

- If an error occurs and goes unnoticed, it doesn't affect any of my "real" accounts. The virtual account here represents the need for a domain with lower-level of scrutiny

I'm not clear on what phantom accounts you use/need. Can you provide an explicit example paycheck transactions - you can remove the amounts - with those virtual postings, as an example?  I don't get it, I need an example.  (What I'm reading from this is that you're using the mirror accounting technique as I do for tracking 401k contributions, and that you're likely doing it on these special accounts, which could probably be defined into Assets/Expenses like I do.)

Phantom accounts: I was referring to what you suggested both in your post and in your documentation. If I understand it right, for IRA limits for instance, you'd suggest:

2015-01-01 * "salary"

    Income:Salary   -100 USD

    Investments:Retirement:401k  20 USD

    Equity:AnnualLimits:401k        20 IRAUSD   ; individual contribution

    Equity:AnnualLimits:Balancer -20 IRAUSD   ; to balance out previous posting

    Assets:Bank

Now at any time, "Equity:AnnualLimits:401k" can be used to compare your contributions to the IRA limit for the year. I was saying that this to me can be much more elegantly expressed with one single what ledger calls an automated transaction:

= expr account=~/^Zero Sum accounts:401k/ and payee=~/Individual Contribution/

   (AnnualLimits:401k)   -1

And thus, the journal remains clean. I guess your mirror method would work for these. But it's nice to be able to write an expression like the above, and have it remain a part of the input data, rather than mess with a plugin, and have it be dependent on the program instead.

The way I look at it, you're describing a nice configuration language for a particular plugin that does something specific. Plugins are more general than this though... in a plugin, you can do anything...

Sounds more complicated to me than it needs to be.

Why can't you just sum over the calendar year?

Because there are like 25 or more of these, and not just a single monolithic 'Expenses:Taxes:Y2013'. And as my example above shows, some have to be subtracted, and some, added.

 

What's not to like about that?

These are the numbers that will appears in the boxes of your W-2.

I don't see a calculation of gross pay, for example. SS pay, medicacare pay, etc. (not taxes).

Gross pay: Just aggregate over the income account.

SocSec / Medicare: These are just expenses accounts. I put mine in taxes (look back at the output) but if you put them somewhere else just aggregate over these accounts.

So to summarize the issues are:

- effective dates  (this seems to be on your to-do list, so not much more discussion needed)

Not in the way that Ledger does it if we can avoid it. One history is a nice property.

 

- automated transactions (hopefully, I've shown why they're useful above. although they can perhaps be solved on the reporting side)

Yes, but I'd like to see that appear in a plugin if the expression language is going to be so specific.

 

- virtual accounts  (I'm not saying virtual accounts are needed, but it's not clear to me as to how to solve the problems above more elegantly. I believe solutions can be found for them via reporting, but I don't have clarity yet).

Long-term maybe, if a good description of how they are to be handled in reports can be spec'ed out.

 

- accounts outside the 5 top-level ones (I didn't talk about this much but this might be related to some of the problems above - more later).

(Same as the previous issue.)

 

It's definitely nice to be able to have a good conversation about these - thanks! :)

Hey, here's a wild idea, just for fun: I add a new root account, let's call it "Virtual" for now. All postings to a Virtual:... account does not participate in the balance of transactions, and they are entirely ignored for building balance sheets and income statements. However, you can use them in the SQL syntax to aggregate over and the linear "balances" report (that includes all accounts without manipulation) includes them. That would not denature the categorization and it would allow you to post amounts to those accounts without having to think about the two sides. Then there is no special syntax for virtual accounts... if it's posted to Virtual:... it doesn't count. Maybe that works. I don't know.

To be continued later... hopefully after some more happy coding time!

[Nathan Grigg]

By "tax report," I really mean a bunch of questions related to taxes. Your (b) above is one of them. Another is: how much have I paid so far for tax-year 2015? The common thing here being, taxes are paid/refunded for given periods outside that period.

Yes, and that's why I identify those transactions by an account dedicated for that year. Another method could be to use tags, of just strings in the narration, or metadata. Just something to attach to the transaction that allows you to aggregate with later on.

Another method is to backdate the tax payment to 2014-12-31 and pay it from e.g. Liabilities:Payable:USTaxes. Then on the day you pay your taxes just pay them into this liabilities account. 

[redst...@gmail.com]

You can do that regardless of the set of accounts you use (the more general method you describe, or my method with per-year accounts). Both methods are valid IMO, but without the per-year accounts, you'd have a more difficult time to figure out the entire history of payments that apply to a particular tax year. I suppose you could also use tags for this, after all, all that's needed is a method for grouping.  #tax2014, for example.

Here, one problem becomes clearer: using effective dates, I can simply mark the exceptions to the norm in my journal, whereas with tags or per-quarter accounts, I'll have to ensure each and every associated transaction has that tag or is put into its correct quarter-based account. With taxes, there are a small set of exceptions to the norm (say 4 estimated tax payments, and one tax filing payment or refund). Which is why using an exception-based mechanism works better.

BTW, I'm not making a case for ledger's effective dates at all. I agree that we can probably find a better solution than that.

 

I just want to clarify something: I like the idea of being able to create multiple transactions in a single one, this idea that then requires "transfer accounts" what we called "limbo" earlier in the discussion with mharris, I think it's clearly useful and that with the transfer account solution it keeps the underlying DE nature of the ledger sane.

IMO creating a separate version of history is a kind of misuse of the feature. In other words, you don't really want the transaction to apply at the "effective" or alternate date, what you want is an aggregation or reporting over a different group of transactions, aggregation which is specified by date. So instead of supporting this odd Ledger feature, I'd rather focus on providing more flexible ways to carry out aggregations on a single version of history.

If you can think of a use case where effective dates enable one to do something realistically useful that cannot be handled by #tags or by a more powerful filtering / aggregation mechanism, I could be swayed, but otherwise adding the feature would break a pretty nice invariant.

Hmm...that definitely makes sense as well. I've never been comfortable with the "separate version of history" either, and I see a lot of sense in not wanting to break the invariant you refer to.

 

In any case, filtering by something else than the date should be solvable how I suggest above. If you like, you can just use tags, e.g. #q1-2015" and have a single date of application (the date when you actually made the payment).

So would there be a an exception-based mechanism to do this?

 

Beancount has no such expression language. Beancount is intended to be a little more "low-level" in that sense. The reason for not including a transformation language is that you should be able to carry this out with plugins that you write (similar to writing the matching expressions above, except a bit more verbose and imperative), and then you can do _anything_ you want in code, including many transformations which are impossible in Ledger. For example, adjusting the cost basis off of matching transactions (you have to find two separate transactions that occur at different points in time and pair them up and then transform them) is something you could not specify with the expression syntax in Ledger. In a Beancount plugin, you receive the entire stream of directives and your plugin outputs a modified stream of directives--you may modify, add or delete directives at will. You can do a lot of crazy stuff. I don't want to put too many limits on the kinds of transformations me or a user might want to carry out... let your imagination run wild. Ledger's approach to this has been an attempt at specifying the nature of the transformations and as such its specifications puts limits on what you can do. So it's a little harder in Beancount: you have to write code. Note that the particulars of the configuration for the plugin may still live in your input file (you insert a blob of Python code that will get evaluated and passed on to your config object).

I like this about Beancount - that it's lower level, and adding a higher level layer can be accomplished via plugins. Whether or not a blob of code can be considered "input" is at best debatable, I think, but that's fine - I think the entire approach is still a step above Ledger, for instance.

 

These two aspects, and a working solution for the specific application you have, can be seen in the beancount.plugins.ira_contribs plugin. The plugin I provide is likely not general enough for all cases (mharris reported so) but you can certainly modify if to make it do what you need it to. You could even prototype a similar expression language in the input configuration of your plugin that does exactly what you describe above (minus the balancing aspect), but that's probably more work than you need to do. In your shoes I would just make a plugin with hardcoded configuration (the plugin file does not have to live in Beancount, anywhere under your PYTHONPATH will do fine).

Good idea, I'll try this first.

 

What's not to like about that?

These are the numbers that will appears in the boxes of your W-2.

I don't see a calculation of gross pay, for example. SS pay, medicacare pay, etc. (not taxes).

Gross pay: Just aggregate over the income account.

SocSec / Medicare: These are just expenses accounts. I put mine in taxes (look back at the output) but if you put them somewhere else just aggregate over these accounts.

By "(not taxes)" part, I meant I wasn't referring to the social security tax, but the social security *wages* - which is very different from social security taxes. Same for medicare. Calculating those is not as trivial as calculating the taxes.

 

So to summarize the issues are:

- effective dates  (this seems to be on your to-do list, so not much more discussion needed)

Not in the way that Ledger does it if we can avoid it. One history is a nice property.

I think I don't have a clear idea of what you mean by effective dates then. I assume you're referring to the limbo account, but if you could give or point me to a limbo account example, that would be great. Specifically, is the limbo account just like any other account, except, when looked at at a long enough interval of time, it always sums to zero?

 

- automated transactions (hopefully, I've shown why they're useful above. although they can perhaps be solved on the reporting side)

Yes, but I'd like to see that appear in a plugin if the expression language is going to be so specific.

This sounds good. I'll put this on my todo list.

Hey, here's a wild idea, just for fun: I add a new root account, let's call it "Virtual" for now. All postings to a Virtual:... account does not participate in the balance of transactions, and they are entirely ignored for building balance sheets and income statements. However, you can use them in the SQL syntax to aggregate over and the linear "balances" report (that includes all accounts without manipulation) includes them. That would not denature the categorization and it would allow you to post amounts to those accounts without having to think about the two sides. Then there is no special syntax for virtual accounts... if it's posted to Virtual:... it doesn't count. Maybe that works. I don't know.

 This is indeed what I'd thought of earlier, but I didn't quite know in what ways making an account like this would violate the assumptions you're making that depend on the five fixed root accounts. Seeing you say it makes me think this is quite doable. At any rate, I think the best way to test it is for me to jump in and try it for a while to see what problems arise. Sounds relatively simple to do as well. Do you have pointers for where I'd have to modify code to do this?

Thank you, and happy coding!

[redst...@gmail.com]

By "tax report," I really mean a bunch of questions related to taxes. Your (b) above is one of them. Another is: how much have I paid so far for tax-year 2015? The common thing here being, taxes are paid/refunded for given periods outside that period.

Yes, and that's why I identify those transactions by an account dedicated for that year. Another method could be to use tags, of just strings in the narration, or metadata. Just something to attach to the transaction that allows you to aggregate with later on.

Another method is to backdate the tax payment to 2014-12-31 and pay it from e.g. Liabilities:Payable:USTaxes. Then on the day you pay your taxes just pay them into this liabilities account. 

I was thinking about this as well, and I like it the best for now (and perhaps as a longer term solution as well), because it is a very portable way do handle this, and also models the real world pretty well. Thanks for mentioning this!

[Martin Blais]

So to summarize the issues are:

- effective dates  (this seems to be on your to-do list, so not much more discussion needed)

Not in the way that Ledger does it if we can avoid it. One history is a nice property.

I think I don't have a clear idea of what you mean by effective dates then. I assume you're referring to the limbo account, but if you could give or point me to a limbo account example, that would be great. Specifically, is the limbo account just like any other account, except, when looked at at a long enough interval of time, it always sums to zero?

To be specific: Ledger allows you to attach a secondary date to transactions. When you enable this effective dates flag the transactions use that date instead of the primary date. This essentially creates an alternative history. (IIRC the effective date is also called the alternative date, it's the same thing, probably some historical legacy.)

Beancount used to have this too, with the same syntax as Ledger, but when I rewrote the entire thing from scratch I removed it. 

Note that if you _really_ badly wanted alternative history, you could you could easily enter alternative dates as metadata (Beancount will recognize and parse a datetime.date type as a value for metadata) and you coudl write _very_ simple plugin that converts all the transactions to use the alternative date where present in the metadata (or otherwise leave the date as is). You could even define yourself multiple different sets of alternative dates by using different metadata fields... you can go crazy if you like and create multiple versions of history that way. But that would be segregated to a plugin so I'm comfortable with it, do whatever you like in plugins, they're perfect for experimentation.

Try it. This could take less than an hour to code. This could be _really_ simple and quick to do... my only guidance would be to point out that you should learn about collections.namedtuple() and use this function to modify your transactions to make sure the back entry on postings points back to the new/modified Transaction namedtuples you'd create:

https://bitbucket.org/blais/beancount/src/3da7a6b4412354bf91b0d3e3a452a13fcf1e1201/src/python/beancount/core/data.py?at=default#cl-426

 

 

Hey, here's a wild idea, just for fun: I add a new root account, let's call it "Virtual" for now. All postings to a Virtual:... account does not participate in the balance of transactions, and they are entirely ignored for building balance sheets and income statements. However, you can use them in the SQL syntax to aggregate over and the linear "balances" report (that includes all accounts without manipulation) includes them. That would not denature the categorization and it would allow you to post amounts to those accounts without having to think about the two sides. Then there is no special syntax for virtual accounts... if it's posted to Virtual:... it doesn't count. Maybe that works. I don't know.

 This is indeed what I'd thought of earlier, but I didn't quite know in what ways making an account like this would violate the assumptions you're making that depend on the five fixed root accounts. Seeing you say it makes me think this is quite doable. At any rate, I think the best way to test it is for me to jump in and try it for a while to see what problems arise. Sounds relatively simple to do as well. Do you have pointers for where I'd have to modify code to do this?

You'd have to add it to the parser to allow it, I'd add a 6th root account:

https://bitbucket.org/blais/beancount/src/3da7a6b4412354bf91b0d3e3a452a13fcf1e1201/src/python/beancount/parser/parser.py?at=default#cl-62

There the code that validates that transactions balances should likely be modified to not take into account those postings:

https://bitbucket.org/blais/beancount/src/3da7a6b4412354bf91b0d3e3a452a13fcf1e1201/src/python/beancount/ops/validation.py?at=default#cl-370

Calls this:

https://bitbucket.org/blais/beancount/src/3da7a6b4412354bf91b0d3e3a452a13fcf1e1201/src/python/beancount/core/interpolate.py?at=default#cl-64

And then some of the reporting code for creating balance sheets and income statements and likely some other code could break, that should make some decision about how to treat those postings. See some code in this file, for example:

https://bitbucket.org/blais/beancount/src/3da7a6b4412354bf91b0d3e3a452a13fcf1e1201/src/python/beancount/ops/summarize.py?at=default

If you just want to hack something together, you can probably do this in an hour or two; to make sure that everything is being dealt in the more sensible manner would require writing a lot of tests and checking things out. I encourage you to play with it in any case, add a Virtual root account and fix whatever break, and let us know what breaks and if it all works well. I bet you will be surprised how easy it is to hack Beancount itself (work with tests, there's good test coverage).

 

[redst...@gmail.com]

Note that if you _really_ badly wanted alternative history, you could you could easily enter alternative dates as metadata (Beancount will recognize and parse a datetime.date type as a value for metadata) and you coudl write _very_ simple plugin that converts all the transactions to use the alternative date where present in the metadata (or otherwise leave the date as is). You could even define yourself multiple different sets of alternative dates by using different metadata fields... you can go crazy if you like and create multiple versions of history that way. But that would be segregated to a plugin so I'm comfortable with it, do whatever you like in plugins, they're perfect for experimentation.

Try it. This could take less than an hour to code. This could be _really_ simple and quick to do... my only guidance would be to point out that you should learn about collections.namedtuple() and use this function to modify your transactions to make sure the back entry on postings points back to the new/modified Transaction namedtuples you'd create:

https://bitbucket.org/blais/beancount/src/3da7a6b4412354bf91b0d3e3a452a13fcf1e1201/src/python/beancount/core/data.py?at=default#cl-426

Thanks, tried it, and I have it working fine for my case. By "it", I don't mean alternative history, but basically what Nathan Grigg above suggested.

My plugin now converts this:

------(

2014-02-01 "Estimated taxes for 2013"

  taxdate: 2013-12-31

  Liabilities:Mastercard    -2000 USD

  Expenses:Taxes:Federal  2000 USD

)------

into this:

------(

2014-02-01 "Estimated taxes for 2013"

  Liabilities:Mastercard     -2000 USD

  Liabilities:Taxes:Federal 2000 USD

2013-12-31 "Estimated taxes for 2013"

  Liabilities:Taxes:Federal    -2000 USD

  Expenses:Taxes:Federal    2000 USD

)------

This works great, and I think there are some patterns that are more apparent now that might help with your limbo account idea to implement per-posting dates that differ from the transaction dates:

- when a posting to an "Expense:" account has a date that is prior to the transaction date, the amount outstanding between the two dates likely needs to be a booked to a Liability account (same as the example above)

- when a posting to an "Expense:" account has a date that occurrs after the transaction date, the amount outstanding between the two dates likely needs to be booked to an Assets account. eg:

------(

2014-02-01 "Insurance for 2 months"

  Liabilities:Mastercard    -2000 USD

  2014-02-01  Expenses:Insurance  1000 USD

  2014-03-01  Expenses:Insurance  1000 USD

)------

should become:

------(

2014-02-01 "Insurance for 2 months"

  Liabilities:Mastercard    -2000 USD

  Expenses:Insurance      1000 USD

  Assets:Holding:Insurance   1000 USD

2014-03-01 "Insurance for 2 months"

  Assets:Holding:Insurance   -1000 USD

  Expenses:Insurance  1000 USD

)------

I'd be happy to contribute my plugin when it works for more general cases, accounts for multiple postings etc., does more error checking, and once I write some test cases for it.

[Martin Blais]

Sweet! :-)

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/36113d44-5f11-4165-a62c-3a0bb1e73713%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.