First opinions, coming from Ledger

[Matthew Harris]

I am a heavy user of Ledger. 'ledger stats' says I have 27k postings spread across 65 files. I had been aware of Beancount before, but I had stayed away because it was focused more on web usage than command line queries and reports, and because the syntax is different. This week I finally read all the Beancount docs and created a simple starter file.

There are a bunch of things I immediately like about Beancount. I am excited that the order of the transactions in the file doesn't matter, so I can finally use balance assertions appropriately. I like that accounts are explicitly opened and closed. (With Ledger, I can sort of get this with --strict and --pedantic, but they didn't always work as expected.) It sounds like rounding support is more intelligent here, so maybe I can eliminate my Expenses:Rounding Error account that absorbs a penny or two per stock transaction. The tagging syntax makes more sense in Beancount, you eliminated the unnecessary support for multiple date formats, and you added dated notes, file links, and events, which sound potentially useful. The documentation is excellent.

I like the idea of using a tag for each trip; I hadn't thought of that. You're much more liberal with tags and accounts and transactions than I am.

The things I miss are all discussed in your TODO file, so maybe they're coming eventually.

The first thing I miss is effective dates. It makes sense to me that when it takes two days to transfer money between accounts or to have a purchase show up on my credit card, both dates should be recorded; the ledger for each account should have the proper dates for that account. And I currently record stock vesting as a single transaction with a posting for each vesting event.

Having posting metadata would be nice but isn't essential; I mostly use it to record check numbers in my checking account.

It would be really useful to associate metadata with accounts. With suitable query support, it would simplify my account hierarchy. Right now I've got accounts like Assets:Investments:Roth IRA:MH:Vanguard, and I use ledger queries to select accounts by owner (MH = me) and type (Roth) and produce CSV files that I plot with R. You also use a separate set of tax expense accounts for each year, which I don't do.

I'd love to be able to track my asset allocation (e.g. stocks vs bonds) over time. This is one thing I can't practically do with Ledger.

I also want to track expected expenses. For example, property tax and insurance premiums are due only once or twice a year. I'd like to have a heads-up so I can plan to keep enough money set aside. I can hack it by creating transactions in advance, but this feels unsolved still.

Matthew

[Martin Blais]

On Thu, Nov 27, 2014 at 5:23 PM, Matthew Harris <mharr...@gmail.com> wrote:

I am a heavy user of Ledger. 'ledger stats' says I have 27k postings spread across 65 files. I had been aware of Beancount before, but I had stayed away because it was focused more on web usage than command line queries and reports, and because the syntax is different. This week I finally read all the Beancount docs and created a simple starter file.

There are a bunch of things I immediately like about Beancount. I am excited that the order of the transactions in the file doesn't matter, so I can finally use balance assertions appropriately. I like that accounts are explicitly opened and closed. (With Ledger, I can sort of get this with --strict and --pedantic, but they didn't always work as expected.) It sounds like rounding support is more intelligent here, so maybe I can eliminate my Expenses:Rounding Error account that absorbs a penny or two per stock transaction.

It hasn't been done yet, but it will be entirely automatic once a plugin is written up to absorb all the residual crumbs remaining from all the balance checks.

The tagging syntax makes more sense in Beancount, you eliminated the unnecessary support for multiple date formats, and you added dated notes, file links, and events, which sound potentially useful. The documentation is excellent.

Thank you! :-)

I like the idea of using a tag for each trip; I hadn't thought of that. You're much more liberal with tags and accounts and transactions than I am.

The things I miss are all discussed in your TODO file, so maybe they're coming eventually.

The first thing I miss is effective dates. It makes sense to me that when it takes two days to transfer money between accounts or to have a purchase show up on my credit card, both dates should be recorded; the ledger for each account should have the proper dates for that account.

I'll agree with you that it would be nicer to have the two dates recorded correctly. The first version of Beancount had support for the effective dates syntax but I removed it when I rewrote it the second time around (the current version).

The reason I removed it is twofold: it wasn't well defined if you put two dates at the transaction level, like this:

  2014-08-23=2014-08-21 * "Transfer from other account"

    ...

    ...

it's not clear what legs are supposed to happen when. This wasn't really useful, this was just another way to attach another attribute to the Transaction object. For the case where you'd put the date on the postings, like this:

  2014-08-23 * "Transfer from other account"

    Assets:Checking              ...

    2014-08-21 Assets:Savings

this was well-defined, the transaction date behaves just like the "default date" for postings without a date, but it breaks an assumption in the new Beancount (which is more strict and principled), that any subset of transactions must sum up to zero. Ledger operates at the level of postings only, so it allows breaking the double-entry rule easily. That's something I don't really want to compromise on--as soon as you begin drawing balance sheet and closing years automatically and doing these high-level operations, it's annoying if the total sums of your accounts don't balance, and this is uglier to me than not having the perfectly correct dates on one side of a transfer. So all transactions, all transfers between accounts are constrained to occur punctually for now (at a single point in time, which is to say, on the same date).

Now I'd really like to solve this problem, because like you, I don't like to have to fudge the dates. I'll have to admit though, that so far, having to choose one date or the date hasn't _really_ been a problem. It pretty much works every time and it hasn't prevented me from recording anything -- perhaps the only bother has been the odd balance assertion that happens to fall in an in-between date that I have to move by a few days or comment out and rely on the next one. Nevertheless, I don't like the approach either, I just have learned to suppress the feeling; in pragmatic terms, choosing just one date so far "works."

Here's an idea I've had some time ago for how to resolve this while keeping the constraint: A transaction with postings that are on different dates could be automatically split into multiple transactions with a "limbo" transfer account to absorb imbalances over time. For example, the following transaction

  2014-08-23 * "Transfer from other account"

    Assets:Checking            210.00 USD

    2014-08-21 Assets:Savings

would be automatically converted by a plugin invoked automatically by the loader, into this:

  2014-08-21 * "Transfer from other account" ^split-478574

    Assets:Savings   -210.00 USD

    Equity:Limbo    210.00 USD

  2014-08-23 * "Transfer from other account" ^split-478574

    Equity:Limbo   -210.00 USD

    Assets:Checking  210.00 USD

(The link would be automatically added and would allow you to later on query for such splits.)

This accomplishes the same thing but each transaction balances and thus leaves the balance sheet to zero. If you're cognizant of this and you don't wink if you happen to look at a period that straddles one of these and you see a non-zero balance for that Equity:Limbo account (and that'll be rare anyhow), that could work.

What do you think?

 And I currently record stock vesting as a single transaction with a posting for each vesting event.

If you're referring to Google context, I don't understand why you do this. It seems to me the right thing to do here is to book each of these as separate transactions on the date that they occur.  If you have multiple grants, I can imagine putting all the vesting events in one transactions (they all occur on the 25th) but even then, I like the fact that each pay stub has a single transaction and document associated with each, and they do issue a separate paystub for each grant.

Can you provide an example and help me understand how it's better, or perhaps why you prefer it?

Having posting metadata would be nice but isn't essential; I mostly use it to record check numbers in my checking account.

In any case, it's pretty much done at this point, you'll have it. I'm just having to assess impact on parsing performance and merge it into "default", perhaps later this weekend.

It would be really useful to associate metadata with accounts.

The metadata I've implemented can be attached to any directlve type. To associate metadata with accounts, the natural place to put it would be on their Open directives.

 

With suitable query support, it would simplify my account hierarchy. Right now I've got accounts like Assets:Investments:Roth IRA:MH:Vanguard, and I use ledger queries to select accounts by owner (MH = me) and type (Roth) and produce CSV files that I plot with R.

This is a great observation indeed. Account names can be treated much like implicit database columns, and various kinds of aggregations make sense. e.g. 

  Assets:Investments:Roth IRA:MH:Vanguard

can be seen as a row with

  type=Assets,activity=Investments,account=Roth IRA,owner=MH,institution=Vanguard

and then you might want to perform aggregations using any values for any subsets of these dimensions.

This is such a pattern, and so common (I use <type>:<country>:<institution>:<account> myself) that I think using metadata is not the right solution for this, it would be too verbose and difficult to manage. In this case, the account name already has all the information you need to create the metadata from it. I've been thinking about other ways to handle this. Just sharing some ideas, your feedback welcome:

- We could add support for formalizing the "schema" of account names. For instance, in my case, I'd have some way to declare that the second component is always a thing called "country", the third a thing called "institution", etc. Then the SQL query syntax (pretty much done, in branch "shell") could automatically add accessors for these and that would naturally support arbitrary filtering and aggregation. The problem with this approach is that not all account follow this schema. Expenses and Equity accounts in particular typically depart from this.

- Another approach could be some sort of declarative language that extracts these values for each account name, e.g., perhaps via metadata, as 

  1980-01-01 open Assets:CA:NatBank:Checking

    attr:country: "CA"

    attr:institution: "NatBank"

    ...

But this is way too verbose.

One could do with just regexps run against all account names, e.g. a list of these

  ('country', '.*:([A-Z][A-Z]):')

if that matches, it would automatically extract a country attribute using the first matching group.

So this is getting a bit complicated, I'd like to let it simmer and get some feedback before coding anything.

One more thing: when I designed the metadata I decided that none of its values would receive special treatment from the base system. The intended purpose of metadata is so far for the user to make use of, and our course the SQL should have some way to query it.  I'm fairly reluctant to change this assumption--I fear a growth in assumptions about special values and what-not, I don't like that. If we can keep it so that metadata is always something that's just carried around and never read and interpreted by the Beancount source code itself, that's a desirable property IMO.  

A viable way to implement something like the above would be to create a plugin that would automatically add the relevant metadata based on account names (and you could use any code or definition language convenient to you in order to do that) and then to provide generic functionality in the SQL query engine that would make use of that metadata, regardless of whether it was added explicitly in the input file or inserted by a plugin. You could go wild with your own special rules about how to categorize your accounts and about how to specify that. (I made plugins _really_ easy to write so I don't see that as a barrier, they're basically just a function that accepts entries and spits out modified entries.)

Finally... if making the account names simpler is your goal, then yes, some of this could just move to the metadata indeed. This would also work: you could write a simple plugin that hunts for all the Open directives and copies metadata fro all the postings for that account. That would work with the above. It could even support you overloading some of the default metadata per posting.  

I do admit though that all this replication may be a little silly... maybe the indirection should be built into the query language, e.g. account.metadata.owner = 'MH' could be a supported filter, and the SQL interpreter itself would gather all the open accounts before running and make it available to the expression evaluator.  Hmmm, something to think about a bit more.  (This is all easy, BTW, fairly small changes to the code.)

Thoughts?

You also use a separate set of tax expense accounts for each year, which I don't do.

This I recommend highly, but it'll work fine without doing that, you don't have to do that at all. If the govt comes after you with an audit, it's nice to be able to track all the activity for that particular year's accounts several years down the road.

I'd love to be able to track my asset allocation (e.g. stocks vs bonds) over time. This is one thing I can't practically do with Ledger.

I'm doing that already, but it's under "beancount/experiments/portfolio" and I have to move that code under the main source code, because I'm using it actively and it's stable. I just have to add some unit tests and some code that invokes it on the example.beancount file to demonstrate some useful output.

I configured mine to output asset class (as you describe), currency exposure, country exposure, liquidity/availability, taxation status (pre/post tax), and a special roll-up for Google stock (I don't do auto-sale but I have a target maximum percentage beyond which I offload shares and this gives me the precise fraction against my net worth. I don't want to lean too heavy on it, I find it nice to have the incentive and some skin in the game, but it's also wise to diversify and not to feel like worrying about the stock price movements). You can define arbitrary dimensions to run on it, it's essentially a dot-product against your holdings, and the selectors required to make this work are only (account, currency, cost-currency)).

Another report I find useful is the "cash" report, which tells you about all the little bits of cash sitting everywhere in all accounts, that may want to be invested.

I also want to track expected expenses. For example, property tax and insurance premiums are due only once or twice a year. I'd like to have a heads-up so I can plan to keep enough money set aside. I can hack it by creating transactions in advance, but this feels unsolved still.

Yes.

Have you seen the forecast plugin example?

https://bitbucket.org/blais/beancount/src/tip/src/python/beancount/plugins/forecast.py?at=returns

This is a simple prototype, but you could easily start writing a custom plugin to insert those transactions automatically just for yourself (plugins don't have to live in the Beancount source code). You could use arbitrary rules to configure it, or just bake everything in the source code-- you have all the power of Python.  I foresee that in the long run I will eventually provide some sort of generic option that you will be able to configure with some rules for generating transactions, but that's a rather low priority on the list of things to do at this moment.

Thanks for taking the time to provide detailed feedback, and looking forward to more discussion.

Matthew

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/76395dc9-768e-43db-a517-8bcc3e20ee8e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthew S. Harris]

On Thu Nov 27 2014 at 5:03:34 PM Martin Blais <bl...@furius.ca> wrote:

On Thu, Nov 27, 2014 at 5:23 PM, Matthew Harris <mharr...@gmail.com> wrote:

The first thing I miss is effective dates. It makes sense to me that when it takes two days to transfer money between accounts or to have a purchase show up on my credit card, both dates should be recorded; the ledger for each account should have the proper dates for that account.

I'll agree with you that it would be nicer to have the two dates recorded correctly. The first version of Beancount had support for the effective dates syntax but I removed it when I rewrote it the second time around (the current version).

The reason I removed it is twofold: it wasn't well defined if you put two dates at the transaction level, like this:

  2014-08-23=2014-08-21 * "Transfer from other account"

    ...

    ...

it's not clear what legs are supposed to happen when. This wasn't really useful, this was just another way to attach another attribute to the Transaction object.

Agreed. 

 

For the case where you'd put the date on the postings, like this:

  2014-08-23 * "Transfer from other account"

    Assets:Checking              ...

    2014-08-21 Assets:Savings

this was well-defined, the transaction date behaves just like the "default date" for postings without a date, but it breaks an assumption in the new Beancount (which is more strict and principled), that any subset of transactions must sum up to zero. Ledger operates at the level of postings only, so it allows breaking the double-entry rule easily. That's something I don't really want to compromise on--as soon as you begin drawing balance sheet and closing years automatically and doing these high-level operations, it's annoying if the total sums of your accounts don't balance, and this is uglier to me than not having the perfectly correct dates on one side of a transfer. So all transactions, all transfers between accounts are constrained to occur punctually for now (at a single point in time, which is to say, on the same date).

Now I'd really like to solve this problem, because like you, I don't like to have to fudge the dates. I'll have to admit though, that so far, having to choose one date or the date hasn't _really_ been a problem. It pretty much works every time and it hasn't prevented me from recording anything -- perhaps the only bother has been the odd balance assertion that happens to fall in an in-between date that I have to move by a few days or comment out and rely on the next one. Nevertheless, I don't like the approach either, I just have learned to suppress the feeling; in pragmatic terms, choosing just one date so far "works."

Here's an idea I've had some time ago for how to resolve this while keeping the constraint: A transaction with postings that are on different dates could be automatically split into multiple transactions with a "limbo" transfer account to absorb imbalances over time. For example, the following transaction

  2014-08-23 * "Transfer from other account"

    Assets:Checking            210.00 USD

    2014-08-21 Assets:Savings

would be automatically converted by a plugin invoked automatically by the loader, into this:

  2014-08-21 * "Transfer from other account" ^split-478574

    Assets:Savings   -210.00 USD

    Equity:Limbo    210.00 USD

  2014-08-23 * "Transfer from other account" ^split-478574

    Equity:Limbo   -210.00 USD

    Assets:Checking  210.00 USD

(The link would be automatically added and would allow you to later on query for such splits.)

This accomplishes the same thing but each transaction balances and thus leaves the balance sheet to zero. If you're cognizant of this and you don't wink if you happen to look at a period that straddles one of these and you see a non-zero balance for that Equity:Limbo account (and that'll be rare anyhow), that could work.

What do you think?

I like that idea very much.

 And I currently record stock vesting as a single transaction with a posting for each vesting event.

If you're referring to Google context, I don't understand why you do this. It seems to me the right thing to do here is to book each of these as separate transactions on the date that they occur.  If you have multiple grants, I can imagine putting all the vesting events in one transactions (they all occur on the 25th) but even then, I like the fact that each pay stub has a single transaction and document associated with each, and they do issue a separate paystub for each grant.

Can you provide an example and help me understand how it's better, or perhaps why you prefer it?

I don't claim this is better, but this is what I eventually settled on (in Ledger syntax), based on what someone else on the ledger list does:

2014/07/02 Stock Vesting

  Assets:Investments:Stock:MH:RSUs:Grant 12345  2 GOOG @ $600.00  ; [2014/07/25]

  Assets:Investments:Stock:MH:RSUs:Grant 12345  2 GOOG @ $600.00  ; [2014/08/25]

  Assets:Investments:Stock:MH:RSUs:Grant 12345  2 GOOG @ $600.00  ; [2014/09/25]

  ...

  Assets:Investments:Stock:MH:RSUs:Grant 12345  2 GOOG  ; [2018/06/25]

  Income:Options:MH

Now, this only shows the shares received after tax withholding. I never did adequately tie together the vesting with the tax event. It'd be great if I could do it without too much work. I see that you do it with transaction links. Ledger doesn't have links, of course, but in general I find link IDs ugly compared to a single transaction with multiple posting dates. I'll have to try it your way and see what happens.

 

It would be really useful to associate metadata with accounts.

The metadata I've implemented can be attached to any directlve type. To associate metadata with accounts, the natural place to put it would be on their Open directives.

Assuming the metadata never changes, this makes sense. If the asset allocation of a mutual fund were stored in metadata, it might need to change over time, such as when a target date fund increases its bond holdings.

I'd love to see other people's examples of useful metadata.

 

With suitable query support, it would simplify my account hierarchy. Right now I've got accounts like Assets:Investments:Roth IRA:MH:Vanguard, and I use ledger queries to select accounts by owner (MH = me) and type (Roth) and produce CSV files that I plot with R.

This is a great observation indeed. Account names can be treated much like implicit database columns, and various kinds of aggregations make sense. e.g. 

  Assets:Investments:Roth IRA:MH:Vanguard

can be seen as a row with

  type=Assets,activity=Investments,account=Roth IRA,owner=MH,institution=Vanguard

and then you might want to perform aggregations using any values for any subsets of these dimensions.

This is such a pattern, and so common (I use <type>:<country>:<institution>:<account> myself) that I think using metadata is not the right solution for this, it would be too verbose and difficult to manage. In this case, the account name already has all the information you need to create the metadata from it. I've been thinking about other ways to handle this. Just sharing some ideas, your feedback welcome:

- We could add support for formalizing the "schema" of account names. For instance, in my case, I'd have some way to declare that the second component is always a thing called "country", the third a thing called "institution", etc. Then the SQL query syntax (pretty much done, in branch "shell") could automatically add accessors for these and that would naturally support arbitrary filtering and aggregation. The problem with this approach is that not all account follow this schema. Expenses and Equity accounts in particular typically depart from this.

- Another approach could be some sort of declarative language that extracts these values for each account name, e.g., perhaps via metadata, as 

  1980-01-01 open Assets:CA:NatBank:Checking

    attr:country: "CA"

    attr:institution: "NatBank"

    ...

But this is way too verbose.

One could do with just regexps run against all account names, e.g. a list of these

  ('country', '.*:([A-Z][A-Z]):')

if that matches, it would automatically extract a country attribute using the first matching group.

So this is getting a bit complicated, I'd like to let it simmer and get some feedback before coding anything.

One more thing: when I designed the metadata I decided that none of its values would receive special treatment from the base system. The intended purpose of metadata is so far for the user to make use of, and our course the SQL should have some way to query it.  I'm fairly reluctant to change this assumption--I fear a growth in assumptions about special values and what-not, I don't like that. If we can keep it so that metadata is always something that's just carried around and never read and interpreted by the Beancount source code itself, that's a desirable property IMO.  

A viable way to implement something like the above would be to create a plugin that would automatically add the relevant metadata based on account names (and you could use any code or definition language convenient to you in order to do that) and then to provide generic functionality in the SQL query engine that would make use of that metadata, regardless of whether it was added explicitly in the input file or inserted by a plugin. You could go wild with your own special rules about how to categorize your accounts and about how to specify that. (I made plugins _really_ easy to write so I don't see that as a barrier, they're basically just a function that accepts entries and spits out modified entries.)

Finally... if making the account names simpler is your goal, then yes, some of this could just move to the metadata indeed. This would also work: you could write a simple plugin that hunts for all the Open directives and copies metadata fro all the postings for that account. That would work with the above. It could even support you overloading some of the default metadata per posting.  

I do admit though that all this replication may be a little silly... maybe the indirection should be built into the query language, e.g. account.metadata.owner = 'MH' could be a supported filter, and the SQL interpreter itself would gather all the open accounts before running and make it available to the expression evaluator.  Hmmm, something to think about a bit more.  (This is all easy, BTW, fairly small changes to the code.)

Thoughts?

This is all sounding quite complicated now.

I would like to point out that I have both Roth and non-Roth portions of my 401k, which is why I need to query for [401k and not Roth], for example, to get the pre-tax portion. I really wouldn't want to have to move the Roth portion outside of my 401k hierarchy.

My initial thought was that we wouldn't constrain the form of account names, but instead we would write declarations like this:

  ;; Everything is post-tax money except a few things.

  setattr Assets "taxstatus" "post-tax"

  setattr Assets:Vanguard:401k "taxstatus" "pre-tax"

  setattr Assets:Vanguard:401k:Roth "taxstatus" "post-tax"

  setattr Assets:Vanguard:IRA "taxstatus" "pre-tax"

  ;; Exclude certain accounts from our net worth, as they're not really ours. Examples: college savings plans, donor-advised funds

  setattr Assets:College "networth" "exclude"

Then my account hierarchy is much flatter.

You also use a separate set of tax expense accounts for each year, which I don't do.

This I recommend highly, but it'll work fine without doing that, you don't have to do that at all. If the govt comes after you with an audit, it's nice to be able to track all the activity for that particular year's accounts several years down the road.

With Ledger I can do that already. [-b 2014 -e 2015] will restrict my query to just this year. I assumed Beancount would get this if it's getting a query language.

 

I'd love to be able to track my asset allocation (e.g. stocks vs bonds) over time. This is one thing I can't practically do with Ledger.

I'm doing that already, but it's under "beancount/experiments/portfolio" and I have to move that code under the main source code, because I'm using it actively and it's stable. I just have to add some unit tests and some code that invokes it on the example.beancount file to demonstrate some useful output.

I configured mine to output asset class (as you describe), currency exposure, country exposure, liquidity/availability, taxation status (pre/post tax), and a special roll-up for Google stock (I don't do auto-sale but I have a target maximum percentage beyond which I offload shares and this gives me the precise fraction against my net worth. I don't want to lean too heavy on it, I find it nice to have the incentive and some skin in the game, but it's also wise to diversify and not to feel like worrying about the stock price movements). You can define arbitrary dimensions to run on it, it's essentially a dot-product against your holdings, and the selectors required to make this work are only (account, currency, cost-currency)).

That all sounds great. I don't really get the feel of it from reading portfolio.py. Examples would be useful.

Have you seen the forecast plugin example?

https://bitbucket.org/blais/beancount/src/tip/src/python/beancount/plugins/forecast.py?at=returns

 

I hadn't seen that example, but that sounds like the direction I was imagining. I agree with the low priority. It'll certainly be nice to have, but I've been getting by without it so far (even though Ledger has something like this).

 

Thanks for taking the time to provide detailed feedback, and looking forward to more discussion.

Likewise.

Matthew

[Martin Blais]

So there are a few problems with your method above:

- Your Income:Options:MH leg will be posted in 2014, for the entire set of legs over multiple years. This is incorrect, from an accounting perspective. At the very least, you should have one such entry for each year, even in Ledger, so that income for that year is posted on that year's income statement. 

- The income you compute with this is also incorrect... as you note, you need to take into account the taxes for it for this to work. That's not really income what you're calculating there, and while this may be a useful quantity for now, this will come around to bite you at the EOY because AFAIK income from RSUs should be reflected on your W-2 (there's an associated paystub that details taxes) and you won't be able to get the right numbers matching the W-2 if you did not calculate it from the paystubs. If you're not entering detail from your paystubs, that might be okay.

- The cost basis which you receive your RSUs at is not pre-determined in any way by the price at which they were set when they were granted to you. So the repeated regular $600.00 above is incorrect. Each vesting event takes place at the market price on that day and so it varies. Because of this, you cannot book it in advance. 

- Nit: RSUs are not options, and they're also not treated as options from a tax perspective (they don't fall under 1256 treatment), you might eventually want to rename this income account. 

The method I propose for dealing with it calculated the pre-tax income, includes tax details, transfers to a cash account the remainder, then reduces that by the integer number of shares acquired (saving cost basis, useful for wash sales later on), leaving cash to be paid later, which then gets booked against the payments for this cash portion, and a final check that the remainder is zero. I'll admit that it's vastly more verbose than the method you suggest above, but it has several advantages: the fees are taken into account, the taxes and income are calculated correctly, it tracks the amount of cash lying in limbo after vesting events, and you can organize the transactions "by account" in your input file, which is nice when you're reconciling history. I'll be checking in an example template for Googlers soon that will demonstrate the full detail of this, so you'll be able to judge for yourself if it's too much of a burden to do.  I'm not sure yet how much of it could be automated, but certainly if any of it can, then we can do it via some plugin. The thing is, once you're gone through one cycle of it, after that it's all just a bit of cut-n-paste from the previous transactions and adjusting the numbers.

It would be really useful to associate metadata with accounts.

The metadata I've implemented can be attached to any directlve type. To associate metadata with accounts, the natural place to put it would be on their Open directives.

Assuming the metadata never changes, this makes sense. If the asset allocation of a mutual fund were stored in metadata, it might need to change over time, such as when a target date fund increases its bond holdings.

Ah... yes, metadata can change over time. I think there's a crossover point where if the particular application requires much configuration data you're meant to offload that configuration to a script and do something custom, metadata will only take you so far. I don't know. One of the distant features I'll implement is the ability for a plugin to define a new directive, which could be potentially be used for something like that within the Beancount syntax.

Apologies... this is all designthink, thinking out loud about different ways this could be implemented.

 

I would like to point out that I have both Roth and non-Roth portions of my 401k, which is why I need to query for [401k and not Roth], for example, to get the pre-tax portion. I really wouldn't want to have to move the Roth portion outside of my 401k hierarchy.

My initial thought was that we wouldn't constrain the form of account names, but instead we would write declarations like this:

  ;; Everything is post-tax money except a few things.

  setattr Assets "taxstatus" "post-tax"

  setattr Assets:Vanguard:401k "taxstatus" "pre-tax"

  setattr Assets:Vanguard:401k:Roth "taxstatus" "post-tax"

  setattr Assets:Vanguard:IRA "taxstatus" "pre-tax"

  ;; Exclude certain accounts from our net worth, as they're not really ours. Examples: college savings plans, donor-advised funds

  setattr Assets:College "networth" "exclude"

Then my account hierarchy is much flatter.

Yes, something like this, with metadata the equivalent would look like this on the Open directives, different syntax but same idea:

YYYY-MM-DD open Assets:Vanguard:401k:Roth

  tax-status: "post-tax"

YYYY-MM-DD open Assets:Vanguard:401k:Roth

  tax-status: "pre-tax"

You also use a separate set of tax expense accounts for each year, which I don't do.

This I recommend highly, but it'll work fine without doing that, you don't have to do that at all. If the govt comes after you with an audit, it's nice to be able to track all the activity for that particular year's accounts several years down the road.

With Ledger I can do that already. [-b 2014 -e 2015] will restrict my query to just this year. I assumed Beancount would get this if it's getting a query language.

Based on your response, I'm not sure we understood each other.  What you suggest wouldn't work in Ledger either, it will only report payments made in 2014, what I mean is that you should be able to look at all tax payments and receipts made "for" taxation year 2014, even those paid in 2015 when you file your taxes and further adjustments to year 2014's accounts made in 2016 in case of a review or if you end up filing a 1040X and further forward. Note that I don't say payments made "in" taxation year 2014, but "for". Payments for tax year 2014 may occur for many years after 2014.  This is how the government actually tracks your taxes AFAIK.

But yes, filtering by year/date is available in the SQL-like query language I've built.

(Also note that restricting by date will also yield incorrect results in Ledger. Ledger does not (AFAIK) transfer retained income and expenses to equity; in general, restricting by date is not sufficient to compute a correct balance sheet, you need to perform a few special operations so that the equity account has the correct amounts. I call these "open" and "close" in the query language, and they are meant to correctly deal with many aspects, e.g. currency conversions and unrealized gains. I have yet to write a detailed explanation about this but I will add this to the doc eventually. The code is under beancount.ops.summarize)

 

I'd love to be able to track my asset allocation (e.g. stocks vs bonds) over time. This is one thing I can't practically do with Ledger.

I'm doing that already, but it's under "beancount/experiments/portfolio" and I have to move that code under the main source code, because I'm using it actively and it's stable. I just have to add some unit tests and some code that invokes it on the example.beancount file to demonstrate some useful output.

I configured mine to output asset class (as you describe), currency exposure, country exposure, liquidity/availability, taxation status (pre/post tax), and a special roll-up for Google stock (I don't do auto-sale but I have a target maximum percentage beyond which I offload shares and this gives me the precise fraction against my net worth. I don't want to lean too heavy on it, I find it nice to have the incentive and some skin in the game, but it's also wise to diversify and not to feel like worrying about the stock price movements). You can define arbitrary dimensions to run on it, it's essentially a dot-product against your holdings, and the selectors required to make this work are only (account, currency, cost-currency)).

That all sounds great. I don't really get the feel of it from reading portfolio.py. Examples would be useful.

Absolutely.... I'll build one that uses the example file and merge the code under beancount.projects soon. (If I'm a little slow it's because I've been insisting on writing unit tests for everything when I move it under the main source tree.)

[Matthew S. Harris]

On Sat Nov 29 2014 at 10:06:02 AM Martin Blais <bl...@furius.ca> wrote:

So there are a few problems with your method above:

- Your Income:Options:MH leg will be posted in 2014, for the entire set of legs over multiple years. This is incorrect, from an accounting perspective. At the very least, you should have one such entry for each year, even in Ledger, so that income for that year is posted on that year's income statement. 

True. I hadn't been relying on the income statement, but maybe if I fix this then I can.

 

- The income you compute with this is also incorrect... as you note, you need to take into account the taxes for it for this to work. That's not really income what you're calculating there, and while this may be a useful quantity for now, this will come around to bite you at the EOY because AFAIK income from RSUs should be reflected on your W-2 (there's an associated paystub that details taxes) and you won't be able to get the right numbers matching the W-2 if you did not calculate it from the paystubs. If you're not entering detail from your paystubs, that might be okay.

Right. I've been entering all of my paystubs too, so the tax calculation is present in a separate transaction. In the past, I was unsuccessful at tying the paystubs to the vesting events.

Similarly, I gave up trying to put my 401(k) contributions in a single transaction spanning my paystub and fund purchase. At least one of the fund purchases was several weeks delayed from the paycheck. I ended up creating a Transfers account in Ledger. The paycheck pays into Transfers and the 401(k) fund buys from Transfers.

 

- The cost basis which you receive your RSUs at is not pre-determined in any way by the price at which they were set when they were granted to you. So the repeated regular $600.00 above is incorrect. Each vesting event takes place at the market price on that day and so it varies. Because of this, you cannot book it in advance. 

Right. I was simplifying there. I do put the correct prices on the vesting events.

 

- Nit: RSUs are not options, and they're also not treated as options from a tax perspective (they don't fall under 1256 treatment), you might eventually want to rename this income account.

Fair point.

 

The method I propose for dealing with it calculated the pre-tax income, includes tax details, transfers to a cash account the remainder, then reduces that by the integer number of shares acquired (saving cost basis, useful for wash sales later on), leaving cash to be paid later, which then gets booked against the payments for this cash portion, and a final check that the remainder is zero. I'll admit that it's vastly more verbose than the method you suggest above, but it has several advantages: the fees are taken into account, the taxes and income are calculated correctly, it tracks the amount of cash lying in limbo after vesting events, and you can organize the transactions "by account" in your input file, which is nice when you're reconciling history. I'll be checking in an example template for Googlers soon that will demonstrate the full detail of this, so you'll be able to judge for yourself if it's too much of a burden to do.  I'm not sure yet how much of it could be automated, but certainly if any of it can, then we can do it via some plugin. The thing is, once you're gone through one cycle of it, after that it's all just a bit of cut-n-paste from the previous transactions and adjusting the numbers.

I look forward to the examples. I seem to be adding my accounts into Beancount one at a time, starting with the simpler accounts, and I'm refactoring the account hierarchy as I go. So the stock accounts will probably be among the last to move. :-)

  

Based on your response, I'm not sure we understood each other.  What you suggest wouldn't work in Ledger either, it will only report payments made in 2014, what I mean is that you should be able to look at all tax payments and receipts made "for" taxation year 2014, even those paid in 2015 when you file your taxes and further adjustments to year 2014's accounts made in 2016 in case of a review or if you end up filing a 1040X and further forward. Note that I don't say payments made "in" taxation year 2014, but "for". Payments for tax year 2014 may occur for many years after 2014.  This is how the government actually tracks your taxes AFAIK.

I see what you're saying. I haven't done any recent transactions outside the year at which they're taxed, but probably I should prepare my hierarchy now anyway. In Ledger I had been putting TaxYear:: [2014] metadata on relevant transactions, but I hadn't been doing anything with it yet.

I have a related question: How can I ask Beancount how much I've contributed to an account in a particular tax year? For 401(k) and 529 accounts, I want to know how close I am to the annual contribution limit. I used to use a Ledger query like

  ledger -Y reg $ACCT_529 and @Contribution

but I think it stopped working at some point.

Matthew 

[Martin Blais]

I think you have already done some: when you file your taxes on year N+1, that's meant to apply on accounts for year N.  So you have already been doing at least two transactions per year on those accounts (federal and state), even if there were no revisions required (yet).

In Ledger I had been putting TaxYear:: [2014] metadata on relevant transactions, but I hadn't been doing anything with it yet.

I have a related question: How can I ask Beancount how much I've contributed to an account in a particular tax year? For 401(k) and 529 accounts, I want to know how close I am to the annual contribution limit. I used to use a Ledger query like

  ledger -Y reg $ACCT_529 and @Contribution

but I think it stopped working at some point.

I track those using mirror accounting, which I automate using a plugin. The examples for Googlers will contain usage of that plugin, and if you want to set it up right now, it's available here:

https://bitbucket.org/blais/beancount/src/tip/src/python/beancount/plugins/ira_contribs.py?at=returns

The docstring has some of the detail.

[Matthew Harris]

My experience of translating a fair amount (not quite half, I think) of my Ledger data to Beancount:

- I'm having to rename most of my accounts as I go, usually because a portion of a name starts with a number ("401k") or contains a space ("1234 Checking"). I'm also creating subaccounts for each commodity, as you recommend, even though it's not clear to me that it's better. I found a few mistakes in my files — two halves of a transaction that should be joined, some expense "accounts" that should be merged — so there is benefit to all the work.

- I still really miss the ability to override the date and payee on postings. I know you don't yet agree, but I'm hoping you can be convinced. :-) I feel that an important principle is that the "journal" report (which is the one I use most often) should match up exactly with the single-entry journal I'd write or receive for the same account. The journal for a bank account should have the bank's dates and descriptions, and the journal for an expenses account should match the dates I incurred the expenses. Balance assertions should make sense, even if I write a check today and it isn't deposited for two months. If I close one bank account account and use the cashier's check to open a new bank account, I don't want to have to fudge the open and close dates. If I'm buying or selling shares, the prices should have the appropriate dates. (It's possible that preserving the two dates provided by the two banks involved in a transaction would help with automated transaction importing ala LedgerHub, but I don't know.)

- I tried to use the ira_contribs plugin to track against annual contribution limits, but it isn't working well for me. I can see how it would work for a 401(k), where I'm going through a fake cash account already (my paycheck deposits into the cash account and then money from the cash account is immediately used to by shares). But right now when I contribute to a 529 plan, I get a transaction like this:

2014-11-07 * "Contribution"

  * Assets:College:Child1:VITPX 45.41522 VITPX @ 46.24 USD

  * Assets:College:Child1:VEMRX 2.74286 VEMRX @ 87.50 USD

  * Assets:College:Child1:VDIPX 33.79416 VDIPX @ 19.53 USD

  * Assets:Bank:Cash  -3000.00 USD

Because ira_contribs takes the numeric value of the contribution and tacks it onto a fake currency, to use ira_contribs would require me to split this transaction into two — one to deposit cash into an Assets:College:Child1:Cash and a second to actually purchase the shares. And if I'm manually duplicating the postings, I might as well do the mirror accounting manually and skip the plugin.

A second problem with the plugin is that it assumes that the tax year of the contribution is the same as the calendar year, which is true for 401(k) and 529 accounts but ironically not for IRAs. If the plugin can work in this case, it's going to need to look at a taxyear metadata field.

Maybe I should just use the plugin for the 401(k) and do manual mirror accounting elsewhere. Certainly the 401(k) is the only place I have a large number of automatic transactions; IRA and 529 contributions tend to be few and manual.

- As an aside, I see that the grammar supports @@ and {{}} syntaxes, but they don't appear to be documented in the language manual.

I'm sure I'll have more comments as I continue. I enjoy watching the log for your newest additions.

[Martin Blais]

Thanks Matt for the fantastic quality feedback. My notes below.

On Mon, Dec 15, 2014 at 3:10 AM, Matthew Harris <mharr...@gmail.com> wrote:

My experience of translating a fair amount (not quite half, I think) of my Ledger data to Beancount:

- I'm having to rename most of my accounts as I go, usually because a portion of a name starts with a number ("401k") or contains a space ("1234 Checking"). I'm also creating subaccounts for each commodity, as you recommend, even though it's not clear to me that it's better. I found a few mistakes in my files — two halves of a transaction that should be joined, some expense "accounts" that should be merged — so there is benefit to all the work.

IMO the conversion of spaces to dashes is a minor compromise, and the resulting account names should also work in Ledger. You should be able to script it if you want, actually. The benefits of it is that it simplifies the syntax and the parser (e.g., no need for something like Ledger's "2 space after account names" rule, for example, and I _really_ like using a parser generator rather than a hand-made one).

- I still really miss the ability to override the date and payee on postings. I know you don't yet agree, but I'm hoping you can be convinced. :-)

Oh, I'm convinced alright :-)  I think on our last discussion we've established it would be good enough to automatically split the transaction in multiple ones with a limbo account and that this would be the preferred solution. All we need to do is add parsing for an optional date in front of accounts on postings and then either post-process the transactions to split them up, or probably easier would be to just create the split transactions in the parser itself and not even store the per-Posting dates on the objects. We need to add an option to specify the limbo account and a reasonable default value. Split transactions should have common unique links too. This is all easy to do.

I'll add this to the TODO file. I estimate this is probably about a day or two of work.

Another idea would be not to split any transaction, but then when you need to apply "cuts" at a particular point in time (such as when preparing to build a balance sheet from date X to date Y) I could just split those transactions whose dates straddle the cut date using a function.

It's not that I'm not convinced, just that I'm still working out how it should be handled best. Using two dates in different contexts breaks assumptions and has many ramifications.

 

I feel that an important principle is that the "journal" report (which is the one I use most often) should match up exactly with the single-entry journal I'd write or receive for the same account. The journal for a bank account should have the bank's dates and descriptions, and the journal for an expenses account should match the dates I incurred the expenses.

I like it too, but it comes at the cost of having a lot of split transaction objects to limbo accounts. The journals will look correct but that's not super elegant from a data representation POV.

 

Balance assertions should make sense, even if I write a check today and it isn't deposited for two months.

In many cases this is not a use case for the second date IMO, many of these are better entered as an explicit entry against Liabilities:AccountsPayable. I suppose these are interchangeable in some sense. What do you think? They are two ways of doing the same thing. One uses an account that shows up on the balance sheet (payable, declared explicitly as two transactions) and the other an account that says "this is in transfer" (probably Equity:Limbo would be the default).

 

If I close one bank account account and use the cashier's check to open a new bank account, I don't want to have to fudge the open and close dates. If I'm buying or selling shares, the prices should have the appropriate dates. (It's possible that preserving the two dates provided by the two banks involved in a transaction would help with automated transaction importing ala LedgerHub, but I don't know.)

For buying and selling shares, the prices aren't very relevant, it's the cost basis and the cash portion that determines the correctness of the amounts. You may actually transact at a different price than the market. i never enter the @ price on my stock transactions, I don't find that useful. Or are you talking about the difference between settlement and trade date? 

- I tried to use the ira_contribs plugin to track against annual contribution limits, but it isn't working well for me. I can see how it would work for a 401(k), where I'm going through a fake cash account already (my paycheck deposits into the cash account and then money from the cash account is immediately used to by shares). But right now when I contribute to a 529 plan, I get a transaction like this:

2014-11-07 * "Contribution"

  * Assets:College:Child1:VITPX 45.41522 VITPX @ 46.24 USD

  * Assets:College:Child1:VEMRX 2.74286 VEMRX @ 87.50 USD

  * Assets:College:Child1:VDIPX 33.79416 VDIPX @ 19.53 USD

  * Assets:Bank:Cash  -3000.00 USD

Because ira_contribs takes the numeric value of the contribution and tacks it onto a fake currency, to use ira_contribs would require me to split this transaction into two — one to deposit cash into an Assets:College:Child1:Cash and a second to actually purchase the shares. And if I'm manually duplicating the postings, I might as well do the mirror accounting manually and skip the plugin.

A second problem with the plugin is that it assumes that the tax year of the contribution is the same as the calendar year, which is true for 401(k) and 529 accounts but ironically not for IRAs. If the plugin can work in this case, it's going to need to look at a taxyear metadata field.

Maybe I should just use the plugin for the 401(k) and do manual mirror accounting elsewhere. Certainly the 401(k) is the only place I have a large number of automatic transactions; IRA and 529 contributions tend to be few and manual.

These are great observations and correct on all accounts. Thanks a lot for trying my little plugin, BTW. What I did is I tracked the mirror entries manually for many many months and after I've observed a pattern that worked for me attempted to generalize it. I certainly did not expect it would cover all possible cases: I think your use case is a little more complex than what I needed at the time and highlights legitimate needs for an extension or some different plugin to be written.

You should probably write your own plugin to cover for the cases you describe. You're welcome to copy mine and extend it if that helps. You can either try to generalize the one I provided or alternatively provide a separate one for 529 accounts or IRA contributions "not for the same year." I'm confident you can automate your mirror postings, but it's not necessarily the case that all plugins can be made generalizable and reusable by others. Maybe you have to hardcode some of the assumptions.

(Note that the path of your plugin does not need to be in the Beancount codebase, anywhere in your own PYTHONPATH will work. You don't have to share it back if you don't want to.)

- As an aside, I see that the grammar supports @@ and {{}} syntaxes, but they don't appear to be documented in the language manual.

Oh I hadn't noticed... I will document those, thank you for reporting this oversight.

 

I'm sure I'll have more comments as I continue. I enjoy watching the log for your newest additions.

Thanks Matt. I've been a little stuck this weekend. I realize that the SQL syntax will require some more work (though it already covers the Ledger functionality pretty much), I thought I was close to done, but no, this will need more time to make it perfect. I think I'll polish the corners and merge its current state into default and leave more for later. It's usable as is, works for balances, journals and select statements. Other work I want to do is soon is integrate that portfolio/dashboard as an example for you to use, and continue work on the booking proposal (to get dated lots and short/long-term reporting and FIFO/LIFO stuff and better booking, which I see as more fundamental and important work). The SQL syntax can be improved more later on.

[Matthew Harris]

On Tuesday, December 16, 2014 5:56:10 AM UTC-8, Martin Blais wrote:

IMO the conversion of spaces to dashes is a minor compromise, and the resulting account names should also work in Ledger. You should be able to script it if you want, actually. The benefits of it is that it simplifies the syntax and the parser (e.g., no need for something like Ledger's "2 space after account names" rule, for example, and I _really_ like using a parser generator rather than a hand-made one).

Yes, I fully agree. I'm not complaining, just describing my experience. If I had had more foresight, I would've done as much of the conversion as possible on the existing Ledger files first.

 

Balance assertions should make sense, even if I write a check today and it isn't deposited for two months.

In many cases this is not a use case for the second date IMO, many of these are better entered as an explicit entry against Liabilities:AccountsPayable. I suppose these are interchangeable in some sense. What do you think? They are two ways of doing the same thing. One uses an account that shows up on the balance sheet (payable, declared explicitly as two transactions) and the other an account that says "this is in transfer" (probably Equity:Limbo would be the default).

That's a fair point, but it doesn't feel right with the way I use the tool. If I get a check from one account and later deposit it into another account, I like the fact that Ledger ties the two postings together in a single transaction. Otherwise I've got two separate transactions (it's more verbose), which may not be near each other in the source file, I have to go searching to find the two halves, and I have to do work to make sure that the two transactions cancel out. (AccountsPayable may not end up zero if I've always got something in flight.) Beancount has links, which do satisfy a need, but then I need a naming scheme for the links, and it's too easy to get it wrong.

Links and AccountsPayable are useful in your reimbursement examples: I pay for travel and lodging for a business trip, and later I get reimbursement in a paycheck. Each of those should be a separate transaction. And if I have few needs for AccountsPayable, it will end up zero most of the time, which makes it easier to verify.

 

If I'm buying or selling shares, the prices should have the appropriate dates.

For buying and selling shares, the prices aren't very relevant, it's the cost basis and the cash portion that determines the correctness of the amounts. You may actually transact at a different price than the market. i never enter the @ price on my stock transactions, I don't find that useful. Or are you talking about the difference between settlement and trade date? 

I don't think I was exactly talking about settlement vs trade dates, but I was talking about having an accurate record of the market price. You're right though that the trade won't necessarily happen at the market closing price, so never mind.

Matthew

[Martin Blais]

On Thu, Dec 18, 2014 at 9:30 PM, Matthew Harris <mharr...@gmail.com> wrote:

On Tuesday, December 16, 2014 5:56:10 AM UTC-8, Martin Blais wrote:

 

Balance assertions should make sense, even if I write a check today and it isn't deposited for two months.

In many cases this is not a use case for the second date IMO, many of these are better entered as an explicit entry against Liabilities:AccountsPayable. I suppose these are interchangeable in some sense. What do you think? They are two ways of doing the same thing. One uses an account that shows up on the balance sheet (payable, declared explicitly as two transactions) and the other an account that says "this is in transfer" (probably Equity:Limbo would be the default).

That's a fair point, but it doesn't feel right with the way I use the tool. If I get a check from one account and later deposit it into another account, I like the fact that Ledger ties the two postings together in a single transaction. Otherwise I've got two separate transactions (it's more verbose), which may not be near each other in the source file, I have to go searching to find the two halves, and I have to do work to make sure that the two transactions cancel out. (AccountsPayable may not end up zero if I've always got something in flight.) Beancount has links, which do satisfy a need, but then I need a naming scheme for the links, and it's too easy to get it wrong.

Links and AccountsPayable are useful in your reimbursement examples: I pay for travel and lodging for a business trip, and later I get reimbursement in a paycheck. Each of those should be a separate transaction. And if I have few needs for AccountsPayable, it will end up zero most of the time, which makes it easier to verify.

I agree with everything you said. I think there's a space to introduce those split transactions, both styles have a place in an input file, and I might myself end up using them a lot once there's some implementation for them.

[captain...@gmail.com]

I do think the expense reimbursements should be done through an Accounts Receivable (not payable) account.  I would take it further and post the actual air/hotel/restaurant charges to an "Unsubmitted Reimbursements" account, then make a posting from UR to AR when you fill out the reimbursement form and "invoice" your employer.

[redst...@gmail.com]

On Thursday, November 27, 2014 at 5:03:35 PM UTC-8, Martin Blais wrote:

Here's an idea I've had some time ago for how to resolve this while keeping the constraint: A transaction with postings that are on different dates could be automatically split into multiple transactions with a "limbo" transfer account to absorb imbalances over time. For example, the following transaction

  2014-08-23 * "Transfer from other account"

    Assets:Checking            210.00 USD

    2014-08-21 Assets:Savings

would be automatically converted by a plugin invoked automatically by the loader, into this:

  2014-08-21 * "Transfer from other account" ^split-478574

    Assets:Savings   -210.00 USD

    Equity:Limbo    210.00 USD

  2014-08-23 * "Transfer from other account" ^split-478574

    Equity:Limbo   -210.00 USD

    Assets:Checking  210.00 USD

Adding my 2 cents on this. I've used double entry bookkeeping for over a decade now, and this a basic problem, along with another problem, for which I've long used a solution.

First, the other problem: I maintain each "real-world" account in its own file. The obvious advantage is that the transactions in that one file will mirror your statements or online download for that account 1-on-1. The problem is with transfers, because each real-world account has a different view of it. Plus, the money is truly missing from all your accounts for a day or two in between. How I've long solved this is what you describe as the Limbo account:

2005-01-01 Transfer

  Assets:Bank_of_A  -20 USD

  ZeroSumAccount:In_Flight

2005-01-03 Transfer

  Assets:Bank_of_B  20 USD

  ZeroSumAccount:In_Flight

This has a few advantages:

a) You can convert each bank's transactions directly into ledger statements. No need to remove the transaction from one of the banks. When you look at your journal files for each account, they match your account statements exactly.

b) Import/conversion (from say, a bank .csv or .ofx) is easier, because your import scripts don't have to figure out where a transfer goes, and can simply assign it to   ZeroSumAccount:In_Flight

c) If there is a problem, your   ZeroSumAccount:In_Flight will have a non-zero value (over time). You can track this down easily.

d) on 2005-01-02, your assets are accurately represented: Bank_A is short by $20, Bank_B still doesn't have it, and the In_Flight account captures that the money is still yours, but is "in flight."

Not sure how this might fit with your Limbo account idea, because that compresses everything into a single transaction. But thought I'd throw this out there.

[Martin Blais]

On Wed, Jan 21, 2015 at 6:13 PM, <redst...@gmail.com> wrote:

On Thursday, November 27, 2014 at 5:03:35 PM UTC-8, Martin Blais wrote:

Here's an idea I've had some time ago for how to resolve this while keeping the constraint: A transaction with postings that are on different dates could be automatically split into multiple transactions with a "limbo" transfer account to absorb imbalances over time. For example, the following transaction

  2014-08-23 * "Transfer from other account"

    Assets:Checking            210.00 USD

    2014-08-21 Assets:Savings

would be automatically converted by a plugin invoked automatically by the loader, into this:

  2014-08-21 * "Transfer from other account" ^split-478574

    Assets:Savings   -210.00 USD

    Equity:Limbo    210.00 USD

  2014-08-23 * "Transfer from other account" ^split-478574

    Equity:Limbo   -210.00 USD

    Assets:Checking  210.00 USD

Adding my 2 cents on this. I've used double entry bookkeeping for over a decade now, and this a basic problem, along with another problem, for which I've long used a solution.

First, the other problem: I maintain each "real-world" account in its own file.

Beancount does not explicitly support include files at the moment but because of the order-independence property of its declaration, a trivial solution is to cat all the files together and run the tools on that; it should work (*). Because of this property, adding the feature would be rather easy and if someone really cares for it -- let me know here if you're craving for it or if that's preventing you from using Beancount -- I can prioritize that and implement it soon (I see this as low-hanging fruit and I normally prioritize the little time I have on the harder and more boring stuff like writing documentation... the fun little feature ideas just appear when I need self-gratification :-) ).

(*) Some options have an effect as from the moment of their parsing onwards, but ideally all your options come first.

 

The obvious advantage is that the transactions in that one file will mirror your statements or online download for that account 1-on-1.

Also note that I organize my file similarly to you but not in separate files: each institution (which involves some subset of accounts) is written down in its own dedicated section. I use org-mode to handle managing this huge file and just expand the sections I'm editing. It works pretty well but I also share your appeal of multiple files and I'd like to have that as an explicit feature.

 

The problem is with transfers, because each real-world account has a different view of it. Plus, the money is truly missing from all your accounts for a day or two in between. How I've long solved this is what you describe as the Limbo account:

2005-01-01 Transfer

  Assets:Bank_of_A  -20 USD

  ZeroSumAccount:In_Flight

2005-01-03 Transfer

  Assets:Bank_of_B  20 USD

  ZeroSumAccount:In_Flight

This works. I think I'll build a new doc summarizing our discussions so far and add this in as "the explicit solution." This works right now in Beancount.

 

This has a few advantages:

a) You can convert each bank's transactions directly into ledger statements. No need to remove the transaction from one of the banks. When you look at your journal files for each account, they match your account statements exactly.

On the flipside, where transfers are settled immediately, it adds some superfluous transactions. (Not a big deal, just saying.)

 

b) Import/conversion (from say, a bank .csv or .ofx) is easier, because your import scripts don't have to figure out where a transfer goes, and can simply assign it to   ZeroSumAccount:In_Flight

This is not too big a problem if you use LedgerHub: each particular importer object is configured with a set of accounts and generally an "external" account can be configured, but it's true that if you have transfers to more than one account you then have to manually correct them. 

With your method indeed no change is necessary. I think if someone wants to make an attempt at _fully_ automating their imports, i.e. minimal oversight or no massaging of imported transactions, I would recommend this method.

 

c) If there is a problem, your   ZeroSumAccount:In_Flight will have a non-zero value (over time). You can track this down easily.

Yes, and I think we can build feature sto help such tracking even further.  We could come up with some sort of creative solution for automatically matching up transactions with the same amounts and having such a plugin insert a link between the matched transactions to take them out and it would result in a list of "pending" unmatched transactions. This would be similar in spirit to the example plugin I created for tagging pending invoices:

https://bitbucket.org/blais/beancount/src/edd13e1cfbce55e1752a23501b89a95f800fc581/src/python/beancount/plugins/tag_pending.py?at=default

 

d) on 2005-01-02, your assets are accurately represented: Bank_A is short by $20, Bank_B still doesn't have it, and the In_Flight account captures that the money is still yours, but is "in flight."

That's nice indeed, though I think in the case of a corporation with lots of payments it probably gets a little bit messy.

Not sure how this might fit with your Limbo account idea, because that compresses everything into a single transaction. But thought I'd throw this out there.

Yes it does. The implementation of the limbo method I favor results in the same thing as this explicit method, only the input is carried out in a single transaction. A user could easily use both types of data entry in a single Beancount input file (the explicit method requires no special feature addition).

Idea: Maybe there should be some way to specify which limbo account to use between pairs of accounts, e.g.

2000-01-10 limbo ZeroSumAccount:In_Flight Assets:Bank_of_A Assets:Bank_of_B 

or perhaps as metadata on its open directive:

2000-01-10 open ZeroSumAccount:In_Flight 

  limbo-src: Assets:Bank_of_A 

  limbo-dst: Assets:Bank_of_B 

This would tell Beancount which particular limbo account to use when a transaction with dates creating limbo amounts occurs between these accounts.

Thanks redstreet0 for your interesting observation and chipping in to the discussion,

 

You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/0068c286-baa6-487e-89a5-62dca9f15f3c%40googlegroups.com.

[redst...@gmail.com]

Beancount does not explicitly support include files at the moment but because of the order-independence property of its declaration, a trivial solution is to cat all the files together and run the tools on that; it should work (*). Because of this property, adding the feature would be rather easy and if someone really cares for it -- let me know here if you're craving for it or if that's preventing you from using Beancount -- I can prioritize that and implement it soon (I see this as low-hanging fruit and I normally prioritize the little time I have on the harder and more boring stuff like writing documentation... the fun little feature ideas just appear when I need self-gratification :-) ).

This would be really nice to have, since my accounts heavily use the filesystem hierarchy. My scripts that download statements depend on the existence of this filesystem hierarchy to provide guarantees (for eg, of not opening/updating the wrong file). It's much easier to see diffs that are limited to a file in my versioning system when I'm hunting down problems. I could go on. But no, this is not preventing me from using Beancount because cat'ing would provide a fix, even if a temporary kludgy one, because of BC's order-independence. I do have other things that are preventing me from using beancount though, but I'll put those in a separate thread.

  

This works. I think I'll build a new doc summarizing our discussions so far and add this in as "the explicit solution." This works right now in Beancount.

That reminds me. Advantage e) This method doesn't depend on the double-entry system being used.

  

This has a few advantages:

a) You can convert each bank's transactions directly into ledger statements. No need to remove the transaction from one of the banks. When you look at your journal files for each account, they match your account statements exactly.

On the flipside, where transfers are settled immediately, it adds some superfluous transactions. (Not a big deal, just saying.)

True. This shouldn't matter if you always have a 1-day granularity, but from a purity perspective, this is not 100% pure.

 

b) Import/conversion (from say, a bank .csv or .ofx) is easier, because your import scripts don't have to figure out where a transfer goes, and can simply assign it to   ZeroSumAccount:In_Flight

This is not too big a problem if you use LedgerHub: each particular importer object is configured with a set of accounts and generally an "external" account can be configured, but it's true that if you have transfers to more than one account you then have to manually correct them. 

With your method indeed no change is necessary. I think if someone wants to make an attempt at _fully_ automating their imports, i.e. minimal oversight or no massaging of imported transactions, I would recommend this method.

For my case:

1) My imports are about 95% automated

2) My scripts to do this work great and are customized to me, and so, ledgerhub is not attractive to me even though I considered it

3) Yes, I have plenty of external accounts,

Which all make the method I use easier, for me. I can't tell how many others it might be useful to, though.

 

c) If there is a problem, your   ZeroSumAccount:In_Flight will have a non-zero value (over time). You can track this down easily.

Yes, and I think we can build feature sto help such tracking even further.  We could come up with some sort of creative solution for automatically matching up transactions with the same amounts and having such a plugin insert a link between the matched transactions to take them out and it would result in a list of "pending" unmatched transactions. This would be similar in spirit to the example plugin I created for tagging pending invoices:

https://bitbucket.org/blais/beancount/src/edd13e1cfbce55e1752a23501b89a95f800fc581/src/python/beancount/plugins/tag_pending.py?at=default

Nice! This has been one of the attractions to beancount for me. I have exactly such a script, but it uses ledger's csv output as its input. With Beancount, it can be integrated via a plugin. BTW, I also use the same script I have (with a different date distance) to track things like reimbursements, mail-in rebates, etc. that are pending. There's other ways of doing this, but I find this way works best for me. Anyway, I'm hoping a bunch of my external helper scripts could be integrated using beancount plugins.

 

or perhaps as metadata on its open directive:

2000-01-10 open ZeroSumAccount:In_Flight 

  limbo-src: Assets:Bank_of_A 

  limbo-dst: Assets:Bank_of_B 

I like this format better, because it seems more human readable, at least to me.

[kljo...@gmail.com]

I wrote a plugin ( https://gist.github.com/kljohann/aebac3f0146680fd9aa5 ) where you have to explicitly
specify the booking to the Limbo account, but can use metadata-sugar for the split postings.
It is triggered by a tag and turns the following (where "Assets:Table" is the limbo account)

plugin "split_transactions"

2010-10-15 open Expenses:Office
2010-08-17 open Assets:Bank
2015-05-09 open Assets:Table

2015-05-09 * "Shop" ""
  Assets:Bank
  Assets:Table                                    43.00 EUR
    obj: "Office Table"

2015-06-01 * "Deprecation" "" #split
  obj: "Office Table"
  Assets:Table
  Expenses:Office                                 10.75 EUR
    date: 2015/06/01
  Expenses:Office                                 10.75 EUR
    date: 2015/07/01
  Expenses:Office                                 10.75 EUR
    date: 2015/08/01
  Expenses:Office                                 10.75 EUR
    date: 2015/09/01

Into

2010-08-17 open Assets:Bank
2010-10-15 open Expenses:Office
2015-05-09 open Assets:Table

2015-05-09 * "Shop" | ""
  Assets:Bank   -43.00 EUR
  Assets:Table   43.00 EUR
    obj: "Office Table"

2015-06-01 * "Deprecation" | ""
  obj: "Office Table"
  Expenses:Office   10.75 EUR
  Assets:Table     -10.75 EUR

2015-07-01 * "Deprecation" | ""
  obj: "Office Table"
  Expenses:Office   10.75 EUR
  Assets:Table     -10.75 EUR

2015-08-01 * "Deprecation" | ""
  obj: "Office Table"
  Expenses:Office   10.75 EUR
  Assets:Table     -10.75 EUR

2015-09-01 * "Deprecation" | ""
  obj: "Office Table"
  Expenses:Office   10.75 EUR
  Assets:Table     -10.75 EUR

Note that in the original transaction all postings except for one have to have a 'date' field.
The posting without a date is used to obtain the limbo account.

[Martin Blais]

This is great! :-)

Very much in line with some of the ideas discussed.

Comments:

- You could potentially do without the need to identify the transactions with a tag (e.g., #split) and instead detect that if at least one of the postings has a "date" metadata field that the transactions is intended to get split.

- Instead of "obj" metadata to link the transactions I tend to use links, e.g. ^office-table on the top line. Eventually there will be better tools to do things on sets of linked transactions, e.g. report on all balances of linked transactions together, for example.

--

You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/06b4c957-65d0-453a-a907-fcea7648f67a%40googlegroups.com.

[Martin Blais]

I've created a new doc for user contributions and added this link to it:

http://furius.ca/beancount/doc/contrib

On Tue, Jun 2, 2015 at 3:11 AM, <kljo...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/06b4c957-65d0-453a-a907-fcea7648f67a%40googlegroups.com.