Attaching custom data to postings...

[ELI]

My understanding is that plugins cannot help Beancount parse entries, so what I was hoping to do was use comments or tags to "activate" a custom plugin and pass it relevant information, for particular postings. What I've found is that tags can't be associated with postings and comments aren't available to plugins. Is there an option I'm overlooking?

What am I trying to do?

Well, for any stock trade, I want to record *all* the information provided to me by my broker: share count, share price, and transaction amount. Beancount only allows me to record share count and any one of price or amount. Additionally, I want my plugin to replace the posting with a posting where one of the three values is calculated using the other two, based on a parameter attached to the posting (or preferably the account).

Thanks,
- Harpreet "Eli" Sangha

[Martin Blais]

On Thu, Nov 20, 2014 at 12:50 PM, ELI <eli...@gmail.com> wrote:

My understanding is that plugins cannot help Beancount parse entries,

Not yet, but I have sketched a plan to make all directives (except Transactions) definable via plugins. Down the very very long road, all directives could be defined by plugins eventually. The plugin would provide a definition of expected data types, to be validated by the parser in Python.

so what I was hoping to do was use comments or tags to "activate" a custom plugin and pass it relevant information, for particular postings. What I've found is that tags can't be associated with postings and comments aren't available to plugins. Is there an option I'm overlooking?

What you want is support for metadata. Here's what I have in mind: I will add support for metadata objects (LISP style, as an attached dict of arbitrary objects) on all transactions and postings. The metadata would be accessible via the Python data structures for you to build custom reports or whatever, and also usable from the SQL interface. There would otherwise be no other semantics associated with it--it would be there for you to use, as extensions (it would support strings, numbers, amounts (number + currency), currency, date, data types.

Now, where is it at? Well, I have already built a prototype that defines a nice syntax and parses the metadata (see branch "metadata"). I haven't yet done the complete work to make it attached to the objects. What I can _quickly_ do is give you parsing capability by merging my branch and this would allow you to immediately start entering all the data in your input file. Attaching the metadata requires some not-so-difficult changes, yet changes that would impact a fair bit of the codebase, so I would rather do that later in a second step.

Would this work for now?

 

What am I trying to do?

Well, for any stock trade, I want to record *all* the information provided to me by my broker: share count, share price, and transaction amount. Beancount only allows me to record share count and any one of price or amount.

The share count, share price and transaction amounts should already be on your transactions.  You can place _both_ cost and price on a single posting (but you don't have to):

  2014-05-02 * "Sell stock"

    Assets:US:Investments:GOOGL       -10 GOOGL {606.12 USD} @ 545.37 USD

    Assets:US:Investments:Cash        5453.70 USD

    Income:CapitalGains          

 Note that you may also already use the "link" feature to attach other infos, e.g. a document filename:

  2014-05-02 * "Sell stock" ^2014-05-02.schwab.pdf

    Assets:US:Investments:GOOGL       -10 GOOGL {606.12 USD} @ 545.37 USD

    Assets:US:Investments:Cash        5453.70 USD

    Income:CapitalGains          

Additionally, I want my plugin to replace the posting with a posting where one of the three values is calculated using the other two, based on a parameter attached to the posting (or preferably the account).

Can you provide a use case? What I think you may be trying to do sounds like it might be better done as a reporting function. The Inventory object has functions to extract number of units, cost, average cost, and market value, that you can use. Let me know what your desired output is.

This also makes me think of an idea I've had for a plugin: a plugin that further verifies that the sum of postings EXCLUDING postings on Income accounts balance to the sum of the postings' "weight" but where the values of postings held-at-cost are calculated using the price instead of the cost value. This would provide yet another check that the input data is calculated correctly, but I'm not sure it's super general yet. It would be an optional feature you could enable through a plugin. Right now, prices on postings held-at-cost are ignored for the purpose of balancing, but they're inserted in the price database (they translate into a Price directive).  Is this what you have in mind?

 

Thanks,
- Harpreet "Eli" Sangha

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAKU2X8Y0X%2Bvr%2BVP9jDYOKWjBjZW-kOp%3DrUjBgaDO%3D24NQo2axw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[ELI]

- Harpreet "Eli" Sangha

On Thu, Nov 20, 2014 at 12:58 PM, Martin Blais <bl...@furius.ca> wrote:

On Thu, Nov 20, 2014 at 12:50 PM, ELI <eli...@gmail.com> wrote:

My understanding is that plugins cannot help Beancount parse entries,

Not yet, but I have sketched a plan to make all directives (except Transactions) definable via plugins. Down the very very long road, all directives could be defined by plugins eventually. The plugin would provide a definition of expected data types, to be validated by the parser in Python.

so what I was hoping to do was use comments or tags to "activate" a custom plugin and pass it relevant information, for particular postings. What I've found is that tags can't be associated with postings and comments aren't available to plugins. Is there an option I'm overlooking?

What you want is support for metadata. Here's what I have in mind: I will add support for metadata objects (LISP style, as an attached dict of arbitrary objects) on all transactions and postings. The metadata would be accessible via the Python data structures for you to build custom reports or whatever, and also usable from the SQL interface. There would otherwise be no other semantics associated with it--it would be there for you to use, as extensions (it would support strings, numbers, amounts (number + currency), currency, date, data types.

Now, where is it at? Well, I have already built a prototype that defines a nice syntax and parses the metadata (see branch "metadata"). I haven't yet done the complete work to make it attached to the objects. What I can _quickly_ do is give you parsing capability by merging my branch and this would allow you to immediately start entering all the data in your input file. Attaching the metadata requires some not-so-difficult changes, yet changes that would impact a fair bit of the codebase, so I would rather do that later in a second step.

Would this work for now?

Sure this works for me.  I also wouldn't mind getting my hands dirty and living on a branch that incorporates with work-in-progress for the purpose of evaluating and providing feedback/contributions.

 

What am I trying to do?

Well, for any stock trade, I want to record *all* the information provided to me by my broker: share count, share price, and transaction amount. Beancount only allows me to record share count and any one of price or amount.

The share count, share price and transaction amounts should already be on your transactions.  You can place _both_ cost and price on a single posting (but you don't have to):

  2014-05-02 * "Sell stock"

    Assets:US:Investments:GOOGL       -10 GOOGL {606.12 USD} @ 545.37 USD

    Assets:US:Investments:Cash        5453.70 USD

    Income:CapitalGains          

Doesn't the above syntax imply that stock bought at 606.12 USD/GOOGL is being sold at 545.37 USD/GOOGL.  What I mean to record is, for example, how many shares my broker reported that I sold, the price they were reportedly sold at, and the reported amount of the total transaction:

2014-05-02 * "Sell stock"

  Assets:US:Investments:GOOGL       -10 GOOGL @ 545.37 USD @@ 5,453.70 USD

  Assets:US:Investments:Cash        5453.70 USD

 Note that you may also already use the "link" feature to attach other infos, e.g. a document filename:

  2014-05-02 * "Sell stock" ^2014-05-02.schwab.pdf

    Assets:US:Investments:GOOGL       -10 GOOGL {606.12 USD} @ 545.37 USD

    Assets:US:Investments:Cash        5453.70 USD

    Income:CapitalGains          

Additionally, I want my plugin to replace the posting with a posting where one of the three values is calculated using the other two, based on a parameter attached to the posting (or preferably the account).

Can you provide a use case? What I think you may be trying to do sounds like it might be better done as a reporting function. The Inventory object has functions to extract number of units, cost, average cost, and market value, that you can use. Let me know what your desired output is.

The use case is for brokerages that allow the purchase of fractional shares, and purchases are made by specifying a dollar amount as opposed to a number of shares.

   2014-09-23 * "Dividend"

     Assets:Vanguard:Taxable:Mutual-Fund    +4.622 VTSAX @ 49.60 USD @@ 229.27 USD

     Assets:US:Investments:Cash             -229.27 USD

The "truth" of the transaction is that Vanguard purchased 229.27 USD for me in fund that cost 49.60 USD, and acquired 4.622379 shares, but reported it as 4.622 shares.  Thus, the goal would be to enter into the bean file these above values that came from my statement, but to tell my plugin to replace this posting with one where the number of shares is calculated by the transaction amount divided by the price per share (229.27/49.60).

This also makes me think of an idea I've had for a plugin: a plugin that further verifies that the sum of postings EXCLUDING postings on Income accounts balance to the sum of the postings' "weight" but where the values of postings held-at-cost are calculated using the price instead of the cost value. This would provide yet another check that the input data is calculated correctly, but I'm not sure it's super general yet. It would be an optional feature you could enable through a plugin. Right now, prices on postings held-at-cost are ignored for the purpose of balancing, but they're inserted in the price database (they translate into a Price directive).  Is this what you have in mind?

I'm not entirely sure I understand this, maybe you can tell me if it relates based on my description above. 

[Martin Michlmayr]

* ELI <eli...@gmail.com> [2014-11-20 13:54]:

> 2014-05-02 * "Sell stock"
> Assets:US:Investments:GOOGL -10 GOOGL @ 545.37 USD @@ 5,453.70 USD
> Assets:US:Investments:Cash 5453.70 USD

FWIW, I remember asking a few years ago if this was possible in ledger
(it wasn't). For some reason, I thought it was a good idea to record
both the share price quoted for one share and the total price (the two
aren't always exactly the same due to rounding).

However, I'm no longer sure there's much value in that. For all
intends and purposes, the total price you paid is what matters and the
price for one share is realy total/quantity and not whatever share
price was quoted.

What I do is to add the share price as a comment just to make it more
readable:
Assets:Pension 35.3414 "World ex UK" @@ 288.28 GBP ; @ 8.1570 GBP

> 2014-09-23 * "Dividend"
> Assets:Vanguard:Taxable:Mutual-Fund +4.622 VTSAX @ 49.60 USD @@ 229.27
> USD
> Assets:US:Investments:Cash -229.27 USD
>
> The "truth" of the transaction is that Vanguard purchased 229.27 USD for me
> in fund that cost 49.60 USD, and acquired 4.622379 shares, but reported it
> as 4.622 shares. Thus, the goal would be to enter into the bean file these
> above values that came from my statement, but to tell my plugin to replace
> this posting with one where the number of shares is calculated by the
> transaction amount divided by the price per share (229.27/49.60).

Are you sure you received 4.622379 shares and not 4.622 shares?

In other words, if you do this transaction three times, does Vanguard
report 13.866 or 13.867 VTSAX? (4.622 * 3 = 13.866; 4.622379 * 3 =
13.867137).

If it's 13.867, then I'd record your purchase as 4.622379 since that's
what you got and Vanguard just didn't show all digits.

If it's 13.866, you may think you got 4.622379 but in reality you got
4.622 due to rounding. Vanguard may say the price is 49.60 USD but
due to rounding what you paid is really 49.6040675...

I don't know anything about VTSAX or US-based Vanguard funds, but at
least in the UK the fraction you can own is usually limited to a
certain digit... so assuming you can only own a fraction with 3 digits
(e.g. 4.622 and next is 4.623), your $229.27 will only get you 4.622
because 4.623 would have cost $229.30.

--
Martin Michlmayr
http://www.cyrius.com/

[Martin Michlmayr]

* Martin Blais <bl...@furius.ca> [2014-11-20 15:58]:

> Can you provide a use case?

I still haven't tried beancount, but since you're looking for use
cases, I have a use case relaed to custom data that ledger doesn't
support. I hope beancount will.

http://bugs.ledger-cli.org/show_bug.cgi?id=1042

The basic idea is that I want to be able to use a meta tag as the
account, so I can run queries on it.

[ELI]

- Harpreet "Eli" Sangha

With Betterment, I've observed it to be true that they maintain a higher precision internally based on the type of calculation you've described above.  At Vanguard, I only suspect it to be the case because it would be simple enough for them to leave 0.002 USD in my account as cash to produce a sane transaction.  I also have dividend reinvestments transactions in my Vanguard account that record the purchase of 0.000 shares.  Also, if two brokerages do things differently, then there is a practical case for the configuration option to the plugin I envision.

If it's 13.867, then I'd record your purchase as 4.622379 since that's
what you got and Vanguard just didn't show all digits.

If it's 13.866, you may think you got 4.622379 but in reality you got
4.622 due to rounding.  Vanguard may say the price is 49.60 USD but
due to rounding what you paid is really 49.6040675...

I don't know anything about VTSAX or US-based Vanguard funds, but at
least in the UK the fraction you can own is usually limited to a
certain digit... so assuming you can only own a fraction with 3 digits
(e.g. 4.622 and next is 4.623), your $229.27 will only get you 4.622
because 4.623 would have cost $229.30.

If they are truly only keeping three decimal places of precision, for this particular share at a price of 49.60 USD, the potential for rounding error is +/- 0.0248 USD.  I also hold a fund with a price of 188.36 USD, thus a potential for rounding error at +/- 0.09418.  These are actionable numbers; they can leave pennies in your account as cash to minimize rounding error rather than claiming that they used every penny.  Then again, maybe I'm giving financial industry more credit than they deserve.

[Martin Blais]

On Thu, Nov 20, 2014 at 4:54 PM, ELI <eli...@gmail.com> wrote:

- Harpreet "Eli" Sangha

On Thu, Nov 20, 2014 at 12:58 PM, Martin Blais <bl...@furius.ca> wrote:

On Thu, Nov 20, 2014 at 12:50 PM, ELI <eli...@gmail.com> wrote:

My understanding is that plugins cannot help Beancount parse entries,

Not yet, but I have sketched a plan to make all directives (except Transactions) definable via plugins. Down the very very long road, all directives could be defined by plugins eventually. The plugin would provide a definition of expected data types, to be validated by the parser in Python.

so what I was hoping to do was use comments or tags to "activate" a custom plugin and pass it relevant information, for particular postings. What I've found is that tags can't be associated with postings and comments aren't available to plugins. Is there an option I'm overlooking?

What you want is support for metadata. Here's what I have in mind: I will add support for metadata objects (LISP style, as an attached dict of arbitrary objects) on all transactions and postings. The metadata would be accessible via the Python data structures for you to build custom reports or whatever, and also usable from the SQL interface. There would otherwise be no other semantics associated with it--it would be there for you to use, as extensions (it would support strings, numbers, amounts (number + currency), currency, date, data types.

Now, where is it at? Well, I have already built a prototype that defines a nice syntax and parses the metadata (see branch "metadata"). I haven't yet done the complete work to make it attached to the objects. What I can _quickly_ do is give you parsing capability by merging my branch and this would allow you to immediately start entering all the data in your input file. Attaching the metadata requires some not-so-difficult changes, yet changes that would impact a fair bit of the codebase, so I would rather do that later in a second step.

Would this work for now?

Sure this works for me.  I also wouldn't mind getting my hands dirty and living on a branch that incorporates with work-in-progress for the purpose of evaluating and providing feedback/contributions.

Thanks Eli,

Right now, that branch would be "metadata" but it's massively out-of-date and I'll just update it and merge it in. It should be harmless to merge into default once I update it, I just have to add a few unit tests for it.

  

What am I trying to do?

Well, for any stock trade, I want to record *all* the information provided to me by my broker: share count, share price, and transaction amount. Beancount only allows me to record share count and any one of price or amount.

The share count, share price and transaction amounts should already be on your transactions.  You can place _both_ cost and price on a single posting (but you don't have to):

  2014-05-02 * "Sell stock"

    Assets:US:Investments:GOOGL       -10 GOOGL {606.12 USD} @ 545.37 USD

    Assets:US:Investments:Cash        5453.70 USD

    Income:CapitalGains          

Doesn't the above syntax imply that stock bought at 606.12 USD/GOOGL is being sold at 545.37 USD/GOOGL.  What I mean to record is, for example, how many shares my broker reported that I sold, the price they were reportedly sold at, and the reported amount of the total transaction:

2014-05-02 * "Sell stock"

  Assets:US:Investments:GOOGL       -10 GOOGL @ 545.37 USD @@ 5,453.70 USD

  Assets:US:Investments:Cash        5453.70 USD

By summing all the postings that aren't the one with the stock on it, you already obtain 5453.70 USD...  why do you need it separately? (I'll venture the question is answered later, that this has to do with rounding...)

 

 Note that you may also already use the "link" feature to attach other infos, e.g. a document filename:

  2014-05-02 * "Sell stock" ^2014-05-02.schwab.pdf

    Assets:US:Investments:GOOGL       -10 GOOGL {606.12 USD} @ 545.37 USD

    Assets:US:Investments:Cash        5453.70 USD

    Income:CapitalGains          

Additionally, I want my plugin to replace the posting with a posting where one of the three values is calculated using the other two, based on a parameter attached to the posting (or preferably the account).

Can you provide a use case? What I think you may be trying to do sounds like it might be better done as a reporting function. The Inventory object has functions to extract number of units, cost, average cost, and market value, that you can use. Let me know what your desired output is.

The use case is for brokerages that allow the purchase of fractional shares, and purchases are made by specifying a dollar amount as opposed to a number of shares.

   2014-09-23 * "Dividend"

     Assets:Vanguard:Taxable:Mutual-Fund    +4.622 VTSAX @ 49.60 USD @@ 229.27 USD

     Assets:US:Investments:Cash             -229.27 USD

The "truth" of the transaction is that Vanguard purchased 229.27 USD for me in fund that cost 49.60 USD, and acquired 4.622379 shares, but reported it as 4.622 shares.  Thus, the goal would be to enter into the bean file these above values that came from my statement, but to tell my plugin to replace this posting with one where the number of shares is calculated by the transaction amount divided by the price per share (229.27/49.60).

Aaah, right, it's the rounding issue.

I would like to call Vanguard to ask them what they actually do. I'm not convinced they actually store 4.622379 (or more precise). Surely they should be able to tell us about the level of precision they actually store the values.

If they do, probably the best thing to do would be to enter that value... but then there's another problem: their reporting only provides 3 fractional digits indeed. Hmmm... this requires some thinking. 

This also makes me think of an idea I've had for a plugin: a plugin that further verifies that the sum of postings EXCLUDING postings on Income accounts balance to the sum of the postings' "weight" but where the values of postings held-at-cost are calculated using the price instead of the cost value. This would provide yet another check that the input data is calculated correctly, but I'm not sure it's super general yet. It would be an optional feature you could enable through a plugin. Right now, prices on postings held-at-cost are ignored for the purpose of balancing, but they're inserted in the price database (they translate into a Price directive).  Is this what you have in mind?

I'm not entirely sure I understand this, maybe you can tell me if it relates based on my description above. 

Well if I understand correctly now, the crux of the issue is the rounding issue. You want to be able to store both the rounded and unrounded (or at least more precisely rounded) values. Correct me if I'm wrong.

[Martin Blais]

On Thu, Nov 20, 2014 at 6:06 PM, Martin Michlmayr <t...@cyrius.com> wrote:

* ELI <eli...@gmail.com> [2014-11-20 13:54]:
> 2014-05-02 * "Sell stock"
>   Assets:US:Investments:GOOGL       -10 GOOGL @ 545.37 USD @@ 5,453.70 USD
>   Assets:US:Investments:Cash        5453.70 USD

FWIW, I remember asking a few years ago if this was possible in ledger
(it wasn't).  For some reason, I thought it was a good idea to record
both the share price quoted for one share and the total price (the two
aren't always exactly the same due to rounding).

However, I'm no longer sure there's much value in that.  For all
intends and purposes, the total price you paid is what matters and the
price for one share is realy total/quantity and not whatever share
price was quoted.

What I do is to add the share price as a comment just to make it more
readable:
  Assets:Pension    35.3414 "World ex UK" @@ 288.28 GBP ; @ 8.1570 GBP

So what you're saying is that the higher precision should be used on the price.

Beancount addresses this in two ways:

- A tolerance in the balancing of postings (currently fixed at 0.01, but which will eventually be a function of the number of digits you enter)

- The new inventory booking syntax will support a "total" price slot, something like this {<per-unit> + <total> <currency}, e.g. {+ 288.28 GBP} would be the equivalent. It'll compute and store the per-unit price automatically. (Accessorily, it will support both per-unit and total together.)  See http://furius.ca/beancount/doc/proposal-booking

 

>    2014-09-23 * "Dividend"
>      Assets:Vanguard:Taxable:Mutual-Fund    +4.622 VTSAX @ 49.60 USD @@ 229.27
> USD
>      Assets:US:Investments:Cash             -229.27 USD
>
> The "truth" of the transaction is that Vanguard purchased 229.27 USD for me
> in fund that cost 49.60 USD, and acquired 4.622379 shares, but reported it
> as 4.622 shares.  Thus, the goal would be to enter into the bean file these
> above values that came from my statement, but to tell my plugin to replace
> this posting with one where the number of shares is calculated by the
> transaction amount divided by the price per share (229.27/49.60).

Are you sure you received 4.622379 shares and not 4.622 shares?

In other words, if you do this transaction three times, does Vanguard
report 13.866 or 13.867 VTSAX?  (4.622 * 3 = 13.866; 4.622379 * 3 =
13.867137).

If it's 13.867, then I'd record your purchase as 4.622379 since that's
what you got and Vanguard just didn't show all digits.

I agree.

The thing is, the downloadable reports do not include that value.

And they also do not report the "total" value attached to the transaction either, you have to back that out manually based on what you know your pre-programmed contribution amount it. (Well, maybe that's not such a big deal actually, I can live with having to do that.)

 

If it's 13.866, you may think you got 4.622379 but in reality you got
4.622 due to rounding.  Vanguard may say the price is 49.60 USD but
due to rounding what you paid is really 49.6040675...

I don't know anything about VTSAX or US-based Vanguard funds, but at
least in the UK the fraction you can own is usually limited to a
certain digit... so assuming you can only own a fraction with 3 digits
(e.g. 4.622 and next is 4.623), your $229.27 will only get you 4.622
because 4.623 would have cost $229.30.

I'd love to know precisely what they do before making design changes indeed.

[Martin Blais]

On Thu, Nov 20, 2014 at 6:12 PM, Martin Michlmayr <t...@cyrius.com> wrote:

* Martin Blais <bl...@furius.ca> [2014-11-20 15:58]:
> Can you provide a use case?

I still haven't tried beancount, but since you're looking for use
cases, I have a use case relaed to custom data that ledger doesn't
support.  I hope beancount will.

http://bugs.ledger-cli.org/show_bug.cgi?id=1042

The basic idea is that I want to be able to use a meta tag as the
account, so I can run queries on it.

Why can't you just add the days as a posting in unit DAYS?

(The bugger is that you'd have to balance it against other accounts.)

What you could eventually do in Beancount is write a plugin that transforms the postings by copying the days from the metadata into new postings (or perhaps even replacing the ones that are there) and then use the usual aggregation methods to produce your reports.

[Martin Blais]

Let's call Vanguard. I can call next week (I'm at the Clojure Conj for a few days), or if you want to call them, these specific example transactions might be a great way to ask them about it.  I'm very curious to know what they're actually doing.

[ELI]

I just had a call with Vanguard. They track holdings to 4 decimal places internally.

[ELI]

How I would apply this to my plugin, in the absence of native support:
- Record shares transacted, price per share, and transaction amount as reported on my statement (if for no other reason, for bookkeeping and my OCD).
- Add meta data to my Vanguard account that tells the plugin to calculate the number shares transacted to 4 decimal places.
- Have the plugin modify the posting with one that uses a calculated share count and the specified price per share.

Replacing this:

   2014-09-23 * "Investment"

     Assets:US:Investments:VIIIX     +3.532 VIIIX @ 188.74 USD @@ 666.67 USD

     Assets:US:Investments:VEMPX     +2.031 VEMPX @ 164.14 USD @@ 333.33 USD

     Assets:US:Investments:Cash      -1,000.00 USD

With this:

   2014-09-23 * "Investment"

     Assets:US:Investments:VIIIX     +3.5322 VIIIX @ 188.74 USD

     Assets:US:Investments:VEMPX     +2.0308 VEMPX @ 164.14 USD

     Assets:US:Investments:Cash      -1,000.00 USD

     Income:RoundingGain             -0.01 USD

- Harpreet "Eli" Sangha

[Martin Blais]

> I just had a call with Vanguard. They track holdings to 4 decimal places internally.

Thanks for calling Eli!

So this solves some of our problems, we can simply enter the data to four decimal places. We just have to find some way to get it without having to do any calculations. 

On Fri, Nov 21, 2014 at 2:07 PM, ELI <eli...@gmail.com> wrote:

How I would apply this to my plugin, in the absence of native support:
- Record shares transacted, price per share, and transaction amount as reported on my statement (if for no other reason, for bookkeeping and my OCD).

If you really want the total amount recorded separately from the cash deposit (it'll be redundant most of the time) you could attach it as metadata, or stash it in the narration.

 

- Add meta data to my Vanguard account that tells the plugin to calculate the number shares transacted to 4 decimal places.

You wouldn't need to do that. If you enter four decimal places, it'll just work, no plugin required.

 

- Have the plugin modify the posting with one that uses a calculated share count and the specified price per share.

Replacing this:

   2014-09-23 * "Investment"

     Assets:US:Investments:VIIIX     +3.532 VIIIX @ 188.74 USD @@ 666.67 USD

     Assets:US:Investments:VEMPX     +2.031 VEMPX @ 164.14 USD @@ 333.33 USD

     Assets:US:Investments:Cash      -1,000.00 USD

With this:

   2014-09-23 * "Investment"

     Assets:US:Investments:VIIIX     +3.5322 VIIIX @ 188.74 USD

     Assets:US:Investments:VEMPX     +2.0308 VEMPX @ 164.14 USD

     Assets:US:Investments:Cash      -1,000.00 USD

     Income:RoundingGain             -0.01 USD

Yes, but you don't need the RoundingGain posting: the sum of those transactions is 1000.00294 USD which is within the accepted tolerance (currently 0.01, but eventually will be automatically inferred from the "1,000.00" number see as 0.005 (half of the last digit's precision).

This currently checks:

2014-09-23 open Assets:US:Investments:Cash

2014-09-23 open Assets:US:Investments:VEMPX

2014-09-23 open Assets:US:Investments:VIIIX

2014-09-23 * "Investment"

  Assets:US:Investments:VIIIX     +3.5322 VIIIX @ 188.74 USD

  Assets:US:Investments:VEMPX     +2.0308 VEMPX @ 164.14 USD

  Assets:US:Investments:Cash      -1,000.00 USD

However, if you want to accumulate all the crumbs (Nathan Grigg, another Googler, recently came up with an idea)  automatically adding the remainders to a configurable Equity:Rounding account and I agreed to do that. You wouldn't have to add a posting manually.

If you really want to add rounding postings _explicitly_ only for particular accounts, you could do that with a plugin you write.

[Martin Blais]

Would this work for now?

Sure this works for me.  I also wouldn't mind getting my hands dirty and living on a branch that incorporates with work-in-progress for the purpose of evaluating and providing feedback/contributions.

Thanks Eli,

Right now, that branch would be "metadata" but it's massively out-of-date and I'll just update it and merge it in. It should be harmless to merge into default once I update it, I just have to add a few unit tests for it.

I just updated and merged branch "metadata" into default.

See the example unittests under beancount.parser.parser_test.TestMetaData for syntax.

(This is just a preliminary syntax proposal, we can always review later.)

 

[Martin Michlmayr]

* Martin Blais <bl...@furius.ca> [2014-11-21 15:17]:

> See the example unittests under beancount.parser.parser_test.TestMetaData
> for syntax.
> (This is just a preliminary syntax proposal, we can always review later.)

Does this mean tags are not possible or would you simply give an empty
value (e.g. test:).

[Martin Michlmayr]

* Martin Blais <bl...@furius.ca> [2014-11-21 11:51]:

> > http://bugs.ledger-cli.org/show_bug.cgi?id=1042
> >
> > The basic idea is that I want to be able to use a meta tag as the
> > account, so I can run queries on it.
>
> Why can't you just add the days as a posting in unit DAYS?
> (The bugger is that you'd have to balance it against other accounts.)

I believe there is information that is worth tracking but that should
not be tracked in the account structure. I will give several examples
in the hope that at least one of them makes sense. :)

I use ledger (and possibly beancount soon) for my personal finances,
so I want to track things that are worth something. I do track
frequent flyer miles and hotel points. Even though it's hard to put a
dollar value on those, they obviously are worth something as I can
get hotel room or flights with them.

On the other hand, something like how many nights I stayed in a hotel
is just some meta information associated with my stay, but it's not
really an asset. You could probably argue with this, because a certain
number of nights per year will give me some status with the hotel, and
that's worth something, but I don't see the number of nights as an
asset, not like the points earned. Maybe I'm wrong but the two just
don't feel the same to me.

A better example might be miles travelled on an airplane. I do want
to track miles or points I receive for trips that I can redeem for
flights. But there is associated information, such as how many miles
I travelled, that don't really make sense in my account structure
whereas they are useful as meta info. Or maybe if I get completely
crazy, I'd want to track the type of aircraft flown and later do some
statistics on that. (I've no need for that personally but I've seen
people do all kinds of interesting stuff.)

Again, you can argue I should track these in some account, but that
leads me again to ask what I'm trying to do. And foremost, I'm trying
to track my personal finances, and the number of miles travelled or
the type of aircraft flown have nothing to do with that.

I guess you could argue that I should just keep a separate ledger with
such info, but this would duplicate info (since I'm tracking flights
in my personal finance anyway due to the points I earn).

One more example: a lot of my trips are for my employer, so I don't
pay for them, but I earn the points. So there's absolutely no
justification for tracking the amount spent on a flight or hotel in my
ledger, but sometimes I wonder what the average spend per point is and
for that having the price in some meta info would be useful.

I don't know if these examples make sense to you, but for me there's
value in separating the financial transaction from additional
information about the transaction. Yet, sometimes I want to run
queries over this non-financial info.

> What you could eventually do in Beancount is write a plugin that
> transforms the postings by copying the days from the metadata into
> new postings (or perhaps even replacing the ones that are there) and
> then use the usual aggregation methods to produce your reports.

Yes, this sounds like what I want. I'm not familiar with the
beancount architecture, but my concern when I hear the word "plugin"
is that everyone will write custom plugins for everything instead of
having generic plugins that everyone can use and are shipped as part
of beancount. I believe my request could be met in a generic plugin
that would be useful to others.

I'm not sure if the question of official plugins shipped with
beancount vs unofficial ones has been discussed already.

(Sorry, I'm behind on your proposals and discussion!)

[Martin Michlmayr]

* Martin Blais <bl...@furius.ca> [2014-11-21 15:11]:

> However, if you want to accumulate all the crumbs (Nathan Grigg, another
> Googler, recently came up with an idea) automatically adding the
> remainders to a configurable Equity:Rounding account and I agreed to do
> that. You wouldn't have to add a posting manually.

Will this feature be optional?

Just so I understand this fully.

Let's say I buy 35.3414 units of AAA for 8.1570 each at a total of 288.28

If I say 35.3414 @ 8.1570 with this feature turned on, will you store
this internally as 288.2797998 and then always tell me that the cost
of one unit was 8.1570?

Whereas if I say 35.3414 @ 8.1570 with this feature turned off, it
will store 288.28 internally as the cost, showing a unit price of
8.157005664744464?

Is this how it will work?

I wonder which one I'd prefer. I've started tracking fund purchases
as the total price with @@ since I feel that what I actually paid is
what matters and not some fictional unit price quoted (i.e. the 8.1570
quoted rather than the 8.157005664744464 I actually paid).

Then again, I like the simplicity of 8.1570 :-)

Actually, I'm wondering if I misunderstood. Maybe it will show the
same results in both cases but only in one is something posted to
Equity whereas in the other it's posted nowhere. But this cannot be
true as things would stop balancing at some point.

[ELI]

- Harpreet "Eli" Sangha

On Fri, Nov 21, 2014 at 12:11 PM, Martin Blais <bl...@furius.ca> wrote:

> I just had a call with Vanguard. They track holdings to 4 decimal places internally.

Thanks for calling Eli!

So this solves some of our problems, we can simply enter the data to four decimal places. We just have to find some way to get it without having to do any calculations. 

On Fri, Nov 21, 2014 at 2:07 PM, ELI <eli...@gmail.com> wrote:

How I would apply this to my plugin, in the absence of native support:
- Record shares transacted, price per share, and transaction amount as reported on my statement (if for no other reason, for bookkeeping and my OCD).

If you really want the total amount recorded separately from the cash deposit (it'll be redundant most of the time) you could attach it as metadata, or stash it in the narration.

Yeah, I'd be happy keeping a record of "what the statement said" as metadata or even a comment.  I prefer to have it somewhere accessible to plugins to quench my scripting whims. :-)  In the example below, though, where the share count is going to be calculated because the statement doesn't give the true precision, isn't the total for each investment *needed* since the cash portion dosn't indicate how much was allocated to each fund?

 

- Add meta data to my Vanguard account that tells the plugin to calculate the number shares transacted to 4 decimal places.

You wouldn't need to do that. If you enter four decimal places, it'll just work, no plugin required.

The problem is I don't have visibility on fourth decimal place. 

 

- Have the plugin modify the posting with one that uses a calculated share count and the specified price per share.

Replacing this:

   2014-09-23 * "Investment"

     Assets:US:Investments:VIIIX     +3.532 VIIIX @ 188.74 USD @@ 666.67 USD

     Assets:US:Investments:VEMPX     +2.031 VEMPX @ 164.14 USD @@ 333.33 USD

     Assets:US:Investments:Cash      -1,000.00 USD

With this:

   2014-09-23 * "Investment"

     Assets:US:Investments:VIIIX     +3.5322 VIIIX @ 188.74 USD

     Assets:US:Investments:VEMPX     +2.0308 VEMPX @ 164.14 USD

     Assets:US:Investments:Cash      -1,000.00 USD

     Income:RoundingGain             -0.01 USD

Yes, but you don't need the RoundingGain posting: the sum of those transactions is 1000.00294 USD which is within the accepted tolerance (currently 0.01, but eventually will be automatically inferred from the "1,000.00" number see as 0.005 (half of the last digit's precision).

This currently checks:

2014-09-23 open Assets:US:Investments:Cash

2014-09-23 open Assets:US:Investments:VEMPX

2014-09-23 open Assets:US:Investments:VIIIX

2014-09-23 * "Investment"

  Assets:US:Investments:VIIIX     +3.5322 VIIIX @ 188.74 USD

  Assets:US:Investments:VEMPX     +2.0308 VEMPX @ 164.14 USD

  Assets:US:Investments:Cash      -1,000.00 USD

However, if you want to accumulate all the crumbs (Nathan Grigg, another Googler, recently came up with an idea)  automatically adding the remainders to a configurable Equity:Rounding account and I agreed to do that. You wouldn't have to add a posting manually.

This would suit me.  These are things I like to have visibility on if I can, even if they go neglected most of the time. 

[ELI]

Awesom! Thanks!  I hope to eventually provide useful feedback and contributions.  I hope you don't mind putting up with my questions in the mean time. :-)

- Harpreet "Eli" Sangha

[Martin Blais]

On Fri, Nov 21, 2014 at 5:40 PM, Martin Michlmayr <t...@cyrius.com> wrote:

* Martin Blais <bl...@furius.ca> [2014-11-21 15:17]:
> See the example unittests under beancount.parser.parser_test.TestMetaData
> for syntax.
> (This is just a preliminary syntax proposal, we can always review later.)

Does this mean tags are not possible or would you simply give an empty
value (e.g. test:).

No it just means that the data is parsed but not stored anywhere for now. I probably just need a few more hours of work on it to make the changes to store it and then we'll have metadata.

[Martin Michlmayr]

* Martin Blais <bl...@furius.ca> [2014-11-22 11:18]:

> > * Martin Blais <bl...@furius.ca> [2014-11-21 15:17]:
> > > See the example unittests under beancount.parser.parser_test.TestMetaData
> > > for syntax.
> > > (This is just a preliminary syntax proposal, we can always review later.)
> >
> > Does this mean tags are not possible or would you simply give an empty
> > value (e.g. test:).
>
> No it just means that the data is parsed but not stored anywhere for now. I
> probably just need a few more hours of work on it to make the changes to
> store it and then we'll have metadata.

Sorry, I wasn't clear. I looked at beancount.parser.parser_test.TestMetaData
and could only find examples in the form of:
test: "Something"
i.e. tag: "value"

I was wondering if tags without value are possible, too.

ledger differentiates between tag values:
; foo: bar
and tags:
; :foo:

[Martin Blais]

I get your feeling... you don't want to create postings for this usage, or at least you don't want to see them by default in your balances, yet you still want to be able to generate them and you want it in one place. That's all good.

How about if.... you stored those informations as metadata, and then wrote a script to create a _new_ set of transactions from your ledger, creating transactions that operate on days by inspecting the metadata? That would work. You would basically create a brand new set of transactions (a new ledger) from the metadata of your main ledger and then you could use the usual reportings & aggregation functions off of that.  Would that be a workable solution?

(Please note: you may not be familiar with it yet, but I assure you that writing scripts off of Beancount data structures is _really_ easy. I could provide an example for this usage once I store the metadata, it would no more than 15 minutes to cook one for you.)

One more example: a lot of my trips are for my employer, so I don't
pay for them, but I earn the points.  So there's absolutely no
justification for tracking the amount spent on a flight or hotel in my
ledger, but sometimes I wonder what the average spend per point is and
for that having the price in some meta info would be useful.

I'm a frequent user of Amtrak, and I track points using Beancount too. I use a special currency for it (AMTKPTS) and when I buy new tickets I attach the new points granted to me on those transactions. I get a monthly balance email and I insert an assertion every time. Works fine. That's the alternative to the metadata method. It works because it's simple enough--I can also see how if you're tracking a whole lot of dimensions it wouldn't really a beautiful solution.

I don't know if these examples make sense to you, but for me there's
value in separating the financial transaction from additional
information about the transaction.  Yet, sometimes I want to run
queries over this non-financial info.

Sure.

Because it's so custom, I think for now I would go with a script generating a new list of transactions.

> What you could eventually do in Beancount is write a plugin that
> transforms the postings by copying the days from the metadata into
> new postings (or perhaps even replacing the ones that are there) and
> then use the usual aggregation methods to produce your reports.

Yes, this sounds like what I want.  I'm not familiar with the
beancount architecture, but my concern when I hear the word "plugin"
is that everyone will write custom plugins for everything instead of
having generic plugins that everyone can use and are shipped as part
of beancount.  I believe my request could be met in a generic plugin
that would be useful to others.

I'm not sure if the question of official plugins shipped with
beancount vs unofficial ones has been discussed already.

Not really, but I'm totally open to adding in plugins if they can be made generic enough (i.e., no account names hardcoded in them).  If they cannot, I could also create a contrib area where people could share their scripts.  I recently created a new "plugin" directive where you can pass in an arbitrary string for configuring your plugin, and that string can contain an arbitrary Python data structure that the plugin can eval. It's not the cleanest design/architecture (it's very loose) but it's unbounded.

On the other hand, custom scripts and plugins are _really_ easy to build (try one, just for fun: try writing a script that prints all the account names of all your transactions, should be 4 or 5 LOC total), and I like the idea of evolving plugins independently for a little while and using them to tease out their interesting bits before moving it into the main repo.

[Martin Blais]

On Fri, Nov 21, 2014 at 6:30 PM, Martin Michlmayr <t...@cyrius.com> wrote:

* Martin Blais <bl...@furius.ca> [2014-11-21 15:11]:
> However, if you want to accumulate all the crumbs (Nathan Grigg, another
> Googler, recently came up with an idea)  automatically adding the
> remainders to a configurable Equity:Rounding account and I agreed to do
> that. You wouldn't have to add a posting manually.

Will this feature be optional?

That's the plan so far.

 

Just so I understand this fully.

Let's say I buy 35.3414 units of AAA for 8.1570 each at a total of 288.28

If I say 35.3414 @ 8.1570 with this feature turned on, will you store
this internally as 288.2797998 and then always tell me that the cost
of one unit was 8.1570? 

Whereas if I say 35.3414 @ 8.1570 with this feature turned off, it
will store 288.28 internally as the cost, showing a unit price of
8.157005664744464? 

Is this how it will work?

I don't really understand what you mean.

Let me rephrase and see if we can understand each other.

If you declare a posting of

  <account>      35.3414 AAA @ 8.1570 USD

The 8.1570 number is only used to compute the "weight" of the postings.

The sum of all the weights must be ~= 0 (the absolute difference must be under a tolerance value).

Those two numbers are stored as they are declared.

The total amount is never stored.

I think what you meant is a declation for units "held at cost", as the preceding would typically only be used for conversions of currencies with currency exchange rates. For stocks and holding the cost basis, you have to declare like this:

  <account>      35.3414 AAA {8.1570 USD}

The two numbers are once again stored as they are declared.

The total cost is never stored.

The "weight" is calculated as the produce of those numbers, and once again, if the magnitude of the sum of the weights is smaller than a small tolerance, the transactions is said to balance.

Now, I think I support this (I don't recall, I never use it):

  <account>      35.3414 AAA {{288.28 USD}}

This will divide the total cost to produce the more precise 81570xxx number, and store that as the cost per-unit.

In both cases, if the plugin is loaded, the residual (the sum of all the weights), if any, will be inserted as a posting against Equity:Rounding, so that the sum of all residuals is available there.  (On a different topic but in a similar fashion, I will also implement a plugin to automatically insert postings to implement the Trading Accounts methods for dealing with conversions.)

I wonder which one I'd prefer.  I've started tracking fund purchases
as the total price with @@ since I feel that what I actually paid is
what matters and not some fictional unit price quoted (i.e. the 8.1570
quoted rather than the 8.157005664744464 I actually paid).

Then again, I like the simplicity of 8.1570 :-)

The precision used to display numbers in many contexts is inferred from the most common precision used to enter the numbers for a particular currency, so even if stored at a higher precision, as long as there are enough numbers in your input file shown at 4 digits of precision, the number would be displayed at that precision anyhow. But that's another topic, some more work will occur on that (for instance, perhaps I'll compute precisions for amounts and prices per-currency separately).

 

Actually, I'm wondering if I misunderstood.  Maybe it will show the
same results in both cases but only in one is something posted to
Equity whereas in the other it's posted nowhere.  But this cannot be
true as things would stop balancing at some point.

I hope my explanations above clear up any misunderstanding. Please let me know if still unclear.

[Martin Blais]

On Fri, Nov 21, 2014 at 10:20 PM, ELI <eli...@gmail.com> wrote:

- Harpreet "Eli" Sangha

On Fri, Nov 21, 2014 at 12:11 PM, Martin Blais <bl...@furius.ca> wrote:

> I just had a call with Vanguard. They track holdings to 4 decimal places internally.

Thanks for calling Eli!

So this solves some of our problems, we can simply enter the data to four decimal places. We just have to find some way to get it without having to do any calculations. 

On Fri, Nov 21, 2014 at 2:07 PM, ELI <eli...@gmail.com> wrote:

How I would apply this to my plugin, in the absence of native support:
- Record shares transacted, price per share, and transaction amount as reported on my statement (if for no other reason, for bookkeeping and my OCD).

If you really want the total amount recorded separately from the cash deposit (it'll be redundant most of the time) you could attach it as metadata, or stash it in the narration.

Yeah, I'd be happy keeping a record of "what the statement said" as metadata or even a comment.  I prefer to have it somewhere accessible to plugins to quench my scripting whims. :-)  

Sure. I think that once you begin using the system more and start to trust its numbers you will end up doing like me and entering the very minimal informations you need :-)

 

In the example below, though, where the share count is going to be calculated because the statement doesn't give the true precision, isn't the total for each investment *needed* since the cash portion dosn't indicate how much was allocated to each fund?

I don't really understand what you mean. 

If you meant that "the postings with stock changes won't sum up to exactly +1000" then there might be a misunderstanding about what it does: the change in the cash account will be exactly -1000 USD because that's what you entered on the posting that changes the cash account. Even if the other legs don't exactly sum up to 1000, it doesn't matter, it won't use that amount to post changes to the cash account. The algorithm is: "sum all the weights, check that the absolute value is within a small tolerance." If the sum of the weights != 0 (but small), the check passes. That's it. The residual is lost, or if you enabled the special "rounding" plugin I discussed in the other email, a posting is inserted to absorb the residual to a global equity account so you can review the total amount of rounding that occurred.

I think you're asking another question. I don't understand it.

 

- Add meta data to my Vanguard account that tells the plugin to calculate the number shares transacted to 4 decimal places.

You wouldn't need to do that. If you enter four decimal places, it'll just work, no plugin required.

The problem is I don't have visibility on fourth decimal place. 

I know. We have to figure out some scheme to get it. I'm far away from my data at the moment, so I can't check what I've imported before, but I think it only has 3 digits of precision (which is stupid from Vanguard's part if they hold it at four). Maybe we can push for them to fix that (that would be the Right Thing to do but I wouldn't hold my breath for it).

[Martin Blais]

I absolutely don't mind at all, on the contrary. Questions are a sign that people are using it and are interested. There are no bad questions IMO. I expect the number of questions to grow over time, and there are still a few rough edges to polish, and it gives me motivation to finish things that aren't as fun as say, computing portfolio returns.

[Martin Blais]

Haaa now I see.

I think the right answer is "do you want it to?"

I don't see any reason we shouldn't be able to add it.

It would be a key-value with a value = None.

BTW, I don't call these tags.

I don't want to make the mistake of confusing those with hashtags on the transaction objects, and create the confusion that exists in the Ledger terminology.

These will be called "key-values", and collectively, "metadata."

In Beancount, "tags" always refer to the #hashtags.

[Martin Blais]

I just did it:

https://bitbucket.org/blais/beancount/commits/2544b61871fda4e8cd8cef908cbba1c56323d126

(In "metadata" branch.)

[ELI]

Regarding the last piece on which there seems to be a misunderstanding between you and I, let me provide an standalone example, outside the context of the plugin.  Since Vanguard only makes three decimal places of precision available me, I need to have the actual share count calculated from the price and transaction amount.

For example, I have a transaction on my activities page with these details:

Shares Transacted: 0.000

Share Price: 48.62

Amount: 0.02

The only way to enter this transaction and have it balance would be to manually calculate the Shares Transacted with four decimal places of precision.  My preferred way of entering this would be to enter the Share Price and Amount and have Beancount calculate Shares Transacted to a precision associated with my Vanguard account.  Additionally, as metadata, I'd record "Shares Transacted: 0.000" as history of "what the statement said".

Maybe you can give me your thoughts on how such a case would be addressed with planned Beancount features or as a plugin I could right?

Thanks,

- Harpreet "Eli" Sangha

[Martin Blais]

On Mon, Nov 24, 2014 at 12:33 PM, ELI <eli...@gmail.com> wrote:

Regarding the last piece on which there seems to be a misunderstanding between you and I, let me provide an standalone example, outside the context of the plugin.  Since Vanguard only makes three decimal places of precision available me, I need to have the actual share count calculated from the price and transaction amount.

For example, I have a transaction on my activities page with these details:

Shares Transacted: 0.000

Share Price: 48.62

Amount: 0.02 

The only way to enter this transaction and have it balance would be to manually calculate the Shares Transacted with four decimal places of precision.  My preferred way of entering this would be to enter the Share Price and Amount and have Beancount calculate Shares Transacted to a precision associated with my Vanguard account.  Additionally, as metadata, I'd record "Shares Transacted: 0.000" as history of "what the statement said". 

Maybe you can give me your thoughts on how such a case would be addressed with planned Beancount features or as a plugin I could right?

What does the downloadable file contain? 0.000 or 0.0004 or even 0.00041?

What this is, most likely, is the quarterly fee that they take, which they price in terms of shares. If this is your Google account, and you sum all the similar such transactions around it, you should observe that it sums to 5$ per quarter.

I would log it like this:

  2014-11-23 * "Quarterly fee"

    Assets:US:Vanguard:VIIPX      -0.00041 VIIPX {* 48.62 USD}

    Expenses:Financial:Fees    0.02 USD

The "*" will be required to merge all the lots into a single one (average cost booking) because the 48.62 USD they declare is _not_ one of your lots, but rather just the price of the asset that happened to be there on the day they took the fee. In other words, they take their fee in terms of units of each asset, priced at the current price, deducting against the cost basis of all your lots together (it doesn't matter which because this is a pre-tax account, so no capital gains are reported).

Now you're asking, how could I avoid calculating 0.00041 myself?

My first answer would be: let's first see if the OFX download from Vanguard includes the precision. That would solve it.  I believe it does (I checked my own history of downloads and I have some numbers in there with 4 fractional digits).

My second answer would be that - if you don't have the number of shares from the file - after I implement the much needed new inventory booking proposal (http://furius.ca/beancount/doc/proposal-booking), the interpolation capabilities will be vastly extended, such that eliding the number of shares would even be possible, like this:

  2014-11-23 * "Quarterly fee"

    Assets:US:Vanguard:VIIPX      VIIPX {* 48.62 USD}

    Expenses:Financial:Fees    0.02 USD

Now this by itself still would not solve the problem: this would store 0.000411353353 which is limited at 12 fractional digits because of the default context. So that's incorrect. What would need to be done to deal with this is to infer the most common number of digits used on units of VIIPX and to quantize the result to that number of digits (I think this could be safe enough). The number of digits appearing in the file for each currency is already tracked in a DisplayContext object that is stored in the options_map from the parser. I'll have to take that into account in the inventory booking proposal.  I'm adding this to the proposal.

Does that make sense?

 

Thanks,

- Harpreet "Eli" Sangha

On Sat, Nov 22, 2014 at 11:51 AM, Martin Blais <bl...@furius.ca> wrote:

I absolutely don't mind at all, on the contrary. Questions are a sign that people are using it and are interested. There are no bad questions IMO. I expect the number of questions to grow over time, and there are still a few rough edges to polish, and it gives me motivation to finish things that aren't as fun as say, computing portfolio returns.

On Fri, Nov 21, 2014 at 10:22 PM, ELI <eli...@gmail.com> wrote:

Awesom! Thanks!  I hope to eventually provide useful feedback and contributions.  I hope you don't mind putting up with my questions in the mean time. :-)

- Harpreet "Eli" Sangha

On Fri, Nov 21, 2014 at 12:17 PM, Martin Blais <bl...@furius.ca> wrote:

Would this work for now?

Sure this works for me.  I also wouldn't mind getting my hands dirty and living on a branch that incorporates with work-in-progress for the purpose of evaluating and providing feedback/contributions.

Thanks Eli,

Right now, that branch would be "metadata" but it's massively out-of-date and I'll just update it and merge it in. It should be harmless to merge into default once I update it, I just have to add a few unit tests for it.

I just updated and merged branch "metadata" into default.

See the example unittests under beancount.parser.parser_test.TestMetaData for syntax.

(This is just a preliminary syntax proposal, we can always review later.)

 

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAKU2X8ZZTTUtnMDP9bWAqFc6snjESef%2B014sUnbp6wUbT3nbdg%40mail.gmail.com.

[Martin Blais]

Alright, some more data points about this: 

- I looked at my Vanguard history, and the data I imported from the OFX download includes up to 6 digits of precision. I have numbers like "48.701290" in them. So I think the full precision data is definitely available for download. You just need to download the OFX format. Let me know if your downloads don't contain the precision, I can produce more precise and specific download instructions (there are many options).

- But... because I don't have the average cost booking syntax yet, I have been living with the fees transactions like the one you described as commented out, and the balance checks also commented out. So I figured I'd test out my data and uncomment those imported-but-so-far-commented-out transactions and make necessary changes to the code to allow summing positive and negative balances, and at least the number of units on the balances should assert correctly, even if there is a mix of positive and negative lots.

So I disabled the balance checks that Beancount applies by commenting out code, which essentially allows a mix of positive and negative units in the same inventory. This results in a model similar to how Ledger accumulates positions. 

I'm happy to report that over my two years of data, uncommenting all the fees transactions and balance assertions, all the balance checks pass (!). This gives me a very warm and very fuzzy feeling that my input data is correct :-)  Also, it validates the method and the fact that you can easily automate the process of importing transactions from Vanguard (those are for me pretty much fully automated, I never need to edit those, I used the OFX importer from LedgerHub).

- The inventory booking proposal for average booking won't be implemented in the next few weeks... I'm tempted to think that maybe I should provide a way to disable the strict balance checks in the interim. This way we could enter the transactions without matching lots strictly... at least all the data would be present and the balance checks would work.  Would people think it's a good idea?  I would do this by extending the default value for the type of booking is intended to take place (as in the inventory proposal) and add a new value for it, i.e,. in addition to STRICT, FIFO, LIFO, AVERAGE, AVERAGE_ONLY, I would add NONE. I would use the proposed syntax extension for the Open directive, e.g.

  2014-08-84 open Assets:US:Vanguard:VIIPX    VIIPX    "NONE"

This also means that you could setup all your accounts to remove all inventory booking, which results in a booking method similar to Ledger (no checks and no errors), by setting the default value for it, like this:

  option "booking_method" "NONE"

This could appeal to those who would like less checks, like Ledger, or who are converting their Ledger ledgers to Beancount.

Down the road, the inventory booking would use that and implement all methods, but for now, only the balance check would consult that value and disable the check if the default booking method is "NONE". I think I could easily hack that in in a few hours.

What do you think? Opinions?

[Martin Blais]

Alright... so I've implemented the parsing of an disambiguation inventory booking method as I describe below and disable the strict booking check if set to NONE. The only supported methods right now are "STRICT" (the current method) and "NONE".

If I were to merge this into default, you would be able to track cost for accounts at handled at average cost basis in the same way that Ledger does, right now, that is, all the lots just accumulate (if there is no match against an existing lot) and you reduce all this to units at the end and sum up all the cost bases. I think it's a good intermediate step.

Unless anyone screams "murder!", I think I'm going to merge this into default soon. This is source compatible with the current default, it only adds a new, optional string parameter to the Open directive.

[Martin Blais]

Ah yeah, a link to source code is always fun, to see what I had to do to make this work:

https://bitbucket.org/blais/beancount/branches/compare/booking..default

[ELI]

This makes sense and look forward to the proposed feature.  I'm not setup to import OFX files yet, I'll start looking into that, but opening an OFX in Vim downloaded from Vanguard via "My Accounts" -> "Transaction history" -> "Download" I only see three decimal places in the <UNITS> field.

- Harpreet "Eli" Sangha

[ELI]

- Harpreet "Eli" Sangha

On Mon, Nov 24, 2014 at 9:13 PM, Martin Blais <bl...@furius.ca> wrote:

Alright, some more data points about this: 

- I looked at my Vanguard history, and the data I imported from the OFX download includes up to 6 digits of precision. I have numbers like "48.701290" in them. So I think the full precision data is definitely available for download. You just need to download the OFX format. Let me know if your downloads don't contain the precision, I can produce more precise and specific download instructions (there are many options).

As noted in the reply to the previous email, I only see 3 digits of precision.  The steps you used would be appreciated. 

- But... because I don't have the average cost booking syntax yet, I have been living with the fees transactions like the one you described as commented out, and the balance checks also commented out. So I figured I'd test out my data and uncomment those imported-but-so-far-commented-out transactions and make necessary changes to the code to allow summing positive and negative balances, and at least the number of units on the balances should assert correctly, even if there is a mix of positive and negative lots.

So I disabled the balance checks that Beancount applies by commenting out code, which essentially allows a mix of positive and negative units in the same inventory. This results in a model similar to how Ledger accumulates positions. 

I'm happy to report that over my two years of data, uncommenting all the fees transactions and balance assertions, all the balance checks pass (!). This gives me a very warm and very fuzzy feeling that my input data is correct :-)  Also, it validates the method and the fact that you can easily automate the process of importing transactions from Vanguard (those are for me pretty much fully automated, I never need to edit those, I used the OFX importer from LedgerHub).

Cool!  I'll have to read up on this an take the same approach: putting in entries that are disabled in anticipation of the feature. 

- The inventory booking proposal for average booking won't be implemented in the next few weeks... I'm tempted to think that maybe I should provide a way to disable the strict balance checks in the interim. This way we could enter the transactions without matching lots strictly... at least all the data would be present and the balance checks would work.  Would people think it's a good idea?  I would do this by extending the default value for the type of booking is intended to take place (as in the inventory proposal) and add a new value for it, i.e,. in addition to STRICT, FIFO, LIFO, AVERAGE, AVERAGE_ONLY, I would add NONE. I would use the proposed syntax extension for the Open directive, e.g.

  2014-08-84 open Assets:US:Vanguard:VIIPX    VIIPX    "NONE"

This also means that you could setup all your accounts to remove all inventory booking, which results in a booking method similar to Ledger (no checks and no errors), by setting the default value for it, like this:

  option "booking_method" "NONE"

This could appeal to those who would like less checks, like Ledger, or who are converting their Ledger ledgers to Beancount.

Down the road, the inventory booking would use that and implement all methods, but for now, only the balance check would consult that value and disable the check if the default booking method is "NONE". I think I could easily hack that in in a few hours.

What do you think? Opinions?

Sounds like a useful option to me. 

[Martin Blais]

On Tue, Nov 25, 2014 at 12:18 PM, ELI <eli...@gmail.com> wrote:

- The inventory booking proposal for average booking won't be implemented in the next few weeks... I'm tempted to think that maybe I should provide a way to disable the strict balance checks in the interim. This way we could enter the transactions without matching lots strictly... at least all the data would be present and the balance checks would work.  Would people think it's a good idea?  I would do this by extending the default value for the type of booking is intended to take place (as in the inventory proposal) and add a new value for it, i.e,. in addition to STRICT, FIFO, LIFO, AVERAGE, AVERAGE_ONLY, I would add NONE. I would use the proposed syntax extension for the Open directive, e.g.

  2014-08-84 open Assets:US:Vanguard:VIIPX    VIIPX    "NONE"

This also means that you could setup all your accounts to remove all inventory booking, which results in a booking method similar to Ledger (no checks and no errors), by setting the default value for it, like this:

  option "booking_method" "NONE"

This could appeal to those who would like less checks, like Ledger, or who are converting their Ledger ledgers to Beancount.

Down the road, the inventory booking would use that and implement all methods, but for now, only the balance check would consult that value and disable the check if the default booking method is "NONE". I think I could easily hack that in in a few hours.

What do you think? Opinions?

Sounds like a useful option to me. 

Hi Eli,

I've implemented it and merged it into default.

Just put "NONE" at the end of your Vanguard accounts and you should be able to assert balance on the parent account, e.g.

  2014-11-27 balance Assets:US:Vanguard         2839.1304 VBMPX

should work, it'll sum all the units in the subaccounts, even if you book them like me, to accounts like

  Assets:US:Vanguard:PreTax401k:VBMPX

  Assets:US:Vanguard:Match401k:VBMPX

  Assets:US:Vanguard:AfterTax401k:VBMPX

  Assets:US:Vanguard:Rollover:VBMPX

(Beancount assertions on parent accounts include the balances on all subaccounts. It allows you to do something like that.)