problem with NONEs and plugin

[Lluís Pàmies i Juárez]

Here is my problem. I modified the "auto_account" plugin to open with "NONE" booking all the subaccounts in my Assets:Vanguard:401k. However, I noticed that this thing doesn't quite work, because the loader module runs booking.book() before run_transformations(), and transactions like this one get transformed to having zero postings:

2017-02-23 * "Exchange"
  Assets:Vanguard:401k:FUNDA  -12.184 FUNDA {32.53 USD}
  Assets:Vanguard:401k:FUNDB   12.188 FUNDB {32.52 USD}

For the small example I'm working with, I can invert the order of these two operations and everything seems to work. Is there any reason why these two operations are executed in this order? Am I going to have other things breaking down the road? and if so, what is the right way to have "NONE" for a bunch of subaccounts, without having to open all of them manually?

--

Lluís Pàmies i Juárez
http://lluis.pamies.cat

[Martin Blais]

I don't have enough information from your description to solve the problem.

That transaction should never have zero postings.

Run bean-doctor context on it and share the output.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAKD2hyHpFsppu3axQcaBXNUyuWL1SbT-RdaYF%3DNsQe7nV5dAcA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Lluís Pàmies i Juárez]

This is the input file "/tmp/a":

plugin "beancount.plugins.auto_accounts"
2017-02-10 * "Contribution"
  Assets:Vanguard:401k:FUNDA      12.184 FUNDA {32.04 USD}
  Assets:Vanguard:401k:Cash   -390.37536 USD
2017-02-23 * "Exchange"
  Assets:Vanguard:401k:FUNDA  -12.184 FUNDA {32.53 USD}
  Assets:Vanguard:401k:FUNDB   12.188 FUNDB {32.52 USD}

Then I run "bean-doctor context /tmp/a 5":

Hash:8960232d31fba5113ed6802ab12d5f44

Location: /tmp/a:5

------------ Balances before transaction

------------ Transaction

2017-02-23 * "Exchange"

------------ Balances after transaction

You can see how the two postings of this transaction have disappeared. I've done some debugging, and "entry.postings" is actually empty.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhMpnGX_umg243Be4Dqy5C_1c13Y4%2BRk81yHC0kSKNuoAg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Martin Blais]

- The reason your transaction has no postings is because is has errors.  When some group of postings cause an error, they are discarded and an error is issued:

/home/blais/r/q/beancount-data/user/lluispamies/2018nov.beancount:5:       No position matches "Posting(account='Assets:Vanguard:401k:FUNDA', units=-12.184 FUNDA, cost=CostSpec(number_per=Decimal('32.53'), number_total=None, currency='USD', date=None, label=None, merge=False), price=None, flag=None, meta={'filename': '/home/blais/r/q/beancount-data/user/lluispamies/2018nov.beancount', 'lineno': 6})" against balance (12.184 FUNDA {32.04 USD, 2017-02-10})

   2017-02-23 * "Exchange"

     Assets:Vanguard:401k:FUNDA  -12.184 FUNDA {32.53 USD}

     Assets:Vanguard:401k:FUNDB   12.188 FUNDB {32.52 USD}

(I've been meaning to make this easier to debug but I haven't chosen a good way / how to keep the postings on failed checks yet. The reason bean-doctor context doesn't show anything is that the transaction stripped of its failed postings doesn't have any accounts to list. This could be improved.)

- You cannot reduce units at cost {32.53 USD} because you don't have any. You deposited units at {32.04 USD}.

- Your change to auto_account is probably buggy, because when I set it up explicitly, e.g., 

2017-01-01 open Assets:Vanguard:401k:FUNDA  "NONE"

2017-01-01 open Assets:Vanguard:401k:FUNDB  "NONE"

I get an expected balance error (so it works).

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAKD2hyF_%2BhssB2JOhbL%2BMVATekxhChDoF7s3JdaBkw4Dm-TRNg%40mail.gmail.com.

[Lluís Pàmies i Juárez]

I don't think it's buggy. What I think it happens is that in loader.py (function _load) there are these lines:

    # Run interpolation on incomplete entries.
    entries, balance_errors = booking.book(entries, options_map)

    # Transform the entries.
    entries, errors = run_transformations(entries, parse_errors, options_map, log_timings)

"booking.book" seems to be stripping out the positions with errors, and it seems that plugins are run in "run_transformations", so my plugin doesn't see any account and doesn't open it. My OP was asking if inverting the order of these two operations is safe in general.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhMMkOQgEXs%2BqsZzLXLST%2BpN6-k6rSnipk%2BcPGwNYer_Pw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Martin Blais]

On Sun, Nov 4, 2018 at 7:53 PM Lluís Pàmies i Juárez <ll...@pamies.cat> wrote:

I don't think it's buggy. What I think it happens is that in loader.py (function _load) there are these lines:

    # Run interpolation on incomplete entries.
    entries, balance_errors = booking.book(entries, options_map)

    # Transform the entries.
    entries, errors = run_transformations(entries, parse_errors, options_map, log_timings)

"booking.book" seems to be stripping out the positions with errors, and it seems that plugins are run in "run_transformations", so my plugin doesn't see any account and doesn't open it. My OP was asking if inverting the order of these two operations is safe in general.

No, this won't work.

transformation have to be run on booked entries, whereby all positions have been resolved and CostSpec have been removed.

The two streams of directives is different, the first one being closer to the unresolved input.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAKD2hyEaX-9NkoRK6DbB%3DX9P8ysdWxL3kswS_qTMpDG0oKyQHg%40mail.gmail.com.

[Lluís Pàmies i Juárez]

I see.

So there is no way to auto-open accounts with "NONE" before booking entries?

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhO%3Da6ex60MYxr5LHQSmff07E%3Dp5Vp%2BPvaSDFbuscx9LJg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Martin Blais]

On Mon, Nov 5, 2018 at 6:17 PM Lluís Pàmies i Juárez <ll...@pamies.cat> wrote:

I see.

So there is no way to auto-open accounts with "NONE" before booking entries?

No.

You can create a plugin to do that.

Another idea is that the current plugin could be modified to look up the metadata up the account name chain for a default value when creating an account.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAKD2hyEFJ3ErEiet%3Dm98wfUaUTnethVMb3jusj3Vph_TMWn8gw%40mail.gmail.com.

[Lluís Pàmies i Juárez]

I'm confused. You say that I can create plugin to do that but I do not see how. Let's say I write a wonderful plugin and use it in this file:

plugin "beancount.plugins.wonderful"
2017-02-10 * "Contribution"
  Assets:Vanguard:401k:FUNDA      12.184 FUNDA {32.04 USD}
  Assets:Vanguard:401k:Cash   -390.37536 USD
2017-02-23 * "Exchange"
  Assets:Vanguard:401k:FUNDA  -12.184 FUNDA {32.53 USD}
  Assets:Vanguard:401k:FUNDB   12.188 FUNDB {32.52 USD}

As I understand it "booking.book" will remove the two postings in the second entry, and then pass the two resulting entries to my plugin (one with two postings, and a second one without any posting). My plugin will never know of the existence of "Assets:Vanguard:401k:FUNDB". Is that correct? What am I missing here?

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhOgseb-7cHd68QnNdpnsgb%3DQO9fXMU%2B3UDKEoCq2Dw%3DTQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Martin Blais]

Ah yes, that's right. There's a fundamental incompatibility with the inability to run plugins before booking.

At some point I thought I would open plugins to be able to affect either stages (before booking and after booking) but it would require a very clear and good specification of both schemas, and I thought it would create more confusion than is necessary.  A few version of Beancount surely would have this.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAKD2hyFYcbTte4gjEUhy3gMXN_69CkTR4f05Sd_xEXO5v7axaQ%40mail.gmail.com.