FIFO without explicit cost basis

[Dave The Happy Singer]

Hi Martin et al,

I'm still trying to understand how to reliably book transactions without explicitly specifying the cost basis on every leg (e.g. with FIFO booking)

Here is a test case with everything explicit, that passes bean-check. I buy 1 BTC for 1000 AUD from an account initialised with a positive AUD balance. Then I transfer a sum to a cold wallet and back to the hot wallet, incurring a fee on each transaction which I book to a BTC-denominated expenses account.

option "operating_currency" "AUD"

2014-01-01 open Equity:Opening-Balances

2014-01-01 open Assets:AU:Cash AUD

2014-01-01 open Expenses:Financial:TransactionFees:BTC BTC

2014-01-01 open Assets:AU:Hotwallet BTC "FIFO"

2014-01-01 open Assets:AU:Coldwallet BTC "FIFO"

2014-01-01 * "Initial cash balance"

  Assets:AU:Cash                             1000.00 AUD

  Equity:Opening-Balances

2014-01-02 * "Initial hotwallet balance"

  Assets:AU:Hotwallet                           1.00 BTC {1000 AUD}

  Assets:AU:Cash                            -1000.00 AUD

2014-01-03 * "Deposit Cold storage"

  Assets:AU:Hotwallet                          -0.050000000 BTC {1000 AUD}

  Assets:AU:Coldwallet                          0.049000000 BTC {1000 AUD}

  Expenses:Financial:TransactionFees:BTC        0.001000000 BTC {1000 AUD}

2014-01-04 * "Withdraw Cold storage"

 Assets:AU:Coldwallet                          -0.049000000 BTC {1000 AUD}

 Assets:AU:Hotwallet                            0.048000000 BTC {1000 AUD}

 Expenses:Financial:TransactionFees:BTC         0.001000000 BTC {1000 AUD}

However, despite making my wallet accounts "FIFO", whenever I try to remove the explicit {1000 AUD} cost basis in my reducing postings, I start getting errors.

For example, if I use {} in the first transaction:

2014-01-03 * "Deposit Cold storage"

  Assets:AU:Hotwallet                          -0.050000000 BTC {}

  Assets:AU:Coldwallet                          0.049000000 BTC {}

  Expenses:Financial:TransactionFees:BTC        0.001000000 BTC {}

I would be hoping that as FIFO is specified for the Coldwallet account, the lot is inferred.

However, bean-check now complains: Failed to categorize posting.

Furthermore, attempting to debug with bean-doctor doesn't quite work for me. The transaction in question starts at line 17. But beandoctor context <filename> 17 returns the context for the previous transaction at line 13.

If I try amending the second transaction to use {} (instead of the first which I set back to {AUD 1000}), bean-check replies "Too many missing numbers for currency group 'AUD". And bean-doctor ... 13 shows the expected balances after the first transaction but beandoctor ... 17 doesn't show any opening balances even though it directly follows the tx at 13.

Any idea what I may have done wrong here? I'm using beancount==2.0rc1 as installed by the latest package on PyPI. I've read the Inventory and Trading docs several times, but obviously I'm still learning so if I've wasted your time I apologise. Still excited by beancount!

Dave

[Martin Blais]

On Thu, Jun 15, 2017 at 11:17 PM, Dave The Happy Singer <sm...@davethehappysinger.com> wrote:

Hi Martin et al,

I'm still trying to understand how to reliably book transactions without explicitly specifying the cost basis on every leg (e.g. with FIFO booking)

Here is a test case with everything explicit, that passes bean-check. I buy 1 BTC for 1000 AUD from an account initialised with a positive AUD balance. Then I transfer a sum to a cold wallet and back to the hot wallet, incurring a fee on each transaction which I book to a BTC-denominated expenses account.

option "operating_currency" "AUD"

2014-01-01 open Equity:Opening-Balances

2014-01-01 open Assets:AU:Cash AUD

2014-01-01 open Expenses:Financial:TransactionFees:BTC BTC

2014-01-01 open Assets:AU:Hotwallet BTC "FIFO"

2014-01-01 open Assets:AU:Coldwallet BTC "FIFO"

2014-01-01 * "Initial cash balance"

  Assets:AU:Cash                             1000.00 AUD

  Equity:Opening-Balances

2014-01-02 * "Initial hotwallet balance"

  Assets:AU:Hotwallet                           1.00 BTC {1000 AUD}

  Assets:AU:Cash                            -1000.00 AUD

2014-01-03 * "Deposit Cold storage"

  Assets:AU:Hotwallet                          -0.050000000 BTC {1000 AUD}

  Assets:AU:Coldwallet                          0.049000000 BTC {1000 AUD}

  Expenses:Financial:TransactionFees:BTC        0.001000000 BTC {1000 AUD}

2014-01-04 * "Withdraw Cold storage"

 Assets:AU:Coldwallet                          -0.049000000 BTC {1000 AUD}

 Assets:AU:Hotwallet                            0.048000000 BTC {1000 AUD}

 Expenses:Financial:TransactionFees:BTC         0.001000000 BTC {1000 AUD}

Side note: Your expenses should probably be best booked in AUD, not in BTC.

In any case, it doesn't make much sense to store anything in an expense account with a cost basis.

However, despite making my wallet accounts "FIFO", whenever I try to remove the explicit {1000 AUD} cost basis in my reducing postings, I start getting errors.

For example, if I use {} in the first transaction:

2014-01-03 * "Deposit Cold storage"

  Assets:AU:Hotwallet                          -0.050000000 BTC {}

  Assets:AU:Coldwallet                          0.049000000 BTC {}

  Expenses:Financial:TransactionFees:BTC        0.001000000 BTC {}

I would be hoping that as FIFO is specified for the Coldwallet account, the lot is inferred.

The reducing lot is inferred (the first one).

However, bean-check now complains: Failed to categorize posting.

For the augmenting lots (the second and third one), there's no way to infer what the cost basis should be, only you know that. e.g., you can use this syntax to do cost basis adjustments. 

Except, you may still be able to skip ONE number for each currency group.

But... you also cannot omit too many currency groups. Consider the case where you might use a different currency on the other leg (this has happened to me before for a stock exchange between CAD and USD if I recall). Anyhow, disambiguating the currency is easy; for example, this works:

2014-01-03 * "Deposit Cold storage"

  Assets:AU:Hotwallet                          -0.050000000 BTC {1000 AUD}

  Assets:AU:Coldwallet                          0.049000000 BTC {999 AUD}

  Expenses:Financial:TransactionFees:BTC        0.001000000 BTC {}

Furthermore, attempting to debug with bean-doctor doesn't quite work for me. The transaction in question starts at line 17. But beandoctor context <filename> 17 returns the context for the previous transaction at line 13.

That's interesting; if there's no error, it works fine.

However, I think in this case, because this error was irrecoverable, the transaction was skipped, and the behavior of the line-number selection is to select the closest previous transaction. It does result in an odd behavior.

Maybe I should always create transactions even when there are errors. 

https://bitbucket.org/blais/beancount/issues/169/consider-always-creating-some-transaction

Which it correctly infers to:

bean-doctor context smile2.beancount 19

Hash:b8a74ca220f3ebc6fce9da5d1ddd8738

Location: /home/blais/r/q/beancount-data/user/smile/smile2.beancount:17

------------ Balances before transaction

  Assets:AU:Hotwallet                                       1.00 BTC {1000 AUD, 2014-01-02}

  Assets:AU:Coldwallet

  Expenses:Financial:TransactionFees:BTC

------------ Transaction

2014-01-03 * "Deposit Cold storage"

  Assets:AU:Hotwallet                     -0.050000000 BTC {1000 AUD, 2014-01-02}  ; -50.000000000 AUD

  Assets:AU:Coldwallet                     0.049000000 BTC {999 AUD, 2014-01-03}   ;  48.951000000 AUD

  Expenses:Financial:TransactionFees:BTC   0.001000000 BTC {1049 AUD, 2014-01-03}  ;   1.049000000 AUD

Tolerances: BTC=5E-10

------------ Balances after transaction

* Assets:AU:Hotwallet                                0.950000000 BTC {1000 AUD, 2014-01-02}

* Assets:AU:Coldwallet                                0.049000000 BTC {999 AUD, 2014-01-03}

* Expenses:Financial:TransactionFees:BTC             0.001000000 BTC {1049 AUD, 2014-01-03}

 

If I try amending the second transaction to use {} (instead of the first which I set back to {AUD 1000}), bean-check replies "Too many missing numbers for currency group 'AUD".

How would you like Beancount to know which currency should be used?

I've been thinking in the past that perhaps it should infer it from the prior inventory--e.g., if there's a single (unambogious) cost currency present in the ante-inventory's list of positions, fill that in automatically-- but that would be more inference than it has done before. I don't dislike the idea.

https://bitbucket.org/blais/beancount/issues/170/try-inferring-currency-from-the-contents

 

And bean-doctor ... 13 shows the expected balances after the first transaction but beandoctor ... 17 doesn't show any opening balances even though it directly follows the tx at 13.

Also probably the same issue, transaction might have been skipped due to error.

Hmm I should improve that, I have seen that problem from time to time, it is a bit of an annoying behavior.

Any idea what I may have done wrong here? I'm using beancount==2.0rc1 as installed by the latest package on PyPI. I've read the Inventory and Trading docs several times, but obviously I'm still learning so if I've wasted your time I apologise. Still excited by beancount!

You're not wasting our time; please keep bringing up interesting corner cases like this.

I'm not really interested in Bitcoins, but I like that it's dredging up things at the edge of the booking code...

This is all good testing and discussion, if you have the patience.

Dave

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/d559299b-9f6e-45f6-bec4-996d2b52b4ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave The Happy Singer]

Hey Martin,

Thanks again for the thorough response. I think I'm struggling to grasp this line from the docs:

>The “FIFO” method automatically select the oldest of the matching lots up to the requested size of the reduction.

In the case I posted, there is only one lot in the Hotwallet inventory, large enough to cover both the deposit to the Coldwallet and the fee. Even if I simplify the case to a single withdrawal with no fee:

option "operating_currency" "AUD"

2014-01-01 open Equity:Opening-Balances

2014-01-01 open Assets:AU:Cash AUD 

2014-01-01 open Expenses:Financial:TransactionFees:BTC BTC 

2014-01-01 open Assets:AU:Hotwallet BTC "FIFO"

2014-01-01 open Assets:AU:Coldwallet BTC "FIFO"

2014-01-01 * "Initial cash balance"

  Assets:AU:Cash                             1000.00 AUD

  Equity:Opening-Balances

2014-01-02 * "Initial hotwallet balance"

  Assets:AU:Hotwallet                           1.00 BTC {1000 AUD}

  Assets:AU:Cash                            -1000.00 AUD

2014-01-03 * "Deposit Cold storage"

  Assets:AU:Hotwallet                          -0.050000000 BTC {}

  Assets:AU:Coldwallet                          0.050000000 BTC {}

This fails with an error, but the following tx works:

2014-01-03 * "Deposit Cold storage"

  Assets:AU:Hotwallet                          -0.050000000 BTC {AUD}

  Assets:AU:Coldwallet                          0.050000000 BTC {}

Given I've specified FIFO (and in my case there is only one lot in the Hotwallet inventory anyway), I can't quite get my head around why I can leave out the number but not the currency.

However, I had a further surprise: the cost basis in the Coldwallet becomes 2014-01-03. I presumed that once a cost basis had been attached to a lot, it would be preserved unless overridden explicitly.

To take another example: I have two stockbroking accounts. One has my margin loan so I normally buy shares with that account. However when I come to sell I often transfer them to another account which has lower brokerage. It's not relevant to my cost basis which account I hold the shares in; I can move them around as I choose but I'd need my cost basis to reflect the reality of when I bought and sold a specific share. I presumed that the <Position> objects created on the augmenting leg(s) would inherit the cost basis of the reducing leg(s) where FIFO is used and unless I explicitly reset the cost basis.

Effectively, it seems the safe way to proceed is to use STRICT and to specify the cost basis explicitly on each tx. But as you noted in our previous thread, these txs are quite small and frequent. But that would also realistically mean running bean-doctor before every entry I write to establish what my inventory is, and rely on my not making mistakes. If I had to correct an historical tx, I'd have to update the cost basis of every subsequent tx.

If this is true, I'd probably write a helper script to 'do FIFO' for me, but before I start going too far down that road it may be that I am still just a bit confused about how beancount's FIFO behaviour works.

Any thoughts on this?

Thanks again,

Dave

Dave

To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/d559299b-9f6e-45f6-bec4-996d2b52b4ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Beancount" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beancount/bMKLxoWJb04/unsubscribe.
To unsubscribe from this group and all its topics, send an email to beancount+...@googlegroups.com.

To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhM3rQdqZ_k1LmY6XLWoGfgm0SpYMNicVHcUxzcOfyv%2Bgg%40mail.gmail.com.

[Martin Blais]

The answer is:

(a) because it's written to be general and support multiple currencies for the cost basis so for augmenting postings it doesn't infer from prior history which currency it could be (i.e., it supports you using a different one), 

or, if you prefer 

(b) because inference is defined mainly "locally", that is, inspecting information only from the transaction itself, in order to keep things simple and understandable without having to refer to other parts in the file; any remote transaction somewhere far in the file could have an impact on the inventory presented as input to your transaction, and if I had designed the interpolation to depend much on that inventory's state you might find odd and surprising errors elsewhere when you add or remove a transaction anywhere. That could create a confusing game of whack-a-mole.

The main effect of your currency addition on the reducing posting is that it disambiguates the currency of the augmenting posting (it's able to infer that it must be AUD since the only other posting is AUD).

Furthermore, I think it's important to realize that reducing postings are treated very differently than augmenting postings. Perhaps I should go back and document that better. Generally, an augmenting posting much provide most of its information or it should otherwise be possible to infer it using interpolation (omitting at most one number per currency group).

Part of what's going on here is that you're using the cost basis in a way that is rather unusual. For stocks, for instance, it's never a problem to insert the full cost basis on purchases, including the currency (well, you wouldn't really have to since the other postings are cash and fees). Transactions where all postings use the cost basis are pretty rare normally, usually only for cost basis adjustments. Using them everywhere like this might require a bit of enhancement to make those more convenient.

Also, qualitatively, I think you're looking at your particular use case and thinking "this should be obvious" but not inspecting all the other possible use cases it has to support. I also think that enhancing the inference code to make use of incoming inventory contents might not be as confusing as I fear it could be and we should experiment with that more.

However, I had a further surprise: the cost basis in the Coldwallet becomes 2014-01-03. I presumed that once a cost basis had been attached to a lot, it would be preserved unless overridden explicitly.

Well by removing it from the Hotwallet, you did the equivalent of "selling it." 

The second posting is like a purchase, so its cost basis inherits that of the transfer transaction.

Again, this is a bit of an oddball case compared to a stock trading scenario.

For cost basis adjustments - which also require the maintenance of the original date - you can override the date manually. I think I must have that example somewhere in my docs.

The issue here is that you're expecting it to treat a cost basis "transfer" differently than it's designed for.

This is designed for stocks and bonds and stocks and suck.

To take another example: I have two stockbroking accounts. One has my margin loan so I normally buy shares with that account. However when I come to sell I often transfer them to another account which has lower brokerage. It's not relevant to my cost basis which account I hold the shares in; I can move them around as I choose but I'd need my cost basis to reflect the reality of when I bought and sold a specific share. I presumed that the <Position> objects created on the augmenting leg(s) would inherit the cost basis of the reducing leg(s) where FIFO is used and unless I explicitly reset the cost basis.

The Transaction objects and algorithms don't know anything about a transfer. They just make sure the total cost basis balances.

If you want to maintain the lot date, you need to override it explicitly. In my experience such transfers are relatively rare (and somewhat difficult to initiate, as the brokers must send share certificates if I'm not mistaken, it's not an every day transaction) and it's designed for the more common case.

Finally, if you are doing day-trading (e.g. 1000 trades/day), Beancount isn't really designed for that. It might work, but if I were to go back to trading electronically I'd write something dedicated for that use case. It's designed more for the individual "investor" in mind (e.g. 10-100 trades/month).

Effectively, it seems the safe way to proceed is to use STRICT and to specify the cost basis explicitly on each tx.

Every augmenting transaction must either be specified explicitly or be inferable (one missing number can be inferred in each currency group).

 

But as you noted in our previous thread, these txs are quite small and frequent. But that would also realistically mean running bean-doctor before every entry I write to establish what my inventory is, and rely on my not making mistakes. If I had to correct an historical tx, I'd have to update the cost basis of every subsequent tx.

If this is true, I'd probably write a helper script to 'do FIFO' for me, but before I start going too far down that road it may be that I am still just a bit confused about how beancount's FIFO behaviour works.

The FIFO works fine AFAIK - that operates on the reducing transactions.

The question is how to specify the augmenting ones.

Any thoughts on this?

Is being restricted to a single augmenting postings per transaction (so its cost can be inferred) and having to put in the currency not working?

 

Thanks again,

Dave

Dave

To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.

To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/d559299b-9f6e-45f6-bec4-996d2b52b4ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Beancount" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beancount/bMKLxoWJb04/unsubscribe.

To unsubscribe from this group and all its topics, send an email to beancount+unsubscribe@googlegroups.com.

To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhM3rQdqZ_k1LmY6XLWoGfgm0SpYMNicVHcUxzcOfyv%2Bgg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Beancount" group.

To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.

To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAHkYY6W-DdPB8n5WS7b%3DB4SiM%3DiOjy%3D67JukaR6%3Dc03bikU5rg%40mail.gmail.com.