Re: How to transfer lots between accounts and how to query lot information?

[Ben Blount]

I saw a few different questions in there:

- "I need to look up/query my lots, but I'm unable to do this. `bean-report FILE holdings` seemed appropriate, but the lot date is missing and the units are trimmed/rounded to two digits:"

Beancount tries to infer the right precision for a particular currency, but provides you options to specify what it should be. See https://beancount.github.io/docs/precision_tolerances.html and https://beancount.github.io/docs/beancount_options_reference.html

I think that should set the precision you are getting in bean-query / report.

- Maintaining basis when transferring lots.

The booking methods as they are now are only used to resolve reductions. So reducing the lots in an account can select several possible lots for a booking method like FIFO, and indeed those get expanded by the booking method to reduce each of the original lots.

However, the augmentation code has no such notion. If you put an augmentation with an empty basis spec, beancount's inference engine will see a single new lot being created and try to infer the missing info by calculating which values would make the transaction balance.

All of this code runs before plugins do, but I believe you could write a fairly simple plugin to do this. Plugins have the opportunity to rewrite the transaction completely, but do run after the booking code has run.

So you would write something like this in your ledger:

2020-12-19 * "Binance" "Transfer BTC Coinbase-Pro => Binance"

Assets:Crypto:BTC:Binance 0.2 BTC {}

Assets:Crypto:BTC:Coinbase-Pro -0.2 BTC {}

Beancount's booking code + FIFO method + inferences would resolve that into something like:

Assets:Crypto:BTC:Binance 0.2 BTC {19000 EUR}

Assets:Crypto:BTC:Coinbase-Pro -0.1 BTC {18000 EUR}

Assets:Crypto:BTC:Coinbase-Pro -0.1 BTC {20000 EUR}

And then a plugin could rewrite that into the original lots:

Assets:Crypto:BTC:Binance     0.1 BTC {18000 EUR}

Assets:Crypto:BTC:Binance     0.1 BTC {20000 EUR}

Assets:Crypto:BTC:Coinbase-Pro -0.1 BTC {18000 EUR}

Assets:Crypto:BTC:Coinbase-Pro -0.1 BTC {20000 EUR}

On Tue, Dec 22, 2020 at 1:25 PM Tobias H <tobias....@gmail.com> wrote:

Is there a way to automatically transfer lots from one account to another using FIFO and keeping the prices paid?

E.g.:

I bought `0.13833333 BTC` for `2500 EUR` and `0.125 BTC` for `2500 EUR` on Coinbase. I transferred `0.2 BTC` to Binance. I sold all `0.2 BTC` for `3500 EUR` on Binance.

*(Which will actually become a swap to another coin. But for taxes I need to account an assumed BTC-EUR sell and an assumed X-Token buy. But since I got that lot issue the final step is omitted.)*

option "booking_method" "FIFO"

option "operating_currency" "EUR"

option "inferred_tolerance_default" "EUR:0.001"

option "inferred_tolerance_default" "BTC:0.00000001"

2020-12-01 open Equity:Opening-Account

2020-12-01 open Assets:Cash:Coinbase-Pro

2020-12-01 open Assets:Cash:Binance

2020-12-01 open Assets:Crypto:BTC:Coinbase-Pro

2020-12-01 open Assets:Crypto:BTC:Binance

2020-12-01 open Expenses:Commission:Coinbase-Pro:EUR

2020-12-01 open Income:Day-trading:EUR

2020-12-17 * "Coinbase" "Opening"

Assets:Cash:Coinbase-Pro 5000 EUR

Equity:Opening-Account -5000 EUR

2020-12-17 * "Coinbase" "Trade EUR => BTC"

Assets:Cash:Coinbase-Pro -2500 EUR

Assets:Crypto:BTC:Coinbase-Pro 0.13833333 BTC {18,000 EUR} @ 18,000 EUR

Expenses:Commission:Coinbase-Pro:EUR 10 EUR

2020-12-18 * "Coinbase" "Trade EUR => BTC"

Assets:Cash:Coinbase-Pro -2500 EUR

Assets:Crypto:BTC:Coinbase-Pro 0.125 BTC {20,000 EUR} @ 20,000 EUR

Expenses:Commission:Coinbase-Pro:EUR 0 EUR

2020-12-19 * "Binance" "Transfer BTC Coinbase-Pro => Binance"

Assets:Crypto:BTC:Binance 0.2 BTC

Assets:Crypto:BTC:Coinbase-Pro -0.2 BTC

2020-12-20 * "Binance" "Trade BTC => USDT"

Assets:Crypto:BTC:Binance -0.2 BTC {} @ 17,500 EUR

Assets:Cash:Binance 3,500 EUR

Income:Day-trading:EUR

The error I'm getting is:

>$ bean-check Krypto.bean

>

>Krypto.bean:38: No position matches "Posting(account='Assets:Crypto:BTC:Binance', units=-0.2 BTC, cost=CostSpec(number_per=<class 'beancount.core.number.MISSING'>, number_total=None, currency='EUR', date=None, label=None, merge=False), price=17500 EUR, flag=None, meta={'filename': 'Krypto.bean', 'lineno': 39})" against balance (0.2 BTC)

>

> 2020-12-20 * "Binance" "Trade BTC => USDT"

> Assets:Crypto:BTC:Binance -0.2 BTC {} @ 17500 EUR

> Assets:Cash:Binance 3500 EUR

> Income:Day-trading:EUR

That's probably because the lot information isn't passed in the 2020-12-19 transfer post.

`--------------------`

According to the manual keeping the lot information could be achieved writing the transfer post like this:

2020-12-16 * "Binance" "Transfer BTC Coinbase-Pro => Binance"

Assets:Crypto:BTC:Binance 0.13833333 BTC {18,000 EUR} ; creates a lot manually. Not FIFO safe

Assets:Crypto:BTC:Coinbase-Pro -0.13833333 BTC {} ; selects the lot automatically. Therefore, Assets:Crypto:BTC:Coinbase-Pro is FIFO safe

But using this means I'm creating a lot in the second account `Assets:Crypto:BTC:Binance` manually. So, I'd need to look up which is/are the lot(s) that get transferred. Is there a syntax to solve automatically? Doing this manually would be cumbersome and error-prone. Dealing with cryptos many transfers between exchanges and privately owned wallets are a common thing. To keep track of them is crucial for any tax declaration.

I think this is related to:

* https://groups.google.com/g/beancount/c/gecmdDlrxWc/m/qRTJniqQAQAJ

* https://groups.google.com/g/beancount/c/Qvbcq-Sk5NY

* https://github.com/beancount/beancount/issues/168

`--------------------`

In order to do the FIFO manually I need to look up/query my lots, but I'm unable to do this. `bean-report FILE holdings` seemed appropriate, but the lot date is missing and the units are trimmed/rounded to two digits:

>$ bean-report Krypto.bean holdings

>

>Account Units Currency Cost Currency Average Cost Price Book Value Market Value

>------------------------------ ----- -------- ------------- ------------ ----- ---------- ------------

>Assets:Crypto:BTC:Binance 0.20 BTC BTC 0.20 0.20

Assets:Crypto:BTC:Coinbase-Pro 0.14 BTC EUR 18,000.00 2,490.00

Assets:Crypto:BTC:Coinbase-Pro 0.12 BTC EUR 20,000.00 2,500.00

Assets:Crypto:BTC:Coinbase-Pro -0.20 BTC BTC -0.20 -0.20

>------------------------------ ----- -------- ------------- ------------ ----- ---------- ------------

I also tried `bean-query`. I could add the date, but the unit size is rounded as well:

>$ bean-query Krypto.bean 'SELECT account, date, position WHERE currency="BTC" ORDER BY account'

>

> account date position

>------------------------------ ---------- --------------------

>Assets:Crypto:BTC:Binance 2020-12-19 0.2 BTC

>Assets:Crypto:BTC:Coinbase-Pro 2020-12-17 0.1 BTC {18000 EUR}

>Assets:Crypto:BTC:Coinbase-Pro 2020-12-18 0.1 BTC {20000 EUR}

>Assets:Crypto:BTC:Coinbase-Pro 2020-12-19 -0.2 BTC

I'd like to have an output with the colums:

account, lot_date, lot_units (using 8 decimal digits), lot_cost (using 8 decimal digits)

Lots which has been transferred/sold should be omitted, of course. (How) Is this possible?

Regards.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/4242c558-c761-4fcb-8703-6b17b9b2a4e5n%40googlegroups.com.

[Martin Blais]

Short answer is: no, but that's quickly becoming a FAQ from crypto users.

It could be designed, but that would be a new feature.

[Justus Pendleton]

On Wednesday, December 23, 2020 at 11:03:44 AM UTC+7 bl...@furius.ca wrote:

Short answer is: no, but that's quickly becoming a FAQ from crypto users.

It could be designed, but that would be a new feature.

I only use Spec ID booking so I might be missing something but isn't moving lots just a small 5 or 6 line python script iterating over beancount.ops.holdings.get_final_holdings? Would this work or is it more complicated? I'm not sure that a plugin is simpler/clearer than just generating the transaction externally and then inserting it into the beancount file.

holdings = beancount.ops.holdings.get_final_holdings(entries, included_account_types=('Assets',))
print(f'{datetime.date.today()} * "ACATS" "Autogenerated lot transfer to new account"')
for h in filter(match, holdings):
   cost_date = None
   print(f' {h.account} -{h.number} {h.currency} {{ {cost_date}, {h.cost_number} {h.cost_currency} }}')
   print(f' {args.destination} {h.number} {h.currency} {{ {cost_date}, {h.cost_number} {h.cost_currency} }}') 

The only problem I see is that get_final_holdings returns a Holding tuple that doesn't include the acquisition date. But there's a note in the code saying the Holding tuple should go away and just be replaced by a Position, which would hold the acquisition date.

[Martin Blais]

On Wed, Dec 23, 2020 at 11:11 PM Justus Pendleton <jus...@ryoohki.net> wrote:

On Wednesday, December 23, 2020 at 11:03:44 AM UTC+7 bl...@furius.ca wrote:

Short answer is: no, but that's quickly becoming a FAQ from crypto users.

It could be designed, but that would be a new feature.

I only use Spec ID booking so I might be missing something but isn't moving lots just a small 5 or 6 line python script iterating over beancount.ops.holdings.get_final_holdings? Would this work or is it more complicated? I'm not sure that a plugin is simpler/clearer than just generating the transaction externally and then inserting it into the beancount file.

There's that. OTOH if this is super common for crypto, I'm thinking we should design it in and make it convenient in the input syntax to make such transfers.

In theory it *could* be done with a plugin today as Ben suggest: I imagined it more like this though:

2020-12-19 T "Binance" "Transfer BTC Coinbase-Pro => Binance"

Assets:Crypto:BTC:Binance 0.2 BTC {}

Assets:Crypto:BTC:Coinbase-Pro 

whereby the plugin would know to select transactions with flag 'T' (or perhaps a tag given in its config, whatever) and convert the posting on Assets:Crypto:BTC:Coinbase-Pro to include all the original matched reductions. I think this should be doable today actually.

holdings = beancount.ops.holdings.get_final_holdings(entries, included_account_types=('Assets',))
print(f'{datetime.date.today()} * "ACATS" "Autogenerated lot transfer to new account"')
for h in filter(match, holdings):
   cost_date = None
   print(f' {h.account} -{h.number} {h.currency} {{ {cost_date}, {h.cost_number} {h.cost_currency} }}')
   print(f' {args.destination} {h.number} {h.currency} {{ {cost_date}, {h.cost_number} {h.cost_currency} }}') 

The only problem I see is that get_final_holdings returns a Holding tuple that doesn't include the acquisition date. But there's a note in the code saying the Holding tuple should go away and just be replaced by a Position, which would hold the acquisition date.

beancount.ops.holdings is gone in v3. It was superfluous anyway (well when I started it I wasn't sure if it would start acquiring more features than just an inventory). Now you can simply aggregate over an inventory and enumerate its contents (positions).

[Martin Blais]

Here, that's what I meant:

https://github.com/beancount/beancount/commit/9e6e8f8c3793982a6de5af5321e3026d326f8170

This plugin does what you want (I think).

[Martin Blais]

Here, refined further to show how it's leveraging the reduction spec:

https://github.com/beancount/beancount/blob/master/experiments/plugins/transfer_lots_test.py

Maybe this should be part of the builtin plugins.

Love my little plugin system

[David Terry]

Hey,

Just getting started with beancount and I ran into this problem when entering some crypto trades. I tried out the transfer_lots plugin in the beanlabs repo, and I ran into some issues that I don't really understand. I can attach cost basis and buy and sell lots within one account without issue (see here), however when I transfer some assets to a new account and then attempt to see them, I'm running into problems (example file).

Once there is a transfer involved, beancount no longer seems to reduce lots according to FIFO, and instead seems to attach a new cost basis to the removed lots based on the price at the time of the sale. The output from running `bean-doctor context` against the last trade in the problematic file is below. You can see that the sale has resulted in an extra 120 lots @ 4EUR being deducted from the inventory. Is this a bug, or am I using the plugin wrong? 

------------ Balances before transaction

  Assets:Binance:XYZ                        100 XYZ {2 EUR, 2020-12-25}

  Assets:Binance:XYZ                         50 XYZ {3 EUR, 2020-12-25}

  Assets:Binance:EUR                                                   

------------ Transaction

2020-12-25 * "Binance" "Sell 120 XYZ"

  Assets:Binance:XYZ  -120 XYZ {4 EUR, 2020-12-25}  ; -480 EUR

  Assets:Binance:EUR   480 EUR                      ;  480 EUR

Basis: (-480 EUR)

------------ Balances after transaction

  Assets:Binance:XYZ                        100 XYZ {2 EUR, 2020-12-25}

  Assets:Binance:XYZ                         50 XYZ {3 EUR, 2020-12-25}

* Assets:Binance:XYZ                       -120 XYZ {4 EUR, 2020-12-25}

* Assets:Binance:EUR                                            480 EUR

[redst...@gmail.com]

This is interesting. From what I can tell:

• Beancount's loader proceeds thus: parse -> booking+interpolation -> plugins -> validate.
• Therefore, when booking+interpolation is run on your parsed source, the transfer plugin has not had a chance to transform the entries and transfer the lots.
• This means, that the reduction in your source (-120 XYZ {}) can't be done because there are no lots to reduce at booking+interpolation. Therefore, it's interpreted as an augmentation (aka buying a short position). (Side note: your "sell" transaction should look like "-120 XYZ {} @@ 480 EUR", or " -120 XYZ {} @ 4 EUR", so it's a reduction, and not an augmentation,  and should include an "Income:" posting to capture gain/loss, which you'll need to do anyway once this problem is fixed)

AFAICT, this means that the transfer lots plugin won't work without some rethinking of the loader, perhaps allowing for pre-booking plugins. Martin may have other ideas or workarounds though.

[Martin Blais]

Ahh yes, indeed. That's a bit of a problem. I don't have a good solution for this yet.

Simply providing parser-level plugin functionality wouldn't work here, because the information for the postings to be transferred is filled in by the interpolation code. This is a feature that would need to be applied in the booking code, e.g., apply the inventory updates based on the result of the transfers.

If this is the defining/key feature that enables working with crypto currencies, we could consider supporting this explicitly in the core booking algos (in v3, not touching v2 much anymore). 

Another idea would be to create a new kind of hook, one that is registered from a plugin, e.g. a callback of yours invoked by the booking code itself, and whose results applied to a transaction are immediately reflected on the state of the affected inventories. Maybe this is the right place to do this (I've never had this idea before.) Now, imagine generalizing this further to provide and implement all opf the current booking mechanisms that are currently built in the core. Call this "customizable booking."

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/58b12132-c3cd-401b-98e5-3035d034846dn%40googlegroups.com.

[redst...@gmail.com]

That makes sense. I was thinking of a system where plugin/booking/interpolation iterate over the same entries until no more modifications occur. This would involve some thought to prove (a) commutativity (order doesn't matter), and (b) convergence (no infinite iterations).

Reg. the other approach -- i.e., supporting this in core booking algos: even outside crypto, isn't the philosophy you've put forth "works on unambiguous source"? Given that, is there a syntax that removes ambiguity? For example:

2020-01-01 * "Transfer"

   Asset:BrokerageA -10 HOOLI {}

   Asset:BrokerageB: 10 HOOLI {}

might be unambiguous for FIFO, LIFO, and STRICT, and arguably for NONE (and AVG in the future). I.e., identical CostSpec after inverting the sign of one. I haven't thought deeply about all cases, and anyway, not the most important thing for v3.

[redst...@gmail.com]

BTW, David: as you can see, transfers work fine when fully specified, so this is a matter of convenience. I personally have a vim plugin that uses bean-doctor context to insert the lots.

Curious: is there anything specific to crypto that makes these transfers common?

[Martin Blais]

On Wed, Dec 30, 2020 at 12:55 AM redst...@gmail.com <redst...@gmail.com> wrote:

That makes sense. I was thinking of a system where plugin/booking/interpolation iterate over the same entries until no more modifications occur. This would involve some thought to prove (a) commutativity (order doesn't matter), and (b) convergence (no infinite iterations).

"Iterate over the same entry until no more modification occurs" seems error prone to me, and a potential nightmare for debugging.

Reg. the other approach -- i.e., supporting this in core booking algos: even outside crypto, isn't the philosophy you've put forth "works on unambiguous source"? Given that, is there a syntax that removes ambiguity? For example:

2020-01-01 * "Transfer"

   Asset:BrokerageA -10 HOOLI {}

   Asset:BrokerageB: 10 HOOLI {}

might be unambiguous for FIFO, LIFO, and STRICT, and arguably for NONE (and AVG in the future). I.e., identical CostSpec after inverting the sign of one. I haven't thought deeply about all cases, and anyway, not the most important thing for v3.

I'm not sure I understand what you mean by "works on unambiguous source", but the essence of the issue is that booking and interpolation are a function of the transaction input AND the state of the inventories to which it is applied beforehand. Providing a hook to modify the output of parsing doesn't provide information that is inferred from the inventories. I'm proposing that deep within the booking code is probably a better place to let the user carry out customizations, providing the parsed transacttion (with missing bits and pieces) and the accumulated state of the inventories. 

Actually... if you allow the user to invoke the booking explicitly in their plugin... well they can then do modifications on the finalized transaction, just like the regular plugins do. Hmm, this *might* be sufficient to completely replace the current plugin system I provide, where you get the full stream of entries in and output modified ones...

 

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/682270b8-812a-4098-a8fc-fcc4655737f9n%40googlegroups.com.

[Martin Blais]

Also: I'd love to gather a set of features that are key to making Beancount more usable for cryptocurrency trading.

Here's a doc where you can insert ideas:

https://docs.google.com/document/d/1taN9lbcNDf8bKgDwprWOhuaOsOgALZzmsfvec-rdaSk/edit#

(FWIW, one of my 2021 goals is to start to dabble with these myself, after years of reticence. Reading the OReilly book on it right now actually. I need Coinbase Pro & Binance importers.)

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/2431d4ee-e998-4260-b98e-a8f801d71cd4n%40googlegroups.com.

[redst...@gmail.com]

On Tuesday, December 29, 2020 at 10:02:15 PM UTC-8 bl...@furius.ca wrote:

On Wed, Dec 30, 2020 at 12:55 AM redst...@gmail.com <redst...@gmail.com> wrote:

That makes sense. I was thinking of a system where plugin/booking/interpolation iterate over the same entries until no more modifications occur. This would involve some thought to prove (a) commutativity (order doesn't matter), and (b) convergence (no infinite iterations).

"Iterate over the same entry until no more modification occurs" seems error prone to me, and a potential nightmare for debugging.

Agreed. Although I've seen it work very well in systems where the key was to identify the constraints to make it work predictably.

 

Reg. the other approach -- i.e., supporting this in core booking algos: even outside crypto, isn't the philosophy you've put forth "works on unambiguous source"? Given that, is there a syntax that removes ambiguity? For example:

2020-01-01 * "Transfer"

   Asset:BrokerageA -10 HOOLI {}

   Asset:BrokerageB: 10 HOOLI {}

might be unambiguous for FIFO, LIFO, and STRICT, and arguably for NONE (and AVG in the future). I.e., identical CostSpec after inverting the sign of one. I haven't thought deeply about all cases, and anyway, not the most important thing for v3.

I'm not sure I understand what you mean by "works on unambiguous source",

What I mean is: even if a CostSpec if incompletely specified, as long as it is unambiguous beancount will process it correctly. For example: there's no need to specify date in a cost specification as long as the price is adequate to uniquely identify the lot. Along those lines, I was making the argument that the transaction above is unambiguous in saying "transfer all lots from BrokerageA to BrokerageB," and thus, it would be nice for the core booking algos to handle it correctly rather than depend on a plugin.

[Martin Blais]

Yes

The challenge is to design those things to be general. I think in this case the addition could be as simple as honoring a special flag on an interpolation posting, telling the interpolation code not to convert to cost.

 

 

 

[David Terry]

Thanks for the detailed answers!!

> BTW, David: as you can see, transfers work fine when fully specified, so this is a matter of convenience. I personally have a vim plugin that uses bean-doctor context to insert the lots.

It seems to me that it's more than a matter of convenience. If the reductions / augmentations are explicitly specified, the booking will be potentially incorrect (i.e. no longer respect FIFO) if transactions that change the state of the inventory are subsequently added to the ledger with a date before that of the transfer.

> Curious: is there anything specific to crypto that makes these transfers common?

Transferring funds between institutions / accounts is very common when working with crypto. For example, it is not generally considered prudent to leave crypto custodied at a centralised exchange, so many users will transfer their assets into their own custody directly after having made a trade. As another example, users of DeFi applications will often move their assets between many different institutions (smart contracts) as the yields offered to depositors change.

> If this is the defining/key feature that enables working with crypto currencies, we could consider supporting this explicitly in the core booking algos (in v3, not touching v2 much anymore)

As mentioned above, these workflows are very common. I would certainly be very happy if these workflows were supported in the core booking algorithms.

> Also: I'd love to gather a set of features that are key to making Beancount more usable for cryptocurrency trading.

> Here's a doc where you can insert ideas:

> https://docs.google.com/document/d/1taN9lbcNDf8bKgDwprWOhuaOsOgALZzmsfvec-rdaSk/edit#

Very happy to hear that you're interested in working to make beancount more friendly for crypto users. I'll keep playing around and see if I can find some other pain points :)

[Ghanashyam Prabhu]

I had a similar use case here and ended up using the plugin to report the transactions and then copied them manually into the transfer posting 

This is my entries list. However I see that when I run bean-check (v2), it errors out with an error 

No position matches "Posting(account='Assets:US:Crypto:Coinbase:BTC', units=-0.20778508 BTC, cost=CostSpec(number_per=Decimal('18078.87'), number_total=None, currency='USD', date=datetime.date(2020, 11, 28), label=None, merge=False), price=None, flag=None, meta={

Do you know why it complains on No Matching position when the Cost basis are exactly the same?

2020-11-28 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC           0.20778508 BTC {} @ 18078.87 USD

  Assets:US:Crypto:Coinbase:Cash            -3500.00 USD

  Assets:US:BofA:Checking                    -256.48 USD

2020-12-17 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC           0.02109060 BTC {} @ 23707.24 USD

  Assets:US:BofA:Checking                    -500.00 USD

2020-12-26 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC            0.01977443 BTC {} @ 25285.18 USD

  Assets:US:BofA:Checking                    -500.00 USD

2020-12-30 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC            0.01741186 BTC {} @ 28716.06 USD

  Assets:US:BofA:Checking                    -500.00 USD

2021-01-01 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC            0.01667888 BTC {} @ 29978.03 USD

  Assets:US:BofA:Checking                    -500.00 USD

2021-01-03 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC            0.01464422 BTC {} @ 34143.16 USD

  Assets:US:BofA:Checking                    -500.00 USD

2021-01-04 balance Assets:US:Crypto:Coinbase:BTC 0.29738506 BTC

2021-01-04 * "Transfer BTC from Coinbase to CoinbasePro"

  Assets:US:Crypto:Coinbase:BTC          -0.20778508 BTC {18078.87 USD, 2020-11-28}

  Assets:US:Crypto:Coinbase:BTC          -0.02109060 BTC {23707.24 USD, 2020-12-17}

  Assets:US:Crypto:Coinbase:BTC          -0.01977443 BTC {25285.18 USD, 2020-12-26}

  Assets:US:Crypto:Coinbase:BTC          -0.01741186 BTC {28716.06 USD, 2020-12-30}

  Assets:US:Crypto:Coinbase:BTC          -0.01667888 BTC {29978.03 USD, 2021-01-01}

  Assets:US:Crypto:Coinbase:BTC          -0.01464422 BTC {34143.16 USD, 2021-01-03}

  Assets:US:Crypto:CoinbasePro:BTC        0.20778508 BTC {18078.87 USD, 2020-11-28}

  Assets:US:Crypto:CoinbasePro:BTC        0.02109060 BTC {23707.24 USD, 2020-12-17}

  Assets:US:Crypto:CoinbasePro:BTC        0.01977443 BTC {25285.18 USD, 2020-12-26}

  Assets:US:Crypto:CoinbasePro:BTC        0.01741186 BTC {28716.06 USD, 2020-12-30}

  Assets:US:Crypto:CoinbasePro:BTC        0.01667888 BTC {29978.03 USD, 2021-01-01}

  Assets:US:Crypto:CoinbasePro:BTC        0.01464422 BTC {34143.16 USD, 2021-01-03}

[Martin Blais]

This looks all wrong, see other thread.

To buy BTC at Coinbase, the money all comes from your Coinbase:Cash account.

Transfers from your bank are separate transactions.

Reflect what's actually going on in the account

--

You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/bd63fee9-2635-4a7f-9d2f-c6be0ab723edn%40googlegroups.com.

[Ghanashyam Prabhu]

Hi Martin, Apologies if this was confusing. Can you elaborate on what is wrong? 

I have added comments to my transactions for your reference - Note the points till which bean-check shows no errors. 

Let me comment the individual transactions. 

;; This transaction purchases BTC at Coinbase from Cash existing in Coinbase account

;; The $3500 already existed and is correct from from the previous transactions. 

;; Also this transaction requires additional $256.48 which is used from my Checking account 

;; There are previous transactions 

;; which I have not listed here.. For our discussion, consider that the transaction starts here with 

2020-11-28 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC           0.20778508 BTC {} @ 18078.87 USD

  Assets:US:Crypto:Coinbase:Cash            -3500.00 USD

  Assets:US:BofA:Checking                    -256.48 USD

;; All the below transactions are purchases by using cash from the checking account

2020-12-17 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC           0.02109060 BTC {} @ 23707.24 USD

  Assets:US:BofA:Checking                    -500.00 USD

2020-12-26 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC            0.01977443 BTC {} @ 25285.18 USD

  Assets:US:BofA:Checking                    -500.00 USD

2020-12-30 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC            0.01741186 BTC {} @ 28716.06 USD

  Assets:US:BofA:Checking                    -500.00 USD

2021-01-01 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC            0.01667888 BTC {} @ 29978.03 USD

  Assets:US:BofA:Checking                    -500.00 USD

2021-01-03 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC            0.01464422 BTC {} @ 34143.16 USD

  Assets:US:BofA:Checking                    -500.00 USD

;; I've added a balance assertion here to sum up the BTC accumulated 

;; because of the above purchases

;; A bean-check until this transaction inclusive of the balance assertion 

;; does NOT show any errors.

2021-01-04 balance Assets:US:Crypto:Coinbase:BTC 0.29738506 BTC

;; The next transaction I want is a transfer transaction from Coinbase to CoinbasePro

;; I initially added the below 

2021-01-04 * "Transfer BTC from Coinbase to CoinbasePro" #transfer

  Assets:US:Crypto:Coinbase:BTC           -0.29738506 BTC

  Assets:US:Crypto:CoinbasePro:BTC        0.29738506 BTC

;; I now added a balance assertion again to check if the transfer is complete

;; and if the balance in CoinbasePro is correct

2021-01-05 balance Assets:US:Crypto:CoinbasePro:BTC 0.29738506 BTC ;; to check transfer complete from Coinbase to CoinbasePro

;; bean-check runs without any error until this point 

;; which means the transfer check is correct even at CoinbasePro account

Now, at this point I want to add a sell transaction 

2021-01-08 * "" "Coinbase dummy sell"

   Assets:US:Crypto:CoinbasePro:BTC       -0.20295107 BTC {} @ 34860.28 USD

   Assets:US:Crypto:CoinbasePro:Cash          7074.93 USD

   Expenses:US:Crypto:CoinbasePro:Fees          35.37 USD

   Income:Capital-gains

When I added the above sell transaction and run bean-check I get the following error

No position matches "Posting(account='Assets:US:Crypto:CoinbasePro:BTC', units=-0.20295107 BTC, cost=CostSpec(number_per=<class 'beancount.core.number.MISSING'>, number_total=None, currency='USD', date=None, label=None, merge=False), price=34860.28 USD, flag=None, meta={file} against balance (0.29738506 BTC)

Ghanashyam

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhOOPwB%3DbQe5GHdtiaZpEUYpYSZsp_Z1D124r0k47eSXZA%40mail.gmail.com.

[Ben Blount]

Beancount has two ways to track any kind of currency/commodity: at cost, or not at cost.

Generally, currency like USD, EUR, etc are not tracked at cost. Stock, securities etc are tracked at cost so you can report capital gains.

Beancounts infers what mode you are working in by the presence of a CostSpec { } when working with that lot (both for reducing and augmenting lots).

It's actually possible for the same account to hold a commodity both at cost, and not at cost. This is strongly not recommended, and beancount.plugins.coherent_cost can be used to verify you don't do it.

That plugin would show you what is wrong here. It's:

2021-01-04 * "Transfer BTC from Coinbase to CoinbasePro" #transfer

  Assets:US:Crypto:Coinbase:BTC           -0.29738506 BTC

  Assets:US:Crypto:CoinbasePro:BTC        0.29738506 BTC

You are trying to work with the BTC in the non-cost-tracking mode. You should at minimum add {}

2021-01-04 * "Transfer BTC from Coinbase to CoinbasePro" #transfer

  Assets:US:Crypto:Coinbase:BTC           -0.29738506 BTC {}

  Assets:US:Crypto:CoinbasePro:BTC        0.29738506 BTC {}

But really what you likely actually want is to recreate all the original lots, just in a new account. That's the original subject of this thread.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CABieCET5KOR5H2LqXBELTWnz4UiD0FEb%3DOYJ8uR6BqOM%2BGT%2BCA%40mail.gmail.com.

[Ghanashyam Prabhu]

Hi Ben, Thanks for pointing out that we'd need {} at a minimum. 

Note that adding this to both the postings shows an error when I run bean-check

Error message is here

Failed to categorize posting 2

   2021-01-04 * "CoinbasePro" "Tranfer BTC from Coinbase to CoinbasePro" #transfer

     Assets:US:Crypto:Coinbase:BTC     -0.29738506 BTC {}

     Assets:US:Crypto:CoinbasePro:BTC   0.29738506 BTC {}

In order to fix this. instead of using #transfer and the transfer plugin I used the below transaction with individual cost basis added to each transaction

2021-01-04 * "Tranfer BTC from Coinbase to CoinbasePro"

  Assets:US:Crypto:Coinbase:BTC          -0.20778508 BTC {18078.87 USD, 2020-11-28}
  Assets:US:Crypto:Coinbase:BTC          -0.02109060 BTC {23707.24 USD, 2020-12-17}
  Assets:US:Crypto:Coinbase:BTC          -0.01977443 BTC {25285.18 USD, 2020-12-26}
  Assets:US:Crypto:Coinbase:BTC          -0.01741186 BTC {28716.06 USD, 2020-12-30}
  Assets:US:Crypto:Coinbase:BTC          -0.01667888 BTC {29978.03 USD, 2021-01-01}
  Assets:US:Crypto:Coinbase:BTC          -0.01464422 BTC {34143.16 USD, 2021-01-03}
  Assets:US:Crypto:CoinbasePro:BTC        0.20778508 BTC {18078.87 USD, 2020-11-28}
  Assets:US:Crypto:CoinbasePro:BTC        0.02109060 BTC {23707.24 USD, 2020-12-17}
  Assets:US:Crypto:CoinbasePro:BTC        0.01977443 BTC {25285.18 USD, 2020-12-26}
  Assets:US:Crypto:CoinbasePro:BTC        0.01741186 BTC {28716.06 USD, 2020-12-30}
  Assets:US:Crypto:CoinbasePro:BTC        0.01667888 BTC {29978.03 USD, 2021-01-01}
  Assets:US:Crypto:CoinbasePro:BTC        0.01464422 BTC {34143.16 USD, 2021-01-03}

Using this too shows another error when I run bean-check 

Error message indicates 

No position matches "Posting(account='Assets:US:Crypto:Coinbase:BTC', units=-0.20778508 BTC, cost=CostSpec(number_per=Decimal('18078.87'), number_total=None, currency='USD', date=datetime.date(2020, 11, 28), label=None, merge=False), price=None, flag=None, meta={'filename': '<filename>', 'lineno': 112})" against balance (0.20778508 BTC {18078.68014392563700916350683 USD, 2020-11-28}, 0.02109060 BTC {23707.24398547220088570263530 USD, 2020-12-17}, 0.01977443 BTC {25285.17889011212965430609125 USD, 2020-12-26}, 0.01741186 BTC {28716.05905400112337223019252 USD, 2020-12-30}, 0.01667888 BTC {29978.03209807852805464155867 USD, 2021-01-01}, 0.01464422 BTC {34143.16365091483192686261201 USD, 2021-01-03})

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CACGEkZsLWdkMLC0LTHy6HuFkfDzGzv2MHwJP0AX6Aje%2B-7Te0w%40mail.gmail.com.

[Ben Blount]

Please carefully review the error message. It has been telling you precisely what is wrong each time, though it does require some understanding which I think you now have.

This time the lot is not matching exactly.

-0.20778508 BTC, cost=CostSpec(number_per=Decimal('18078.87'  ... USD

You can see the balance it's trying to match against in the same error:

0.20778508 BTC {18078.68014392563700916350683 USD, 2020-11-28}, 

An exact match is required. I have a pull request open that would've made this work as you expected it to. Note, I don't know if Martin agrees with me that lot matching should accept small variations, so that pull request might not be merged.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CABieCES1xujaP%2BqZbMFHBXAMnknOpYcOJkGgdcBf3VGFPW8vDg%40mail.gmail.com.

[Ben Blount]

One more thing to add, if you want to match the lot with 2 digits of precision, try to specify it that way when you first create the lot. Unfortunately sometimes that means you can't use expressions that end up generating lots of digits or else you'll end up with a reduction like  -0.20778508 BTC {18078.68014392563700916350683 USD}.

Alternatively, Beancount does let you match using the date, assuming that is unambiguous. 

[Ghanashyam Prabhu]

I did note the precision from the error message 

Are you suggesting using something like 

option "inferred_tolerance_default" "USD:0.001" 

I have it already - with and without this makes no difference 

The alternative method of using only dates, is it something like this below? This too shows an error with bean-check but the error message isn't that informative though

Failed to categorize posting 7 

2021-01-04 * "Transfer BTC from Coinbase to CoinbasePro"

  Assets:US:Crypto:CoinbasePro:BTC        0.20778508 BTC {2020-11-28} ;;;; 18078.87 USD,
  Assets:US:Crypto:Coinbase:BTC          -0.20778508 BTC {2020-11-28} ;;;; 18078.87 USD,
  Assets:US:Crypto:CoinbasePro:BTC        0.02109060 BTC {2020-12-17} ;;;; 23707.24 USD,
  Assets:US:Crypto:CoinbasePro:BTC        0.01977443 BTC {2020-12-26} ;;;; 25285.18 USD,
  Assets:US:Crypto:CoinbasePro:BTC        0.01741186 BTC {2020-12-30} ;;;; 28716.06 USD,
  Assets:US:Crypto:CoinbasePro:BTC        0.01667888 BTC {2021-01-01} ;;;; 29978.03 USD,
  Assets:US:Crypto:CoinbasePro:BTC        0.01464422 BTC {2021-01-03} ;;;; 34143.16 USD,
  Assets:US:Crypto:Coinbase:BTC          -0.02109060 BTC {2020-12-17} ;;;; 23707.24 USD,
  Assets:US:Crypto:Coinbase:BTC          -0.01977443 BTC {2020-12-26} ;;;; 25285.18 USD,
  Assets:US:Crypto:Coinbase:BTC          -0.01741186 BTC {2020-12-30} ;;;; 28716.06 USD,
  Assets:US:Crypto:Coinbase:BTC          -0.01667888 BTC {2021-01-01} ;;;; 29978.03 USD,
  Assets:US:Crypto:Coinbase:BTC          -0.01464422 BTC {2021-01-03} ;;;; 34143.16 USD,

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CACGEkZszOgBvRfmfzv5ymuNXyxqXJ-9BmFdKH4yWL9MmTmx2Dg%40mail.gmail.com.

[Ben Blount]

Beancount Precision & Tolerances

Beancount can be smart about finding the original lot when reducing. If you do it by price it must be exact. If you do it by date, it must be exact. Either way it must be unambiguous in the end. For multiple lots in a single day you'd need to provide both cost and date. Tolerances and precision are not in any way involved when matching lots.

Please look carefully at your transaction, it makes no sense. Remember Beancount has no idea that you are moving the same lot to CoinbasePro so trying to put a posting like this will not work:

Assets:US:Crypto:CoinbasePro:BTC        0.20778508 BTC {2020-11-28}.

This has no cost attached to it.

Please review the docs provided in earlier replies, I think they will help you if you take the time to grok them fully.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CABieCEQ7YOVhkoG7jDB%2BS8D51MR3fKieDv-6Eu19RVSPQSRRuA%40mail.gmail.com.

[Ghanashyam Prabhu]

Thanks Ben, I was able to go through this documentation and made some updates to the file. 

I wasn't documenting the Buy Posts correctly. I now have it such that it adds a cost basis with the date at buy time. With this I was able to even write the transfer as well and bean-check runs clean!

Reference here for folks who visit this thread. (posted a couple of transactions)

Thanks

Ghanashyam

2020-11-28 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC           0.20778508 BTC {18078.87 USD, 2020-11-28}
  Assets:US:Crypto:Coinbase:Cash            -3756.52 USD

2020-12-17 * "Coinbase" "Buy BTC at Coinbase"

  Assets:US:Crypto:Coinbase:BTC           0.02109060 BTC {23707.24 USD, 2020-12-17}

  Assets:US:BofA:Checking                    -500.00 USD

2021-01-04 * "Tranfer BTC from Coinbase to CoinbasePro"

  Assets:US:Crypto:CoinbasePro:BTC        0.20778508 BTC {18078.87 USD, 2020-11-28}

  Assets:US:Crypto:Coinbase:BTC          -0.20778508 BTC {18078.87 USD, 2020-11-28}

  Assets:US:Crypto:CoinbasePro:BTC        0.02109060 BTC {23707.24 USD, 2020-12-17}

  Assets:US:Crypto:Coinbase:BTC          -0.02109060 BTC {23707.24 USD, 2020-12-17}
  

You received this message because you are subscribed to a topic in the Google Groups "Beancount" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beancount/vDX1oA2mJXA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CACGEkZuPp55sqs5Ei5jRtUXRNx5he2nts_CE-ZjhXq82yRs%2BPw%40mail.gmail.com.

[Martin Blais]

This "transfer with cost" problem is becoming a FAQ for crypto users.

We're going to have to implement something automatic eventually.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CABieCERk8rcwab2m56u3wdNvyycp_zFuoSLHZpPVO8TkyVObCA%40mail.gmail.com.

[Martin Blais]

https://github.com/beancount/beancount/issues/614

[sm...@davethehappysinger.com]

Apologies for bumping an old thread with an off-topic question, but Redstreet, you said:

> I personally have a vim plugin that uses bean-doctor context to insert the lots.

This is really interesting; I wonder if this is something that's openly available or if it's a personal hack whether you'd be interested in sticking it on Github?

[redst...@gmail.com]

On Friday, June 4, 2021 at 8:40:54 PM UTC-7 sm...@davethehappysinger.com wrote:

Apologies for bumping an old thread with an off-topic question, but Redstreet, you said:

> I personally have a vim plugin that uses bean-doctor context to insert the lots.

This is really interesting; I wonder if this is something that's openly available or if it's a personal hack whether you'd be interested in sticking it on Github?

For conversions, it's trivial: it's a 2 or 3 line python script that scales the units and cost fields. When I have an imported transaction that looks like:

2000-01-01 * "Upgrade shares"

  Assets:XTrade:ABC -234.15 ABC

  Assets:XTrade:DEF 175.45 DEF

Then what I do is: get beancount context (I've mapped this to '\g' in vim-beancount), which opens the context in a split. I then select (visual mode) all the ABC lots, and pipe it through the python script. An equivalent awk script would look roughly like:

:'<,'>!awk '{print $1, -$2*(175.45/234.15), $3, $4, $5*(234.15/175.45), $6, $7}'

My python script just made this easier by spitting out both sides of the transaction (DEF as well), and accepting parameters for the scale value.

[sm...@davethehappysinger.com]

Thanks, some neat ideas there! :D

[redst...@gmail.com]

Glad it helped. I'll post the python file if I find it. Meanwhile, here is a dummy skeleton of a vim function written in python to serve as an example. I find these useful to save for somewhat rare (say once/twice a year) but repeated transformations. Eg: stock splits. I can never remember the syntax of vimscript since I rarely use it, and hence, writing these in python helps.

# to be run on bean-doctor context output inside vim
function! CapGains()
python3 << EOF
import vim
from decimal import Decimal

line = vim.current.line
s = line.split()
account = s[0].replace('Assets', 'Income')
vim.current.line = f'{line} ; Hello world: {account}'
EOF
endfunction

[Eric Altendorf]

Just found this thread.  I think I'm encountering similar issues; difficulties tracking assets that have been transferred.

Have there been any developments on this front since?

[Martin Blais]

On Wed, Jul 19, 2023 at 6:44 AM Eric Altendorf <erical...@gmail.com> wrote:

Just found this thread.  I think I'm encountering similar issues; difficulties tracking assets that have been transferred.

Have there been any developments on this front since?

No

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/2a9684b7-8c10-47ee-8bec-bd58c1095023n%40googlegroups.com.

[Eric Altendorf]

On Sun, Jul 30, 2023 at 8:52 AM Martin Blais <bl...@furius.ca> wrote:

On Wed, Jul 19, 2023 at 6:44 AM Eric Altendorf <erical...@gmail.com> wrote:

Just found this thread.  I think I'm encountering similar issues; difficulties tracking assets that have been transferred.

Have there been any developments on this front since?

No

FWIW, I *think* things are working for me by using both:

1) The {USD} cost hack to pass cost basis info

2) The ZeroSum account pattern to decouple the outgoing half of the transfer from the incoming.  I haven't actually plugged in RedS's plugin but the balances in the intermediating

virtual "buffer" accounts show me how close I am to properly accounting for both the incoming and outgoing amounts.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhMisGxYyhdEk2RpFB-UWPz40T3wv7seWog-qyaC5qJEew%40mail.gmail.com.

[Eric Altendorf]

[rereading this thread more carefully now]

On Monday, January 18, 2021 at 6:57:52 PM UTC-8 bl...@furius.ca wrote:

This looks all wrong, see other thread.

To buy BTC at Coinbase, the money all comes from your Coinbase:Cash account.

FWIW, Coinbase (and other exchanges) also offer (for better or worse) services where you can buy crypto with a debit or credit card, in which case, I think a debit posting in another institution's account could be appropriate?  I of course have no idea if that's what happened here.

[Eric Altendorf]

On Tue, Aug 1, 2023 at 12:11 PM Eric Altendorf <erical...@gmail.com> wrote:

On Sun, Jul 30, 2023 at 8:52 AM Martin Blais <bl...@furius.ca> wrote:

On Wed, Jul 19, 2023 at 6:44 AM Eric Altendorf <erical...@gmail.com> wrote:

Just found this thread.  I think I'm encountering similar issues; difficulties tracking assets that have been transferred.

Have there been any developments on this front since?

No

FWIW, I *think* things are working for me by using both:

1) The {USD} cost hack to pass cost basis info

2) The ZeroSum account pattern to decouple the outgoing half of the transfer from the incoming.  I haven't actually plugged in RedS's plugin but the balances in the intermediating

virtual "buffer" accounts show me how close I am to properly accounting for both the incoming and outgoing amounts.

I spoke too soon, its not working :)

I reread the thread, here's my understanding of the current state:

- You implemented at least some aspects of the solution in a plugin in this patch

- That plugin appears to have ended up in beanlabs here

- This issue was also filed

I'm willing to spend some cycles on this problem if I can understand a bit more.  Some initial questions:

What does the beanlabs plugin implement, and what is left to do?

What does the filed issue cover, and how does that relate to the patch?

Do you want this behavior in a plugin?  Should transferring cost be a default behavior (personally, I would think so; transfers may be more common for crypto, but it seems more correct for any asset -- I have transferred equities between brokerages as well and I needed to transfer lots and cost basis there too)