bean-price --update feature

[Vivek Gani]

Hi, I've tried extending bean-price to have a --update option which can be found at https://bitbucket.org/seltzered/beancount/branch/bean-price-update . Essentially it will look at commodity lifetimes and the last fetched price to fetch up to (but not including) the present date.

The motivations for this were:

  - Previously it seemed like most would write their own bespoke script for price updates.

  - It seemed like others had interest in a --update feature- see Johannes Harms comments in https://bitbucket.org/blais/beancount/issues/329/bean-price-support-fetch-over-range-of I initially had a branch for specifying a "--date-range" option but having something that looked into commodity lifetimes and prices to find the dates to update seemed more useful.

  - There was some hinting of a making a 'historical price fetcher' routine in lifetimes.py already. There's also been experiments like fetch-lifetime-prices.py.

Features:

  - By default it fetches for 'weekday' prices, but one can set the --update-rate to 'weekly' or 'daily' instead.

  - You can use the existing --date option to fetch up to a specific final date. Could be helpful if you really want to fetch inclusive of today.

  - You can use the --update-compress or --inactive options to fetch prices during times where you're not holding a given commodity.

Edge cases & current drawbacks (much of the logic is in the 'get_price_jobs_up_to_date' method in find_prices.py):

  - For a given commodity, the update looks for the latest price date to start updating from, it doesn't try to do any advance 'filling' across multiple gaps of missing prices, though that doesn't seem like a typical usecase.

  - If you use --inactive with a commodity that has never been held, it will currently try to fetch prices from the commodity directive's date. Open to feedback as to whether this should be done or not.

  - I'm not sure how to handle commodities that don't have a price source listed - right now those will just throw up error messages. Perhaps an --ignore-undeclared-source option?

So far the testing has just been through unit tests & my personal ledger file, so would love to hear if anyone else tries this out. I don't seem have pullrequest access so perhaps Martin can merge it in if it seems suitable.

-Vivek

[jus...@ryoohki.net]

Vivek,

This looks interesting. The one thing I note is that it is based on the "commodity lifetime". Does that mean that start date is the date used when the commodity is defined? So if I define a commodity as

1976-08-31 commodity VFINX

then it will try to fetch every daily price from 1976 up to the present?

[Vivek Gani]

Will need to double check, but I believe that case should only happen if you use update with the —inactive option AND have a commodity that has never been held.

[Andrzej]

Hi, this would be very useful to me, unfortunately it seems to be waiting for review/merge. What's the status? Is there anything I could do to help?

Alternatively, does anyone know if there a tool that does something similar but is outside of beancount main repository? (I'm not a fan of maintaining my own fork of beancount for this patch)

[Justus Pendleton]

On Monday, January 20, 2020 at 5:14:13 AM UTC+7, Andrzej wrote:

Alternatively, does anyone know if there a tool that does something similar but is outside of beancount main repository? (I'm not a fan of maintaining my own fork of beancount for this patch)

Vivek mentioned in his original post that people wrote their own scripts, so you could just do that. Depending on what you want, it isn't especially complicated. Here's one I had for when I wanted to populate the price map with historical data.

If uses gdate because I usually run it on a mac. If you wanted to update prices weekly instead of daily, just change the "+ 1 day" to something else.

#!/bin/bash -e

DATE=gdate

if [ -z $1 ]; then
  echo "Must supply start date, e.g. 1983-05-22"
  exit 1
else
  from=$1
fi

if [ -z $2 ]; then
  to=$(${DATE} -I)
else
  to=$2
fi

d=$from
while [ "$d" != "$to" ]; do
  echo Fetching prices on $d...
  bean-price --date $d my.beancount >> prices.beancount
  d=$(${DATE} -I -d "$d + 1 day")
done

[Vivek Gani]

Hi Andrzej! Nice to hear you're interested in the price update feature. There was some discussion between me and Johannes Harms late last year in a related issue ( https://bitbucket.org/blais/beancount/issues/329/bean-price-support-fetch-over-range-of )  We may want to continue discussion there - last thing we were looking into was optimizing how a bunch of historical prices are fetched.

I honestly don't know how the review process works for beancount, I've just been using my own fork in the meantime, so a script like Justus uses may be a better solution for your needs.

[Andrzej Kotulski]

Thank you, script was useful to me. It was painfully slow to import 2 years of prices (partially due to inefficient price source implementation), but from now on I should run this more frequently so it should be fine.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/28d0651b-d5ad-4a5d-9944-7944cf1ed217%40googlegroups.com.

[Ghanashyam Prabhu]

Hi Vivek, Do you have this update? The bitbucket link is not available anymore. Curious if this feature was ever reviewed and accepted, seems like a very good feature

[vive...@gmail.com]

Hi Ghanashyam,
Yes, during the migration from bitbucket to github Martin Blais moved it to https://github.com/beancount/beancount/tree/pr128_seltzered_beanpriceupdate . It hasn't been reviewed/accepted yet, but perhaps it could be updated so it's easier to consider again. I haven't had time to touch beancount this year so that is the latest commit.

[Martin Blais]

Vivek, the repository has moved to https://github.com/beancount/beanprice

Would you like to resubmit it there?

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/b471b460-1e64-47f1-91a0-4aba5789aa59n%40googlegroups.com.

[Martin Blais]

Had a quick look; updating the branch in v2 isn't trivial (I deleted and revived beanprice during the move by accident).

I'll merge manually the changes to ops/lifetimes.py and create a new change in the new repo.

[vive...@gmail.com]

Thanks Martin!

[Martin Blais]

Merged.

beancount (master):

https://github.com/beancount/beancount/commit/00c0b71fb03c074be08ad10f709c9b6f7e99aa70

beanprice:

https://github.com/beancount/beanprice/commit/f1bcfea1c217c460ef419e05aa7b9ab2dff0bab2

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/d52cdaea-f55e-49b7-b543-8fdac7fb9823n%40googlegroups.com.

[Ghanashyam Prabhu]

Apologies for the trivial question. I re-installed the latest beancount and beanprice from the repository using 

sudo pip3 install git+https://github.com/beancount/beancount#egg=beancount

sudo pip3 install git+https://github.com/beancount/beanprice#egg=beanprice 

With this I do see --update-rate switch when I run bean-price -h

However, I could not figure out the switch combinations to use to get a dump of the price on a specific range 

An example that I tried out is here 

bean-price --update --update-rate daily --data 2020-10-26 -e USD:coinbase/BTC-USD

Please suggest the correct command to fetch prices from a date (say 2020-10-26) to present day. 

Thanks

Ghanashyam

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhPG%2B5-9rfzu93GCRXaKVi865D4bhABfiZVXNOCut9LsYg%40mail.gmail.com.

[vive...@gmail.com]

I'll be honest I haven't had time to use beancount or run the latest, but looking briefly at the code there's a few things that comes to mind:

• I wasn't expecting one to use the --update feature without a beancount file (i.e. how you're using -e). --update was really meant for a workflow with an existing beancount file, where there's dates defined around the commodity and prior prices and transactions to help figure out.
  
  Is this a use case you see yourself regularly using? I know earlier I considered changing things to have a specified --start-date and --end-date but that would've likely broken how so many others already use bean-price in conjunction with a basic shell script to grab multiple prices. Another option is to just mock up a beancount file with the initial commodity entry date (e.g. "2020-10-26 commodity BTC") then run bean-price --update (no --date needed).
  
  
• "--data" - just as a sanity check I assume you meant "--date" here.
  
  
• Regarding the combination of using --update AND --date, Per the documentation, the way I have things setup is that when --update is used, it will "Fetch prices from most recent price for each source up to present day or specified --date" . So in this scenario it will look at the last date fetched and fill in the gaps up till the specified date (2020-10-26). If you already have a price after 2020-10-26 it's not going to do anything.
  
  
  

-Vivek

[Ghanashyam Prabhu]

Hi Vivek, I am currently exploring beancount to track an investment portfolio. In this example, I want to track buy/sell transactions of BTC from Coinbase.

I exactly do not understand the various switches that beanprice utility provides. 

I plan to use bean price to populate a prices.db file which I plan to include in the beancount file and later use to track the value of the portfolio at different times. Is this a workflow that can be achieved with beancount?

What exactly does the --update option do if my beancount file already has transactions made at corresponding prices in USD. Below is an example from my beancount file

2020-10-31 * "Buy BTC at Coinbase"
  Assets:US:Crypto:Coinbase:BTC           0.03528803 BTC {} @ 14169.11 USD
  Assets:US:BofA:Checking                    -500.00 USD

2020-11-03 * "Buy BTC at Coinbase"
  Assets:US:Crypto:Coinbase:BTC           0.03598271 BTC {} @ 13895.56 USD
  Assets:US:BofA:Checking                    -500.00 USD

Regarding the combination of using --update AND --date, Per the documentation, the way I have things setup is that when --update is used, it will "Fetch prices from most recent price for each source up to present day or specified --date" . So in this scenario it will look at the last date fetched and fill in the gaps up till the specified date (2020-10-26). If you already have a price after 2020-10-26 it's not going to do anything.

Can you point me to this documentation that describes the behavior of the bean-price utility? Also where are the prices stored? Is that in cache? Can I use it to populate an explicit file? 

Thank you for taking time to go through my (lengthy) query.

Ghanashyam 

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/224355f3-4f1e-4047-a3fd-5d556d41077dn%40googlegroups.com.

[Ghanashyam Prabhu]

And yes I mean to say --date here, was a typo

bean-price --update --update-rate daily --data 2020-10-26 -e USD:coinbase/BTC-USD

[vive...@gmail.com]

>Can you point me to this documentation that describes the behavior of the bean-price utility?

The best documentation at the moment specifically on bean-price --update is the code itself. There's two core parts of interest within beanprice/price.py:

• the process_args function (sortof an entry point for the tool) where it gets into the update function.
  
• the get_price_jobs_up_to_date function
  

Some of the earlier documentation is:

• Bean-price's original documentation: https://docs.google.com/document/d/1thYRAMell_QT1Da1F_laprSs6BlROZjyK_h3V8qHW9c/edit (doesn't have information about --update yet).
  
• Earlier archived discussion about how --update emerged from originally wanting to just get prices over a range of dates: https://web.archive.org/web/20200621145827/https://bitbucket.org/blais/beancount/issues/329/bean-price-support-fetch-over-range-of)

>Also where are the prices stored? Is that in cache? Can I use it to populate an explicit file?

Prices can be stored in an explicit file, you just pipe the output of the bean-price command to the file. A typical use might be:
bean-price --update --update-rate weekday myLedger.beancount > myPrices.beancount

Prices can also just be referenced in your beancount file with an include line like:
include "myPrices.beancount"

[Ghanashyam Prabhu]

Thanks Vivek for the details. 

I used the bean-price --update btc-price.beancount and it works well

As a reference I also want to update the btc-price.beancount file contents here.

; -*- mode: org; mode: beancount; coding: utf-8; fill-column: 120
* Options
option "title" "crypto pf ledger"
option "operating_currency" "USD"
option "booking_method" "FIFO" 

2020-10-26 commodity BTC
  export: "Crypto"
  name: "Bitcoin"
  price: "USD:coinbase/BTC-USD" 

1980-01-01 commodity USD
  export: "CASH"
  name: "US Dollar" 

* Equity Accounts
1980-01-01 open Equity:Opening-Balances 

* Banking Accounts
1980-01-01 open Assets:US:BofA:Checking
2020-10-24 * "Opening Balance for checking Account"
  Assets:US:BofA:Checking                    7100.00 USD
  Equity:Opening-Balances                   -7100.00 USD 

* Taxable Investments
1980-01-01 open Assets:US:Crypto:Coinbase:BTC BTC 

* Crypto Accounts
2021-01-05 * "Buy BTC at Coinbase"
  Assets:US:Crypto:Coinbase:BTC           0.03 BTC {} @ 15000 USD

  Assets:US:BofA:Checking                    -500.00 USD

> bean-price btc-price.beancount --update 

2021-01-05 price BTC                              34043.91 USD
2021-01-06 price BTC                              36859.26 USD
2021-01-07 price BTC                              39510.55 USD
2021-01-08 price BTC                              40642.15 USD
2021-01-11 price BTC                              35456.89 USD
2021-01-12 price BTC                              34035.53 USD
2021-01-13 price BTC                              37393.67 USD
2021-01-14 price BTC                              39123.05 USD
2021-01-15 price BTC                              36754.60 USD

I noticed that this update to bean-price was made on the master branch of beancount repository. It looks like the master branch now has v3 dev updates but does not have bean-report or bean-web utility. I switched to the v2 branch in beancount repository and for this update of bean-price, I cloned a separate repo for it and installed bean-price on it.

- Ghanashyam

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/d4bff4b8-2de3-4308-9710-32e8163d2439n%40googlegroups.com.

[vive...@gmail.com]

Thanks for the update. I think dropping bean-report & bean-web was intentional as the fava project has taken over those needs, there's a discussion about that here: https://groups.google.com/g/beancount/c/pAQqGwgC4NI/m/VQ7clBSVAQAJ .