The Five-Minute Ledger Update

[redst...@gmail.com]

A few years ago, I found Beancount and very quickly understood how well it solved many problems in personal finance software that I'd faced for years prior. Beancount's extensibility was a core attraction for me, and it was clear the software was worth investing in. I started writing automation around it for my needs. Today, a bunch of that automation works surprisingly well, though it's taken effort to get here.

I've started writing up some of my Beancount workflows in the hope that it saves others tons of time. The first in this series is ledger updates. I'd previously posted this in a thread somewhere, but am creating a separate thread here, so I can post updates and such.

Link to article series: The Five-Minute Ledger Update

Extract follow:

Problem: Updating Your Ledger is a Pain!

That’s right, updating your ledger with data from your financial institutions is the most laborious and frustrating part of personal finance. It doesn’t need to be so with Beancount, which is the point of this series of articles.

With a little bit of effort upfront, open source tooling can actually be way better than commercial solutions, and far more flexible and extensible.

Zero Effort Updates

The ultimate vision of this set of articles is to have your ledger updated automatically with zero effort from you. How close can we get to that vision? When I started out, each update would take hours of frustrating effort and reconciliation across 60+ accounts at institutions. So much so, I only did updates once in a few months. After understanding why, and developing solutions, I am now at a point where my ledger updates take well less than five minutes.

Bringing it down to under five minutes was critical to making personal finance productive and fun, because it lets me get away from tinkering, and enables me to focus on the actual finances.

[cha...@gmail.com]

Very well put-together. I will say that for me the thing that made it easy to stick with Beancount, and plain text accounting in general, is to simplify my financial life.

I have one login for my checking/savings/brokerage/retirement/main credit card acounts, another couple CCs, my mortgage, and just don't track much other stuff (like airline miles).

This simplicity also probably stops me from getting into things that wouldn't be good for my financial life, like day trading :)

[Ben Blount]

Thanks for putting this together!

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/af881cfd-58fc-4f81-aa82-0d9135dd4163n%40googlegroups.com.

[redst...@gmail.com]

Thank you for the kind words :).

Reg. simplifying: I heartily agree. Once upon a time (a long time ago now), I had it all down to a single banking institution (bank, credit card, investments), and it was amazing. But then, that didn't work out. Two adults and several employers each across time wrecked that beautiful simplicity.

I show the reasons here, which you probably read. That article was originally titled "Simplifying your financial life," and was about how to consolidate and simplify. Which I then changed because when I counted 61 accounts, that didn't fit my dream of simplicity at all :(.

Despite all the above, I still make a strong effort to keep things as simple as possible. No day trading for me either, though I still do all the things that I wrote fava_investor for. No point writing software to solve the unnecessary complexity :).

[Daniele Nicolodi]

On 19/04/2021 11:37, redst...@gmail.com wrote:
> I show the reasons here
> <https://reds-rants.netlify.app/personal-finance/how-many-accounts-might-a-person-have/>,

You forgot to count the grocery store fidelity cards and the library
cards, with those you could easily bring the total count to 100...

:)

Cheers,
Dan

[redst...@gmail.com]

Happy it's helpful! Feel free to let me know what else might be helpful to discuss here or as comments in the articles.

[Jason Chu]

Keeping up with my own finances was lagging behind until I found this set of posts. They helped me to reduce the overall time it will take to process everything (though I doubt I'll get it down to 5 minutes like you).

I'm excited to continue reading about your setup as you publish, but I'm especially interested in the posts about you testing framework and how you're automating downloads.

Thanks for writing this!

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/1a944b17-2a0d-459b-a681-f19fb8bef056n%40googlegroups.com.

[redst...@gmail.com]

Cool, happy to hear this!

I'll write these articles up soon. Meanwhile, here's a preview of both:

My testing framework is rather rudimentary, but works well: it's just a few shell scripts to run my importers across every ofx I've ever downloaded, and compare the output to reference beancount files. Details, limitations, and such soon, in a post.

I use ofxget (from ofxtools) to automate the downloads. I'll post soon on some minimal code I use around it to store credentials in `pass`, and such.

Perhaps I'll look into to the possibility of doing a screen GIF of my entire workflow so people can compare and see their sticking points in truly getting it down to under 5mins. Privacy is the main issue.

[Ben Blount]

I'm interested in how you do password management for ofxget.

--

You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/41514faa-8e6a-4be9-944d-7d8c2f65999an%40googlegroups.com.

[redst...@gmail.com]

Given the interest in this topic, here is a new article: Direct Downloads

> I'm interested in how you do password management for ofxget.

Ah, I nearly forgot about that, thanks for asking :). You will need this patch (see this thread) until it is released.

[Bman Q]

Red, thanks for the write up. It's motivating.

Currently i have ingesting part fully automated, but have to manually download csv files.

It takes some time, and i am thinking about automating it as well.

But hesitating on ROI (time needed to maintain downloading scripts vs manually downloading) cause ingesting, categorizing, etc was most time consuming part. Downloading is relatively quick. (though number of my institutions is growing :) )

Can ofget download csv files as well? Or only ofx?

Can you specify date range of transactions for download?

How reliable / breakable it is? Is there somewhere a list of institutions ofxget works with?

[Ben Blount]

Sadly ofxget compatible endpoints seem to be dying - every year less of the institutions I use support it. It's very easy to use though so if one of your banks works with it, definitely worth trying it.

Several of them at least still support ofx export, but from their website, requiring selenium to get. I have to agree that selenium is not always worth the roi.

--

You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/0bb17295-2b13-456f-82f7-813fbf50c34fn%40googlegroups.com.

[Martin Blais]

re. ROI: it's really a question of frequency IMO.

I automate as much of the accounts that I update frequently.

The other ones I just do a bit more manual work.

Be pragmatic; the goal is not to fully automate everything, the goal is to keep your books in order and minimize the total time (manual + programming)

--

You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/0bb17295-2b13-456f-82f7-813fbf50c34fn%40googlegroups.com.

[Bman Q]

"Sadly ofxget compatible endpoints seem to be dying - every year less of the institutions I use support it."

- Could you please explain why are they dying? Is there another new format / api coming that replaces it?

"Several of them at least still support ofx export, but from their website, requiring selenium to get. I have to agree that selenium is not always worth the roi."

- agree, i don't see anything selenium related as long term solution.

Basically i am trying to understand what's the most future proof solution (where industry is headed),

I saw mentions of Yodlee / Plaid but don't like idea of (1) being dependent on 3rd party $ company (2) it storing my credentials

[Ben Blount]

Ofx is still the best, the institutions that have removed their API endpoints said they did so for security reasons.

Vanguard and ibkr for example still offer ofx, just have to download it from the web with a browser or use selenium.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/682b03a7-7fc3-49e2-b64b-59ee3c698314n%40googlegroups.com.

[redst...@gmail.com]

Bman,

What Martin says below is key: trying to automate a 100% of your ingest isn't worth it. This was going to be the point of an article called "Being Smart About Which Accounts to Update When" that I haven't yet published on The Five-Minute Ledger Update.

To answer your other question: the ofx standard is not going away anywhere. What Ben has seen erode a bit is the downloading mechanism (scriptable API for getting the ofx vs. downloading them manually by logging in to the website). I personally have seen one institution remove it and one add it over the past couple years. YMMV.

> Can ofget download csv files as well? Or only ofx?

I've never seen csv, only ofx

> Can you specify date range of transactions for download?

Yes, but YMMV on the range institution accepts. See ofxget scripts.

> How reliable / breakable it is? Is there somewhere a list of institutions ofxget works wi

Very reliable for me. I've seen it break once over the last 3-4 years when the institution decided to make an API change. ofxget pushed a fix quickly.

Yes, on the list, use this command:

ofxget list

[Bman Q]

I quickly looked at ofxtools, and runned "ofxget prof INSTITUTION_NICKAME" to see which ones (of important for me) i will be able to download.

amex works, chase works, schwab not

so i looked at issues, and found this discussion insightful:

https://github.com/csingley/ofxtools/issues/93#issuecomment-736192102

likely soon transaction data for more and more institutions will be available only through middle men ($)

[redst...@gmail.com]

I wouldn't count on it happening any time soon. Inertia works to our advantage here :). Quicken has struck deals with many institutions, and it's that interface that ofxget typically uses.

Regardless, using ofxget is only a small part of the ingest, and is fairly quick and easy to setup, and modular enough to easily replace in the future.

[Bman Q]

I wouldn't count on it happening any time soon. Inertia works to our advantage here :).

-absolutely, i just wanted to tell that i my opinion there is clear headwind for ofx and it's not the future, for several institutions i checked:

   capital one dropped ofx ~2018

   schwab dropped ofx in 2020

   us bank dropped ofx in 2021

and at ofxhome's list of institutions more and more are dead

Quicken has struck deals with many institutions, and it's that interface that ofxget typically uses

-Intuit has sold off Quicken to a PE firm which is not working to improve ofx but to develop replacement fdx. (not compatible / not open)

plus for institutions it's easier/secure/less liability to have 1-3 big "connectors (plaid/etc)" than to have thousands of individuals to connect

https://github.com/csingley/ofxtools/issues/118#issuecomment-778504788

Regardless, using ofxget is only a small part of the ingest, and is fairly quick and easy to setup, and modular enough to easily replace in the future.

-agree. If your institution supports ofxget and you already have/use ofx importers than using it is no-brainer,

however if you are currently using cvs reports / importers then one should consider pros / cons.

To me it's looking now that plaid / etc (aggregators) are most likely to result in most future proof way to "download" data,

but it's not clear now what are the terms / pricing will be for individual users. Cause current developer account access approach is questionable.

(example US Bank no longer allows non-production plaid accounts to get data)