importers and get going

[Jonathan Goldman]

Hi,

I have done a bunch of exploration of beancount over the last few months and really like its capabilities and now ready to get serious about getting it working for me. So far this is what I have done:

• Used the sample files to familiarize myself with the commands and beancount's capabilities.
• Created accounts and manually entered some real data for my personal finances.
• Tried this sem-auto-categorization web-based import tool. It's good but I think I'll mostly go the manual route.
• Tried various import methods
• Tried importing a quicken file...have had some trouble with this.
• Tried a csv2ofx tool
• Used reds_importers for Vanguard. This has worked successfully. This generates output that has two-sided.
• Wrote my own csv importer for my bank that is based on this csv importer.. (I'm not a software engineer by training so this took me some time but I'm very happy with this now and understand the code much more). This generates output that is only one-sided. 

My own conclusion is that I want go the route suggested by Martin and effectively have my own importers for each source and with text-based editor with auto-completion do the categorization. For the brokerages there are a limited number of accounts/categories you can have so doing the two-sided entry can be done here so it is auto-categorization if you will. (e.g. dividends from account xyz always go to this income account, etc). For credit cards and banks it rarely helps as there are so many expense categories. Perhaps a subset (e.g. interest income from the bank can be autocategorized or mortgage payment of specific amount) can be automated. 

Q1. Is doing two-side entry for brokerages and single-side for banks/credit cards a good approach the general practice that most here use? 

Q2. Is there other import approaches to consider? I think I'm finding this the hardest part to get going on and that is why other tools like Quicken/Mint/GnuCash are easier to get going. So not sure how, but perhaps data collection and import is something already solved that is not beancount specific?

I also noticed that Martin mentioned this in the import documentation:

A list of things I’d really want to add, beyond fortifying what’s already there:

• A generic, configurable CSV importer which you can instantiate. I plan to play with this a bit and build a sniffer that could automatically figure out the role of each column.

• A hook to allow you to register a callback for post-processing transactions that works across all importers.

Q3. Is there any update/development on these two items?

thanks,

Jonathan 

 

[Patrick Ruckstuhl]

Hi Jonathan,

Q1

I would say whenever you have both (or more) sides of a transactions then it's the best to import all legs. The thing is that in most cases you don't have them. On thing that I also do quite often is, that if I can import both sides of a transaction form two different sources (e.g. two different banks), then I use the transaction ids of each bank to add them to the same transaction, so I know that I have it imported from both sides.

Q2

Not sure if you've come across smart importer https://github.com/beancount/smart_importer, this is a wrapper around your importer that uses existing transactions to guess what the missing other leg could be. For me this works extremely well.

I use it together with the web ui fava https://github.com/beancount/fava

when you then run an importer you get basically the list of all your transactions and you can still modify if the smart importer guess wrong.

What I also started doing more and more (and also works quite nicely), is to actually not even have a file to import but instead directly use an available api (e.g. truelayer, revolut, transferwise, interactive brokers, ...), you can see some examples in my beancounttools https://github.com/tarioch/beancounttools

Q3

There were some enhancements done for hooks: https://github.com/beancount/beancount/issues/458

Regards,

Patrick

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/b2b52ac6-b019-4375-adcb-e3f7e6ab0c3bo%40googlegroups.com.

[er...@ericglanz.com]

Jonathan,

Are you looking to only import investment account information, or bank/credit card transactions as well?  If it's the latter, I've built a utility that does the following:

• Reads a config file containing accounts to process
• For each account, use ofxtools to download the account transaction data
• Use the smart importer hook to remove duplicates and auto-assign expense categories based on previous transactions
• Writes all new transactions to your beancount file, with a tag denoting the transaction download timestamp
• Email a daily report with all new transactions

I've been using it daily for almost a year and it works great for my use, but has some specific customization code for my Chase credit card accounts.

I'd be open to cleaning up the code and publishing it on GitHub if your interested in collaborating.

Regards,

Eric 

[Jonathan Goldman]

Thanks this is very helpful. I was not aware of the smart importer and will try it out. And I’ll check out beancounttools. Thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/10207a15-f6f1-d6d6-994c-33e9520e2c83%40ch.tario.org.

[Red S]

FYI, I have importers for banks and credit cards that support ofx, based on my beancount_reds_importers library. I'm happy to add that to my python package that you're using if there's interest. I wrap it with smart importer, which is very well designed and written, and gets it right well over 90% of the time.

You're right: I've found that building dependable automation around my import/ingest made took that part from a 3 hour task to a 3min task, and was a key piece in making my beancount/fava experience fruitful.

This is why I get my imports to extract balances and prices and convert them to balance check and price entries. Taking automation a step further, I also have some scripts to auto-download my statements for a few banks/cards/brokerages that support quicken direct connect. This automates the downloading step as well, and stores your passwords securely (encrypted with a master password).

To unsubscribe from this group and stop receiving emails from it, send an email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/b2b52ac6-b019-4375-adcb-e3f7e6ab0c3bo%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bean...@googlegroups.com.

[Bman Q]

Eric hi, did you end up pushing your project on github?