Importing transactions from bank.

[mpl...@gmail.com]

Guys hi,

How do you import transactions from bank(checking account, credit cards, etc)? What set up do you use? I've found several ledger CLI related projects for this on github, but nothing for beancount.

p.s. I am in USA, and have a chase bank if it matters.

[Michael Droogleever]

You'll need to use something like this, https://gist.github.com/mterwill/7fdcc573dc1aa158648aacd4e33786e8#file-importers-chase-py, or make your own: https://docs.google.com/document/d/11EwQdujzEo2cxqaF5PgxCEZXWfKKQCYSMfdJowp_1S8/edit

[mpl...@gmail.com]

Thank you Michael, that's exactly what i was looking for!

[Daniele Nicolodi]

That's very nice! I think it is time I rewrite some of the import
scripts I wrote for ledger to take advantage of the beancount facilities.

Cheers,
Dan

[mpl...@gmail.com]

I've started playing with import scripts,the sample files provided in repo are very useful, just have couple of additional questions i was not able to figure out/not 100% sure about:

1 bean-extract does not write transactions in main beancount file (personal.beancount), it generates output into command line or tmp.beancount and we enter them manually, right?

2 when running  "bean-extract -e BEANCOUNT_FILE" what is the algorithm used to check duplication? Does it check only for same amounts? Or also takes into consideration additional clues (payee, date range, etc)?

3 When filing files bean-file example.import ../Downloads/ -o documents/ how are name and date generated?
Seems it is picks up date from original file name (example UTrade20160215.csv), but when it's not in file name how ofxdownload.ofx turns to 2013-12-18.bofa.ofx

4 Main value of regression testing is to insure that new changes/additions to importers still work with old input files,
It's not so useful to help importers work with new formats if format was changed by bank, right?

On Monday, April 9, 2018 at 4:28:30 PM UTC-4, Michael Droogleever wrote:

[mpl...@gmail.com]

5 Also what's the right way to calculate balance?

Chase have the following CSV structure,

Balance fields are included in checking CSV, so i guess i can use balance of the last transactions for balance.
They are not included in credit card CSV however. Is there a way to calculate them?

And how important "balance" in the first place? Cause for example this import script https://gist.github.com/mterwill/7fdcc573dc1aa158648aacd4e33786e8#file-importers-chase-py does not have "balance".

[Martin Blais]

On Sun, Apr 15, 2018 at 8:35 PM, <mpl...@gmail.com> wrote:

I've started playing with import scripts,the sample files provided in repo are very useful, just have couple of additional questions i was not able to figure out/not 100% sure about:

Really glad the example files are useful.

1 bean-extract does not write transactions in main beancount file (personal.beancount), it generates output into command line or tmp.beancount and we enter them manually, right?

Correct. You redirect its output to a temp file and using your editor, cut/paste them to your main file wherever makes most sense for you (and complete them, e.g., categorize the expenses).

 

2 when running  "bean-extract -e BEANCOUNT_FILE" what is the algorithm used to check duplication? Does it check only for same amounts? Or also takes into consideration additional clues (payee, date range, etc)?

Not so great, definitely could use some improvement:

https://bitbucket.org/blais/beancount/src/cd5fe599fdc80d47caa8a2618f5ee489ce88d101/beancount/ingest/similar.py

https://bitbucket.org/blais/beancount/issues/185/make-duplicate-detection-configureable

3 When filing files bean-file example.import ../Downloads/ -o documents/ how are name and date generated?
Seems it is picks up date from original file name (example UTrade20160215.csv), but when it's not in file name how ofxdownload.ofx turns to 2013-12-18.bofa.ofx

From the file_name() and file_date() methods of your importer implementation:

https://bitbucket.org/blais/beancount/src/cd5fe599fdc80d47caa8a2618f5ee489ce88d101/beancount/ingest/importer.py

If you use the example CSV importer it does what it can.

Typically you'd write your own and choose what you want it to be.

 

4 Main value of regression testing is to insure that new changes/additions to importers still work with old input files,
It's not so useful to help importers work with new formats if format was changed by bank, right?

Both.

Those importer codes are typically the last thing you want to do on a Saturday night, so they tend to be kludgey.

- When you need to support something new, you want to make sure you don't break old functionality.

- When the Java programmers at your favorite bank change the output format without warning you (or realize 10 years after that escaping the commas in the CSV fields might have been a good idea), there's no faster way to test a new / updated importer than to plot a newer file in the directory and run regression tests

On Mon, Apr 16, 2018 at 8:30 AM, <mpl...@gmail.com> wrote:

5 Also what's the right way to calculate balance?

Ideally you don't calculate it. Ideally you find it somewhere and enter it.

They're optional.

The reason for Balance directives is to assert (and I mean that in the computer programming sense as well) that your ledger's calculated balance matches the institution's idea of what it should be. The picture perfect idea of a Balance directive is a fuddy duddy accountant type wearing brown socks staring at the bottom line of his paper statement carefully through bifocals and manually copying the number to a Balance directive in Beancount.

My understanding is that most systems work on the basis of "reconciliation", which is a fancy way of saying that you're meant to eyeball what you've entered and confirm it perfect, and after that it's frozen. I preferred granting my users the superpowers of changing the past at will, so I decided to encode that as allowing you to assert balances at particular points in time - Beancount will always automatically complain is a computed balance fails to match your bank's expectation. All the assertions are always checked. Adding Balance directives often also allows you to zero in on errors faster when you're debugging things (e.g. some missing transaction which for whatever reason you failed to enter - it happens).

If your downloaded file contains a parseable balance field, I encourage you to make that importer automatically generate a corresponding Balance directive.

Otherwise, you can manually insert one manually every now and then when you log into that website (e.g. to make a payment) and copy the number on the screen to the current date. The idea is to enter there a number which you know (or assume) is correct.

 

On Monday, April 9, 2018 at 4:28:30 PM UTC-4, Michael Droogleever wrote:

You'll need to use something like this, https://gist.github.com/mterwill/7fdcc573dc1aa158648aacd4e33786e8#file-importers-chase-py, or make your own: https://docs.google.com/document/d/11EwQdujzEo2cxqaF5PgxCEZXWfKKQCYSMfdJowp_1S8/edit

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/57580546-5574-4004-9794-4c3dd6f99812%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[mpl...@gmail.com]

Martin, thank you for your detailed reply! Now it's super clear :)

To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To post to this group, send email to bean...@googlegroups.com.