Beancount_reds_importers questions

[jrgo...@gmail.com]

Also want to give a shout out to Red for the really helpful posts and great contribution to getting going quickly with importing. I was able to get 3 importers working using this framework (one US bank that has OFX, one overseas bank with OFX), and one investment account. 

I had a few questions regarding the code:

1) in the importer class you have "self.account_number_field = "account_id". How is that used? The OFX file has this: <BANKID>123456789<ACCTID>123456<ACCTTYPE>CHECKING. Is this relevant?

2) In your examples folder you have 2 entries for "self.filename_identifier_substring". Is this an "AND" match or is it a "OR" match? Do I hard code entries in my bank importer or is there a way to use the config file to pass those parameters. For example, in your importer for Vanguard you have        self.filename_identifier_substring = 'OfxDownload.qfx' so does that mean I have to have the Vanguard files be called '***OfxDownload.qfx'? 

3) My OFX for my bank has entries like this

"<STMTTRN><TRNTYPE>DEP<DTPOSTED>20200131000000<TRNAMT>4.22<FITID>123<NAME>Credit Interest<MEMO>Deposit, Processed</STMTTRN>

"

Your config example has entries like "income' : 'INCOME:INTEREST:BANKNAME"

Is this idea that this type of transaction automatically gets processed and put into that account? Out of the box it didn't happen has perhaps banking.py in libimport is looking for a different string. My question is how should I configure this or is this simply better to handle with the smart_importer and mostly ignore those configurations?

thanks,

Jonathan

[redst...@gmail.com]

That's great!

1. Ah, I need to clean/document that better. You can ignore it assuming you're not having trouble. ofxparse should pick up your account number fine from that input.

2. Also code that needs to be cleaned up. There should be only a single entry. Correct, that is how your Vanguard files should be named. That's how the website names them when you download them, BTW, so this should work out of the box.

3. Correct, the banking transaction builder relies on smart_importer to generate that Income posting. See this article. You'll have to setup smart_importer (very easy), and point it at your existing beancount file.

[john.a.r...@gmail.com]

Apologies for piggy-backing on this thread, but it fits -- especially since I agree with Jonathan that Red has been a huge help on importing records. Thanks!

I have three investment accounts with Fidelity, and I've used Red's set-up to import records from two of them. However, I can't get records for the 3d account, though I know that transactions for this account are in the download. It fails with this message:

Error: fund info not found for: []

Just to be clear -- nothing is inside the brackets. This appears to be the same response that calls for addition CUSIP info in my import file. With the other accounts, once I'd added the requested CUSIP numbers and related fund information the processing continued.

Any hints on what it's missing and how I can trouble-shoot this? (It's surely clear from the way I've described this and asked my question that I'm not a programmer!)

Thanks --

John

[redst...@gmail.com]

Glad you find the importers helpful!

Thanks for reporting this. Someone else had a case of this too. Filed:

https://github.com/redstreet/beancount_reds_importers/issues/9

Made a fix that should help. Could you please try re-running after:

pip3 install git+https://github.com/redstreet/beancount_reds_importers

[john.a.r...@gmail.com]

Hi, Red --

Thanks very much for the prompt response.

I installed the plug-in as instructed above, and unfortunately I've taken a step back. Bean-identify still associates these files with my import settings, but now I can't extract any files. Instead, I get this error message:

ERROR:root:Importer beancount_reds_importers.fidelity.Importer.extract() raised an unexpected error: 'capgains_lt'

Traceback (most recent call last):
  File "/Users/johnrakestraw/.pyenv/versions/3.9.1/lib/python3.9\\\\\\/site-packages/beancount/ingest/extract.py", line 182, in extract
    new_entries = extract_from_file(
  File "/Users/johnrakestraw/.pyenv/versions/3.9.1/lib/python3.9/site-packages/beancount/ingest/extract.py", line 67, in extract_from_file
    new_entries = importer.extract(file, **kwargs)
  File "/Users/johnrakestraw/.pyenv/versions/3.9.1/lib/python3.9/site-packages/beancount_reds_importers/libtransactionbuilder/investments.py", line 311, in extract
    self.initialize(file)
  File "/Users/johnrakestraw/.pyenv/versions/3.9.1/lib/python3.9/site-packages/beancount_reds_importers/libtransactionbuilder/investments.py", line 45, in initialize
    self.build_account_map()  # TODO: avoid for identify()
  File "/Users/johnrakestraw/.pyenv/versions/3.9.1/lib/python3.9/site-packages/beancount_reds_importers/libtransactionbuilder/investments.py", line 57, in build_account_map
    "capgains_lt": self.config['capgains_lt'],

KeyError: 'capgains_lt'

And there's a further mystery (at least, a mystery to me). I get this error message on records I downloaded from Fidelity last night. (And, as I said earlier, I successfully extracted transactions for two of my accounts but not for the third.)

The mystery is that I downloaded fresh files from Fidelity just now, after getting that error message, using the same commands I used last night. But bean-identify doesn't identify these using my my.import file and bean-extract returns no error messages but instead merely the header -- ;; -*- mode: beancount -*-.

Why would it generate the error message with last night's files but not even recognize the ones I just now downloaded? Makes no sense to me.

[redst...@gmail.com]

Apologies, this is unreleased, bleeding edge code, and I should've warned that you're bound to run into these types of things. The fix for this one is simple: just add 'capgains_lt' and 'capgains_st' to your config. The updated example file shows how:

https://github.com/redstreet/beancount_reds_importers/blob/master/beancount_reds_importers/example/my.import

Optional: You might also want to pip3 install again (I just updated the example code), and run the example code, which should help as a reference for troubleshooting future issues.

I wouldn't be surprised if you ran into a couple more problems on this bleeding edge code: simply post here and I'll fix them.

Regarding the ofx file you just downloaded:

- is it named appropriately (includes 'fidelity' in the filename)?

- does it include transactions for the account number in your config? ("grep <acc_num> fidelity.ofx" should tell you)

[john.a.r...@gmail.com]

Hi, Red --

Thanks again, and definitely no need to apologize.

It's now working. One challenge along the way is that my data file includes the transaction types (or keys?) "buyother" and "sellother". The extract choked with error messages to that effect, giving lines in the file "investments.py." As a short-term fix (or perhaps a misguided move?), I added these two terms to the list at line 245 in investments.py and also in the account_map beginning on line 51. I'm now sure how these buys and sells are different from the standards "buystock" and "sellstock." They're associated with transactions involving funds with a CUSIP.

Thanks again for your responses to my questions and also for the files and "rants." All of it has been immensely helpful.

Now to play with smart importer.

[john.a.r...@gmail.com]

And you were right -- I didn't have fidelity in the file names of today's downloads.

[jrgo...@gmail.com]

Thanks @John for raising this issue and to @reds for fixing it so quickly. I also, coincidentally, encountered this issue today and followed this thread and updated to the git version and it works now too! I got error messages related to CUSIPs that I didn't do a normal buy from at Vanguard. But the error message me the CUSIP and I added them to the fund_info file. These CUSIPS were transferred in from another institution and not buys. I inspected the OFX and pasted the info for that CUSIP if that helps make the importer more robust. If there are parts missing I can send that over to you as well. But I can see that it is a transfer.

>>>

Error: fund info not found for 00206R102

ERROR:root:Importer beancount_reds_importers.vanguard.Importer.extract() raised an unexpected error: name 'hasatrr' is not defined

Traceback (most recent call last):

>>>

OFX snippet:

<MEMO>TRANSFER OF ACCOUNTTRANSFER OF ACCOUNT</INVTRAN><SECID><UNIQUEID>00206R102<UNIQUEIDTYPE>CUSIP</SECID><SUBACCTSEC>CASH<UNITS>7.0<TFERACTION>IN<POSTYPE>LONG</TRANSFER><TRANSFER><INVTRAN>

[redst...@gmail.com]

> ERROR:root:Importer beancount_reds_importers.vanguard.Importer.extract() raised an unexpected error: name 'hasatrr' is not defined

Drat, fixed that typo, thanks!

[redst...@gmail.com]

Great! And thank you for the kind feedback. It's helpful to know this is useful to folks!

Reg sellother/buyother: good to know. Across years and many brokerages, I don't have a single sellother/buyother transactions, strangely enough. This is a challenge in developing ofx importers, since I'm restricted to my own ofx file for examples. So thanks for reporting this.

The OFX Spec says this should be used for money market transactions. If (and only if) you feel comfortable sharing if it was indeed a money market fund, that'll give me some certainty in updating the code to treat it that way. Meanwhile, you did the right thing wrt the fix.

On Sunday, May 23, 2021 at 6:07:27 PM UTC-7 john.a.r...@gmail.com wrote:

Hi, Red --

[john.a.r...@gmail.com]

Both of these transactions (one buyother and one sellother) involved a REIT.

[redst...@gmail.com]

That helps, thanks. I updated the code in the same way you did.