New version of F2O

[Cal Learner]

I uploaded f2o_beta10.zip to public folder https://drive.google.com/drive/folders/1-4fBbRxaPhNYZK0gNHuZ3n-Bdoncy2aR replacing previous versions.

1. Reason for change: Fidelity changed the Accounts_History.csv to add a new column-- account name and account number became separate The previous F2O versions will not work with new history CSV files.
2. What code changed: only F2O.py

I hope that some try this new release. While it seems to work for me, I would like info from others.

If you ran before, you already used make_new_F2O_DBgather.bat.bat already made F2O_DBgather.bat. Run F2O_DBgather.bat again to add new symbols to your F2O_DB.csv. F2O_DB.csv is the database that maps symbols to CUSIPS. Otherwise F2O.py will ask about each new symbol, and it is best to avoid that.

Note that the IGNORE_BEFORE_YYYYMMDD variable sets a date telling F2O to ignore transactions before a date. This is because generated transitions will have a different FITID than those downloaded from the Fidelity server. No need to set that if you are only checking for Python errors or OFX errors.

If you are new to F2O,  this message will not tell you how to get started. But there are other posts that get into that.

Questions or comments are welcome.

[Cal Learner]

I would also comment that there is no new feature, other than working with the new Accounts_History*.csv format.

[Cal Learner]

Thanks Kevin! I did not test the generation of  new entries into  F2O_DB.csv by F2O.py since  F2O_DBgather.bat got the securities. I suspect that  I had not gotten rid of investments that I acquired  since my  F2O_DB.csv was previously done  on 07/14/2024. In other words, new stuff I still held, so the info was available in the OFX files for 

Your generated file clearly shows the problem. I will try to address this when I get inspired.

On Friday, January 31, 2025 at 6:36:41 PM UTC-6 kmnu...@hotmail.com wrote:

HI Cal,

I extracted F2o_Beta10.zip to "%USERPROFILE%\documents\F2O"

I ran make_new_F2O_DBgather.bat.bat. It located my Pocketsense xfr folder and it created the F2O_DBgather.bat file as expected.

I ran F2O_DBgather.bat.  It created F2O_DB.csv as expected.

I downloaded Account_History.csv & Portfolio_Positions_Jan-31-2025.csv from Fidelity to the Downloads folder.

I ran f2o.bat.  The command window is asking to create new entries based on the securities held in Account_History.csv.  The securities already exist in F2O_DB.csv

I created the new entries in F2O_DB.csv

I've attached F20.DB.csv for your perusal.  The last 20 entries in F2O_DB.csv are the new entries entered by f2o.bat.  

11 *.ofx files were created based on the transaction types for the securities in Account_History.csv instead of the expected 5 *.ofx files based on the actual individual Accounts held at Fidelity.

I have to confess that it's been a while since I've tinkered with f2o.bat.  I think I've done everything correctly but if I missed something to cause this odd behavior, please let me know.

-Kevin N.

[-Kevin N]

Hi Cal,

Excellent, Thank you.

-Kevin  N.

[Cal Learner]

Uploaded f2o_beta10A.zip

The f2o_beta10.zip had the old F2O.py dated 07/16/2024. New one has F2O.py dated  02/01/2025.

So I was running my revised file, and somehow missed copying the revised file over to be zipped up.

Thanks. I was doing a lot of pdb, trying to replicate the bug, before I popped my head out...

[-Kevin N]

Hi Cal,

f2o_beta10A looks to be perfect - again :)

Thanks for looking into it.

-Kevin N.

[Cal Learner]

Kevin: I considered trying to make the code work on both Python 2.7.18 etc, but also Python 3.something. During F2O writing I had the feeling that we could lose the Fidelity OFX server at any time.
 
PocketSense from Robert runs on the newer versions of 2.7, and it makes some things were easier. Among the easier things were some print statements. It was not as easy as just adding parens to make things compatible. So I stuck with Python 2.7 that also runs PocketSense.
 
I did make a bit of note as to things that would need to change if ported to Python 3. Some will be fairly common. A harder one is that hashing gives different results with Python 3 vs 2. Hashing is important in FITID generation. If a change is made in hashing, then it is important to not come up with a different FITID for transactions already in Money.

The FITID generation is currently in the def gen_FITID() starting at line 103.

A change could be dealt with by adjusting the IGNORE_BEFORE_YYYYMMDD before running the new version of F2O.py on Python 3. But even better would be code that would produced the same results with Python2 and Python3.

https://stackoverflow.com/questions/57335387/how-to-make-update-function-of-hashlib-md5-work-in-both-python-2-and-python-3 talks of a way to get the same hash values from 2 and 3.

An advantage of Python 2.7.18 is shared with Microsoft Money: no new bugs.

[-Kevin N]

Hi Cal,

In case you didn't already know it, there is a user who ported Pocketsense to Python 3.

He posted it on GitHub. https://github.com/magnuspyke/ofxscripts

I tried using the Python 3 Pocketsense scripts on a Linux laptop and they do work well enough.  

There are just some minor functions that don't work but no show-stoppers.

If it comes down to Fidelity dropping OFX direct-connect and I have to use Python 2.7 in order to run f2o then, so be it.

-Kevin N.