New version of F2O

[Cal Learner]

I uploaded f2o_beta10.zip to public folder https://drive.google.com/drive/folders/1-4fBbRxaPhNYZK0gNHuZ3n-Bdoncy2aR replacing previous versions.

1. Reason for change: Fidelity changed the Accounts_History.csv to add a new column-- account name and account number became separate The previous F2O versions will not work with new history CSV files.
2. What code changed: only F2O.py

I hope that some try this new release. While it seems to work for me, I would like info from others.

If you ran before, you already used make_new_F2O_DBgather.bat.bat already made F2O_DBgather.bat. Run F2O_DBgather.bat again to add new symbols to your F2O_DB.csv. F2O_DB.csv is the database that maps symbols to CUSIPS. Otherwise F2O.py will ask about each new symbol, and it is best to avoid that.

Note that the IGNORE_BEFORE_YYYYMMDD variable sets a date telling F2O to ignore transactions before a date. This is because generated transitions will have a different FITID than those downloaded from the Fidelity server. No need to set that if you are only checking for Python errors or OFX errors.

If you are new to F2O,  this message will not tell you how to get started. But there are other posts that get into that.

Questions or comments are welcome.

[Cal Learner]

I would also comment that there is no new feature, other than working with the new Accounts_History*.csv format.

[Cal Learner]

Thanks Kevin! I did not test the generation of  new entries into  F2O_DB.csv by F2O.py since  F2O_DBgather.bat got the securities. I suspect that  I had not gotten rid of investments that I acquired  since my  F2O_DB.csv was previously done  on 07/14/2024. In other words, new stuff I still held, so the info was available in the OFX files for 

Your generated file clearly shows the problem. I will try to address this when I get inspired.

On Friday, January 31, 2025 at 6:36:41 PM UTC-6 kmnu...@hotmail.com wrote:

HI Cal,

I extracted F2o_Beta10.zip to "%USERPROFILE%\documents\F2O"

I ran make_new_F2O_DBgather.bat.bat. It located my Pocketsense xfr folder and it created the F2O_DBgather.bat file as expected.

I ran F2O_DBgather.bat.  It created F2O_DB.csv as expected.

I downloaded Account_History.csv & Portfolio_Positions_Jan-31-2025.csv from Fidelity to the Downloads folder.

I ran f2o.bat.  The command window is asking to create new entries based on the securities held in Account_History.csv.  The securities already exist in F2O_DB.csv

I created the new entries in F2O_DB.csv

I've attached F20.DB.csv for your perusal.  The last 20 entries in F2O_DB.csv are the new entries entered by f2o.bat.  

11 *.ofx files were created based on the transaction types for the securities in Account_History.csv instead of the expected 5 *.ofx files based on the actual individual Accounts held at Fidelity.

I have to confess that it's been a while since I've tinkered with f2o.bat.  I think I've done everything correctly but if I missed something to cause this odd behavior, please let me know.

-Kevin N.

[-Kevin N]

Hi Cal,

Excellent, Thank you.

-Kevin  N.

[Cal Learner]

Uploaded f2o_beta10A.zip

The f2o_beta10.zip had the old F2O.py dated 07/16/2024. New one has F2O.py dated  02/01/2025.

So I was running my revised file, and somehow missed copying the revised file over to be zipped up.

Thanks. I was doing a lot of pdb, trying to replicate the bug, before I popped my head out...

[-Kevin N]

Hi Cal,

f2o_beta10A looks to be perfect - again :)

Thanks for looking into it.

-Kevin N.

[Cal Learner]

Kevin: I considered trying to make the code work on both Python 2.7.18 etc, but also Python 3.something. During F2O writing I had the feeling that we could lose the Fidelity OFX server at any time.
 
PocketSense from Robert runs on the newer versions of 2.7, and it makes some things were easier. Among the easier things were some print statements. It was not as easy as just adding parens to make things compatible. So I stuck with Python 2.7 that also runs PocketSense.
 
I did make a bit of note as to things that would need to change if ported to Python 3. Some will be fairly common. A harder one is that hashing gives different results with Python 3 vs 2. Hashing is important in FITID generation. If a change is made in hashing, then it is important to not come up with a different FITID for transactions already in Money.

The FITID generation is currently in the def gen_FITID() starting at line 103.

A change could be dealt with by adjusting the IGNORE_BEFORE_YYYYMMDD before running the new version of F2O.py on Python 3. But even better would be code that would produced the same results with Python2 and Python3.

https://stackoverflow.com/questions/57335387/how-to-make-update-function-of-hashlib-md5-work-in-both-python-2-and-python-3 talks of a way to get the same hash values from 2 and 3.

An advantage of Python 2.7.18 is shared with Microsoft Money: no new bugs.

[-Kevin N]

Hi Cal,

In case you didn't already know it, there is a user who ported Pocketsense to Python 3.

He posted it on GitHub. https://github.com/magnuspyke/ofxscripts

I tried using the Python 3 Pocketsense scripts on a Linux laptop and they do work well enough.  

There are just some minor functions that don't work but no show-stoppers.

If it comes down to Fidelity dropping OFX direct-connect and I have to use Python 2.7 in order to run f2o then, so be it.

-Kevin N.

[Cal Learner]

Note that Fidelity  Accounts_History*.csv files currently messed up, so that would prevent F2O from working in this interim. I expect they will fix this, so I am not trying to make F2O adapt for now.

Fortunately the Fidelity OFX/QFX server still works great, so PocketSense still works great for Fidelity and some others.  https://pocketsense.blogspot.com/ has the downloads and discussions. But this is also a good place for PocketSense discussions. If you want to discuss PocketSense, I suggest you make a new "Conversation" with PocketSense in the Subject, or add to one of the existing threads with PocketSense in the Subject.  https://groups.google.com/g/microsoft-money/search?q=pocketsense

[Cal Learner]

Studying the current Accounts_History*.csv files, the bug is that the data for the Quantity column is missing, and the Quantity column now just contains USD for me. USD should be under the Currency header.

Quantity is an important number, so I don't see a workaround for what I presume is this temporary problem.

NetBenefits users are currently having a PocketSense problem. I don't know if this is a precursor, or just a temporary glitch.  For those who have not done anything about this, I suggest at least keeping a copy of all of your latest Fidelity OFX files from your xfr folder. Such a copy can let F2O create the database that relates symbols and CUSIPs for your holdings. I keep them all, having my batch file that runs PocketSense copy those files. 

If/when the OFX server goes away, things will be hopping. It will take a while to get back into the project. For now, keeping a copy of your OFX or better still, do the F2O_DBgather process.

[Cal Learner]

Turns out NetBenefits was not a problem. User just had to add the 443 to the  url : https://ofx.fidelity.com:443/ftgw/OFX/clients/download  in sites.dat.

[Cal Learner]

To further correct, Harold says the working URL for NetBenefits on PocketSense is https://nbofx.fidelity.com:443/netbenefits/ofx/download

[Cal Learner]

 F2O_beta11.zip uploaded today into folder https://drive.google.com/drive/folders/1-4fBbRxaPhNYZK0gNHuZ3n-Bdoncy2aR

Changes are minor, but mainly recognizing that the "morning after" has happened. 

[Cal Learner]

Up to F2O_beta11A.zip, and there are still deficiencies. Same folder

[-Kevin N]

Hi All,

Try this...

In sites.dat add the following to the Fidelity <site> parameters:

userAgent : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0

-Kevin N.

[Cal Learner]

Kevin... It WORKED. PocketSense. Thanks!!!

<site>
    SiteName : FIDELITY
    AcctType : INVSTMT
    fiorg : fidelity.com
    url  :  https://ofx.fidelity.com:443/ftgw/OFX/clients/download
    fid  : 7776
    bankid :
    brokerid : fidelity.com
    appid :
    appver :

    userAgent : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0

    mininterval : 20
    timeOffset: # do not use withy my qq scrubber. Maybe -34 with released.
</site>

[Tom Meyer]

Nice.  Works on WIn11 and Debian13

Regards,
Tom Meyer


------------------------

--
You received this message because you are subscribed to the Google Groups "Microsoft Money" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microsoft-mon...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/microsoft-money/70d26ca3-53f8-449d-a253-01ac9ce04d91n%40googlegroups.com.