Sorry, I solved it. For people facing the same issue as this here is how I diagnosed it and how I solved it. Not sure why I had the problem in the first place though.
I tested it in Edge and it was just as slow as Chrome but I was able to profile it in Edge and saw that it was idle 99.6% of the time, which was strange because there's an infinite loop until the file is completely parsed. From an older version of the codebase I had a setTimeout with a time of 0 for the infinite loop, so I changed this to a while (true) and this completely fixed the slowness issue. So, for some reason a setTimeout 0 inside a web worker in both Chrome and Edge is extremely slow but in Firefox it's fast.
Thanks.