Hi Jonah,
I'm not sure what's going on, but I can suggest some ways to start debugging it.
If you haven't already, make sure that the Chrome debugger is configured to stop on all exceptions, not just handled ones.
First, make sure you're getting to the transformGeometry function in ArcGisImageServerTerrainProvider. If you are, that means the image is being retrieved successfully. Then, stop in the HeightmapTessellator in the web worker. It loops over the heightmap and creates vertices for each height. Make sure that the heights are as you expect. If all that looks good, let me know and I can suggest some next steps.
Kevin