After running a few big jobs with this change, I still see that
walking the hash takes longer and longer the more keys are accessed.
For example, at the start, the first 100 keys took 80sec to access and
output (<1sec/key), while 100 keys around the 72K mark took 300sec
(3sec/key). This seems to progress quite steadily from start to
finish.
The output is just normal text piped to gzip. Other processes running
don't seem to phase it (4 cpu system, lots of memory). I can only
suggest that this is some kind of disk access problem.
Any other suggestions?