I never did figure out how to limit either:
- The amount of RAM GAM uses for a file buffer, or
- The number of items it processes at once.
Either would have made it easily workable on any machine.
I wound up switching to a Ubuntu 20 box with 32GB of RAM and very fast CPUs. On this machine, the first part that generates the raw file data took around 5 hours, the file was 1.7GB. This was the step that was crashing my 16GB laptop.
The second command reduced that to a 243MB file with 1.65M lines!
The third command took 8 DAYS to process all the files!
Watching the resource monitor, the only real impact was to RAM. The CPUs were hardly doing anything. Network traffic was pretty light for most of the time as well (surprisingly).
I'm posting this for the next person to run into this issue, and needs some idea of what to do: Get more RAM.