Huge, huge thanks to Ross, we figured it out!
csvkmd to the rescue!
If you run this command you will avoid all but the occasional passing backoff that doesn't hurt anything:
gam csvkmd users (csv-file-name) keyfield (user csv field) datafield (contact id csv field) delete othercontacts csvdata (contact id csv field)
I did generate my CSV with only the otherContacts data to avoid false lookups from the People entries, you could do the same or tweak the csvkmd commands to only return those rows. So mine, using the standard gam contacts csv output looks like this:
gam csvkmd inactive_contacts_filtered.csv keyfield User datafield resourceName delete othercontacts csvdata resourceName
Hope this helps the next person that needs to delete 100,000 contacts!