For the gory details:
* The main disk of the server ran out of free disk at some point. The database (maria 10) detected this and was, I think, reasonably happy, but it just wouldn't update any data,
* Early today UTC, two events happen: First a new partition is added to the 'report' table (this is the big table with one row per reported spot). This failed (no space). But it appears to have taken 6 hours to figure that out based on timestamps. Normally, this is essentially instantaneous.
* Then a second script files up (5 minutes after the first one starts) that dumps all data in the 'report' table into a file, and then sends that file to s3 for archiving. Those partitions are then dropped. The partition drop is normally quick. However, today, the dump failed and it certainly didn't drop any partitions.
* The database is not currently executing any queries.
* I have file_per_table set to on.
My guess is that the recovery from the the 'add partition' ddl did not go well and there is a window when running out of disk space is fatal.
I think that I now know what I should do:
0) move the partition .ibd files somewhere safe.
1) restart the db server
2) Recreate the 'report' table schema using a reasonably recent definition that I have.
3) add/remove partitions from this new empty table until the partitions match the files that I have saved
4) shut down the database
5) move the .ibd files back (replacing the new emtpy files)
6) start the database again.
7) see if the table has re-appeared
However, I really don't want to make this worse. by doing the wrong things (or even the right things in the wrong order).
I really would appreciate a mysql expert help me with validating this approach. Please contact me directly.
Philip