I figured it out. The only SMS-verified account was listed as a developer. I verified one of the owners and then it let me remove a different owner.
So, what it should actually say is "None of the owners are SMS-verified. At least one of the owners must be SMS verified before you will be able to remove any of the owners."
Weird but that's how it works.