I've spoken to the head of our software team, and this is by design, because removing the last item means you would be saving it with nothing selected, which isn't permitted for a required field. You can add a second item then remove the first.
We could probably improve the UX around this. Because we auto-save, you have to get the warning right away.