Looks like an issue with the Git Plugin. On 1.x, it simply has "Delete Repository" buttons that do the same.
For comparison, Subversion Plugin removes that button when there's only one location, and Jenkins 2.0 handles this correctly.
not specific 2.x but annoying ...
Fixed in git plugin 2.5.1 and later