Unfortunately there isn't currently a way to deny signup while allowing signin. One common approach is to use a system like yours (custom claims set though the admin SDK) for security, and couple that with removing any UI links to signup.
You can also setup an http cloud function running the Admin SDK to screen incoming users. If they meet your criteria, use the admin SDK to create an account for them (and set the custom claims token).
The alternative, though slightly more involved, approach is to use
custom auth. If you have a database of users, you can use the admin SDK to mint tokens that the user can exchange for a firebase auth token.