I've inherited a project where a core model was done like this, and let me tell you right away: don't do this.
The performances don't seem too much affected (see comment below the answer), but I do know there are a few limitations that come with it, and changing it later is not
a straightforward operation. The first limitation that comes to my mind is with
GenericForeignKey. It expects the object ID to be an integer. Even if you don't use that feature directly, some libraries that you might need, will, and you'll be blocked from using them.