JDBC doesn't have any actual requirements for returned table types, the provided list is only an example.
H2 returns table types as required by the SQL Standard for INFORMATION_SCHEMA.TABLES.TABLE_TYPE column.
But each JDBC driver should return its own table types from DatabaseMetaData.getTableTypes() and H2 does it correctly.
I don't know what do you mean by system table and why you need to distinguish them from normal tables and views. The INFORMATION_SCHEMA and pg_catalog (it exists only when PostgreSQL compatibility mode was enabled) normally have only very special virtual tables, but they aren't expected to be found in other places.
You can use non-standard DB_OBJECT_SQL function, it definitely returns NULL for various special tables.
SELECT TABLE_SCHEMA, TABLE_NAME, DB_OBJECT_SQL('TABLE', TABLE_SCHEMA, TABLE_NAME) FROM INFORMATION_SCHEMA.TABLES;
These is also non-standard INFORMATION_SCHEMA.TABLES.TABLE_CLASS column, but you should understand that names of some implementations may be changed at any moment, even it patch release, some implementations may appear only under certain conditions and it is possible to add an own implementation of a table with a custom table engine.