file: and nio: have no special meaning in modern versions of H2, jdbc:h2:file:SOME_PATH, jdbc:h2:nio:SOME_PATH, and plain jdbc:h2:SOME_PATH have exactly the same meaning.
Unfortunately, invocation of Thread.interrupt() during disk I/O closes the underlying file channel, so it isn't safe to interrupt a thread that executes some database command in embedded persistent database. If you use a separate server process you can interrupt client threads safely.
async: file system uses asynchronous I/O on Windows and it may work a little bit faster, but better performance is not guaranteed for all cases.
On POSIX systems in simply performs I/O in separate threads, so performance may be slightly reduced. These systems, unlike Windows, don't need any special optimizations for multi-threaded access to the same file.
In both cases case Thread.interrupt() on thread that works with database doesn't perform disk I/O by itself, so if you cannot prevent interrupts and cannot use a separate server, you need to use this file system abstraction layer.