Sorry for the delay, I wear far too many hats! My biggest concern about a custom file chooser that is NOT built atop the platform file chooser is that you lose out when the OS updates and new features are added top the platform, as well as diverging from the user's expectations. I got thrown last year when Windows 11 changed the file chooser experience, but I still think it is better to stick to the platform vs. a cross-platform experience such as in Swing.
What I wish we could do is add a node, as you say, such as what Adobe does with some of their apps. I wish I knew how they do it, as I don't think just anybody can gain access to what are probably private API's at the OS level so that they can add their hooks. maybe a licensing fee to Apple and Microsoft?
Technically, it is possible to do what you ask, and I did so for stuff that is published in my GitHub libraries, but the preview is AFTER the file is selected and it loads the appropriate parser or viewer for the file type (I wrote one for AutoCAD DWG files; image formats and vector graphics are of course built into JavaFX).
It is nice to see a thumbnail before selecting a file, of course, especially if it is a large one that may take a few minutes to parse and that may exhaust memory upon being converted to a viewable scene graph in JavaFX. Many file formats have raster image based thumbnails, of course, but these tend to interact with the native file system and in most cases I think this gives them the advantage of being displayable in the native file chooser. A home-grown solution might not.