A few quick pointers upon a quick glance, as your setup has several intertwined issues:
- with dx=dy=10m you are not resolving all of your waves; you are asking the wavemaker to generate waves with min period of Tp=1sec (max freq of 1), which would yield not even two points per shortest wave
- with your peak frequency of 0.03 Hz (roughly 404m peak wavelength) do you need to generate waves up to 1 Hz?
- it’s not easy to generate such long peak wavelengths with the internal wavemaker, as you are essentially in the infragravity (IG) band; is your wave field really dominated by IG waves with peak frequency of 0.03 Hz? In similar applications we select min freq as 0.03, max as 0.3 and if you have say 10s waves we impose peak freq of 0.1; lower harmonics are naturally generated
- also, you are trying to generate 5m wave heights in 15m depths, could your waves be immediately breaking inside the wavemaker?
- you set PERIODIC=T, but is your domain truly periodic north-south? It doesn’t appear to be
- finally, if instability persists, you may consider lowering your CFL, then you might even apply BATHY_CORRECTION