Dear Wizards/Colleagues,
I am presently working on a hi-res (200m) GETM setup for a deep-water Greenland fjord, and I have a met a few challenges, that I would like to share. The first concerns 2D scaling, and the second stability/wiggles. Both issues seem problematic, but maybe some of you already have solutions or ideas to pursue.
I submit this message to both getm-users and getm-devel as the issues fall somewhere in the middle. Feel free to reply only to one of the lists.
2D Scaling.
For the fjord in question, the some sections exceed 700m water depth, and with a 200m bathy, the CFL condition (internally computed in GETM) gives a limit on the time step less than ~1.1s – for simulations timestep=1.0 has been chosen.
Given this time step, we can use a fairly large split factor of at least 60 to 120.
This means that the computations to a larger extend than what I am used to, is dominated by 2D computations and associated data exchanges. So far my limited experience with 2D GETM indicates that it does not at all scale well with number of subdomains. In normal (3D) GETM, the amount of work on the 3D side is so large, that the poor 2D performance is of little consequence. But in hires setups with deep areas, the game may change.
Did anyone look at the parallel scaling for 2D GETM? If so, are there any results/recommandations?
Stability and wiggles.
If An is not used (An=0), then wiggles grow really fast. In order to suppress the wiggles, I have to use An of around 30 or more. This seems high to me – as I a priori expected a scaling with cell area. In a new paper submitted by Markus Reinert et al, a similar fjord setup w/ dx=500m resolution use An=50. And in a 600m setup of the actual fjord in question here, I can use An=30. So, I hoped that I could get away with An in the order of 1-10, but that is not at all sufficient to dampen the wiggles.
The problem with high An here, is that it actually modifies the results. I can see differences in the timing of tidal waves going through the fjord. It may only be a few cm – or minutes in the timing, but it is significantly more than what I am comfortable with.
Are there any know/good ways to improve the stability wiggle-wise, so that we may use a lower value for An? I expect that the wiggle issue may be related to dx/D, but I have not done the math. If it holds true, then maybe An could be defined (as a file field) as somehow dependent on bathy depth.
If you have experience with this – or ideas on how to improve GETM in this respect, then please share.
Thanks,
/Bjarne
Joint GEOMETOC Support Centre
Lautrupbjerg 1-5 - 2750 Ballerup
Danish Defence Acquisition and Logistics Organisation
MOBIL: +45 4078 7320
FIIN: FMI-MA-OPD04
http://WWW.FCOO.DK/
Hi Bjarne,
I guess for a 200 m resolution at 700m water depth, the shallow-water approximation in the governing equations of GETM doesn’t hold anymore?
So a fundamental problem (regardless of whether this is related to the wiggles you’re experiencing)? Or am I mistaken?
Best wishes,
Johan
--
You received this message because you are subscribed to the Google Groups "GETM-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
getm-devel+...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/getm-devel/876e75e5c13d40fbafd6ae2bbe8c024e%40fcoo.dk.
Hi Johan,
>I guess for a 200 m resolution at 700m water depth, the shallow-water
>approximation in the governing equations of GETM doesn’t hold anymore?
Even if the mathematical approximation of the physics doesn’t hold, we should still be able to solve the mathematical/numerical problem, ie check convergence etc.
In principle, I could enable the non-hydro approximations in GETM, but I am not sure that it will actually help with the wiggles(?)
Any experiences?
/Bjarne
Joint GEOMETOC Support Centre
MOBIL: +45 4078 7320
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-devel/PRAP192MB150537F529A1EE8574143076E7899%40PRAP192MB1505.EURP192.PROD.OUTLOOK.COM.
Hi Johan,
>I guess for a 200 m resolution at 700m water depth, the shallow-water
>approximation in the governing equations of GETM doesn’t hold anymore?
Even if the mathematical approximation of the physics doesn’t hold, we should still be able to solve the mathematical/numerical problem, ie check convergence etc.
In principle, I could enable the non-hydro approximations in GETM, but I am not sure that it will actually help with the wiggles(?)
Any experiences?
In our nonhydrostatic benchmark test cases our horizontal resolution was also much smaller than the water depth. Still we could produce stable hydrostatic simulations. So the hydrostatic approximation only caused wrong results, but does not cause instabilities.
The problem is with high-resolution and deep water and runtype=4.
Bjarne, can you confirm that up to runtype=3 everything is fine?
--
---
You received this message because you are subscribed to the Google Groups "GETM-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to getm-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-users/4513de9c27f5409ba2815d9273b518dc%40fcoo.dk.
Hi Knut,
> Bjarne, can you confirm that up to runtype=3 everything is fine?
I’ve made a preliminary test, and that seems to be true.
I need to make a second test with more realistic S/T fields, I will report back ASAP.
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-devel/f3560cc6-e1e1-4ee9-b9e6-0a3eaff49226%40gmx.net.
you could try with:
cmake ... -DGETM_FLAGS="-D_DELAY_SLOW_IP_"
and in getm.inp:
smooth_bvf_hor=.true.
(smooth_bvf_ver=.true.)
this helped Adolf for MedSea.
(and I think he scaled An with water depth)
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-devel/c43180dc285544d4aa44680a50c1f17d%40fcoo.dk.
Hi Yannik,
Could you give info on dx, dy and max depth for your case?
>actually it was 'smooth_bvf_hor = .true.' that helped in my case.
The cases I have been running so far have all used
smooth_bvf_hor = .true.
smooth_bvf_ver = .false.
I am now trying the -D_DELAY_SLOW_IP_ compile option and also smooth_bvf_ver=.true.
From looking at the code I am not sure why DELAY_SLOW_IP would work, but I’ll try it for sure.
Thank you for chipping in!
From: Yannik Muche [mailto:yannik...@uni-rostock.de]
Sent: Wednesday, March 29, 2023 15:07
To: Bjarne Büchmann; getm-...@googlegroups.com
Cc: Getm Users; marvin...@io-warnemuende.de
Subject: AW: [getm-devel:5274] Issues for deep-water highres fjord model(s)
Hi Bjarne and Hans (cc Marvin & Knut),
actually it was 'smooth_bvf_hor = .true.' that helped in my case. I had been dealing with a checkerboard pattern in salinity near the plume and increasing An didn't help. Now it is working fine with An=5.
Best,
Yannik
Von: Bjarne Büchmann <b...@fcoo.dk>
Gesendet: Mittwoch, 29. März 2023 14:05:55
An: getm-...@googlegroups.com
Cc: Getm Users; marvin...@io-warnemuende.de; Yannik Muche
Betreff: RE: [getm-devel:5274] Issues for deep-water highres fjord model(s)
________________________________
Achtung! Externe E-Mail: Klicken Sie erst dann auf Links und Anhänge, nachdem Sie die Vertrauenswürdigkeit der Absenderadresse geprüft haben.
________________________________
Hi Bjarne,
I now remember I’ve been through a similar thing together with Knut for our northwest european shelf setup a few years ago.
We ended up using -D_SLR_V26_ and -D_DELAY_SLOW_IP_, and also a depth-varying An field: zero for water depths up to 100 m, and 0.2*depth elsewhere.
Hope this helps,
Johan
--
---
You received this message because you are subscribed to the Google Groups "GETM-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
getm-users+...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/getm-users/f8f32f44dae5406fa96276be81bcb593%40fcoo.dk.
just a a side-note: _SLR_V26_ is not needed anymore
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-devel/PRAP192MB15051DAAB101F20A97BC5A35E7899%40PRAP192MB1505.EURP192.PROD.OUTLOOK.COM.
Hi Johan,
> We ended up using -D_SLR_V26_ and -D_DELAY_SLOW_IP_,
Yes, that is what I try out now.
>a depth-varying An field: zero for water depths up to 100 m, and 0.2*depth elsewhere.
Right. I’m thinking about something similar, albeit maybe with a min(An) larger than zero.
For comparison, do you recall the spatial resolution (approx. dx) and max depth for the setup?
As we have depth~700[m], it would mean An up to in the order of 140m^2/s, which seems to be a lot. It will definitely impact the results.
Presently, I’ll just run a single month with various settings, starting with constant to see what we need, then in a second test, I’ll try something slightly more fancy.
Am I correct to assume, that An in principle should scale with dx*dy, ie finer grid means smaller An?
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-users/PRAP192MB15051DAAB101F20A97BC5A35E7899%40PRAP192MB1505.EURP192.PROD.OUTLOOK.COM.
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-users/PRAP192MB15051DAAB101F20A97BC5A35E7899%40PRAP192MB1505.EURP192.PROD.OUTLOOK.COM.
Hi Bjarne,
Dx approx. 5 km (it’s a lat-long grid so it varies).
Max depth of the order of 4 km.
The problem was off-shelf, and our main region of interest the North Sea, hence the choice for An zero for smaller water depth so as not to impact the solution where it mattered if at all possible.
I don’t know if An should scale with the grid, but in principle I aimed for the smallest values I could get away with. Yes the values get quite high. Let me know if you’re interested in the python script I used to generate the .nc file.
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-users/a5b75970173f4901bc1081555b41b5a6%40fcoo.dk.
Hey Johan,
>Dx approx. 5 km
Thanks
>(it’s a lat-long grid so it varies).
Right, but not a lot..
>Max depth of the order of 4 km.
OK, so in the order of DX.
> Let me know if you’re interested in the python script I used to generate the .nc file.
Thanks, but no thanks.
It is super-easy with NCO/ncap2. Just a few lines in bash.
Best,
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-devel/PRAP192MB1505953288E77906D5A048D6E7899%40PRAP192MB1505.EURP192.PROD.OUTLOOK.COM.
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-users/ae25edfe8ce941dfba595caecf041f87%40fcoo.dk.
Hi Adolf,
For sure good points, but I have not looked at these numbers. If somebody already wrote a (python) utility script to check – eg from bathy (rx0) or from a hotstart files (rx1), then we should share it – eg on the getm-utils repo.
I don’t have the experience working with these numbers. However, using dx=200m, and rx0_max=0.4, the depth difference between neighboring grid cells may be up to 80m, right? I think that should be OK, although I did not actually compute it.
What I *have* done is ensure that the neighboring grid depth ratio is limited, and smoothed the topo by reducing the local curvature (5-point stencil).
In general, I am a strong advocate for ensuring good bathy (and “good” does not mean “as close to real data as possible”).
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-devel/CAEXw-aaaDggo%2BwdjjS6HKQMORJp1pvV7EFMmBFyRWBqzbBFmbg%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-devel/8a1d1914f0ba9602dc1f584f30e13e90%40io-warnemuende.de.
what is your reasoning for the proposed scaling?
Why proportional to D instead of sqrt(D). If related, the latter might be motivated by CFL...
Why proportional to dx instead of 1/dx? Sure, the former would be
equivalent to LES and could be motivated by internal pressure
gradient errors, but can you confirm less wiggles for higher
resolutions in your case?
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-devel/CAEXw-aYBL4G3Ox2QoCVaUJhbdY3L%2BkarM-kMfunCuydu9NwwUg%40mail.gmail.com.
Hi Knut,
Scaling proportional to D:
Nothing other than that a large An was needed to stabilise the model the largest depths, and that this could go to zero for smaller depths.
Sqrt(D): of course you could try, but you’d probably need the same maximum value, so overall, over the whole domain, An values would then be larger.
It might make more sense to try D^2.
Best wishes,
Johan
To view this discussion on the web visit https://groups.google.com/d/msgid/getm-devel/723d1fd3-396f-3376-f1c5-8929d74477eb%40gmx.net.