Problem with recent builds

[slab...@gmail.com]

Dear dedalus experts,

I encountered a really annoying issue for which I am totally at loss and only you can help me. A few months ago, I ran parameter space swipe on compressible convection with the D3 version of the script we developed for a previous paper (doi: 10.1093/gji/ggac102) and I needed to run a few more cases. For various reasons (including system update), I installed a new version (3.0.2, latest) of dedalus on our cluster and I can’t get the same results. As an example, I show the time evolution of several scalar diagnostics originally obtained in a simple case which reaches a periodic solution and the continuation using the new install. I checked on my Mac laptop, on which I still have a rather old install (3.0.0a), and I get the proper continuation. The most annoying part is that when I plot the vertical total heat flow, which should be constant when time-averaged, I indeed get the proper result in the old results but not at all with the new install. Moreover, I installed the latest version as a different conda environment on my laptop and it does not run at all (with the same parameters) : after a few time steps, it blows off with NaNs. I checked that the problem does not arise in a Boussinesq model: this is really associated with some compressible terms.

That situation brings several questions:

- The most important one: what makes that issue arise and what can be done to solve it?
- I trust more the results obtained with the old build since it conserves heat but should I?
- How can I go back to the previous build. On the cluster, I use a somewhat custom install because of modules to be charged etc, using git to get the source. I guess I could checkout the one used before but how can I know which one it was ? On my laptop, I also tried another install (in a different env) using conda install -c conda-forge dedalus=3.0.0=py310h5fe63e2_0 but that did not work. Since I have a working (?) version, can I recover some reference to the exact build to use by git and/or conda?

I attach, to this mail, the script and par file to replicate the run, as well as post-treatment scripts. And the figures. Any help would be greatly appreciated.

Stephane

[Keaton Burns]

Hi Stephane,

We’re looking into this now. In the meantime, it would be helpful if you could try the different pip release versions (in your new build) to help narrow down when the change was introduced. You should be able to get versions 3.0.0, 3.0.1, and 3.0.2 from pip.  If its still broken on the 3.0.0 release, we’ll have to go through older commits on github to narrow it down.

Thanks,

-Keaton

--
You received this message because you are subscribed to the Google Groups "Dedalus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/64184e70-141b-4a85-864d-7f6256e55088n%40googlegroups.com.

[Daniel Lecoanet]

Also can you send the analysis script so we can test?

Thanks,

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/CADZXxBi%2B5qaBw3fsxjq8poQv6wJ5gUTbD%2B%2ByQWbdA1F__gtiUQ%40mail.gmail.com.

[slab...@gmail.com]

Thanks a lot for looking into that!

A short news since my previous mail : Louis Couston (sorry for putting you in the spot, Louis), ran my script successfully and I attach the figures I got from his results. He used an old custom build on the same cluster I use, possibly the same version as I previously used. I also attach the analysis script (sorry for forgetting in my earlier mail). 

First run:

python plot_tseries.py --tstart=2.5

Then

python plot_mean_profiles.py

@Keaton: I never could make an install using conda to run through our scheduler on the cluster so it seems difficult to accomplish what you mention. Louis forwarded me a mail with a possible solution and I will try that and tell you if I can find a build that works. I can try pip instead but it might be tough. On the other hand, I have a working version on my laptop but when I tried to reinstall it in another conda env, it did not run properly. The version I have on my macbook is apparently 3.0.0a but are there different flavors of that version? How can I determine which one it is exactly? 

[Daniel Lecoanet]

Hi Stephane,

If you have a custom build, can you look at which git commit you’re running? It would help to find a version where the code works. I tried running on the current version and can reproduce the issues you’re having with the newest version.

Daniel

On Jun 12, 2024, at 12:42 PM, slab...@gmail.com <slab...@gmail.com> wrote:

Thanks a lot for looking into that!

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/05f93c45-fac5-444d-9ee6-257e8f2f8055n%40googlegroups.com.

[slab...@gmail.com]

The custom build I have does not work, it's the most recent version. I unfortunately lost the one that worked. Louis has one that works, and he may tell us what version it is (I cannot access it myself).

It tried to install using pip without success. I am trying conda now to see if I can find a version that works.

S

[Keaton Burns]

Once you have a conda stack that works, it should be possible to remove just dedalus (keeping all the dependencies) and then install different versions with pip to test.  Or building different conda stacks for each version should work too.

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/5b68c85e-d42c-4497-8fc0-9f2933509c52n%40googlegroups.com.

[slab...@gmail.com]

OK, I'll try that on my laptop, working on a clone of the working conda install to keep the working one safe.

[Daniel Lecoanet]

If you try running with a commit before this one:

https://github.com/DedalusProject/dedalus/commit/f26341574fbd5951d716f1ad6e6d5a54576a7d65

the simulation crashes within the first 10 time steps. When using that commit and later, the script runs, but does not evolve the system to thermal equilibrium.

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/df960489-a479-4a86-9eca-9a54586c0cadn%40googlegroups.com.

[Daniel Lecoanet]

This is the commit that causes the code to crash within the first 10 time steps:

https://github.com/DedalusProject/dedalus/commit/1218c62fe8f3f758853246ac15855cd25409ab81

It looks like the more recent commit from my previous email fixed the blow up. However, the results from the simulations that don’t blow up are different prior to commit 1218c62 and after f263415. So I wonder if the bug fix in commit f263415 might not actually completely solve the issue introduced in commit 1218c62.

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/69B77CCD-8A76-4927-B81F-B5911B8A4678%40northwestern.edu.

[slab...@gmail.com]

Just to be sure, Daniel: there are two issues.

- blow up : that happens with all the recent install on my mac. I still have the 3.0.0a that works though.

- In some cases (with the recent custom build on our linux cluster), it runs apparently properly but the solution does not conserve heat. Is that what you mean by no thermal equilibrium? 

[Ben Brown]

I suggest opening an issue on GitHub and moving this discussion there so that we can track and narrow down any potential bugs here. 

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/3b908dfb-b3db-4b31-9292-9042cdf5ae34n%40googlegroups.com.

[Daniel Lecoanet]

Hi Stephane,

I can confirm that commit ff3aea3 works, see attached flux plot. So for now you could switch to that commit. However, I don’t think the bug is too complicated, so I think we might be able to fix it early next week.

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/3b908dfb-b3db-4b31-9292-9042cdf5ae34n%40googlegroups.com.

[Stephane Labrosse]

Hi Daniel,

Just to confirm: I installed that commit on our cluster and I get the proper results, with the example I provided and some others. I will then hold on this build for the moment. Thanks a lot! 

Stéphane

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/05C6B7E9-EB2E-4BF8-869D-DDB529F48472%40northwestern.edu.
<prof_Fluxes.pdf>