KINSOL_KRYLOV_FAILURE when using solid file

[P. James Dennedy-Frank]

Hi All,

I am trying to get a previously-functioning ParFlow model of the Cosumnes watershed working, but I have been hitting a wall when trying to use the solid file previously built for that model. I haven't been able to find a lot of documentation about the solidfile setup, and the failures are sufficiently vague that I haven't been able to figure them out, so I appreciate any help or thoughts folks have!

Whenever I try to use the solid file with the correct ComputationalGrid.Lower.X,Y,Z I get a pair of unclear errors: 1) in the slurm output file (so effectively stdout)  it starts with a set of errors about “Unable to add metadata” for each of the climate variables, then an error for a failed timestep on the first timestep (or actually 1/5 of that, since it tries halving the timestep 4 times), and then “Problem Solved”; 2) the kinsol log it shows only a single nonlinear iteration and then KINSOL_KRYLOV_FAILURE for each of those attempts. These show a full set of 300 linear iterations per attempt, but linear convergence fails in each case, as shown in the table below (and others like it on previous attempts at timestep division). 

                    Iteration             Total

Nonlin. Its.:               1                 5

Lin. Its.:                300              1500

Func. Evals.:               1                 5

PC Evals.:                  1                 5

PC Solves:                300              1500

Lin. Conv. Fails:           1                 5

Beta Cond. Fails:           0                 0

Backtracks:                 0                 0

However, the error message is sufficiently abstract that I’m not sure what it means and I haven’t been able to figure out what is causing this error. A few things I've checked:

* The rest of the parts of the model work with a simple box input. 

* The solid file looks well formed, and has the X and Y values I'd expect when I load a .vtk version in Paraview. 

Strangely the model seems to run when I set ComputationalGrid.Lower.X,Y,Z  all to 0.0 (rather than the correct X,Y,Z values for the solid file), but I'm pretty concerned that won't work when linking in other input files with the correct spatial information. 

I'd appreciate any thoughts or experience folks have on this kind of problem with a solid file and with KINSOL_KRYLOV_FAILURE errors. Please let me know if any additional information would be useful--I've tried to explain the problem relatively concisely, but there are obviously lots of setup details that I can explain more fully. 

Thank you for your help!

Be Well,

James

[Reed M. Maxwell]

Hi James-

 

Thanks for your question.  I have two questions and a request that might help diagnose what is going on.

1. Was this exact problem working prior with an earlier version of ParFlow? (or)
2. Are you trying to re-set up a problem based on prior datasets?

 

If it’s (1) can you provide the version of ParFlow for which this ran, which will help identify changes that are now breaking things.  The metadata bug may not be stopping execution and is listed as an issue here, thanks for also bringing it up again.  If this is (2), or in any case, can you include the exact version of PF that you are running, compilers used to build, OS, etc, that it’s running on?  Can you also include your python or TCL script that is setting up the problem and what code / workflow was used to create the solid file?  The origin coordinates for the solid file *should* be the same as the origin coordinates for the computational grid, and you are correct that this should match other inputs that have an origin as well.

 

Thanks in advance for providing additional information and hopefully this will help us diagnose what is going on.

 

Reed

 

Reed M. Maxwell, Ph.D.
Professor

Department of Civil and Environmental Engineering

High Meadows Environmental Institute

Director, Integrated GroundWater Modeling Center

Princeton University

maxwell.princeton.edu

igwmc.princeton.edu

--
You received this message because you are subscribed to the Google Groups "ParFlow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to parflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/parflow/f71198cc-df72-4643-a4e1-a8890a6e76b4n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Reed M. Maxwell]

Also, as an additional follow up / clarification.  Each of the points in the solid file need to be placed in the computational grid so it depends on the origin.  If the origin doesn’t match what is used to create the PFSOL there is no way for ParFlow to match these up.  You can use the mask-to-pfsol and pfsol-to-vtk tools to generate your solid files and VTK files that can be easily visualized either with the patches (shown here) or the vertices and lines of the mesh (shown below) in e.g. ParaView.

 

 

To view this discussion on the web visit https://groups.google.com/d/msgid/parflow/DM6PR04MB6697CABFB80DA6EB9C2A2ED0A6439%40DM6PR04MB6697.namprd04.prod.outlook.com.

[P. James Dennedy-Frank]

Hi Reed,

Thanks for your quick replies and your help on this! To reply to your questions:

1. *Basically* this exact problem was working with an earlier version of ParFlow — there are a few files that were corrupted so were re-built, though, so it’s not quite as simple as just switching ParFlow versions. However, that model was running on version v3.3.1 (specifically the log says v3.3.1-10-gd319600). It may have run on some other versions as well, but that was definitely one of them. 

2. I’m now using v3.9.0 I’m not certain which compilers were used because it’s the version Steve Smith set up in /global/project/projectdirs/parflow/ on Cori, but I’m sure we can track that down. I’ve attached here the .tcl script setting up the problem for a version that fails. There’s a lot commented out because I’ve been trying to simplify it to track down the problem, let me know if you have any questions. I can’t, at the moment, let you know about the solid file because I’m waiting to hear on that myself, but I can let you know as soon as I do. 

I did convert the solid file to a vtk using pfsol-to-vtk, and looking at that in Paraview shows the range is within the computational grid using the correct coordinates. I will note that I’m not sure how to show the patches from the solid file—it’s a well-formed polygonal mesh, but I’m not seeing how to select the patch indexes. 

Thank you so much for your help!

Be Well,

James

—
P. James Dennedy-Frank
Postdoctoral Scholar
Lawrence Berkeley Lab
pj...@lbl.gov

[Reed M. Maxwell]

Hi James-

 

Thanks for the clarification and for sending the script.  Can you send along the pfsolid file too?  To visualize the patches you just select “patch index” in Paraview

 

Thanks

To view this discussion on the web visit https://groups.google.com/d/msgid/parflow/72079996-62de-445b-9df9-4584537101eb%40pjdf-m71.hsd1.ca.comcast.net.

[P. James Dennedy-Frank]

Hi Reed,

Thanks again for your help on this!

The link to the solid file is https://www.dropbox.com/s/kyptkvbc1obu6f9/domain8c80_02.pfsol?dl=0 — unfortunately it’s too large to send via lab email. Weirdly I don’t see the “patch index” option in the file, so maybe there’s a problem with the solid file? The format for the patches looks correct to me, with the number of triangles in the next patch occurring at about the correct line, but I’m not positive. 

 

Be Well,

James

—
P. James Dennedy-Frank
Postdoctoral Scholar
Lawrence Berkeley Lab
pj...@lbl.gov

[Reed M. Maxwell]

Thanks James.  I have the file.  How did you generate it?  The Z range of the file runs from 0 to 500 so this will only work with the PF Z origin (you have this set as a TCL variable then set in the computational grid) of Zero, not the 2910 value I think you want.  The issue here is likely some solid file / computational grid synch up.  Have you plotted the mask output by PF in Paraview too?

 

Happy to help

[P. James Dennedy-Frank]

Hi Reed,

I got a response that the attachment is too large. I *think* you should have gotten it through the direct response, but in case you didn’t the link is: https://www.dropbox.com/s/tu6ikjtdn3d0p93/e01.out.mask.pfb?dl=0

Be Well,
James 

—
P. James Dennedy-Frank
Postdoctoral Scholar
Lawrence Berkeley Lab
pj...@lbl.gov

On Wed, Dec 29, 2021 AM , P. James Dennedy-Frank wrote:

Hi Reed,

I didn’t generate this file—I’m trying to get the details on how it was done, but haven’t been able to get those yet. 

The 0 value for z is actually correct—my apologies, the note about “Should be” is from some previous work. The problem for the location is with the ComputationalGrid.Lower.X,Y values — the problem runs if those are 0.0, but not with the correct x0,y0 values of 619696.3427 and 4220655.0278. I will note that these x0,y0 values don’t match the solid file exactly because the setup had that solid file in a slightly larger background box, but I find it odd that I get the KINSOL_KRYLOV_FAILURE when the solid file is inside the ComputationalGrid but not when it isn’t. 

I’ve attached here the mask created during the ParFlow run. It looks to have the right min and max x and y values (and z values, but that isn’t a problem), and the mask looks to be of the correct solid file area. However, this is the case where the KINSOL_KRYLOV_FAILURE happens. I can take a look at what happens with the mask when running with the ComputationalGrid.Lower.X,Y values are 0,0 but I’ll need to do that in a bit. Let me know what else I might be able to do to diagnose this.

Thanks,

James

[Reed M. Maxwell]

Hi James-

 

Thanks, got it.

 

Reed

 

Reed M. Maxwell, Ph.D.
Professor

Department of Civil and Environmental Engineering

High Meadows Environmental Institute

Director, Integrated GroundWater Modeling Center

Princeton University

maxwell.princeton.edu

igwmc.princeton.edu

 

 

From: P. James Dennedy-Frank <pj...@lbl.gov>

Date: Wednesday, December 29, 2021 at 3:47 PM
To: P. James Dennedy-Frank <pj...@lbl.gov>

[P. James Dennedy-Frank]

Hi Reed,

I wanted to follow up with this and see if you have any thoughts on this. I looked at the mask when I run the simulations with ComputationalGrid.Lower.X,Y = 0,0 and while the simulation runs, it looks like it’s doing so without the solid file—the mask is just a box, all with value 0. This differs from when I use the correct ComputationalGrid.Lower.X,Y = 619696.3427, 4220655.0278 (the correct values)—in this case the mask contains what certainly looks like the correct solid file shape, but I get the KINSOL_KRYLOV_FAILURE. The one thing I’m working on is to get more details on the solid file construction, but as mentioned previously it *looks* like it’s well-formed, and the mask looks okay. I wanted to see if you have any other thoughts on how we might solve this.

Thank you very much for your help!

Be Well,
James

—
P. James Dennedy-Frank
Postdoctoral Scholar
Lawrence Berkeley Lab
pj...@lbl.gov