Removing anti-support for AIJ matrix types in PFLOTRAN?

1 view
Skip to first unread message

Mills, Richard Tran

unread,
Jan 6, 2026, 4:08:37 PM (4 days ago) Jan 6
to pflotr...@googlegroups.com
Dear PFLOTRAN developers,

We currently have some checks in PFLOTRAN that cause it to bail out if it the user has specified that a PETSc AIJ matrix should be used, e.g., in pmc_subsurface.F90:

      if (solver%M_mat_type == MATAIJ .and. &
          option%iflowmode /= RICHARDS_MODE) then
        option%io_buffer = 'AIJ matrix not supported for current &
          &mode: '// option%flowmode
        call PrintErrMsg(option)
      endif

I want to be able to use AIJ and related matrix classes, because currently all of the GPU support in PETSc relies on subclasses of MATAIJ (though I want to add BAIJ support soon), and because many types of preconditioners work only with AIJ matrices.

I would like to submit an MR to remove these nearly all of these checks (the CPR preconditioner ones should remain, since that one depends explicitly on the BAIJ layout); I believe that I have pretty much everything (except the aforementioned CPR preconditioner) working with AIJ now. Right before the holiday break, I spent a few days tracking down and fixing a very difficult bug in PETSc that has been the cause of mysterious problems with AIJ matrices in PFLOTRAN for several years: See PETSc merge request https://gitlab.com/petsc/petsc/-/merge_requests/8910

Before submitting a pull request to remove these checks, I want to ask

  1. Does anyone on the pflotran-dev list remember specific problems they previously ran into with AIJ, so that I can check to see that these have been resolved?
  2. Is it OK to change (in the PR) the supported version of PETSc from v3.24.0 to v.3.24.3? This will bump the PETSc patchlevel up to the latest tag in the 3.24 release that includes my fix; because this is only a change in the patchlevel, there should be no API changes. (And if this is OK, am I correct in believing that this entails changing the contents of .gitlab/petsc-git-version.txt and PETSC_VERSION_PATCH in pflotran_constants.F90?)

Thanks,
Richard

Hammond, Glenn E

unread,
Jan 7, 2026, 1:11:33 PM (3 days ago) Jan 7
to pflotr...@googlegroups.com
Richard,

I have no concerns with removing these error messages if the PFLOTRAN infrastructure also supports AIJ. For RICHARDS (without the well model), this is a non-issue as there is a single dof per cell. For other flow modes, we would need to modify routines in petsc_utility.F90 (the wrapper for PETSc MatSetValueXXX routines) to unpack the blocks and call the AIJ variants. 

See my other comments below in red.

Glenn

From: 'Mills, Richard Tran' via pflotran-dev <pflotr...@googlegroups.com>
Date: Tuesday, January 6, 2026 at 1:08 PM
To: pflotr...@googlegroups.com <pflotr...@googlegroups.com>
Subject: [pflotran-dev: 6363] Removing anti-support for AIJ matrix types in PFLOTRAN?

Check twice before you click! This email originated from outside PNNL.

Dear PFLOTRAN developers,

We currently have some checks in PFLOTRAN that cause it to bail out if it the user has specified that a PETSc AIJ matrix should be used, e.g., in pmc_subsurface.F90:

      if (solver%M_mat_type == MATAIJ .and. &
          option%iflowmode /= RICHARDS_MODE) then
        option%io_buffer = 'AIJ matrix not supported for current &
          &mode: '// option%flowmode
        call PrintErrMsg(option)
      endif

I want to be able to use AIJ and related matrix classes, because currently all of the GPU support in PETSc relies on subclasses of MATAIJ (though I want to add BAIJ support soon), and because many types of preconditioners work only with AIJ matrices.

I would like to submit an MR to remove these nearly all of these checks (the CPR preconditioner ones should remain, since that one depends explicitly on the BAIJ layout); I believe that I have pretty much everything (except the aforementioned CPR preconditioner) working with AIJ now. Right before the holiday break, I spent a few days tracking down and fixing a very difficult bug in PETSc that has been the cause of mysterious problems with AIJ matrices in PFLOTRAN for several years: See PETSc merge request https://gitlab.com/petsc/petsc/-/merge_requests/8910

Before submitting a pull request to remove these checks, I want to ask

  1. Does anyone on the pflotran-dev list remember specific problems they previously ran into with AIJ, so that I can check to see that these have been resolved?
  1. These error messages pre-date most of the current devs. So, I doubt it.
  1. Is it OK to change (in the PR) the supported version of PETSc from v3.24.0 to v.3.24.3? This will bump the PETSc patchlevel up to the latest tag in the 3.24 release that includes my fix; because this is only a change in the patchlevel, there should be no API changes. (And if this is OK, am I correct in believing that this entails changing the contents of .gitlab/petsc-git-version.txt and PETSC_VERSION_PATCH in pflotran_constants.F90?)
  1. This should should be “OK”. However, I need to re-build the docker image used in CI so that it uses 3.24.3. We do not re-build PETSc every time we run CI on GitLab.

Thanks,
Richard
--
You received this message because you are subscribed to the Google Groups "pflotran-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pflotran-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pflotran-dev/CO6PR09MB7352F23C7C9BD9B0DA74F7F1B887A%40CO6PR09MB7352.namprd09.prod.outlook.com.

Mills, Richard Tran

unread,
Jan 7, 2026, 2:24:38 PM (3 days ago) Jan 7
to pflotr...@googlegroups.com
Thanks, Glenn. Regarding the routines in petsc_utility.F90, which ones in there do you think will not work with AIJ matrices? Even when we are using AIJ matrices in PETSc, they get an associated matrix block size assigned and all of the PETSc routines for using the blocked interfaces are supposed to work with them. (That is, if you give PETSc blocks it should internally handle the unpacking and calling the appropriate things for AIJ.) If there is something that we've missed, then I should fix it in PETSc. Or am I misunderstanding you?

Aside: I was trying to remember when we first added support for using AIJ matrices with the blocked matrix APIs and who did it. Looking through the PETSc commit logs, I was surprised to find that the answers is... me! I had absolutely forgotten doing that. And it was way, waaaay back in 2008. I can no longer ignore it: I am old.

Cheers,
Richard
From: 'Hammond, Glenn E' via pflotran-dev <pflotr...@googlegroups.com>
Sent: Wednesday, January 7, 2026 10:11 AM
To: pflotr...@googlegroups.com <pflotr...@googlegroups.com>
Subject: [pflotran-dev: 6364] Re: Removing anti-support for AIJ matrix types in PFLOTRAN?
 
Richard, I have no concerns with removing these error messages if the PFLOTRAN infrastructure also supports AIJ. For RICHARDS (without the well model), this is a non-issue as there is a single dof per cell. For other flow modes, we would need
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
 
ZjQcmQRYFpfptBannerEnd

Mills, Richard Tran

unread,
Jan 7, 2026, 4:46:32 PM (3 days ago) Jan 7
to pflotr...@googlegroups.com
Actually, perhaps what I should really have asked is: What are some regression tests (or shortcourse problems) that you expect might break when I try to use AIJ matrices?

--Richard
From: 'Mills, Richard Tran' via pflotran-dev <pflotr...@googlegroups.com>
Sent: Wednesday, January 7, 2026 11:24 AM
To: pflotr...@googlegroups.com <pflotr...@googlegroups.com>
Subject: [pflotran-dev: 6365] Re: Removing anti-support for AIJ matrix types in PFLOTRAN?
 
Thanks, Glenn. Regarding the routines in petsc_utility. F90, which ones in there do you think will not work with AIJ matrices? Even when we are using AIJ matrices in PETSc, they get an associated matrix block size assigned and all of the PETSc

Hammond, Glenn E

unread,
Jan 8, 2026, 1:04:45 PM (2 days ago) Jan 8
to pflotr...@googlegroups.com
If PETSc handles the mapping of BAIJ to AIJ under the hood. There should be no issues on the PFLOTRAN side.

Glenn

From: 'Mills, Richard Tran' via pflotran-dev <pflotr...@googlegroups.com>
Date: Wednesday, January 7, 2026 at 11:24 AM
To: pflotr...@googlegroups.com <pflotr...@googlegroups.com>
Subject: [pflotran-dev: 6365] Re: Removing anti-support for AIJ matrix types in PFLOTRAN?

Thanks, Glenn. Regarding the routines in petsc_utility.F90, which ones in there do you think will not work with AIJ matrices? Even when we are using AIJ matrices in PETSc, they get an associated matrix block size assigned and all of the PETSc routines for using the blocked interfaces are supposed to work with them. (That is, if you give PETSc blocks it should internally handle the unpacking and calling the appropriate things for AIJ.) If there is something that we've missed, then I should fix it in PETSc. Or am I misunderstanding you?

Aside: I was trying to remember when we first added support for using AIJ matrices with the blocked matrix APIs and who did it. Looking through the PETSc commit logs, I was surprised to find that the answers is... me! I had absolutely forgotten doing that. And it was way, waaaay back in 2008. I can no longer ignore it: I am old.

Cheers,
Richard
From: 'Hammond, Glenn E' via pflotran-dev <pflotr...@googlegroups.com>
Sent: Wednesday, January 7, 2026 10:11 AM
To: pflotr...@googlegroups.com <pflotr...@googlegroups.com>
Subject: [pflotran-dev: 6364] Re: Removing anti-support for AIJ matrix types in PFLOTRAN?
 
This Message Is From an External Sender
This message came from outside your organization.
 

Hammond, Glenn E

unread,
Jan 8, 2026, 1:14:33 PM (2 days ago) Jan 8
to pflotr...@googlegroups.com
From: 'Mills, Richard Tran' via pflotran-dev <pflotr...@googlegroups.com>
Date: Wednesday, January 7, 2026 at 1:46 PM
To: pflotr...@googlegroups.com <pflotr...@googlegroups.com>
Subject: [pflotran-dev: 6366] Re: Removing anti-support for AIJ matrix types in PFLOTRAN?

Actually, perhaps what I should really have asked is: What are some regression tests (or shortcourse problems) that you expect might break when I try to use AIJ matrices?

--Richard
From: 'Mills, Richard Tran' via pflotran-dev <pflotr...@googlegroups.com>
Sent: Wednesday, January 7, 2026 11:24 AM
To: pflotr...@googlegroups.com <pflotr...@googlegroups.com>
Subject: [pflotran-dev: 6365] Re: Removing anti-support for AIJ matrix types in PFLOTRAN?
 
This Message Is From an External Sender
This message came from outside your organization.
 
Thanks, Glenn. Regarding the routines in petsc_utility.F90, which ones in there do you think will not work with AIJ matrices? Even when we are using AIJ matrices in PETSc, they get an associated matrix block size assigned and all of the PETSc routines for using the blocked interfaces are supposed to work with them. (That is, if you give PETSc blocks it should internally handle the unpacking and calling the appropriate things for AIJ.) If there is something that we've missed, then I should fix it in PETSc. Or am I misunderstanding you?

Aside: I was trying to remember when we first added support for using AIJ matrices with the blocked matrix APIs and who did it. Looking through the PETSc commit logs, I was surprised to find that the answers is... me! I had absolutely forgotten doing that. And it was way, waaaay back in 2008. I can no longer ignore it: I am old.

Cheers,
Richard
From: 'Hammond, Glenn E' via pflotran-dev <pflotr...@googlegroups.com>
Sent: Wednesday, January 7, 2026 10:11 AM
To: pflotr...@googlegroups.com <pflotr...@googlegroups.com>
Subject: [pflotran-dev: 6364] Re: Removing anti-support for AIJ matrix types in PFLOTRAN?
 
This Message Is From an External Sender
This message came from outside your organization.
 
Reply all
Reply to author
Forward
0 new messages