Automatic Differentiation in Zapdos

[Shane Keniley]

Hello all,

As Zapdos and Crane start being used for more complicated systems (e.g. gas mixtures), it's going to be very difficult to generalize the code to handle all possible jacobians. Our jacobians seem to be pretty much perfect for our use cases so far, but if we started adding more capabilities, like a linear model using streamline-upwind petrov-galerkin stabilization or a mixture-averaged diffusion model (like COMSOL!) where every heavy species diffusion coefficient depends on the concentration of every other species, I think it would be beneficial for everyone if we can write code that just works rather than spending days or weeks trying to get the perfect Jacobians.

Point being, I've started rewriting Crane and Zapdos with AD functionality (keniley1/zapdos, ad_kernels branch and keniley1/crane, ad_kernels branch). It seems to be working as advertised so far, which is really nice! I just have a couple questions:

1.) Does AD work with scalar variables and scalar kernels? I use Crane in 0D a lot so this could be useful.

2.) Does AD work with ParsedMaterials? I see ADFunctionDirichletBC in moose so I know it can work with functions in general, but I can't find any examples of it being used in a material. (Many rate coefficients are written in terms of functions that depend on electron temperature, so I would need AD to be able to account for that.)

3.) I noticed that there is no ADSplineInterpolation. Using ADLinearInterpolation to interpolate rate coefficients only causes a slight difference in the final densities so it's not a huge deal, but I'm assuming splines are more accurate. How hard would it be to add AD splines to moose?

-Shane

[Alexander Lindsay]

On Sat, Feb 22, 2020 at 12:54 PM Shane Keniley <keni...@illinois.edu> wrote:

Hello all,

As Zapdos and Crane start being used for more complicated systems (e.g. gas mixtures), it's going to be very difficult to generalize the code to handle all possible jacobians. Our jacobians seem to be pretty much perfect for our use cases so far, but if we started adding more capabilities, like a linear model using streamline-upwind petrov-galerkin stabilization or a mixture-averaged diffusion model (like COMSOL!) where every heavy species diffusion coefficient depends on the concentration of every other species, I think it would be beneficial for everyone if we can write code that just works rather than spending days or weeks trying to get the perfect Jacobians.

Point being, I've started rewriting Crane and Zapdos with AD functionality (keniley1/zapdos, ad_kernels branch and keniley1/crane, ad_kernels branch). It seems to be working as advertised so far, which is really nice! I just have a couple questions:

1.) Does AD work with scalar variables and scalar kernels? I use Crane in 0D a lot so this could be useful.

There is support for doing AD with scalar variables in ADKernels, AD boundary conditions, and ADMaterials, (see moose/test/src/bcs/ADMatchedScalarValueBC.C) but there is not currently an ADScalarKernel. You could create an issue on idaholab/moose and assign me to it. It wouldn't be that much work to implement (you could even try it yourself if you wanted to).

2.) Does AD work with ParsedMaterials? I see ADFunctionDirichletBC in moose so I know it can work with functions in general, but I can't find any examples of it being used in a material. (Many rate coefficients are written in terms of functions that depend on electron temperature, so I would need AD to be able to account for that.)

I actually don't know the answer to the question about ParsedMaterials. I am cc'ing the MOOSE users list, hoping Daniel will answer this. It looks like he added some support to the FunctionParser contrib in libMesh. 

Daniel does have this ticket open on github, but I'm not quite yet sure what he wants yet :-)

3.) I noticed that there is no ADSplineInterpolation. Using ADLinearInterpolation to interpolate rate coefficients only causes a slight difference in the final densities so it's not a huge deal, but I'm assuming splines are more accurate. How hard would it be to add AD splines to moose?

Maybe Daniel's ticket actually is related to this. For all the Interpolation objects other than the LinearInterpolationTempl template, `sample` is hard-coded to accept `Real` arguments, and hence you cannot call it with an AD argument. But even for ADLinearInterpolation you're not going to get the derivative propagation that you might expect from an AD object. The reason for this is that the LinearInterpolation data is still stored as vectors of Reals. So the right way to get derivatives (conceptually) with any Interpolation object (including linear) is to apply the chain rule. E.g. if you have some property:

property = interpolation.sample(coupled_var);

then in order to get the Jacobian right you would need to do:

property = interpolation.sampleDerivative(coupled_var.value()) * coupled_var.derivatives();

The best example of this is in ADPiecewiseLinearInterpolationMaterial.C

-Shane

--
You received this message because you are subscribed to the Google Groups "zapdos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zapdos-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zapdos-users/574bbdd5-c298-4b36-83ac-72cda7249e46%40googlegroups.com.

[Alexander Lindsay]

Thanks Daniel!

On Mon, Feb 24, 2020 at 10:06 AM Daniel Schwen <dan...@schwen.de> wrote:

ParsedMaterials are not yet working with AD. There are two obstacles here. The Function Parser library currently only supports explicit instantiations, and at the time of compiling Function parser the actual MOOSE dual number type is not known. Changing the code organization to allow for late instantiation during the MOOSE build is a bit challenging and dual number support would require a bunch of specializations (FParser frequently casts floats into integers which is forbidden for dual numbers - it requires the use of raw_value).

The other hurdle is the just in time compilation. That one I have already addressed through a recent refactoring of the FParser JIT code. My current way forward is to start supporting AD in parsed materials _only_ for evaluation through the JIT compiled functions. This should be the way functions are evaluated on most systems anyways (if JIT compilation fails we do an interpretation to evaluate the expression - this fallback will only work if FParser is instantiated on dual numbers per the first point).

In the long run I'd love to get rid of FParser entirely and replace it with a new system. I have a parser component written, I have explored multiple approaches to just in time compilation, I'm working on my own function optoimizer/simplification system, I'm working on vector and tensor valued expression etc., but this is not ready yet.

TLDR: AD support for parsed functions is coming, but it will require the JIT compilation to work on your system.

You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CANFcJrE_GzmXuyPu_qBQWXCdsEaL7oXiUSft9t%3D0XbBwrdb3fA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAPxoKqen2B6Ui%3D3W9o1Ud_8PuX_K%3Dizf5wJzuWsTz3Apb0-zSA%40mail.gmail.com.

[Shane Keniley]

Hey Alex,

Sorry for the late response, but thanks for the info! 

There is support for doing AD with scalar variables in ADKernels, AD boundary conditions, and ADMaterials, (see moose/test/src/bcs/ADMatchedScalarValueBC.C) but there is not currently an ADScalarKernel. You could create an issue on idaholab/moose and assign me to it. It wouldn't be that much work to implement (you could even try it yourself if you wanted to).

Alright, I'll take a look at it myself. You're right, it doesn't seem too tricky!

 Daniel's response was understandable, but a little disappointing. One of my primary motivations for getting AD running on Crane and Zapdos was the fact that I don't want to spend time figuring out how to get my parsed rate coefficients into jacobians. I know it's possible with DerivativeParsedMaterial; I was just trying to minimize work. Oh well. Time to get my hands dirty! 

The other reason I'm interested in AD is I want to try to get a linear formulation working instead of this log form we have currently. I believe convergence would be better, but more importantly, I could stop using this log stabilization term. I've always disliked how something that feels so ad hoc has such a strong affect on the simulation performance, and on top of that I think it might be artificially inflating my liquid phase species. Have you given any thought into a linear formulation with Zapdos, or do you advise against it for some reason? I've heard about SUPG and there's also the RDG module in moose which apparently has good performance for advection-dominated problems, but I've never used either so I wasn't sure where to start. 

Thanks Daniel!

To unsubscribe from this group and stop receiving emails from it, send an email to zapdos...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/zapdos-users/574bbdd5-c298-4b36-83ac-72cda7249e46%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to moose...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CANFcJrE_GzmXuyPu_qBQWXCdsEaL7oXiUSft9t%3D0XbBwrdb3fA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to moose...@googlegroups.com.

[Alexander Lindsay]

Daniel, especially if we add GeomReal to libMesh, then libMesh will know the MOOSE DualReal type after configure and would theoretically be able to explicitly instantiate.

On Mon, Feb 24, 2020 at 10:06 AM Daniel Schwen <dan...@schwen.de> wrote:

ParsedMaterials are not yet working with AD. There are two obstacles here. The Function Parser library currently only supports explicit instantiations, and at the time of compiling Function parser the actual MOOSE dual number type is not known. Changing the code organization to allow for late instantiation during the MOOSE build is a bit challenging and dual number support would require a bunch of specializations (FParser frequently casts floats into integers which is forbidden for dual numbers - it requires the use of raw_value).

The other hurdle is the just in time compilation. That one I have already addressed through a recent refactoring of the FParser JIT code. My current way forward is to start supporting AD in parsed materials _only_ for evaluation through the JIT compiled functions. This should be the way functions are evaluated on most systems anyways (if JIT compilation fails we do an interpretation to evaluate the expression - this fallback will only work if FParser is instantiated on dual numbers per the first point).

In the long run I'd love to get rid of FParser entirely and replace it with a new system. I have a parser component written, I have explored multiple approaches to just in time compilation, I'm working on my own function optoimizer/simplification system, I'm working on vector and tensor valued expression etc., but this is not ready yet.

TLDR: AD support for parsed functions is coming, but it will require the JIT compilation to work on your system.

On Sun, Feb 23, 2020 at 2:02 PM Alexander Lindsay <alexlin...@gmail.com> wrote:

You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CANFcJrE_GzmXuyPu_qBQWXCdsEaL7oXiUSft9t%3D0XbBwrdb3fA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAPxoKqen2B6Ui%3D3W9o1Ud_8PuX_K%3Dizf5wJzuWsTz3Apb0-zSA%40mail.gmail.com.

[Alexander Lindsay]

On Tue, Mar 10, 2020 at 12:17 PM Shane Keniley <keni...@illinois.edu> wrote:

Hey Alex,

Sorry for the late response, but thanks for the info! 

There is support for doing AD with scalar variables in ADKernels, AD boundary conditions, and ADMaterials, (see moose/test/src/bcs/ADMatchedScalarValueBC.C) but there is not currently an ADScalarKernel. You could create an issue on idaholab/moose and assign me to it. It wouldn't be that much work to implement (you could even try it yourself if you wanted to).

Alright, I'll take a look at it myself. You're right, it doesn't seem too tricky!

 Daniel's response was understandable, but a little disappointing. One of my primary motivations for getting AD running on Crane and Zapdos was the fact that I don't want to spend time figuring out how to get my parsed rate coefficients into jacobians. I know it's possible with DerivativeParsedMaterial; I was just trying to minimize work. Oh well. Time to get my hands dirty! 

Hopefully we can make this easier in the future (see my response to Daniel today).

The other reason I'm interested in AD is I want to try to get a linear formulation working instead of this log form we have currently. I believe convergence would be better, but more importantly, I could stop using this log stabilization term. I've always disliked how something that feels so ad hoc has such a strong affect on the simulation performance, and on top of that I think it might be artificially inflating my liquid phase species. Have you given any thought into a linear formulation with Zapdos, or do you advise against it for some reason? I've heard about SUPG and there's also the RDG module in moose which apparently has good performance for advection-dominated problems, but I've never used either so I wasn't sure where to start. 

With the fact that oftentimes our problems are advection-dominated, we are susceptible to concentration oscillations when using continuous finite elements. When using a linear form, this means that concentration can go negative, which I believe can be bad/fatal in some cases; I know I started with the linear form and then moved to the log form; I like to think there was a good reason.

I understand your distaste for the log form and I share it. It would be great to stop using it. I'm fully supporting of adding SUPG and seeing how it goes. We may also choose to add shock-capturing terms. You can look at Benjamin Kirk's dissertation for how to include those terms: https://repositories.lib.utexas.edu/handle/2152/13302. It's not too crazy.

The benefit of using AD is most apparent if you do indeed pursue SUPG (and possibly shock-capturing): those stabilization terms can be non-linear and it would be a lot easier to just think about coding up the residuals and let AD worry about the Jacobians.

I believe that RDG has really only been vetted in 1D.

Finite volumes are coming down the pipeline, but that's probably going to be a while until it's ready for widespread use.

Alex

To unsubscribe from this group and stop receiving emails from it, send an email to zapdos-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zapdos-users/8d7e3745-7f34-43c5-9ac5-fc14449a20d6%40googlegroups.com.