Nested reverse and forward-mode Sacado memory leak

[Doug Shi-Dong]

Hello everyone,

I found that Sacado's nested reverse and forward mode AD leads to a memory leak when use multiple times. I have opened an issue on their end:

https://github.com/trilinos/Trilinos/issues/7741

and the steps to reproduce are pretty simple.

However, I've found that the community here is pretty good at responding. I was wondering if anyone here has previously encountered/fixed this. I know that J-P Perlteret's code-gallery example Quasi_static_Finite_strain_Compressible_Elasticity with this AD type turned on by using

set Automatic differentiation order = 2

does lead to a memory leak.

Basically, for every cell's computation, a little bit of memory leaks, which quickly blows up for large problems. 

Please let me know what you think. It's likely completely in the hand of the Sacado group unless I start digging in their code. However, if it is something that will likely not get fixed, we might want to put a warning on the "Supported automatic differentiation libraries" page.

Best regards,

Doug

[Jean-Paul Pelteret]

Hi Doug,

Yes, sadly I have experienced the same but wasn’t able to allot the time to figure out if it was indeed a bug in Sacado (and maybe something that’s already been fixed upstream in a newer version of Trilinos than I was using) or something that we did. I’m really glad that you managed to supply them with a MWE that definitely reproduces the issue, since this disambiguates their code from our drivers. Thank you for that. For your interest, I’ve attached the MWE that I made at the time. I can only suppose that back then I wasn’t confident that I’d recreated the conditions for the leak, because the valgrind logs that I still have didn’t report and memory as being “definitely lost”. I guess the trick would have been to have stuck all of this in a loop, which I clearly didn’t think to do :-/

Since you’ve confirmed that you’ve experienced the same, I think that it would be worthwhile to add a note to this effect in our documentation. If you'd like to do so yourself, then free to make a suggestion where you think it would be appropriate to be. My first inclination would have been where the enumerated type sacado_rad_fad is documented, i.e.

https://github.com/dealii/dealii/blob/master/include/deal.II/differentiation/ad/ad_number_types.h#L87

but of course that’s completely up for discussion. Either way, I think that this should be documented because this issue may persist for some time, and it would be good to have a warning in place until we know which version of Trilinos has a fix in place. Perhaps we should also add a link there to your issue on their GitHub page so that the reader can easily find where to check the status of the issue?

I hope that you’ve spotted the sacado_fad_fad alternative that should work without issue. In many cases I would expect it to be less performant than the (buggy) Rad-Fad implementation but they both still have, at the very least, some use in debugging or prototyping.

Best,

Jean-Paul

[Doug Shi-Dong]

Thanks for the answer! I am currently using the master branch of Trilinos, to make sure that a newer version did not fix this.

I had some second-guessing with how I had configured Sacado itself since there seems to be some options that changes how the reverse mode reuses the memory through RAD_REINIT. However, I've been unable to compile with this flag's options. Hopefully they can shed some light on the issue.

Sounds good. I'll create a pull-request to update the documentation in  

https://github.com/dealii/dealii/blob/master/include/deal.II/differentiation/ad/ad_number_types.h#L87

and in 

https://github.com/dealii/dealii/blob/master/doc/doxygen/headers/automatic_and_symbolic_differentiation.h#L303

referring to the AD number type.

I've actually implemented sacado_fad_fad first because I understood it theoretically. Turns out, it was a bottleneck in my code, so I need sacado_rad_fad. Might just switch the AD package.

Thanks again, following-up with a documentation PR https://github.com/dealii/dealii/pull/10818

Doug

[Jean-Paul Pelteret]

Hi Doug,

Many thanks for the patch that documents this issue. I hope that the Trilinos folk are willing and able to help you (and us) out. If you get any further with this problem then please do let us know — especially if there’s something that we can do on our side (e.g. if it turns out that threes some missing call to free memory, or a required compile-time setting for Sacado that we can check for, etc.).

Best,

Jean-Paul

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/3551c670-3e01-4cd4-afa6-8d06e92c03den%40googlegroups.com.