Enzyme Multisource C++ Builds

11 views
Skip to first unread message

Mark Abate

unread,
Jul 14, 2026, 5:42:24 PM (7 days ago) Jul 14
to Enzyme AD
Hi all,

I've been working on figuring out the build process for multisource projects with Enzyme. I tried to isolate a small issue in a dummy project (source files at the bottom of this email) and was wondering if anyone had any thoughts on the build process I'm using.

These are the build commands:
clang++ -flto -c multisource.cpp -o multisource.o
clang++ -flto -c math_utils.cpp -o math_utils.o
clang++ -fuse-ld=lld -flto -Wl,-mllvm -Wl,-load=/global/homes/m/mabate/src/Enzyme/build/Enzyme/LLDEnzyme-21.so -Wl,--load-pass-plugin=/global/homes/m/mabate/src/Enzyme/build/Enzyme/LLDEnzyme-21.so  multisource.o math_utils.o -o multisource

My goal here is to build each source file separately without running the Enzyme derivative pass, and then link at the end with the LLDEnzyme plugin.

The error you see during the link step is:
ld.lld: error: <unknown>:0:0: in function main i32 (): Enzyme: Cannot cast __enzyme_autodiff primal argument 1, found i32 0, type i32 (simplified to i32 0 )  - to arg 0, double

It seems like Enzyme is not picking up on enzyme_const/out variable names correctly in this case. The build works fine if you copy the function in math_utils.cpp into multisource.cpp (i.e. for single source builds, the derivative call above seems to be correct).

Another known issue we've had with the above build process is that since the compile step doesn't use the Enzyme plugin, it's not clear if attributes like enzyme_inactive are handled correctly (the compiler says in a warning that it's ignoring them). So we've also tried compiling with the ClangEnzyme with enzyme-enable=0 (so the derivative calculation doesn't happen until the link step). But we're not sure if that's the correct way to do this, so any advice on that would be appreciated too. Here is an example of what that build process looks like:
clang++ -flto -fplugin=/global/homes/m/mabate/src/Enzyme/build/Enzyme/ClangEnzyme-21.so -mllvm -enzyme-enable=0 -c multisource.cpp -o multisource.o
clang++ -flto -fplugin=/global/homes/m/mabate/src/Enzyme/build/Enzyme/ClangEnzyme-21.so -mllvm -enzyme-enable=0 -c math_utils.cpp -o math_utils.o
clang++ -fuse-ld=lld -flto -Wl,-mllvm -Wl,-load=/global/homes/m/mabate/src/Enzyme/build/Enzyme/LLDEnzyme-21.so -Wl,--load-pass-plugin=/global/homes/m/mabate/src/Enzyme/build/Enzyme/LLDEnzyme-21.so  multisource.o math_utils.o -o multisource


## Files in the dummy project:

multisource.cpp
#include <iostream>
#include "math_utils.h"

int enzyme_const, enzyme_out, enzyme_dup;
template <typename RT, typename... T>
RT __enzyme_autodiff(void*, T...);

int main() {
    double x = 2.0;
    double y = 3.0;
    double dmult_dy = __enzyme_autodiff<double>((void*) mult, enzyme_const, x, enzyme_out, y);
    std::cout << dmult_dy << "\n";
    return 0;


math_utils.cpp
#include "math_utils.h"

double mult(double x, double y) {
    return x*y;
}


math_utils.h
#ifndef MATH_UTILS_H
#define MATH_UTILS_H

double mult(double x, double y);

#endif

Gregory Dushkin

unread,
Jul 15, 2026, 4:37:01 AM (6 days ago) Jul 15
to Mark Abate, Enzyme AD
Hi!

I haven't worked with Enzyme in a while, but when I did, I remember
declaring `enzyme_dup`, `enzyme_const` etc. as `extern`. Not sure about
the exact reason (I'd guess that if you don't do that, you're
essentially referring to a different entity? But the devs will probably
correct me on that one) - but this seems to also be the way the in-tree
C++ bindings do it as well:
https://github.com/EnzymeAD/Enzyme/blob/main/enzyme/include/enzyme/utils

And for some unsolicited advice (which I might also get corrected on
xD): I'd just use CMake and hook Enzyme from there if I were you instead
of manually handling the compilation process, and I'd also use the
Enzyme's C++ headers instead of declaring my own.

Multi-TU setup from CMake was pretty easy, it looked something like
that:
```
cmake_minimum_required( VERSION 3.18 )

project( MultiFile LANGUAGES CXX )

include( CPM.cmake/cmake/CPM.cmake )

CPMAddPackage(
GITHUB_REPOSITORY "EnzymeAD/Enzyme"
GIT_TAG "main"
SOURCE_SUBDIR "enzyme"
)
# This doesn't get automaticlly added with LLDEnzymeFlags
include_directories( "${Enzyme_SOURCE_DIR}/enzyme/include" )

# Here I had a single main file that would call autodiff
# on a function declared in a header and provided by either
# lib1 or lib2
add_executable( out1 src/main.cc src/lib1.cc )
add_executable( out2 src/main.cc src/lib2.cc )
target_link_libraries( out1 PRIVATE LLDEnzymeFlags )
target_link_libraries( out2 PRIVATE LLDEnzymeFlags )
```
and it probably didn't change much (if at all).

Hope this helps!

On Wed Jul 15, 2026 at 12:42 AM MSK, Mark Abate wrote:
> Hi all,
>
> I've been working on figuring out the build process for multisource
> projects with Enzyme. I tried to isolate a small issue in a dummy project
> (source files at the bottom of this email) and was wondering if anyone had
> any thoughts on the build process I'm using.
>
> These are the build commands:
> clang++ -flto -c multisource.cpp -o multisource.o
> clang++ -flto -c math_utils.cpp -o math_utils.o
> clang++ -fuse-ld=lld -flto -Wl,-mllvm
> -Wl,-load=/global/homes/m/mabate/src/Enzyme/build/Enzyme/LLDEnzyme-21.so
> -Wl,--load-pass-plugin=/global/homes/m/mabate/src/Enzyme/build/Enzyme/LLDEnzyme-21.so
> multisource.o math_utils.o -o multisource
>
> My goal here is to build each source file separately without running the
> Enzyme derivative pass, and then link at the end with the LLDEnzyme plugin.
>
> The error you see during the link step is:
> *ld.lld: error: <unknown>:0:0: in function main i32 (): Enzyme: Cannot cast
> __enzyme_autodiff primal argument 1, found i32 0, type i32 (simplified to
> i32 0 ) - to arg 0, double*
>
> It seems like Enzyme is not picking up on enzyme_const/out variable names
> correctly in this case. The build works fine if you copy the function in
> math_utils.cpp into multisource.cpp (i.e. for single source builds, the
> derivative call above seems to be correct).
>
> Another known issue we've had with the above build process is that since
> the compile step doesn't use the Enzyme plugin, it's not clear if
> attributes like enzyme_inactive are handled correctly (the compiler says in
> a warning that it's ignoring them). So we've also tried compiling with the
> ClangEnzyme with enzyme-enable=0 (so the derivative calculation doesn't
> happen until the link step). But we're not sure if that's the correct way
> to do this, so any advice on that would be appreciated too. Here is an
> example of what that build process looks like:
> clang++ -flto
> -fplugin=/global/homes/m/mabate/src/Enzyme/build/Enzyme/ClangEnzyme-21.so
> -mllvm -enzyme-enable=0 -c multisource.cpp -o multisource.o
> clang++ -flto
> -fplugin=/global/homes/m/mabate/src/Enzyme/build/Enzyme/ClangEnzyme-21.so
> -mllvm -enzyme-enable=0 -c math_utils.cpp -o math_utils.o
> clang++ -fuse-ld=lld -flto -Wl,-mllvm
> -Wl,-load=/global/homes/m/mabate/src/Enzyme/build/Enzyme/LLDEnzyme-21.so
> -Wl,--load-pass-plugin=/global/homes/m/mabate/src/Enzyme/build/Enzyme/LLDEnzyme-21.so
> multisource.o math_utils.o -o multisource
>
>
> ## Files in the dummy project:
>
> *multisource.cpp*
> #include <iostream>
> #include "math_utils.h"
>
> int enzyme_const, enzyme_out, enzyme_dup;
> template <typename RT, typename... T>
> RT __enzyme_autodiff(void*, T...);
>
> int main() {
> double x = 2.0;
> double y = 3.0;
> double dmult_dy = __enzyme_autodiff<double>((void*) mult, enzyme_const,
> x, enzyme_out, y);
> std::cout << dmult_dy << "\n";
> return 0;
> }
>
>
> *math_utils.cpp*
> #include "math_utils.h"
>
> double mult(double x, double y) {
> return x*y;
> }
>
>
> *math_utils.h*

Mark Abate

unread,
Jul 16, 2026, 7:05:45 PM (5 days ago) Jul 16
to Gregory Dushkin, Enzyme AD
Hi Greg,

Thanks for the response! Adding the extern keyword fixed the build for that example. I guess without extern the variable name is not propagated correctly to LLDEnzyme. The linker flags we're using should agree with what Enzyme sets through cmake (link), but we will try to hook into the cmake configuration in the future.

We're running into another slightly different problem now, but I thought I would continue the conversation here to see if anyone had any ideas.

We're ultimately trying to differentiate functions that use constructs from the AMReX library for adaptive mesh refinement (many physical simulation codes at Berkeley Lab are built on top of this framework).   

I've been trying to build this AMReX example with Enzyme. This is adapted from their heat equation example, but I've pared down the logic so the function run_simulation just creates an MFIter object and then returns. When we try to build this, Enzyme's type analysis seems to enter an infinite loop (runs for > 4 hours). Another thing we tried was adding __attribute__((enzyme_inactive)) in different locations (e.g. to the MFIter constructor/destructor), but that didn't seem to help.

Here's our build flags and the output:
LLVM_VERSION = 21
ENZYME_CXXFLAGS = -flto -fplugin=/global/homes/m/mabate/src/Enzyme/build/Enzyme/ClangEnzyme-21.so -mllvm -enzyme-enable=0
ENZYME_LDFLAGS = -fuse-ld=lld -flto -Wl,-mllvm,-load=/global/homes/m/mabate/src/Enzyme/build/Enzyme/LLDEnzyme-21.so -Wl,--load-pass-plugin=/global/homes/m/mabate/src/Enzyme/build/Enzyme/LLDEnzyme-21.so -Wl,--lto-O1 -Wl,-mllvm,-enzyme-print=1 -Wl,-mllvm,-enzyme-print-activity=1 -Wl,-mllvm,-enzyme-print-type=1 -Wl,-mllvm,-enzyme-globals-default-inactive=1



Ideally we would like Enzyme to ignore the MFIter object entirely since we know it doesn't contribute to the gradient of the function, but we're not clear on how to do that in practice. There are functions we would be interested in differentiating that would require a loop over this type of iterator, so we wouldn't be able to extract it out of the function.

Best,
Mark

Gregory Dushkin

unread,
Jul 17, 2026, 10:42:23 AM (4 days ago) Jul 17
to Mark Abate, Gregory Dushkin, Enzyme AD
Great! Glad I could be of help!

It just so happens that I also can chip in a little when it comes to the
"infinite compilation" issue. That is, I can confirm that I've also
encountered it around two years ago. Sadly, I've never found a simple
reproducer, so I never reported it, and looking at the issues now I'm
guessing no one else has found a sensible reproducer too - or maybe it's
a completely avoidable issue that I just didn't know how to solve :)

What I did find about it was the following (from memory and whatever
little chatlogs I have found atm):
1. This does not appear to be an infinite loop, because, when I added
debug output to it, it didn't look looped, but rather like just a
very complex algorithm that just grew inifnitely large on the input
code. Although it could be that the looping part itself was just so
large that I just couldn't possibly see the repeating structure
2. I tried profiling the compiler (I won't remember with what tools now)
to see where it was spending the most time in - it pointed me to the
function bool checkedOrIn(const TypeTree&, bool, bool) - apparently,
the compiler spent 67% (I'm not kidding, two years ago this meme
didn't even exist) of the long time I left it running for in it.

IIRC it also did some kind of recursive execution during the analysis
with either no caching or with the inputs juts different enough to not
allow caching - but at this point I already bit off more than I could
chew at the moment and I don't think I really returned to it
afterwards and focused on trying to gradually roll out my own code to
avoid falling in that pit like I did with an external library I was
using.

That's about all I can remember about this issuesoff the top of my head,
I maybe will look more in my private repos with Enzyme experiments or on
my PC with some projects I've never uploaded anywhere when I return to
it in a week or so.

On Fri Jul 17, 2026 at 2:05 AM MSK, Mark Abate wrote:
> Hi Greg,
>
> Thanks for the response! Adding the extern keyword fixed the build for that
> example. I guess without extern the variable name is not propagated
> correctly to LLDEnzyme. The linker flags we're using should agree with what
> Enzyme sets through cmake (link
> <https://github.com/EnzymeAD/Enzyme/blob/3c65e661f643c1594e293e1e796e577beb18dd32/enzyme/Enzyme/CMakeLists.txt#L216-L220>),
> but we will try to hook into the cmake configuration in the future.
>
> We're running into another slightly different problem now, but I thought I
> would continue the conversation here to see if anyone had any ideas.
>
> We're ultimately trying to differentiate functions that use constructs from
> the AMReX <https://github.com/AMReX-Codes/amrex> library for adaptive mesh
> refinement (many physical simulation codes at Berkeley Lab are built on top
> of this framework).
>
> I've been trying to build this
> <https://github.com/BLAST-WarpX/gradient-harness/blob/97e1b8b9a499fe95450eb15d23f4e41826de53f1/src/amrex_heat_equation/main.cpp#L136>
> AMReX
> example with Enzyme. This is adapted from their heat equation example
> <https://github.com/AMReX-Codes/amrex-tutorials/blob/main/ExampleCodes/Basic/HeatEquation_EX0_C/Source/main.cpp#L18>,
> but I've pared down the logic so the function run_simulation just creates
> an MFIter object and then returns. When we try to build this, Enzyme's type
> analysis seems to enter an infinite loop (runs for > 4 hours). Another
> thing we tried was adding __attribute__((enzyme_inactive)) in different
> locations (e.g. to the MFIter constructor/destructor), but that didn't seem
> to help.
>
> Here's our build flags and the output:
> LLVM_VERSION = 21
> ENZYME_CXXFLAGS = -flto
> -fplugin=/global/homes/m/mabate/src/Enzyme/build/Enzyme/ClangEnzyme-21.so
> -mllvm -enzyme-enable=0
> ENZYME_LDFLAGS = -fuse-ld=lld -flto
> -Wl,-mllvm,-load=/global/homes/m/mabate/src/Enzyme/build/Enzyme/LLDEnzyme-21.so
> -Wl,--load-pass-plugin=/global/homes/m/mabate/src/Enzyme/build/Enzyme/LLDEnzyme-21.so
> -Wl,--lto-O1 -Wl,-mllvm,-enzyme-print=1 -Wl,-mllvm,-enzyme-print-activity=1
> -Wl,-mllvm,-enzyme-print-type=1
> -Wl,-mllvm,-enzyme-globals-default-inactive=1
>
> Function we are differentiating
> <https://github.com/BLAST-WarpX/gradient-harness/blob/97e1b8b9a499fe95450eb15d23f4e41826de53f1/src/amrex_heat_equation/main.cpp#L136>
>
> Build output (no enzyme_inactive attributes set)
> <https://gitlab.com/-/snippets/6010801>
> Build output (enzyme_inactive set on m_TheTileArrayCache)
> <https://gitlab.com/-/snippets/6010802>
> Build output (enzyme_inactive set on MFIter constructor/destructor)
> <https://gitlab.com/-/snippets/6010803>
Reply all
Reply to author
Forward
0 new messages