System test: please verify you can build the HelloSwift target in the j2objc-2.7/examples/HelloSwift/HelloSwift.xcodeproj Xcode project. It's a horrible app (we're not UI engineers), but demonstrates how to set up a simple Swift app that references a Java class. If that builds and runs, compare its settings with your project's.
An XCFramework should be treated as an opaque building block, even though you can peek inside it from the command line. It uses separate libraries for each support architecture/platform, but its Info.plist provides the information so that the application linker can pick which library it needs without specific configuration. Previously, Static Frameworks combined all libraries into a "fat-library" for distribution, but conflicts between platforms and CPU types grew as platform/CPU combinations expanded. For example, the MacOS, Mac Catalyst and iOS Simulator builds all target the x86_64 CPU, so they couldn't be in the same fat-library.
Although header files can be individually assessed if you hack your include paths, normally all of the framework's declarations are included with a single header that has the same name as the framework. For example:
#import "java/lang/System.h"
#import "javax/crypto/KeyAgreement.h"
#import "org/xml/sax/Parser.h"
can all be replaced with:
#import <JRE/JRE.h>
So there's normally no reason to look inside a framework or add build references to its internals, except when debugging a build failure.