Hi, Martin,
Your scenario is localization, and it is great use for ISAM2, but structureless factors are actually a concept in SLAM. You're still estimating the landmarks there.
My answer on how to do this would depend on what language you're trying to do this in. It's in Python or MATLAB, probably creating a Pose3 (or NavState) custom factor for each timestamp that evaluates the re-projection error for all known landmarks seen from
that timestamp in parallel is going to be the fastest and most convenient.
Frank