I am using a reference position over the whole graph assuming that my scenarios are quite localized (within a few km max).
Going over longer distances where a fixed reference would not hold over the whole graph is an interesting topic to discuss. My first thought is to update the reference periodically and store the references over windows, this for sure would affect the whole optimization (jumps when you update)... If the graph is "chained" I guess the impact would not be so big and you could do more sub-graphs, one with its own reference.
The classical approach to define the state as ECEF/LLH, attitude to ENU, biases,.. etc I assume would require a "custom" IMU factor that "knows" to work with LLH and that attitude... that might not be very straightforward. I am interested in the topic, I will come back when I get more time to think about it.