Ran into similar problem recently and this is the only related post I search out. Just post my understanding (I am not sure this solution is completely correct) here in case someone else with similar questions can search it. I know this is an outdated post.
1. 2DoF Rotation
This can be obtain by modifying the QuaternionParameterization s.t. [delta_x, delta_y, delta_z] is changed to [delta_x, delta_y, 0]. The jacobian computation should also be modified (basically remove the third column).
2. Rotation in local or global frame
I am optimizing a R, s.t. X_global = Q * X_local. The ceres QuaternionParameterization perturbs Q as dQ * Q. Since Q * X_local is a point in global frame, dQ * Q * X_local should be pertubed in the global frame. If we want to rotate in local frame, we probably should perturb Q as Q * dQ.
3. Rotation of a gravity
Should probably rotate (0, 0, 1) by dQ = [delta_x, delta_y, 0] in the local frame, I guess.