Aha, got it. Just posting to help others.
Suppose you create a problem variable called prob, add stuff to it, solve it etc.
Now you want to change prob, and then resolve it (since building it again from scratch can be time-consuming).
To add variables, do it normally
To remove variables, can just ignore them in the constraints or set them equal to zero.
To add constraints, do it normally
To remove constraints, do: del prob.constraints[constraint_name]
To change the objective, do it normally
Something to watch out for: duplicating problem variables leads to problems