You cannot. You either have to be clever and re-parameterize your model so it is expressed in terms of that inverse instead completely, and thus use that as the decision variable
The only way to use an inverse inv(X) is to introduce a new variable Y and add the constraint X*Y == eye(n), which of course leads to a horrible nonconvex bilinear model