Hello,
Since the variable is x and that is a real number, the problem as stated would not fit as an optimization problem on symfixedrankYYcomplexfactory.
You might reformulate it by introducing a new variable Z and optimize this:
min || G - xB - Z ||^2
for the variables x in R and Z in symfixedrankYYcomplexfactory.
If somehow you can minimize this down to 0, then indeed G - xB = Z, and Z satisfies the constraints, so you would be fine.
I don't know if that's the best approach though. This is really a one-dimensional search problem, with quite a bit of structure. Also, the matrix G - xB is the form that comes up naturally when considering the
matrix pencil of (G, B). Since you want G-xB to have rank k (presumably that's < n), then we know det(G - xB) = 0. In other words, x must be an "eigenvalue" for the pencil (G, B). In Matlab, you can compute those with eig(G, B). Then, you are left with a finite number of candidate values for x that you can try one by one. If none of those succeed, then you know there is no solution.
Best,
Nicolas