Hi,
It is not possible in full generality because an inequality between two quadratic forms (or norms) like that is not convex.
Perhaps you can heuristically optimize for some tradeoff combination of
factor_risk + lambda*spec_risk
and depending on the other constraints it may lead to a desired balance, but directly comparing the difference or quotient of those is not possible.
Best,
Michal