I'm relatively new to the V8 code base, but I recently noticed that the x64 code-generation logic for instructions which involve both vector (XMM) and integer (GPR) registers seems to have a flaw -- although the code works b/c the number of integer GPRs and XMM registers (in AVX2-class machines) are less-than or equal at 16; and for an AVX3/AVX-512 machine there will be a difference (16 vs. 32), but the number of XMM registers is still larger than the number of integer GPRs.
The issue occurs in-and-around callers of vinstr(), an example of which is vcvtlsi2sd()
void vcvtlsi2sd(XMMRegister dst, XMMRegister src1, Register src2) {
XMMRegister isrc2 = XMMRegister::from_code(src2.code());
vinstr(0x2a, dst, src1, isrc2, kF2, k0F, kW0);
Here, the integer GPR register argument (src2) is treated as an XMM register when calling from_code(). This means that the incoming register code is "subject to" the limits of the XMMRegister incarnation of RegisterBase.
i.e. it's subject to the kDoubleAfterLast limit, instead of the kRegAfterLast limit.
If the XMMRegister space is equal to or larger than the integer GPR space, then this "works" (b/c the check is a simple less-than) -- but if there is ever an inversion (where the number of GPRs exceeds the number of XMM registers), then the code will fail.
This isn't a functional bug right now, but it seems like the logic which is coercing a Register to XMMRegister is incorrect. Is this worth a fix? It affects downstream code of course, as the type signatures of functions called from here "down" all assume XMMRegisters as arguments (for the most part).