WMT GenMT constrained model size clarification

[cep...@gmail.com]

Hi all,
I have a question about the 20B model limit and rescoring for constrained track. If I have multiple translation models under 20B producing hypotheses and a rescoring model to select the best hypothesis (under 20B as well), does that count as a constrained submission?
If not, what about the situation where all of the models are the same base model with different small LoRA adapters on top? E.g. an 8B model with multiple adapters (tens of Ms of parameters each, under 20B parameters combined) and the same base model with a different LoRA adapter for rescoring, is that constrained?
Thanks,
Josef

[Kocmi T.]

Hi Josef,

That is a great question, firstly we want to highlight that there is not a simple definition that would fit all scenarios nicely (same point is unfairness for MoE models vs. dense models). For simplicity, we have a simple rule: sum of all parameters of all used models must be below 20B to participate in a constrained track.

However and most importantly in your paper submission, you should highlight the situation that it is closer to constrained track. And same in the GenMT Findings we will highlight the differences as well, do explain it in the Model card poll.

For your specific situation, while the ensemble should be clearly a sum of all parameters, the Lora is a more challenging situation and theoretically closer to constrained (parameters of single model + tiny bit for Lora). However, at the end to use multiple models with various Lora adapters for inference one would still need to have all models deployed on GPUs (could be done by model swapping though) technically turning it into a similar situation like with MoE. Thus we won't be updating the rule about constrained/unconstrained track.

Thank you for participating and have a great day,

Kocmi

--
You received this message because you are subscribed to the Google Groups "WMT: Workshop on Machine Translation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wmt-tasks+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/wmt-tasks/40b3277a-068b-46f4-ab46-8090e7aa67e1n%40googlegroups.com.