SFINAE is applied when template argument deduction fails to produce a valid signature (§14.8.3). So the question is whether argument deduction is required to observe the defect, required not to, or that there is no requirement.
It appears that the the invalid type is required to be formed, hence SFINAE is guaranteed. §14.8.2/5:
When all template arguments have been deduced or obtained from default template arguments, all uses of template parameters in the template parameter list of the template and the function type are replaced
with the corresponding deduced or default argument values. If the substitution results in an invalid type,
as described above, type deduction fails.
If that isn't convincing, then try ¶7 (although "used" perhaps needs to be "occurs"; I'm not sure what it means for an expression to be used):
The substitution occurs in all types and expressions that are used in the function type and in template
parameter declarations.
It doesn't matter whether a use of a template parameter in the template parameter list is ultimately unused. All substitutions must be performed.
Perhaps you could weasel your way through this with selective interpretation. But although the pack is allowed to match zero arguments, when it does so it doesn't cease to exist. The pack isn't replaced with a finite sequence of parameters which then match arguments. Something like that would need to happen to make it go away before SFINAE could fail.