First, one typically never works explicitly with a det operator since it is such an awful mathematical object. For that reason, YALMIP does not construct an explicit representation of the det by default. What you want is
Then you get the degree 20 and 3260 term polynomial you want.
A degree 20 polynomial in 10 variables requires O(10^(20/2)) monomials in the basis, thus working with matrices of size 10^10 x 10^10. Ain't gonna happen.
To get anywhere here, you have to do a lot of research on reduced representations for this particular instance, or simply solve it using some other strategy than moments which is nice in theory but horrible in practice most often.