Hello! I am one of the authors of InvariantRing. Thank you for using our package!
I think the unexpected result you are seeing may be related to the fact that a^3-1 is not a minimal polynomial for a over QQ.
The adjoin a cubic root of unity to the rational numbers you can do the following:
i1 : loadPackage "InvariantRing"
o1 = InvariantRing
o1 : Package
i2 : K = toField(QQ[a]/(a^2+a+1))
o2 = K
o2 : PolynomialRing
i3 : S=K[x,y]
o3 = S
o3 : PolynomialRing
i4 : myAction = finiteAction(matrix{{a,0},{0,a}},S)
o4 = S <- {| a 0 |}
| 0 a |
o4 : FiniteGroupAction
i5 : invariants myAction
3 2 2 3
o5 = {y , x*y , x y, x }
o5 : List
As far as I recall, the characteristic zero methods we wrote should work just fine over finite extensions of QQ (though they may run slower).
Also, if you are interested in invariants for actions of abelian groups, I recommend setting up that kind of action since the specialized algorithms for abelian groups are much faster than those for arbitrary finite groups.
Cheers,
Fred