It may be not your problem but this same error happened if you are with Java and using GAE SDK 1.5.5 because Google included jar datanucleus-appengine.1.0.9 by mistake (where XG transactions are not supported). They notice and ask you to update that jar to datanucleous-appengine.1.0.10final , you can find it
here.
Another reason may be that XG transactions are limited to operations between a max of five entity groups, but I don't know if this is the exception returned in that case.