It's not entirely clear what you're doing, but here's how PTR records work in VegaDNS:
In the case of IPv4, you must create the domain ending in in-addr.arpa that has been delegated to your DNS servers, and then add the records to that domain. The Hostname will end in the domain name, including the in-addr.arpa suffix, and the value will be what you want it to resolve to, i.e.
server1.mydomain.com.
In the case of IPv6, you can use AAAA+PTR, and the AAAA as well as PTR records will be created, but the caveat is that it won't actually resolve unless you separately create the PTR domain (ending in ip6.arpa in this case). You can also just use a PTR record for your ip6.arpa domain.
So what you describe sounds like you're trying to add a PTR record to a non in-addr.arpa domain, which will fail validation, and even if it didn't, wouldn't resolve anyway.
Cheers,
Bill