Hi Tjark,
you can happily ignore the warning about the map function. This is
relevant only if you use the "old" Quotient package for constant
definitions and proving and Lifting/Transfer.
Concerning the other warning. You can be fine if you don't want do
lifting/transferring that would change 'a in your type 'a Q. Nesting is
an example when this can happen: 'a Q Q => 'a T T.
If you want to support/use this, you have to provide some semantic
information for 'a T (a relator is a part of this). Just look at
HOL/Lifting_Set.thy where such a thing is provided for the type 'a set
and follow analogously. As Andreas already mentioned, this is not
necessary if 'a T is a datatype without "dead" variables.
Hope this helps,
Ondrej