I have never been able to replicate the masks that persist after surface mask field is cleared problem. There are however some oddities with this mechanism anyway. Perhaps if the code for surface masks were rewritten to address these, the problem might be caught in the process.
Surface masks are not bundled.
If a surface mask is not present when a file is opened no warning is given
If a surface mask is not present, then the surface mask field is blank there is no indication a mask has ever been there let alone a red x
The workspace has still got the mask path stored, even if it is resaved with the mask missing and the field blank.
If the mask is present the next time the workspace is launched it is found again and appears in the field.
On the file format issue I can find no difference on either my 10.10 mac mini or 10.8.5.MacBook Pro any difference between the behaviours of the updating of surface masks regardless of forma (psd jpg or png). They update when the surface mask field is clicked, immediately.
I only use jpgs and pngs in qlab and on other video servers, but the reasons for this are probably historical with older video servers and OSs rather for any good modern reason. But I use psds for lots of other things. They open in the finder preview, any software of any printer I have sent them to, and in tons of really bad shareware apps without problems. Mind you I do always save them with maximise compatibility checked (this saves a hidden flattened version as an additional layer, which makes the file size a bit bigger). They don't seem to cause any problems when used as normal cues in Qlab. Is there any reason why they should be any more of a problem in the surface settings? Presumably when Qlab opens and applies the surface mask it flattens it to a grayscale single layer representation of the original file straight away.
I always try and minimise the amount of times I need to redraw a mask to finesse it. I generally output the photoshop canvas full screen to the projector and trace the set directly. I normally find if there aren't too many awkward bezier curves that I can get the mask within 3mm on the first attempt.
Mic