Since there is no way to 'invert' the mask, I use this workaround:
Let's say you have a mask in the image going clockwise. When drawing
this mask, instead of closing it when you're done, continue a bit
beyond the first point. Then move outside the image, and set mask
points so that you get a frame all along outside of the image, going
counterclockwise. Again you go a little further than where you first
stepped outside the image. Now you can close the mask, and lo and
behold, the margin is excluded and the center remains. six more
clicks, but I got used to it and it works. If you only want a simple
rectangular mask, use 'cropping' instead. A simple example for the
first technique would be this:
# w2856 h4290
k i0 t0 p"617 1639 1028 778 2087 796 2418 1594 2105 2490 841 2436 635
1334 -81 1307 -63 4345 2956 4336 2929 -63 -99 -54 -36 1630"
You can save it as xx.msk and load it into hugin in the masking
dialog. (You may want an image of the right size 2856x4290 to see it
right, easy to make one with the gimp)
Kay