Ive thought about hiding a luma keyed frame used to subtract alpha from
the source in an alt-ref-frame.
Possibly the normal decoder would not be told to reference the alt frame
in the bytestream, whereas your player could assume alt-ref's we to be
used to subtract alpha.
For instance, the alt ref frame could contain the image of the
bluescreen that was used without any actors, and then subtract the
values in real time.
I did a similar operation using flv feeds that required a 32bit depth
from a 24 bit source. the blue/green screen stage was sent in a jpg and
used to subtract the alpha , converting the 24 bit image to a 32bit
image in the process.
Fun times
Andy