Doing a little research on the net about Quicktime Player 7 (the 32bit version) and macOS Catalina, it comes out, the Catalina stopped supporting 32bit apps. I think, that's why there is no mov format in the output settings of OpenToonz in macOS Catalina.
On Github is a bug report for this too, but it points more in a development direction, that's not tested.
But, this said, I think there are other possibilities for rendering an alpha channel video:
»In this case no image has to be used as background, and the output
file format has to support alpha channel information, e.g. TIF at 32 or
64 bits per pixel, TGA at 32 bits per pixel, or MOV with codecs
supporting alpha (i.e. Animation).
To render animation with alpha channel information:
- Choose Xsheet → Scene Settings….
- Set the alpha channel of the Camera BG Color to 0 (i.e. transparent).
- Choose an output file format supporting alpha channel information.«