Auditory masking is stimului specific. There aren't any formulas.
Auditory masking is modeled in IEC60268-16 for the calculation of
speech intelligibility (STI).
The amount of auditory masking is captured in tabular form for some
relevant octave band levels.
The standard refers to: "Zwicker, E. and Feltkeller, Das Ohr als
Nachtrichtenempfanger, Hirzel Verlag, Stuttgart,
1967, p. 187-200." as a source for this information.