FWIW
Most of the time I just say "flat" or "Sharp".
E.g. "It's a b9."
"It's a #5."
"Flat that fifth."
"Curiously, for sharps, I'd usually say "It's a raised fifth" or "Raise
that fifth" rather than "Sharp that fifth" or "Sharpen that fifth."
Etc.
What really matters is that the other musicians you're talking to
understand what you're saying.