The real problem is that \div is a LaTeX base macro. Consequently, it will be shadowed by physics and even loading amsmath after physics won't bring it back.
So the behaviour is similar to LaTeX where you could use a \let construct to save the old version.
However, for convenience I'll add a \divisionsymbol to the base package.