The issue is an interaction with JavaScript, when a growable SharedArrayBuffer is grown. There can be some overhead there, on the JavaScript side.
If you aren't writing any JavaScript yourself, then the danger of a slowdown is small. Though there are some places Emscripten itself uses JavaScript (when you need to call Web APIs, say to render or download a file), so the danger is not zero.
If you don't see a slowdown (say, comparing the non-pthread version to a pthread version that uses only one thread), then there is nothing to worry about.
But if your users are on new-enough browsers, I would recommend building with
-sGROWABLE_ARRAYBUFFERS=2
to avoid any danger of a slowdown.