Sometimes extensions don't have a choice, it depends on what the intent is. If it's possible to isolate your extension to a subset of domains however then this is advised over a expansive permissive scope. Extensions which have broad Permissions are subject to tighter review scrutiny however, so bare this in mind.
You should be able to scope out IF you require it or not, it's more less a simple case of "do I need to run on all websites, or only some". It's not always that easy though, Chrome's permissions with MV3 may not be granular if you are calling a certain API or otherwise.
Generally it's just about sound judgement - the prompt does NOT mean the extension is modifying your data on every website, just that it HAS to have this for some design reason, or another.
Thanks,
Deco