As such, it's generally a good idea to have fallback options in place, such as [Code] in PrepareToInstall that checks if your app is still running and to automatically close it, and/or using AppMutex to advise the user to close the app themselves. (Both of these may require changes to your app's code. Which one is preferable depends on whether your app can be in a state where the user has unsaved documents.)
Failing all else, you can slap the "restartreplace" flag on the [Files] entry. This is not very user-friendly, but might be considered better than an error.