I've just tested this out on the public demo site
and had no problems displaying an apostrophe in IE11, or updated versions of Chrome, Firefox, and Edge on a Windows computer. Is it possible that you cut and pasted the site title from elsewhere to add to the settings field? My suspicion is that the current apostrophe in your site title is not a standard UTF-8 apostrophe, but rather a curly apostrophe, like the kind that Microsoft products such as Word are so fond of using. Before we dig much deeper, I would suggest that you try deleting the current apostrophe and typing in a new one, and then re-saving to see if that solves the issue.