This sort of thing comes up a lot in discussions of fantasy writing. There's no reason why a fantasy setting should have, say, potatoes, or cathedrals - and if it does, there's no reason it should know them by those names. That's the logical argument.
The practical argument is that we're not writing the fantasy novel in Elvish or Norcretan or whatever language exists within the fantasy realm itself - I mean you could, but not many people would read it - so using common English words is fine; you're theoretically translating everything anyway, and it doesn't take people out of the narrative the way it could if they have to stop and think what it means for a character being thrown out of the Highfane for eating fried earthfruit.
You can split the difference by deciding that although the High Kingdom of Norcrete does not in fact have potatoes, the word "potato" is the closest equivalent in English and although not strictly accurate conveys the meaning well enough for understanding.
There's no definitively correct answer. Most authors try to avoid words with a particularly strong association to a culture or place - unless they specifically want to evoke that culture or place - and keep the setting-specific vocabulary for describing things that don't exist outside the setting.
R.