Since you didn't provide any example, this is my theoretical advice list.
First of all, displacement is always in world units. The difference between TGA and EXR should be only in how precise data you have, so you don't have
banding problem. TGA is 8bits per channel, EXR can be 16-32bits. Another thing, if you want to be memory friendly, convert your textures to tx.
If you want to flush textures you can this MEL command:
arnoldFlushCache -textures true;
If you want to flush everything:
arnoldFlushCache -flushall true;
When doing IPR your texture gets loaded into memory, and if it doesn't find it, it uses the default value of 0.5.
Flushing can solve the problem since it rereads data into memory.
Also, a way you could debug your displacement values is connect your displacement shader's displacement attribute into a custom channel(
alSurface, for example, has this) and you can color pick to see what values you get.(For example Arnold RenderView's Pixel Information on the custom AOV)