Hi Joshua,
I'll need a little more information.
That behavior you've described more closely matches nitrogen 3.0's behavior.
But even then, no part of it should be breaking the templates and site directory.
Can you go into some detail about what is happening?
If your app is explicitly designed to run 2.4, make sure you're targeting the 2.x branch of nitrogen_core (in rebar.config) and that your app's skeleton is generated with the 2.x branch of nitrogen.
This sounds like some kind of upgrade from a previous version, or is this a new deployment or something?
-Jesse