From my recollection, the "edit" object link only lets you edit the metadata (usually MODS or DC datastreams). I know, there is an element on the form where it *looks like* you can upload a different object, but it is misleading because it doesn't work. At least in d7, the "edit" processing code is different from the "ingest" processing code, and the file gets ignored.
If you want to change the actual image stored in OBJ, you have to go into "Detailed list of content" at the bottom of Object Details, and delete the old object .jpg, then add it again as a new datastream (using DSID: OBJ). However this won't trigger the thumbnail to be re-generated, you'll have to do that manually too. It's a bit of a pain.
If I may rant for a moment...
This is one of the ways that Islandora acts like a "repository" and not a "content management system" - it expects you to have a digital file on hand, which you throw in the repository and never modify. This is good for some use cases (like historical document scans) but maybe not ideal for hosting all digital collections, when you might want to edit/modify the files themselves. So welcome to the edge use cases!
Also, I've found Islandora 7 much, much easier to modify than 6. The hook structure is fabulous, for example, for causing thumbnails to be regenerated when OBJ is replaced.
-R